KR101929471B1 - Method for tracking marker, apparatus and system for executing the method - Google Patents

Abstract

A marker tracking method and apparatus and system for performing the same are disclosed. The marker tracking apparatus according to the exemplary embodiment includes an infrared camera for photographing a marker, a visible light camera for photographing a marker, an infrared ray member detecting unit for detecting an infrared ray member provided on the marker in an image captured by the infrared camera, An image processing area restricting part for restricting an image processing area for detecting a code part included in the marker based on the position of the infrared ray member in one image and a code part detecting part for detecting the code part in the image processing area.

Description

Technical Field [0001] The present invention relates to a marker tracking method, and a device and system for performing the marker tracking method.

Embodiments of the invention relate to marker tracking techniques.

Augmented Reality is a technology that shows a virtual object overlaid on the real world that the user sees. In order to realize the augmented reality, the position of the marker attached to the object is tracked using an RGB camera. Such augmented reality technology has recently been applied to medical fields such as surgical navigation.

Generally, the size of the marker used in the augmented reality is a rectangular shape with a side of about 3 cm. If this is used for surgical navigation, there is a problem that the doctor's vision is obscured due to the marker or the marker covers the affected part of the patient .

In order to prevent such a problem, the size of the marker must be reduced. When the size of the marker is reduced, the recognition accuracy of the marker corner point with respect to the marker size should be improved. In order to detect the marker, it is necessary to perform image processing on all the pixels of the image, so that it takes a lot of computation time in accordance with the image processing. As a result, real-time operation navigation can not be realized, Which leads to an extension of time.

Korean Patent Registration No. 10-1652888 (2016.09.01)

The disclosed embodiment is intended to provide a marker tracking technique that can reduce the computational processing for detecting a marker while minimizing the size of the marker.

According to an embodiment of the present invention, there is provided a marker tracking apparatus comprising: an infrared camera for photographing a marker; A visible light camera for photographing the marker; An infrared ray component detector for detecting an infrared ray component provided in the marker on the image taken by the infrared ray camera; An image processing area restricting unit for restricting an image processing area for detecting a code part included in the marker based on the position of the infrared ray member in the image photographed by the visible light camera; And a code portion detector for detecting the code portion in the image processing region.

Wherein the image processing area limitation unit converts the coordinates of the infrared ray member of the image taken by the infrared ray camera into coordinates of the image taken by the visible ray camera and detects the coordinates of the infrared ray member in the image taken by the visible ray camera Can be set as the image processing area.

The image processing area limitation unit may convert the coordinates of the infrared ray component of the image taken by the infrared camera into coordinates of the image taken by the visible ray camera through the following equation.

(Equation)

S: Coordinate system of the image taken by the visible light camera

K _S : Intrinsic Matrix of visible ray camera

: 가시광선 카메라의 외적 매트릭스(Extrinsic Matrix)

: Extrinsic Matrix of visible light camera

R _S : Matrix that represents the rotation transformation between the coordinate system of the image taken by the visible light camera and the world coordinate system

t _S : Matrix that represents the motion transformation between the coordinate system of the image captured by the visible light camera and the world coordinate system

P ₂ : Coordinates in the world coordinate system of a predetermined point photographed by a visible ray camera

M: Conversion matrix between the coordinate system of the image captured by the infrared camera and the coordinate system of the image captured by the visible light camera

 I: The coordinate system of the image taken by the infrared camera

K _I : Intrinsic Matrix of Infrared Camera

: 적외선 카메라의 외적 매트릭스(Extrinsic Matrix)

: Extrinsic Matrix of Infrared Camera

R _I : Matrix indicating the rotation transformation between the coordinate system of the image taken by the infrared camera and the world coordinate system

t _I : A matrix indicating the shift transformation between the coordinate system of the image taken by the infrared camera and the world coordinate system

P ₁ : Coordinates in the world coordinate system of a predetermined point photographed with an infrared camera

The code portion detection unit detects edge pixels in the image processing region and merges edge pixels whose direction of change in the direction in which the brightness variation of the pixel value is the largest among the detected edge pixels is within a predetermined range, Detects an internal space formed by the detected lines as a position of the code unit, and extracts marker information from the code unit.

The marker tracking apparatus may further include a marker ID detector for extracting IDs of the respective markers based on the infrared rays detected from the images captured by the infrared camera and identifying the respective markers.

Wherein the marker ID detection unit clusters the detected infrared members into a plurality of groups based on the position of the detected infrared members, extracts IDs of the markers corresponding to the number of infrared members belonging to each group, can do.

The marker includes: a base member; A code unit formed on one surface of the base member and including marker information; And a plurality of infrared light members surrounding and spaced apart from the code unit on the outer side of the code unit on one side of the base member.

A marker tracking method in accordance with one disclosed embodiment is a method performed in a computing device having one or more processors and a memory storing one or more programs executed by the one or more processors, Detecting an infrared ray member provided on the marker; Limiting an image processing region for detecting a code portion included in the marker based on a position of the infrared ray member in a visible ray photographing image of the marker; And detecting the code portion in the image processing region.

Wherein the step of limiting the image processing region includes the steps of: converting coordinates of the infrared ray member of the infrared ray image into coordinates of the visible ray ray image; And setting an area of the visible ray photographing image positioned within the coordinates of the infrared ray member as the image processing area.

Converting the coordinates of the infrared ray member of the infrared ray image into coordinates of the visible ray ray image using the following equation.

(Equation)

S: Coordinate system of visible light image

K _S : Intrinsic Matrix of visible ray camera

: 가시광선 카메라의 외적 매트릭스(Extrinsic Matrix)

: Extrinsic Matrix of visible light camera

R _S : matrix representing the rotation transformation between the coordinate system of the visible light image and the world coordinate system

t _S : Matrix representing the motion transformation between the coordinate system of the visible light image and the world coordinate system

P ₂ : Coordinates in the world coordinate system of a predetermined point photographed by a visible ray camera

M: Conversion matrix between the coordinate system of the infrared ray image and the coordinate system of the visible ray image

 I: Coordinate system of infrared image

K _I : Intrinsic Matrix of Infrared Camera

: 적외선 카메라의 외적 매트릭스(Extrinsic Matrix)

: Extrinsic Matrix of Infrared Camera

R _I : matrix representing the rotation transformation between the coordinate system of the infrared image and the world coordinate system

t _I : Matrix that represents the motion transformation between the coordinate system of the infrared image and the world coordinate system

P ₁ : Coordinates in the world coordinate system of a predetermined point photographed with an infrared camera

Detecting the code portion comprises: detecting edge pixels within the image processing region; Detecting lines by merging edge pixels whose direction of change in a direction in which the brightness variation of the pixel value is the largest among the detected edge pixels is within a predetermined range; And detecting the internal space formed by the detected lines as the position of the code unit, and extracting the marker information from the code unit.

The marker tracking method may further include extracting an ID of each marker based on the infrared rays detected in the infrared radiographic image, and identifying each marker.

Wherein identifying the marker comprises: clustering the detected infrared elements into a plurality of groups based on the location of the detected infrared elements; And extracting an ID of a marker corresponding to the number of infrared ray members belonging to each group.

The marker tracking system according to an embodiment disclosed herein includes a base member, a code portion formed on one surface of the base member and including marker information, and a code portion surrounding the code portion on the outer side of the code portion on one surface of the base member, A marker including a plurality of infrared light members which are provided so as to be arranged; And an infrared ray member provided on the marker in an infrared radiographic image of the marker and limits an image processing region for detecting a code portion included in the marker based on the position of the infrared ray member in the visible light ray image of the marker, And a marker tracking device for detecting the code part in the image processing area.

According to an embodiment of the present invention, an infrared ray component is detected in an infrared radiographic image to limit an image processing performing area for detecting a code part in a visible light radiographic image, and then the detection of a code part is performed only in a limited area (i.e., It is possible to reduce the time required for the entire processor, thereby improving the image quality by improving the shutter speed of the visible light camera and improving the accuracy of the marker position tracking.

In addition, by improving the accuracy of the marker tracking, it is possible to overcome the limitations of the existing marker size, thereby preventing the markers from masking the doctor's view or obstructing the affected part of the patient. In addition, it is possible to reduce the amount of computation by detecting the code unit by limiting the image processing execution region, thereby realizing real-time surgical navigation.

1 is a view for schematically explaining a marker tracking method according to an embodiment of the present invention;
2 is a diagram showing the configuration of a marker tracking apparatus according to an embodiment of the present invention.
3 is a diagram showing a state of detecting a line in a code unit according to an embodiment of the present invention;
4 is a flowchart for explaining a marker tracking method according to an embodiment of the present invention.
5 is a block diagram illustrating and illustrating a computing environment including a computing device suitable for use in the exemplary embodiments.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. The following detailed description is provided to provide a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, this is merely an example and the present invention is not limited thereto.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail. The following terms are defined in consideration of the functions of the present invention, and may be changed according to the intention or custom of the user, the operator, and the like. Therefore, the definition should be based on the contents throughout this specification. The terms used in the detailed description are intended only to describe embodiments of the invention and should in no way be limiting. Unless specifically stated otherwise, the singular forms of the expressions include plural forms of meanings. In this description, the expressions "comprising" or "comprising" are intended to indicate certain features, numbers, steps, operations, elements, parts or combinations thereof, Should not be construed to preclude the presence or possibility of other features, numbers, steps, operations, elements, portions or combinations thereof.

In the following description, terms such as " transmission ", "transmission "," transmission ", "reception ", and the like, of a signal or information refer not only to the direct transmission of signals or information from one component to another But also through other components. In particular, "transmitting" or "transmitting" a signal or information to an element is indicative of the final destination of the signal or information and not a direct destination. This is the same for "reception" of a signal or information. Also, in this specification, the fact that two or more pieces of data or information are "related" means that when one piece of data (or information) is acquired, at least a part of the other data (or information) can be obtained based thereon.

On the other hand, directional terms such as the top, bottom, one side, the other, and the like are used in connection with the orientation of the disclosed figures. Since the elements of the embodiments of the present invention can be positioned in various orientations, directional terms are used for illustrative purposes and not limitation.

Also, the terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms may be used for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

FIG. 1 is a view for schematically explaining a marker tracking method according to an embodiment of the present invention.

Referring to FIG. 1, the marker 10 may include a base member 11, an infrared ray member 13, and a code unit 15. In an exemplary embodiment, the marker 10 may be affixed to at least one of a surgical tool, a perimeter of a patient, a camera lens, or the like. For example, the marker 10 may be for providing position and orientation information in an image-based surgical system such as a surgical navigation system or the like.

The base member (11) serves to support the marker (10). A pressure-sensitive adhesive may be provided on the bottom surface of the base member 11 so that the marker 10 can be attached to a surgical instrument, an affected part of a patient, a camera lens, a fixing device for fixing the camera, or the like.

The infrared ray member (13) may be provided on the upper surface of the base member (11). A plurality of the infrared ray members 13 may be spaced apart from each other at an outer edge of the base member 11 (for example, a rim or an edge region). The infrared ray member 13 may be, for example, an infrared light emitting diode (LED) or an infrared ray emitting material. The infrared ray member 13 can be used to detect the approximate position of the code portion 15. [ That is, the infrared ray member 13 can be used to limit the area where the marker 10 performs image processing to detect the code portion 15 in the photographed image. The infrared ray member 13 may be provided to surround the code unit 15 on the outer side of the code unit 15.

The code portion 15 may be formed on the upper surface of the base member 11. The code unit 15 may include marker information (e.g., position and orientation information, etc.) that the marker 10 desires to provide. In an exemplary embodiment, the code unit 15 may be one in which the marker information is coded (for example, a bar code or a QR code) and printed. For example, the code unit 15 may have a rectangular shape, but the shape thereof is not limited thereto.

In the disclosed embodiment, after the infrared ray member 13 is detected by the infrared camera and the visible ray camera limits the image processing performing area for detection of the code unit 15 in the image of the marker 10, It is possible to reduce the time required for the entire processor by performing the image processing for detecting the code unit 15 only in the region (i.e., the image processing performing region), thereby improving the shutter speed of the visible light camera, It is possible to improve the accuracy of the marker position tracking.

In addition, by improving the accuracy of the marker tracking, it is possible to reduce the size of the marker itself, thereby preventing the marker 10 from covering the field of view of the doctor or covering the affected part of the patient. In addition, it is possible to limit the image processing performing region to reduce the amount of calculation according to the detection of the code unit 15, thereby realizing real-time surgical navigation.

2 is a block diagram of a marker tracking apparatus according to an embodiment of the present invention.

2, the marker tracking apparatus 100 includes an infrared camera 102, a visible light camera 104, an infrared ray member detection unit 106, a marker ID detection unit 108, an image processing region restriction unit 110, And a code portion detecting unit 112.

The infrared camera 102 is a first photographing means for photographing the marker 10. Specifically, the infrared camera 102 is a photographing means used for detecting the infrared ray member 13 in the marker 10. [ The infrared camera 102 can receive light of an infrared wavelength longer than the wavelength of visible light and photograph the marker 10.

The visible light camera 104 is a second photographing means for photographing the marker 10. Specifically, the visible light camera 104 is a photographing means used for detecting the code unit 15 in the marker 10. [ The visible light camera 104 receives light of a visible light wavelength and can photograph the marker 10. For example, the visible light camera 104 may be, but is not limited to, a stereo camera.

The infrared ray member detection unit 106 can detect the infrared ray member 13 in an image taken by the infrared ray camera 102 (hereinafter, referred to as an infrared ray image). The infrared ray member detection unit 106 can detect the position (i.e., coordinates) of the infrared ray members 13 in the infrared ray image.

The marker ID detector 108 can detect the ID of the marker 10 based on the infrared image. Specifically, when there are a plurality of markers 10 on the image taken by the infrared camera 102 (for example, when the first marker is attached to the surgical tool and the second marker is attached around the affected part of the patient, etc. ), Each marker 10 must be distinguishable and identifiable. Thus, the marker ID detection unit 108 can detect which marker each of the infrared ray members 13 belongs to in the infrared ray image. In the exemplary embodiment, the marker ID detection unit 108 may identify the respective markers 10 according to the number of the clustered infrared members 13, and clustering the infrared members 13 in the infrared image . At this time, the ID of each marker 10 can be stored by matching with the number of the infrared ray members 13 formed on the corresponding marker 10.

In an exemplary embodiment, when a first marker having three infrared members is attached to a surgical tool and a second marker having four infrared members are attached around the affected part of the patient, the marker ID detector 108 detects The infrared ray members 13 (i.e., seven infrared ray members) in the infrared ray image can be clustered into two groups. At this time, the marker ID detecting unit 108 can cluster the infrared ray members 13 into two groups, for example, through the K-Means clustering technique.

The marker ID detection unit 108 can identify the number of objects (infrared members) belonging to each group and extract the ID of the marker 10 corresponding thereto. That is, when the number of objects belonging to the first group is three, the marker ID detecting unit 108 detects the ID of the marker 10 (i.e., the first marker) matching the three infrared ray members 13 Can be extracted. In the case where the number of objects belonging to the second group is four, the marker ID detection unit 108 detects the ID of the marker 10 (i.e., the second marker) matched with four infrared ray members 13 Can be extracted.

However, the present invention is not limited to this, and the infrared rays 13 may be grouped by applying different frequencies to the infrared rays 13 on a marker-by-marker basis. In other words, the infrared ray members 13 are arranged to emit infrared rays of different wavelengths (or frequencies) for each marker, and the infrared ray members 13 which emit the same wavelength (or frequency) It can be detected which marker belongs to which marker. Each of the markers 10 may be identified according to the number of the grouped infrared rays 13.

As described above, the infrared ray component 13 is used not only for limiting the performing region of the image processing when detecting the code portion 15 of the marker 10, but also for identifying the ID of the marker 10, It is not necessary to separately add identification information of the marker 10 to the code unit 15, thereby minimizing the size of the code unit 15.

In other words, the marker 10 according to the disclosed embodiment does not add additional identification information to the code unit 15 by matching the number of infrared members 13 with the ID of the corresponding marker 10, 15 can be minimized. By designating the approximate position of the code unit 15 by the infrared ray member 13 while reducing the size of the code unit 15, real-time operation navigation can be realized by reducing the amount of calculation required for detecting the code unit 15 .

The image processing region restricting unit 110 is a region for performing image processing for detecting the code unit 15 in an image photographed by the visible light camera 104 (hereinafter referred to as a visible light photographing image) , Image processing area) can be limited. Specifically, the image processing region restricting unit 110 converts the coordinates of the infrared ray members 13 of the infrared ray image into coordinates of the visible ray ray image, and then, based on the coordinate information of the infrared ray members 13, The image processing area can be set in the image.

The coordinates (p _I ) of a pixel corresponding to a predetermined point in the infrared radiographic image can be expressed by the following equation (1).

(1)

는 적외선 카메라의 외적 매트릭스(Extrinsic Matrix)로서, R_I는 적외선 촬영 영상의 좌표계와 월드 좌표계 사이의 회전 변환을 나타내는 매트릭스이고, t_I는 적외선 촬영 영상의 좌표계와 월드 좌표계 사이의 이동 변환을 나타내는 매트릭스이다. P1는 월드 좌표계(실제 공간에서의 좌표계)에서 소정 포인트에 대응하는 좌표를 나타낸다.Here, I represents a coordinate system of an infrared radiographic image. K _I is an Intrinsic Matrix of an infrared camera and is a matrix having characteristic parameters of an infrared camera (for example, focal length and principal point, etc.).

R _I is a matrix representing the rotation transformation between the coordinate system of the infrared image and the world coordinate system, t _I is a matrix representing the transformation between the coordinate system of the infrared image and the world coordinate system, to be. P1 represents a coordinate corresponding to a predetermined point in the world coordinate system (coordinate system in actual space).

The coordinates (p _S ) of the pixel corresponding to a predetermined point in the visible light radiographic image can be expressed by the following equation (2).

(2)

는 가시광선 카메라의 외적 매트릭스(Extrinsic Matrix)로서, R_S는 가시광선 촬영 영상의 좌표계와 월드 좌표계 사이의 회전 변환을 나타내는 매트릭스이고, t_S는 가시광선 촬영 영상의 좌표계와 월드 좌표계 사이의 이동 변환을 나타내는 매트릭스이다. P2는 월드 좌표계(실제 공간에서의 좌표계)에서 소정 포인트에 대응하는 좌표를 나타낸다.Here, S represents a coordinate system of the visible ray photographing image. K _S is an Intrinsic Matrix of a visible light camera and is a matrix having characteristic parameters of a visible light camera (e.g., focal length and principal point, etc.).

R _s is a matrix representing the rotation transformation between the coordinate system of the visible ray image and the world coordinate system, t _s is a moving transformation between the coordinate system of the visible ray photographing image and the world coordinate system, ≪ / RTI > P2 represents a coordinate corresponding to a predetermined point in the world coordinate system (coordinate system in actual space).

The image processing region restricting unit 110 may convert the coordinates (p _I ) of the pixel corresponding to the infrared ray member 13 to the coordinates (p _S ) of the visible ray photographing image in the infrared ray photographing image using the following equation (3) have.

(3)

Here, M is a homography, which is a transformation matrix between the coordinate system of the infrared image and the coordinate system of the visible light image. In an exemplary embodiment, M may be computed using a DLT (Direct Linear Transform) algorithm. Since this is a known technique, a detailed description thereof will be omitted.

The image processing region restricting unit 110 sets the image processing region for detecting the code unit 15 based on the coordinates of the pixel corresponding to each infrared ray member 13 of the visible light ray image calculated through Equation (3) . The image processing region restricting unit 110 may set a region located in the coordinates of each infrared ray member 13 in the visible light ray image as the image processing region. For example, the marker tracking apparatus 100 may set an internal region formed by connecting the coordinates of each infrared ray member 13 in the visible light radiography image to the image processing region. At this time, the coordinate value of each infrared ray member 13 can be used as the center coordinate value of the infrared ray member 13.

The code portion detection unit 112 can detect the code portion 115 of the marker 10 within the image processing region of the visible light ray image. In an exemplary embodiment, the code portion detection unit 112 may detect edge pixels within the image processing region set by the image processing region restriction unit 110. [ The code portion detection unit 112 can detect edge pixels by checking a rate of change in brightness of a pixel value while moving a window of a predetermined size in each direction within an image processing region.

For example, the code portion detection unit 112 obtains a difference in brightness values of pixels located in the neighborhood (left, right, top, and bottom) of the center pixel of the window in the image processing region, Can be calculated. The code portion detection unit 112 may determine the pixel as an edge pixel when the rate of change of brightness of the pixel is equal to or greater than a preset threshold value.

Next, the code portion detecting unit 112 may detect a line by collecting edge pixels whose direction of change in the direction in which the rate of change of brightness is the greatest among the edge pixels (that is, the changing angle) is within a predetermined range. That is, the code portion detecting unit 112 can detect the line by collecting pixels having the same or similar change direction in the direction in which the brightness change rate of the detected edge pixels is the largest, while moving the window in the left and right direction in the image processing region .

3, when the code unit 15 is formed in a rectangular shape, the code unit detecting unit 112 detects that the direction of change in the direction in which the brightness change rate of the edge pixels is the largest in the image processing region 20 is the same Or similar edge pixels to detect the first line 21, the second line 23, the third line 25, and the fourth line 27, respectively. The code portion detecting unit 112 detects the code portion of each line 21, 23, 25, 27 when the inner space formed by each line 21, 23, 25, 27 forms a closing loop. And the marker information can be extracted from the code unit 15. In this case,

4 is a flowchart illustrating a marker tracking method according to an embodiment of the present invention. The method shown in Fig. 4 can be performed, for example, by the marker tracking apparatus 100 described above. In the illustrated flow chart, the method is described as being divided into a plurality of steps, but at least some of the steps may be performed in reverse order, combined with other steps, performed together, omitted, divided into detailed steps, One or more steps may be added and performed.

Referring to FIG. 4, the marker tracking apparatus 100 detects the infrared ray 13 from the image of the marker 10 captured by the infrared ray camera 102 (S 101). At this time, the marker tracking apparatus 100 can detect the position (coordinate) of the infrared ray member 13 in the infrared ray image.

Next, the marker tracking apparatus 100 groups the infrared rays 13 of the infrared radiographic image (S 103). In an exemplary embodiment, the marker tracking device 100 may group a plurality of infrared elements 13 using a K-Means clustering technique. Alternatively, a plurality of infrared ray members 13 may be grouped using the wavelength (or frequency) of the light emitted from the plurality of infrared ray members 13.

Next, the marker tracking apparatus 100 identifies each marker 10 based on the number of the infrared rays 13 classified into each group (S 105). The marker tracking apparatus 100 can identify the marker 10 by extracting the ID of the stored marker 10 by matching with the number of the infrared ray members 13 belonging to each group.

Next, the marker tracking apparatus 100 converts the coordinates of each infrared ray member 13 of the infrared ray image into coordinates of the visible ray ray image (S 107). The marker tracking apparatus 100 can convert the coordinates of each infrared ray member 13 of the infrared ray image into the coordinates of the visible ray ray image using Equations (1) to (3).

Next, the marker tracking apparatus 100 detects an image for detection of the code unit 15 based on the coordinates of the respective infrared ray members 13 in an image (i.e., a visible light ray image) photographed by the visible ray camera 104 A process area is set (S 109). The marker tracking apparatus 100 can set a region located within the coordinates of each infrared ray member 13 in the visible light ray image as the image processing region.

Next, the marker tracking apparatus 100 detects edge pixels in the image processing region (S 111). In an exemplary embodiment, the marker tracking apparatus 100 can detect edge pixels by detecting a change in brightness of a pixel value while moving a window of a predetermined size in each direction within an image processing region.

Next, the marker tracking apparatus 100 detects the line by collecting the edge pixels having the direction of the change in the direction in which the brightness change rate is the largest among the detected edge pixels within the predetermined range (S 113).

Next, the marker tracking apparatus 100 detects the internal space formed by each line as the position of the code unit 15, and stores marker information (e.g., position and attitude information) in the code unit 15, (S115).

5 is a block diagram illustrating and illustrating a computing environment 10 including a computing device suitable for use in the exemplary embodiments. In the illustrated embodiment, each of the components may have different functions and capabilities than those described below, and may include additional components in addition to those described below.

The illustrated computing environment 10 includes a computing device 12. In one embodiment, computing device 12 may be an apparatus (e.g., marker tracking device 100) for tracking markers.

The computing device 12 includes at least one processor 14, a computer readable storage medium 16, The processor 14 may cause the computing device 12 to operate in accordance with the exemplary embodiment discussed above. For example, processor 14 may execute one or more programs stored on computer readable storage medium 16. The one or more programs may include one or more computer-executable instructions, which when executed by the processor 14 cause the computing device 12 to perform operations in accordance with the illustrative embodiment .

The computer-readable storage medium 16 is configured to store computer-executable instructions or program code, program data, and / or other suitable forms of information. The program 20 stored in the computer-readable storage medium 16 includes a set of instructions executable by the processor 14. In one embodiment, the computer-readable storage medium 16 may be any type of storage medium such as a memory (volatile memory such as random access memory, non-volatile memory, or any suitable combination thereof), one or more magnetic disk storage devices, Memory devices, or any other form of storage medium that can be accessed by the computing device 12 and store the desired information, or any suitable combination thereof.

Communication bus 18 interconnects various other components of computing device 12, including processor 14, computer readable storage medium 16.

The computing device 12 may also include one or more input / output interfaces 22 and one or more network communication interfaces 26 that provide an interface for one or more input / output devices 24. The input / output interface 22 and the network communication interface 26 are connected to the communication bus 18. The input / output device 24 may be connected to other components of the computing device 12 via the input / output interface 22. The exemplary input and output device 24 may be any type of device, such as a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touch pad or touch screen), a voice or sound input device, An input device, and / or an output device such as a display device, a printer, a speaker, and / or a network card. The exemplary input and output device 24 may be included within the computing device 12 as a component of the computing device 12 and may be coupled to the computing device 12 as a separate device distinct from the computing device 12 It is possible.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, . Therefore, the scope of the present invention should not be limited to the above-described embodiments, but should be determined by equivalents to the appended claims, as well as the appended claims.

10: Marker
11: Base member
13: Infrared ray member
15: Code section
100: marker tracking device
102: Infrared camera
104: Visible light camera
106: Infrared ray member detection unit
108: marker ID detection unit
110: image processing area limitation unit
112: Code section detection section

Claims (14)

An infrared camera for photographing the marker;
A visible light camera for photographing the marker;
An infrared ray component detector for detecting an infrared ray component provided in the marker on the image taken by the infrared ray camera;
An image processing area restricting unit for restricting an image processing area for detecting a code part included in the marker based on the position of the infrared ray member in the image photographed by the visible light camera; And
And a code section detection section for detecting the code section in the image processing area,
The image processing apparatus according to claim 1,
The infrared ray camera converts the coordinates of the infrared ray member of the image photographed by the infrared ray camera into coordinates of the image photographed by the visible ray camera and displays an area of the image photographed by the visible ray camera located in the coordinates of the infrared ray member, Processing area,
Wherein the coordinates of the infrared ray member of the image photographed by the infrared camera are converted into the coordinates of the image photographed by the visible ray camera through the following equation.
(Equation)

S: Coordinate system of the image taken by the visible light camera
K _S : Intrinsic Matrix of visible ray camera

: Extrinsic Matrix of visible light camera
R _S : Matrix that represents the rotation transformation between the coordinate system of the image taken by the visible light camera and the world coordinate system
t _S : Matrix that represents the motion transformation between the coordinate system of the image captured by the visible light camera and the world coordinate system
P ₂ : Coordinates in the world coordinate system of a predetermined point photographed by a visible ray camera
M: Conversion matrix between the coordinate system of the image captured by the infrared camera and the coordinate system of the image captured by the visible light camera
I: The coordinate system of the image taken by the infrared camera
K _I : Intrinsic Matrix of Infrared Camera

: Extrinsic Matrix of Infrared Camera
R _I : Matrix indicating the rotation transformation between the coordinate system of the image taken by the infrared camera and the world coordinate system
t _I : Matrix indicating movement transformation between the coordinate system of the image taken by the infrared camera and the world coordinate system
P ₁ : Coordinates in the world coordinate system of a predetermined point shot by an infrared camera.
delete delete The method according to claim 1,
Wherein the code portion detection unit comprises:
Detecting edge pixels in the image processing region, detecting lines by merging edge pixels whose direction of change in a direction in which the brightness variation of the pixel value is the largest among the detected edge pixels is within a predetermined range, Detects the internal space formed by the lines as the position of the code portion, and extracts marker information from the code portion.
The method according to claim 1,
The marker tracking device includes:
Further comprising a marker ID detecting unit for extracting IDs of the respective markers based on the infrared rays detected from the images taken by the infrared camera and identifying the respective markers.
The method of claim 5,
Wherein the marker ID detector comprises:
Wherein the detected infrared ray members are clustered into a plurality of groups based on the detected positions of the infrared ray members, and the IDs of the markers corresponding to the number of infrared ray members belonging to each group are extracted to identify the corresponding markers.
The method according to claim 1,
The marker
A base member;
A code unit formed on one surface of the base member and including marker information; And
And a plurality of infrared light members surrounding and spaced apart from each other on the outer side of the code portion on one side of the base member.
One or more processors, and
A method performed in a computing device having a memory storing one or more programs executed by the one or more processors,
Detecting an infrared ray member provided on the marker in an infrared ray image of the marker;
Limiting an image processing region for detecting a code portion included in the marker based on a position of the infrared ray member in a visible ray photographing image of the marker; And
And detecting the code portion in the image processing area,
Wherein the step of limiting the image processing area comprises:
Converting the coordinates of the infrared ray member of the infrared radiographic image into coordinates of the visible radiographic image; And
Setting an area of the visible light radiographic image located in the coordinates of the infrared ray member as the image processing area,
Wherein the step of converting the coordinates of the visible ray photographing image into coordinates of the visible ray photographing image comprises:
Wherein the coordinates of the infrared ray member of the infrared ray image are converted into the coordinates of the visible ray ray image through the following equation.
(Equation)

S: Coordinate system of visible light image
K _S : Intrinsic Matrix of visible ray camera

: Extrinsic Matrix of visible light camera
R _S : matrix representing the rotation transformation between the coordinate system of the visible light image and the world coordinate system
t _S : Matrix representing the motion transformation between the coordinate system of the visible light image and the world coordinate system
P ₂ : Coordinates in the world coordinate system of a predetermined point photographed by a visible ray camera
M: Conversion matrix between the coordinate system of the infrared ray image and the coordinate system of the visible ray image
I: Coordinate system of infrared image
K _I : Intrinsic Matrix of Infrared Camera

: Extrinsic Matrix of Infrared Camera
R _I : matrix representing the rotation transformation between the coordinate system of the infrared image and the world coordinate system
t _I is a matrix representing the motion transformation between the coordinate system of the infrared image and the world coordinate system
P ₁ : Coordinates in the world coordinate system of a predetermined point shot by an infrared camera.
delete delete The method of claim 8,
Wherein the step of detecting the code portion comprises:
Detecting edge pixels in the image processing region;
Detecting lines by merging edge pixels whose direction of change in a direction in which the brightness variation of the pixel value is the largest among the detected edge pixels is within a predetermined range; And
Detecting an internal space formed by the detected lines as a position of the code portion, and extracting marker information from the code portion.
The method of claim 8,
The marker tracking method includes:
Further comprising the step of identifying each marker by extracting an ID of each marker based on infrared rays detected in the infrared radiographic image.
The method of claim 12,
Wherein identifying the marker comprises:
Clustering the detected infrared members into a plurality of groups based on the detected positions of the infrared members; And
And extracting an ID of a marker corresponding to the number of infrared members belonging to each group.
1. A marker comprising: a base member; a code portion formed on one surface of the base member, the code portion including marker information; and a plurality of infrared members surrounding the code portion and spaced apart from each other outside the code portion on one side of the base member; And
Detecting an infrared member provided in the marker in an infrared radiographic image of the marker and restricting an image processing region for detecting a code portion included in the marker based on a position of the infrared ray member in a visible ray photographing image of the marker, And a marker tracking device for detecting the code part in the image processing area,
The marker tracking device includes:
Converting the coordinates of the infrared ray component of the infrared radiographic image into coordinates of the visible ray radiographic image and setting an area of the visible ray radiographic image located in the coordinates of the infrared ray component as the image processing area,
Wherein the coordinates of the infrared ray member of the infrared ray image are converted into the coordinates of the visible ray ray image through the following equation.
(Equation)

S: Coordinate system of the image taken by the visible light camera
K _S : Intrinsic Matrix of visible ray camera

: Extrinsic Matrix of visible light camera
R _S : Matrix that represents the rotation transformation between the coordinate system of the image taken by the visible light camera and the world coordinate system
t _S : Matrix that represents the motion transformation between the coordinate system of the image captured by the visible light camera and the world coordinate system
P ₂ : Coordinates in the world coordinate system of a predetermined point photographed by a visible ray camera
M: Conversion matrix between the coordinate system of the image captured by the infrared camera and the coordinate system of the image captured by the visible light camera
I: The coordinate system of the image taken by the infrared camera
K _I : Intrinsic Matrix of Infrared Camera

: Extrinsic Matrix of Infrared Camera
R _I : Matrix indicating the rotation transformation between the coordinate system of the image taken by the infrared camera and the world coordinate system
t _I : Matrix indicating movement transformation between the coordinate system of the image taken by the infrared camera and the world coordinate system
P ₁ : Coordinates in the world coordinate system of a predetermined point shot by an infrared camera.

Images