KR102313319B1 - AR colonoscopy system and method for monitoring by using the same - Google Patents

Abstract

An AR endoscope system that can conveniently monitor the shape of an endoscope inserted into the body and provide accurate location information when a lesion is found and a monitoring method using the same are provided. This AR endoscope system recognizes the position information (actual position coordinates) of the endoscope scope on the first coordinate system (fixed coordinate system), and then converts it into virtual position coordinates on the second coordinate system (variable coordinate system) defined based on the AR glasses. It is characterized by rendering the virtual scope corresponding to the scope on the AR glass using the virtual position coordinates.

Description

AR colonoscopy system and method for monitoring by using the same}

The present invention relates to an augmented reality (AR) endoscope system and a monitoring method using the same, and more particularly, a virtual image of the same shape as a real scope inserted into a patient's body through AR glasses can be checked by overlapping the patient's body. It relates to an AR endoscope system and a monitoring method using the same.

Colonoscopy is a test method that inserts a special camera called an endoscope into the anus to observe the inside of the large intestine and the distal end of the small intestine adjacent to the large intestine. Most of the endoscopes have a camera attached to the end of the scope, so the operator can see the inside of the large intestine and make a diagnosis with the eye, and if necessary, hemostasis, histological examination, or removal of the lesion.

Although the camera is installed in the endoscope as described above, the operator can only grasp where the scope is located in the abdominal cavity only with the endoscope screen and the feeling of the hand, and it is difficult to grasp the exact position or shape of the scope. In particular, if there are adhesions or loops in the intestine, or the curvature of the intestine is severe, there is a risk of perforation even if the scope punctures the intestine, causing pain to the patient or severe pain even for an experienced operator.

In addition, even if a polyp or carcinoma is found during colonoscopy, the scope's exact coordinates in the body cannot be stored, so it is often difficult to find it again when trying to remove the tumor with a subsequent procedure or surgery.

SUMMARY OF THE INVENTION An object of the present invention is to provide an AR endoscope system that can conveniently monitor the shape of an endoscope inserted into the body and accurately store the location information along with the shape of a virtual scope when a lesion is found.

Another problem to be solved by the present invention is to provide a monitoring method using such an AR endoscope system.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

An AR endoscope system according to an embodiment of the present invention for achieving the above object is an AR endoscope system using an endoscope in which a plurality of markers are formed on a scope: a first coordinate system is referenced while the scope is inserted into a patient's body a marker detection sensor for recognizing the actual position coordinates of the marker; and setting a second coordinate system based on the AR glasses worn by the operator, converting the real position coordinates into virtual position coordinates on the second coordinate system, and generating a virtual scope corresponding to the scope using the virtual position coordinates It may include an image processing device that renders on the AR glass after doing so.

The second coordinate system may be set based on the gaze direction of the AR glass, and the image processing apparatus may calculate the virtual position coordinates using the relative position between the marker and the AR glass and the gaze direction of the AR glass. .

the marker detection sensor recognizes location information of a reference marker disposed outside the patient's body; The image processing apparatus calculates a display position of the reference marker on the AR glass using the image information of the reference marker photographed from the AR glass; The image processing apparatus may render the virtual scope by reflecting the relative positions of the marker and the reference marker based on the display position of the reference marker on the AR glass.

When a lesion is found during monitoring, the image processing apparatus may store shape information of the virtual scope at a location where the lesion is found.

An AR endoscope monitoring method according to an embodiment of the present invention for achieving the above another object is an AR endoscope monitoring method using an endoscope in which a plurality of markers are formed on a scope: The marker detection sensor is inserted into the body of a patient recognizing the actual position coordinates of the marker on the basis of the first coordinate system; and the image processing device sets a second coordinate system based on the AR glasses worn by the operator, converts the actual position coordinates into virtual position coordinates on the second coordinate system, and uses the virtual position coordinates to correspond to the scope After generating the virtual scope, it may include rendering on the AR glass.

recognizing, by the marker detection sensor, location information of a reference marker disposed outside the patient's body; calculating, by the image processing device, a display position of the reference marker on the AR glass using image information of the reference marker photographed from the AR glass; and rendering, by the image processing apparatus, the virtual scope by reflecting the relative positions of the marker and the reference marker based on the display position of the reference marker on the AR glass.

Specific details of other embodiments are included in the detailed description and drawings.

As described above, according to the AR endoscope system and the monitoring method using the same according to the present invention, the operator can perform the operation while monitoring the 3D virtual scope image of the colonoscope in the body in real time through AR glasses using augmented reality. Therefore, through the shape of the virtual scope, it is possible to identify the case of adhesions, loops, or severe curvature in the intestine, so that pain or perforation of the patient due to excessive treatment can be prevented in advance. In addition, when a lesion is found, the location of the lesion can be accurately confirmed from the shape information of the virtual scope when the tumor is removed by a subsequent procedure or surgery by storing the shape information of the virtual scope at the location.

1 is a diagram schematically showing an AR endoscope system according to an embodiment of the present invention.
2 is a diagram showing the configuration of an AR endoscope system according to an embodiment of the present invention.
FIG. 3 is a view showing the configuration of the endoscope of FIG. 2 .
FIG. 4 is a view for explaining an operation of the image processing apparatus of FIG. 2 .
5 is a flowchart illustrating an AR endoscope monitoring method according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in a variety of different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention belongs It is provided to fully inform the possessor of the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout.

The endoscope of the present invention is a medical device that allows direct viewing of internal organs or body cavities, and is designed to insert and observe organs that cannot directly see lesions without surgery or autopsy. As such an endoscope, a bronchoscope, an esophageal endoscope, a gastroscope, a duodenum, a rectal endoscope, a cystoscope, a laparoscope, a thoracoscopy, a mediastinoscope, a cardiscope, or a colonoscope may be used. In this embodiment, for convenience of description, a colonoscope is described as an example, but the present invention is not limited thereto and may be substantially equally applied to other types of endoscopes.

Hereinafter, an AR endoscope system according to an embodiment of the present invention will be described with reference to FIGS. 1 to 4 . 1 is a diagram schematically showing an AR endoscope system according to an embodiment of the present invention. 2 is a diagram showing the configuration of an AR endoscope system according to an embodiment of the present invention. FIG. 3 is a view showing the configuration of the endoscope of FIG. 2 . FIG. 4 is a view for explaining an operation of the image processing apparatus of FIG. 2 .

The AR endoscope system 100 includes an endoscope 40 having a plurality of markers 48 formed therein, a marker detection sensor 50 for detecting the position and shape of the endoscope 40, and AR glasses 22 worn by the operator; , an image processing device 60 for rendering a virtual scope image corresponding to the endoscope 40 inserted into the body on the AR glasses 22 . Optionally, the AR endoscope system 100 may include a monitor 30 that displays images captured by the endoscope 40 .

When the patient 10 is lying on the bed 12 , the operator 20 inserts the endoscope 40 into the body of the patient 10 . The endoscope 40 includes a handle portion 44 for controlling the movement of the endoscope held by the operator by hand, a scope 42 connected to the handle portion 44 and inserted into the body and having a camera formed at the end thereof, and a scope 42 a plurality of markers 48 formed thereon. In addition, the endoscope 40 may further include a cable 46 connected to the handle portion 44 to transmit image information photographed through the camera to the monitor 30 .

The plurality of markers 48 may be formed on the scope 42 at predetermined intervals (eg, 10 cm). In a state in which the scope 42 is inserted into the body, the material or configuration of the marker 48 may be controlled so that the marker detection sensor 50 may detect accurate position information of the marker 48 . For example, when the marker 48 is composed of an electromagnetic coil, the marker detection sensor 50 collects magnetic pulses generated from the marker 48 to determine the exact position information of each marker 48, and based on this, The direction and shape of the scope 42 can be calculated. However, the present invention is not limited thereto, and various non-contact type position sensors may be used as markers.

The marker detection sensor 50 is the actual position coordinates (x, y, z) of the marker 48 based on the first coordinate system (X, Y, Z) while the scope 42 is inserted into the body of the patient 10 . can recognize For example, the marker detection sensor 50 may define a first coordinate system based on the couch 12 and recognize the actual position coordinates (x, y, z) of the marker 48 on the first coordinate system. The first coordinate system is preferably a fixed coordinate system that does not change even with changes in the surrounding environment. The marker detection sensor 50 may be configured with a plurality of sensor modules to generate accurate position coordinates of the marker 48 .

In addition, the marker detection sensor 50 may recognize position information of the reference marker 49 disposed outside the body of the patient 10 . Here, the reference marker 49 may be any object disposed outside the body of the patient 10 , and for example, may be composed of a specific part of the endoscope 40 , a handle, a scope, and the like. The reference marker 49 may be a reference when rendering an augmented reality image or a virtual scope on the AR glasses 22 . Furthermore, the marker detection sensor 50 may recognize the location information of the AR glasses 22 . A position sensing module may be separately installed in the reference marker 49 and the AR glass 22 . As such, the marker detection sensor 50 may define a relative positional relationship between the marker 48 , the reference marker 49 , and the AR glass 22 based on the first coordinate system. In this embodiment, one marker detection sensor 50 is described as recognizing the position information of the marker 48, the reference marker 49, and the AR glass 22, but the present invention is not limited thereto, and the marker detection sensor 50 is composed of a plurality of sensor modules, each of which may be integrated after recognizing the position information of the marker 48, the reference marker 49, and the AR glasses 22 individually.

The AR glass 22 is a device in the form of glasses worn on the face of the operator 20 and may include a camera unit for photographing an object and a display unit for displaying an augmented reality image. The display unit is disposed between the operator's 20's eye and the object, and may be implemented in any form that can simultaneously recognize the augmented reality image while the operator 20 looks at the object. For example, when the display unit is composed of a spectacle lens and the augmented reality image is displayed on the lens, the operator 20 may recognize the augmented reality image while viewing the object through the lens.

The AR glasses 22 may photograph the reference marker 49 through the camera unit and transmit the image information to the image processing device 60 . The AR glasses 22 may include an acceleration sensor, a gyro sensor, a motion sensor, a geomagnetic sensor, and the like to transmit information about a gaze direction of the AR glasses 22 to the image processing device 60 .

The image processing device 60 includes the actual position coordinates (x, y, z) of the marker 48 recognized from the marker detection sensor 50 based on the first coordinate system, and the reference marker 49 and the AR glass 22 receive location information.

The image processing device 60 sets the second coordinate system (X', Y', Z') based on the AR glass 22 , and preferably sets the second coordinate system based on the gaze direction of the AR glass 22 . can be defined Unlike the first coordinate system, the second coordinate system is preferably a variable coordinate system that is changed according to the gaze direction of the AR glass 22 . The image processing apparatus 60 converts the actual position coordinates (x, y, z) of the marker 48 into virtual position coordinates (x', y', z') on the second coordinate system. The image processing apparatus 60 may calculate virtual position coordinates (x', y', z') using the relative position between the marker 48 and the AR glass 22 and the gaze direction of the AR glass 22 . . The image processing apparatus 60 generates a virtual scope 142 corresponding to the scope 42 using the virtual position coordinates (x', y', z') and renders it on the AR glass 22 .

The AR endoscope system 100 of the present invention can selectively use the reference marker 49 so that the virtual scope 142 expressed in augmented reality is displayed at the correct position on the AR glasses 22 . The image processing apparatus 60 calculates the display position of the reference marker 49 on the AR glass 22 by using the image information of the reference marker 49 photographed from the AR glass 22 . For example, a plurality of cameras are installed on the AR glass 22 , and while these cameras shoot the reference marker 49 , the distance between the AR glass 22 and the reference marker 49 , and the AR glass 22 . By using information about the direction in which the reference marker 49 is located from the gaze direction, the position at which the reference marker 49 is displayed on the AR glass 22 may be calculated. Subsequently, the image processing device 60 reflects the relative positions of the marker 48 and the reference marker 49 to render the virtual scope 142 based on the display position of the reference marker 49 on the AR glass 22 . can

When a lesion is found while the operator performs monitoring using the endoscope, the image processing apparatus 60 may store shape information of the virtual scope 142 at the location where the lesion is found. The stored information on the formation of the virtual scope 142 may be used as location information of the lesion when the tumor is to be removed by a later procedure or surgery.

Hereinafter, an AR endoscope monitoring method according to an embodiment of the present invention will be described with reference to FIG. 5 . 5 is a flowchart illustrating an AR endoscope monitoring method according to an embodiment of the present invention.

When the operator 20 inserts the endoscope 40, particularly the scope 42, into the body of the patient 10 while the patient 10 is lying on the bed 12, the marker detection sensor 50 is the first coordinate system. The actual position coordinates (x, y, z) of the marker 48 are recognized based on (X, Y, Z) (S10). In addition, the marker detection sensor 50 may also recognize the location information of the reference marker 49 and the AR glasses 22 together.

The image processing apparatus 60 sets the second coordinate system X', Y', Z' based on the AR glass 22, preferably based on the gaze direction of the AR glass 22 (S20). The image processing apparatus 60 converts the actual position coordinates (x, y, z) of the marker 48 into virtual position coordinates (x', y', z') on the second coordinate system (S30). At this time, the image processing apparatus 60 calculates virtual position coordinates (x', y', z') using the relative position between the marker 48 and the AR glass 22 and the gaze direction of the AR glass 22 . can do.

Then, the image processing apparatus 60 generates a virtual scope 142 corresponding to the scope 42 using the virtual position coordinates (x', y', z') and renders it on the AR glass 22 ( S40). At this time, the reference marker 49 may be selectively used so that the virtual scope 142 is displayed at an accurate position on the AR glasses 22 . That is, the image processing apparatus 60 may calculate the display position of the reference marker 49 on the AR glass 22 using image information of the reference marker 49 photographed from the AR glass 22 . Subsequently, the image processing device 60 reflects the relative positions of the marker 48 and the reference marker 49 to render the virtual scope 142 based on the display position of the reference marker 49 on the AR glass 22 . have.

Although the embodiments of the present invention have been described above with reference to the accompanying drawings, those of ordinary skill in the art to which the present invention pertains can realize that the present invention can be embodied in other specific forms without changing its technical spirit or essential features. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

10: patient
12: Couch
20: operator
22: AR glass
30: monitor
40: endoscope
42: scope
44: handle part
46: cable
48: marker
49: reference marker
50: marker detection sensor
60: image processing unit
100: AR endoscope system
142: virtual scope

Claims (5)

An augmented reality (AR) endoscope system using an endoscope in which a plurality of markers are formed on a scope, comprising:
a marker detection sensor for recognizing the actual position coordinates of the marker based on a first coordinate system that is a fixed coordinate system while the scope is inserted into the patient's body; and
Set a second coordinate system that is a variable coordinate system based on the gaze direction of the AR glasses worn by the operator, convert the actual position coordinates into virtual position coordinates on the second coordinate system, and use the virtual position coordinates to correspond to the scope After generating a virtual scope to include an image processing device for rendering on the AR glass,
The marker detection sensor recognizes the location information of the AR glasses to define a relative position between the marker and the AR glasses,
The image processing device calculates the virtual position coordinates using the relative position between the marker and the AR glass and the gaze direction of the AR glass,
The virtual scope is displayed on the lens so that the operator can overlap the virtual scope with the patient while looking at the patient through the lens of the AR glass,
When a lesion is found during monitoring, the image processing device stores shape information of the virtual scope at the location where the lesion is found and uses it as location information of the lesion. delete According to claim 1,
the marker detection sensor recognizes location information of a reference marker disposed outside the patient's body; The image processing device calculates a display position of the reference marker on the AR glass using the image information of the reference marker photographed from the AR glass; AR endoscope system, characterized in that the image processing device renders the virtual scope by reflecting the relative positions of the marker and the reference marker based on the display position of the reference marker on the AR glass. delete An augmented reality (AR) endoscope monitoring method using an endoscope in which a plurality of markers are formed on a scope, comprising:
Recognizing the actual position coordinates of the marker based on a first coordinate system that is a fixed coordinate system while the scope is inserted into the body of the patient by the marker detection sensor; and
The image processing apparatus sets a second coordinate system that is a variable coordinate system based on the gaze direction of the AR glasses worn by the operator, converts the actual position coordinates into virtual position coordinates on the second coordinate system, and uses the virtual position coordinates After generating a virtual scope corresponding to the scope, comprising the step of rendering on the AR glass,
The marker detection sensor recognizes the location information of the AR glasses to define a relative position between the marker and the AR glasses,
The image processing device calculates the virtual position coordinates using the relative position between the marker and the AR glass and the gaze direction of the AR glass,
The virtual scope is displayed on the lens so that the operator can overlap the virtual scope with the patient while looking at the patient through the lens of the AR glass,
When a lesion is found during monitoring, the image processing device stores shape information of the virtual scope at the location where the lesion is found and uses it as location information of the lesion.

Images