WO2016190607A1

WO2016190607A1 - Smart glasses system for providing surgery assisting image and method for providing surgery assisting image by using smart glasses

Info

Publication number: WO2016190607A1
Application number: PCT/KR2016/005312
Authority: WO
Inventors: 김법민; 김현구; 김민지; 오유진; 허정석
Original assignee: 고려대학교 산학협력단
Priority date: 2015-05-22
Filing date: 2016-05-19
Publication date: 2016-12-01

Abstract

The present invention relates to a smart glasses system for providing a surgery assisting image and a method for providing the surgery assisting image by using smart glasses. The smart glasses system according to the present invention comprises: an image processing module; smart glasses having a transparent display unit for displaying an image, a visible ray camera, and a glasses communication unit for transmitting, to the image processing module through wireless communication, a visible ray image photographed by the visible ray camera; and a near infrared ray photographing module having a near infrared ray camera, and a module communication unit for transmitting a near infrared ray fluorescent image captured by the near infrared ray camera module to the image processing module, wherein: the image processing module generates a real-time converted fluorescent image by converting, in real time, the near infrared ray fluorescent image on the basis of the visible ray image by means of one of a photographing direction and a size of the visible ray image, and transmits the real-time converted fluorescent image to the smart glasses; and the transparent display unit of the smart glasses displays the real-time converted fluorescent image received through the glasses communication unit such that the real-time converted fluorescent image can be overlapped with the view of an operating surgeon wearing the smart glasses. Therefore, the operating surgeon can proceed with surgery while visually checking, in real time, a resection site, including a sentinel lymph node, to which cancer cells have spread, and an effect of a near infrared ray fluorescent image appearing to be displayed to an actual surgical site of a patient is provided, thereby enabling the operating surgeon to concentrate the operating surgeon's own view only on a surgical site of a patient and proceed with surgery while checking the surgical site.

Description

Smart Glass System Providing Surgical Support Image and Surgical Support Image Provision Method Using Smart Glass

The present invention relates to a smart glass system for providing a surgical support image and a method for providing a surgical support image using a smart glass, and more particularly, to provide a surgical support image that allows a surgeon to more accurately identify a surgical site through a fluorescent image. The present invention relates to a method for providing a surgical support image using a smart glass system and smart glass.

Sentinel lymph node (SLN) is a lymph node in which cancer cells preferentially metastasize in the primary tumor and is an important indicator for determining lymph node metastasis. Therefore, if cancer cells are not found through the biopsy of the monitored lymph nodes, other lymph nodes are considered to have no metastases and no further surgery is performed.

In-vivo examination of lymph node through accurate search of the lymph node, which is an important indicator in determining the metastasis of cancer, can reduce postoperative sequelae such as lymphedema and minimize scarring on the patient's body. . As a result, surveillance lymph nodes using targeted drugs have been used as a standard technique in early breast cancer or melanoma surgery.

In order to detect the lymph nodes in the body by using a target drug, a method of obtaining a visible light image using a blue dye and a visible light camera, and a near infrared fluorescence image using a near infrared fluorescent dye and a near infrared camera A method of obtaining and a method of obtaining a radiographic image by using a gamma imaging apparatus of radiopharmaceuticals integrated in a surveillance lymph node have been proposed.

Recently, with the approval of IOG (Indocyanine Green) among the near-infrared fluorescent dyes, the exploration of surveillance lymph nodes using near-infrared fluorescent dyes has been prepared for clinical use.

On the other hand, when performing the operation to remove the actual tumor with the exact search for the lymph nodes, the surgeon sees the surgical site of the actual patient and the surgical lymph nodes and the surgical site of the actual patient Matching can determine the extent to be excised.

This is a problem that the cancer cells are metastasized after surgery and some of the remaining cancer cells are not resected until the metastasis of the monitored lymph nodes, and reoperation is performed. Due to the problem of deteriorating the quality of life of the patient, not only the precise search of the lymph nodes, but also the correct resection during surgery.

Therefore, a method of injecting near-infrared fluorescent dye (Fluorescent dye) into cancer cells during surgery, photographing the surgical site with a visible light camera and a near-infrared fluorescent camera, matching the visible and near-infrared images and displaying them on a monitor installed in the operating room has been proposed. .

However, as shown in FIG. 1, the surgeon checks the fluorescence image on the monitor installed in the operating room with the eyes, and proceeds to the operation while looking at the operation site of the patient lying on the operating table. There is discomfort to operate. In particular, when looking at the actual surgical site of the patient there is a limit that can not be made precisely because the fluorescent portion is not identified.

In order to solve such a problem, in Korean Patent Registration No. 10-1355348, in the 'guided surgery imaging system and method thereof', the gynecologist wears an affected image of a patient photographed by CT, MRI, and X-ray previously photographed. The present invention discloses a technique for displaying surgery on a transparent display in a shape and looking at the affected part of the gynecologist and the actual affected part passing through the transparent display.

The method disclosed in the Korean Patent Registration uses a gyro sensor or converts the affected image to a specific point of the patient in order to match the pre-photographed affected image and the actual affected part seen through the transparent display.

However, when detecting the movement of the gyros using a gyro sensor or the like, there is a disadvantage that the affected image cannot be exactly matched to the actual image because it cannot reflect the movement of the patient.

In addition, in the case of using the pre-recorded affected image, if the position change of the patient's surgical site, i.e., moving the organs, etc. during the operation, the affected image does not exactly match the actual affected area, which is why Rather, it may interfere with recognition.

Accordingly, the present invention has been made to solve the above problems, a smart glass that provides a surgical support image that can proceed with the operation while visually screening the resection of cancer cells metastasized including the monitoring lymph nodes in real time It is an object of the present invention to provide a method for providing a surgical support image using a system and smart glasses.

In addition, the near-infrared fluorescence image provides the same effect as the actual surgical site of the patient by providing a surgeon to provide a surgical support image that allows the surgeon to focus the field of vision only on the patient's surgical site to check the surgical site Another object is to provide a surgical support image providing method using a glass system and a smart glass.

According to the present invention, in the smart glass system for providing a surgical support image, wireless communication between the image processing module, the transparent display unit to display the image, the visible light camera, the visible light image captured by the visible light camera And a near-infrared photographing module having a smart glass having a glass communication unit for transmitting to the image processing module through the camera, a near-infrared camera, and a module communication unit for transmitting the near-infrared fluorescent image photographed by the near-infrared camera module to the image processing module. ; The image processing module generates a real-time converted fluorescence image by converting the near-infrared fluorescence image to at least one of a photographing direction and a size of the visible light image based on the visible light image, and transmits the real-time converted fluorescence image to the smart glass. To; The real-time conversion fluorescence image received through the glass communication unit is displayed on the transparent display unit of the smart glass, the operation support image, characterized in that the real-time conversion fluorescence image overlaps the field of view of the house wearing the smart glass. Achieved by a smart glass system.

Wherein at least three color fluorescent markers are arranged in a geometric structure at a position that can be photographed by the near infrared camera and the visible light camera around the surgical site of the patient; The image processing module extracts the color fluorescent markers from the visible light image and the near infrared fluorescence image, respectively; The real-time conversion fluorescence image may be generated by converting the near infrared fluorescence image so that the position of the color fluorescent marker extracted from the near infrared fluorescence image overlaps with the position of the color fluorescent marker extracted from the visible light image.

In addition, the near-infrared imaging module may be provided in the form of a head mount that can be worn on the head of the jikdo.

The near-infrared imaging module may further include a near-infrared light source for irradiating near-infrared light to enable imaging of the near-infrared fluorescent image by the near-infrared camera.

In addition, the visible light camera may be installed on the smart glass so as to photograph the position corresponding to the gaze of the house when wearing the smart glass.

On the other hand, the above object according to another embodiment of the present invention, in a method for providing a surgical support image using a smart glass having a transparent display unit, a visible light camera and a glass communication unit, the image is displayed, (a) to the visible light camera Photographing the visible light image in real time; (b) transmitting the visible light image to the image processing module through the glass communication unit; (c) photographing a near infrared fluorescence image by a near infrared camera; (d) transmitting the near infrared acid fluorescence image to the image processing module; (e) converting the near-infrared fluorescent image in real time to at least one of a direction and a size of the visible light image based on the visible light image in the image processing module to generate a real-time converted fluorescent image; (f) transmitting the real-time converted fluorescence image from the image processing module to the smart glass; and (g) displaying the real-time converted fluorescent image received through the glass communication unit on the transparent display unit of the smart glass.

Wherein at least three color fluorescent markers are arranged in a geometric structure at a position that can be photographed by the near infrared camera and the visible light camera around the surgical site of the patient; Step (e) comprises: (e1) extracting the color fluorescent markers from the visible light image and the near infrared fluorescence image, respectively; (e2) generating the real-time converted fluorescence image by converting the near infrared fluorescence image so that the position of the color fluorescent marker extracted from the near infrared fluorescence image overlaps with the position of the color fluorescent marker extracted from the visible light image can do.

In addition, the near-infrared camera may be provided in the form of a head mount that can be worn on the head of the gypsy.

According to another embodiment of the present invention, in the smart glass system for providing a surgical support image, the image processing module, the transparent display unit for displaying the image, the near-infrared camera, and the image taken by the near-infrared camera module Smart glass having a glass communication unit for transmitting the near-infrared fluorescent image to the image processing module; The image processing module generates a real-time converted fluorescent image by real-time converting the size of the near-infrared fluorescent image transmitted from the smart glass based on pre-registered reference information, and transmits the real-time converted fluorescent image to the smart glass; The real-time conversion fluorescence image transmitted from the glass communication unit is displayed on the transparent display unit of the smart glass, and the real-time conversion fluorescence image overlaps the field of view of the house wearing the smart glass. It is achieved by providing a smart glass system.

Here, a fluorescent marker that can be photographed by the near infrared camera is disposed at a position that can be photographed by the near infrared camera around a surgical site of a patient; The image processing module may generate the real-time converted fluorescence image by extracting the fluorescent marker from the near-infrared fluorescent image, comparing the extracted fluorescent marker with the reference information in real time to convert the size of the near-infrared fluorescent image.

In addition, the fluorescent markers are at least three or more arranged in a geometric structure; The reference information includes at least two reference marker information for the fluorescent markers respectively extracted from an image photographed at different distances from the fluorescent markers; The image processing module detects a distance from the near infrared camera by comparing the change in the geometry of the fluorescent marker extracted from the near infrared fluorescence image with the reference marker information, and detects the distance from the near infrared camera based on the distance from the near infrared camera. The size of the image can be converted in real time.

The smart glass may further include a near-infrared light source for irradiating near-infrared light to enable imaging of the near-infrared fluorescent image by the near-infrared camera.

In addition, the near-infrared camera may be installed on the smart glass so as to photograph the position corresponding to the gaze of the house when wearing the smart glass.

On the other hand, the above object is, according to another embodiment of the present invention, in a method for providing a surgical support image using a smart display having a transparent display unit, a near-infrared camera and a glass communication unit, the image is (a) the reference information is an image Registering with a processing module; (b) photographing a near infrared fluorescence image by the near infrared camera; (c) transmitting the near-infrared fluorescent image from the smart glass to the image processing module; (d) generating, by the image processing module, a real-time converted fluorescence image by converting the size of the near infrared fluorescence image based on the reference information; (e) transmitting the real-time converted fluorescence image from the image processing module to the smart glass; and (f) displaying the real-time converted fluorescence image received through the glass communication unit on the transparent display unit of the smart glass.

Here, a fluorescent marker that can be photographed by the near infrared camera is disposed at a position that can be photographed by the near infrared camera around a surgical site of a patient; The step (d) may include (d1) extracting the fluorescent marker from the near infrared fluorescence image, and (d2) comparing the fluorescent marker extracted from the step (d1) with the reference information to determine the size of the near infrared fluorescence image. It may include the step of generating a real-time conversion fluorescence image by real-time conversion.

And at least three or more fluorescent markers are arranged in a geometric structure; Step (a) is (a1) the at least two or more fluorescent markers are photographed at different distances from the fluorescent marker by the near-infrared camera, and (a2) from two or more images captured in the (a1) step, respectively. At least two reference marker information for the extracted fluorescent marker is registered as the reference information; In the step (d2), (d21) comparing the reference marker information with the change in the geometry of the fluorescent marker extracted in the step (d1) and detecting the distance from the near infrared camera, (d22) the ( and converting the size of the near-infrared fluorescent image in real time based on the distance from the near-infrared camera detected in step d22) to generate the real-time converted fluorescent image.

According to the present invention according to the configuration as described above, using the smart glass system and smart glass to provide a surgical support image that can proceed with the operation while visually screening the resection of cancer cells metastasized including the monitoring lymph nodes in real time Provided is a method for providing a surgical support image.

In addition, the near-infrared fluorescence image provides the same effect as that displayed on the actual surgical site of the patient, the surgeon can proceed to the operation while confirming the surgical site by concentrating his vision only on the surgical site of the patient.

1 is a view showing an example of a surgical environment of a conventional operating room,

2 is a view showing the configuration of a smart glass system for providing a surgical support image according to a first embodiment of the present invention,

3 is a view showing an example of the configuration of a smart glass according to the first embodiment of the present invention,

4 is a diagram illustrating an example of a configuration of an image processing module according to a first embodiment of the present invention.

5 and 6 are views for explaining a surgical support image providing method using a smart glass according to a first embodiment of the present invention,

FIG. 7 is a diagram illustrating an example of an actual image provided by a method for providing a surgical support image using smart glasses according to a first embodiment of the present invention.

8 is a diagram showing the configuration of a smart glass system for providing a surgical support image according to a second embodiment of the present invention,

9 is a view showing an example of the configuration of a smart glass according to a second embodiment of the present invention,

10 is a diagram illustrating an example of a configuration of an image processing module according to a second embodiment of the present invention.

11 and 12 are views for explaining a surgical support image providing method using a smart glass according to a second embodiment of the present invention.

The present invention relates to a smart glass system for providing a surgical support image and a method for providing a surgical support image using a smart glass. The smart glass system for providing a surgical support image according to the present invention comprises an image processing module, a transparent display unit displaying an image, a visible light camera, and a visible light image captured by the visible light camera through the image processing module. And a near-infrared photographing module having a smart glass having a glass communication unit for transmitting a light source, a near-infrared camera, and a module communication unit for transmitting a near-infrared fluorescent image photographed by the near-infrared camera module to the image processing module; The image processing module generates a real-time converted fluorescence image by converting the near-infrared fluorescence image to at least one of a photographing direction and a size of the visible light image based on the visible light image, and transmits the real-time converted fluorescence image to the smart glass. To; The real-time conversion fluorescence image received through the glass communication unit is displayed on the transparent display unit of the smart glass, characterized in that the real-time conversion fluorescence image overlaps the field of view of the house wearing the smart glass.

Hereinafter, with reference to the accompanying drawings will be described embodiments of the present invention;

2 is a view showing the configuration of a smart glass system for providing a surgical support image according to a first embodiment of the present invention. As shown in FIG. 2, the smart glass system according to the first embodiment of the present invention includes a smart glass 100, a near infrared ray imaging module 200, and an image processing module 310.

As shown in FIG. 1, the smart glass 100 is provided in the form of glasses that can be worn by a doctor. That is, the smart glass 100 may include a glass frame 150 that forms a skeleton and a lens 160 that is fixed to the glass frame 150 to be positioned in front of a human eye when the glass frame 150 is worn. .

In addition, as shown in FIG. 3, the smart glass 100 includes a transparent display unit 110, a visible light camera 120, and a glass communication unit 140. In addition, the smart glass 100 may include a glass control unit 130 for controlling the re-function of the transparent display unit 110, visible light camera 120 and the glass communication unit 140.

The transparent display unit 110 is made of a transparent material to display an image on the surface. In the first exemplary embodiment of the present invention, the transparent display unit 110 includes the entire lens 160 constituting the smart glass 100. For example, the transparent display unit 110 may be formed only in a part of the lens 160. Of course, it can be prepared.

According to the configuration of the transparent display unit 110, even if the wearer wears the smart glass 100 or the image is displayed on the transparent display unit 110, the actual image coming through the transparent display unit 110, that is, The actual image viewed by the eye can be viewed together with the image displayed on the transparent display unit 110. That is, when an image is displayed on the transparent display unit 110, an image displayed on the transparent display unit 110 may be overlapped with an actual image to obtain an effect.

The visible light camera 120 is installed in the smart glass 100 to take a visible light image. In the first exemplary embodiment of the present invention, as shown in FIG. 2, the visible light camera 120 is installed in the glass frame 150 of the smart glass 100. It is installed to take a visible light image in the same direction as the eye gaze direction. Through this, in the case of the visible light image captured by the visible light camera 120 is matched with the actual image viewed by the chief physician, a description of the effect will be described later.

The glass communication unit 140 is connected to the image processing module 310 through wireless communication. In the first embodiment of the present invention, the glass communication unit 140 communicates with the image processing module 310 through TCP / IP-based Wi-Fi communication or Bluetooth communication, and other wireless communication is applicable. Of course.

According to the above configuration, the glass controller 130 transmits the visible light image captured by the visible light camera 120 to the image processing module 310 in real time through the glass communication unit 140, the image processing module 310 The real-time conversion fluorescence image transmitted from the glass communication unit 140 and received through the transparent display unit 110 will be described, which will be described later.

Meanwhile, as shown in FIG. 3, the NIR module 200 may include a NIR camera 210 and a module communicator 230.

The near infrared camera 210 photographs a near infrared fluorescence image of the surgical site of the patient. The module communicator 230 transmits the near infrared fluorescence image captured by the near infrared camera 210 to the image processing module 310.

In the first exemplary embodiment of the present invention, as shown in FIG. 2, the near-infrared imaging module 200 is provided in the form of a head mount that can be worn on the head. Accordingly, the gynecologist wears the smart glass 100 in the form of glasses, and proceeds to surgery while wearing the head-mounted near infrared imaging module 200 on the head.

In addition, as shown in Figures 2 and 3, the near-infrared light source 220 for irradiating near-infrared to enable the imaging of the near-infrared fluorescent image by the near-infrared camera 210 is installed in the near-infrared imaging module 200 of the head-mount type For example,

2 and 3 illustrate that the smart glass 100 and the near-infrared photographing module 200 are configured as independent devices. In addition, the near-infrared imaging module 200 and the smart glass 100 in the form of a head mount may be provided in one device form. At this time, as shown in FIG. 3, the glass communication unit 140 and the module communication unit 230, which were independently configured, are integrated into one communication module 400 and have a near infrared fluorescent shape captured by the near infrared camera 210. Under the control of the glass control unit 130 may be provided to be transmitted to the image processing module 310 through one communication module 400. In addition, the on / off of the near infrared light source 220 may also be provided to operate under the control of the glass controller 130.

Meanwhile, the image processing module 310 processes the visible light image transmitted from the smart glass 100 and the near infrared fluorescence image transmitted from the near infrared imaging module 200 to display on the transparent display unit 110 of the smart glass 100. Generate real-time converted fluorescence images to be.

In the first exemplary embodiment of the present invention, as shown in FIG. 2, the image processing module 310 and the field display unit 320 are configured as an information processing apparatus 300 such as a computer. 4 is a diagram illustrating an example of a configuration of an image processing module 310 according to a first embodiment of the present invention. As illustrated in FIG. 4, the image processing module 310 may include a first communication unit 311, a second communication unit 312, an image matching unit 314, and a main control unit 313.

The first communication unit 311 is connected to the glass communication unit 140 of the smart glass 100 through wireless communication to receive a visible light image transmitted from the smart glass 100, the real-time conversion fluorescence image to the smart glass 100 send. The second communication unit 312 is connected to the module communication unit 230 of the near infrared imaging module 200 to receive a near infrared fluorescence image transmitted from the near infrared imaging module 200.

Here, as described above, when the smart glass 100 and the near-infrared imaging module 200 are provided in the form of an independent device, the first communication unit 311 and the second communication unit 312 are glass of the smart glass 100. The communication unit 140 and the near infrared imaging module 200 may be provided in accordance with the communication method of the module communication unit 230, respectively. For example, the first communication unit 311 may be connected to the glass communication unit 140 through wireless communication as described above, and the second communication unit 312 may be connected to the module communication unit 230 through wired communication or wireless communication. Can be.

In addition, even if the glass communication unit 140 of the smart glass 100 and the module communication unit 230 of the near-infrared photographing module 200 are provided as one communication module 400 or independently provided as shown in FIG. 3. When using the same communication scheme, the first communication unit 311 and the second communication unit 312 may be provided as one communication module.

Referring to FIG. 4 again, the image matching unit 314 may display the near-infrared fluorescent image received through the second communication unit 312 based on the visible light image received through the first communication unit 311. A real-time converted fluorescent image is generated by converting at least one of the photographing direction and the size in real time.

As described above, the visible light camera 120 is configured to photograph the visible light image in the same direction of view as the eye direction of the house of the house, so that the visible image captured by the visible camera 120 may be matched with the actual image viewed by the house of the house. have.

On the other hand, in the case of the near-infrared camera 210, the installation position is different so that the distance between the image photographed by the visible light camera 120 and its direction or size, that is, the surgical site. Therefore, in order to match the near-infrared fluorescence shape photographed by the near-infrared camera 210 to the gaze of the gimmick, the near-infrared fluorescence image is converted in accordance with the visible light image corresponding to the gaze of the gimmick.

In the first exemplary embodiment of the present invention, at least three color fluorescent markers are disposed in a geometric structure at a position that can be photographed by the near infrared camera 210 and the visible light camera 120 around the surgical site of the patient, and the image processing module is used therein. An example in which the image matching unit 314 of 310 generates a real-time converted fluorescence image by converting a near infrared fluorescence image will be described below in more detail with reference to FIGS. 5 to 7.

First, when the operation is in progress, as described above, the visible light image is photographed by the visible light camera 120 (S10), the near infrared fluorescence image is photographed by the near infrared camera 210 (S30). The visible light image photographed by the visible light camera 120 and the near infrared fluorescence image photographed by the near infrared camera 210 are transmitted to the image processing module 310 (S11 and S31).

The image matching unit 314 of the image processing module 310 extracts a color fluorescent marker from the visible light image and the near infrared fluorescence image (S20). The color fluorescent marker is made of a color fluorescent material, and the visible light image extracts the position of the color fluorescent marker based on the color of the fluorescent color marker. In the near infrared fluorescent image, the position of the color fluorescent marker is determined based on the fluorescent material of the fluorescent color marker. Will be extracted.

In the first embodiment of the present invention, as shown in FIG. 6, three color fluorescent markers are arranged in a triangular geometric structure around the patient's surgery, and FIG. 6A is extracted from the visible light image. The position of the color fluorescent marker M1 is shown, and FIG. 6B shows the position of the color fluorescent marker M2 extracted from the near infrared fluorescence image.

As shown in FIG. 6, as the positions of the visible light camera 120 and the near infrared fluorescence camera are different, the color fluorescent markers M1 and M2 in the visible light image and the near infrared fluorescence image do not coincide with each other. The near-infrared fluorescent image is converted so that the position of the color fluorescent marker M2 extracted from the image overlaps the position of the color fluorescent marker M1 extracted from the visible light image.

For example, when the near-infrared fluorescent image shown in FIG. 6B is rotated by a counterclockwise angle, the color fluorescent marker M1 extracted from the near-infrared fluorescent image is extracted from the visible light image. ) Will match.

Here, the conversion of the near-infrared fluorescent image may be rotated about three axes, or may be converted in size, and may be converted to three axes. This can be calculated through the geometric relationship between the color fluorescent markers, that is, the geometric relationship such as distance and angle.

As described above, when the near-infrared fluorescence image is converted, a real-time converted fluorescence image that matches the line of sight of the focus is generated (S22). In this case, the real-time conversion fluorescence image may go through various types of image processing processes known in advance so that the near-infrared fluorescence image of the original form can be easily visually recognized by the gynecologist.

The generated real-time converted fluorescence image as described above is transmitted to the smart glass 100 by the main control unit 313 through the first communication unit 311 (S23), the glass control unit 130 of the smart glass 100 is glass The real-time conversion fluorescent image received through the communication unit 140 is displayed on the transparent display unit 110 (S12).

When displayed on the transparent display unit 110 of the real-time conversion fluorescence image as described above, by looking at the real-time conversion fluorescence image displayed on the transparent display unit 110 to see the surgical site of the actual patient over the transparent display unit 110 By visually recognizing that the real-time converted fluorescence image displayed on the transparent display unit 110 is overlapped on the actual surgical site of the patient, the same effect as that of fluorescent colored material is labeled on the actual surgical site of the patient is obtained. do.

FIG. 7A illustrates a surgical site of a patient who is visible when a real-time conversion fluorescence image is not displayed on the transparent display unit 110, and FIG. 7B is displayed on the transparent display unit 110. It is a real-time conversion fluorescence image, Figure 7 (c) is a surgical site of the patient visible to the eye of the patient in the state in which the real-time conversion fluorescence image is displayed on the transparent display 110. As shown in (c) of FIG. 7, the surgeon is able to proceed with the operation as if the fluorescent material is actually displayed on the surgical site of the patient, while performing the operation while watching the screen of the monitor provided in the operating room. The inconvenience of coming out of the way can be eliminated.

In addition, even if the gimmick moves during surgery, the gaze of the gimmick that changes according to the movement of the gimmick is corrected based on the visible light image taken by the visible light camera 120, thereby enabling more accurate display of the near infrared fluorescence image. do.

Here, the main control unit 313 matches the real-time converted fluorescence image generated in step S22 and the visible light image with each other and displays them on the field display unit 320 installed at the surgery site, whereby other personnel or assistants at the surgery site are additionally added. It may be prepared to check this.

In addition, in the above-described embodiments, the near-infrared light source 220 is installed in the near-infrared imaging module 200 of the head mounted type as shown in FIG. 2, but the outside of the near-infrared imaging module 200 is an example. For example, it may be installed in a specific space inside the operating room.

Hereinafter, a smart glass system and a method for providing a surgical support image for providing a surgical support image according to a second embodiment of the present invention will be described in detail with reference to FIGS. 8 to 12.

8 is a view showing the configuration of a smart glass system for providing a surgical support image according to a second embodiment of the present invention. As shown in FIG. 8, the smart glass system according to the second embodiment of the present invention includes a smart glass 1000 and an image processing module 3100.

As shown in FIG. 8, the smart glass 1000 is provided in the form of glasses that can be worn by a zip doctor. That is, the smart glass 1000 may include a glass frame 1500 that forms a skeleton and a lens 1600 that is fixed to the glass frame 1500 to be positioned in front of the human eye when the glass frame 1500 is worn. .

In addition, as illustrated in FIG. 9, the smart glass 1000 includes a transparent display unit 1100, a near infrared camera, and a glass communication unit 1400. Here, the smart glass 1000 may include a glass control unit 1300 for controlling the functions of the transparent display unit 1100, the near infrared camera, and the glass communication unit 1400.

The transparent display unit 1100 is made of a transparent material and displays an image on the surface. According to the second embodiment of the present invention, the transparent display unit 1100 is composed of the entire lens 1600 constituting the smart glass 1000, but only a part of the lens 1600 may be provided with the transparent display. Of course.

According to the configuration of the transparent display unit 1100, even if the wearer wears the smart glass 1000 or the image is displayed on the transparent display unit 1100, the actual image passing through the transparent display unit 1100, that is, The actual image viewed by the eye of the house may be viewed together with the image displayed on the transparent display unit 1100. That is, when an image is displayed on the transparent display unit 1100, an image displayed on the transparent display unit 1100 overlaps an actual image, thereby obtaining an effect.

The near infrared camera is installed in the smart glass 1000 to take a near infrared fluorescence image of the surgical site of the patient. Here, the glass controller 1300 transmits the near infrared fluorescence image captured by the near infrared camera to the image processing module 3100 through the glass communication unit 1400.

In the second embodiment of the present invention, as shown in FIG. 8, it is assumed that the near-infrared camera is installed at a position capable of photographing a position corresponding to the gaze of the gimmick when the gimmick wears the smart glass 1000.

According to the above configuration, the glass control unit 1300 transmits the near-infrared fluorescent image captured by the near infrared camera in real time to the image processing module 3100 through the glass communication unit 1400, and transmits the image from the image processing module 3100. The real-time conversion fluorescent image to be described later received through the glass communication unit 1400 is displayed on the transparent display unit 1100, which will be described later.

Meanwhile, the image processing module 3100 processes the near-infrared fluorescent image transmitted from the smart glass 1000 to generate a real-time converted fluorescent image to be displayed on the transparent display unit 1100 of the smart glass 1000.

According to the second embodiment of the present invention, as shown in FIG. 8, the image processing module 3100 and the field display unit 3200 are configured as an information processing apparatus 3000 such as a computer. 10 is a diagram illustrating an example of a configuration of an image processing module 3100 according to a second embodiment of the present invention. Referring to FIG. 10, the image processing module 3100 may include a communication unit 3110, an image processing unit 3140, and a main control unit 3130.

The communication unit 3110 is connected to the glass communication unit 1400 of the smart glass 1000 through wireless communication to receive a near-infrared fluorescent image transmitted from the smart glass 1000, and transmits a real-time converted fluorescent image to the smart glass 1000. do. Here, the communication unit 3110 may be connected to the glass communication unit 1400 through wireless communication, for example, Wi-Fi or Bluetooth communication based on TCP / IP.

The image processor 3140 processes the near-infrared fluorescent image received through the glass communication unit 1400 to generate a real-time converted fluorescent image. The real-time converted fluorescence image generated by the image processor 3140 is transmitted to the smart glass 1000 through the glass communication unit 1400.

Here, the main controller 3130 controls the image processor 3140 such that the image processor 3140 generates a real-time converted fluorescent image based on pre-registered reference information. Hereinafter, the image processor 3140 may convert the real-time converted image. How to create.

The image processor 3140 generates a real-time converted fluorescence image by real-time converting at least one of the photographing direction and size of the near infrared fluorescence image transmitted from the smart glass 1000 based on the reference image.

As described above, the near-infrared camera is installed in the smart glass 1000 so as to correspond to the direction of eye gaze of the house, and according to the position of the near-infrared camera of this position, the near-infrared camera is in the same direction of gaze as the eye of the house of the house. The near-infrared fluorescence image is taken, and the near-infrared fluorescence image photographed by the near-infrared camera coincides with the direction of gaze and the actual image seen by the surgeon.

Accordingly, a real-time conversion fluorescence image is generated by adjusting the size of an image portion (hereinafter referred to as a 'fluorescent image') represented by the near infrared fluorescent dye in the near infrared fluorescence image photographed by the near infrared camera, which is determined by the smart glass 1000. When the display is displayed on the transparent display unit 1100, the real-time conversion image overlaps the field of view of the gyro, and the fluorescent material may be displayed on the surgical site of the patient.

Here, the image processing module 3100 according to the second embodiment of the present invention uses a fluorescent marker and pre-registered reference information to generate a real-time converted image.

The fluorescent marker is placed at a position that can be photographed by a near infrared camera around the surgical site of the patient. In the second embodiment of the present invention, as shown in Fig. 11, three fluorescent markers are arranged in a geometric structure, for example, a triangular structure.

The image processor 3140 extracts the fluorescent marker from the near infrared fluorescence image and compares the extracted fluorescent marker with reference information to convert the size of the near infrared fluorescence image, that is, the above-described fluorescent image in real time. Here, the reference information may include at least 8 reference marker information for the fluorescent markers extracted from the image, which is drawn at different distances from the fluorescent marker.

In more detail, when the surgeon photographs the fluorescent marker disposed around the surgical site of the patient at a close distance while wearing the smart class before surgery, the image processor 3140 is transmitted from the smart glass 1000. The fluorescent image is extracted from the image and transmitted to the smart plastic to be displayed on the transparent display unit 1100.

In this case, the fluorescence image is displayed in the state overlapping with the surgical site, and the gynecologist adjusts the size of the fluorescence image while checking whether the fluorescence image overlaps with the actual size at the current position, and finally the size and extraction of the finally determined fluorescence image The size of the fluorescent marker or the distance between the fluorescent markers is set as one reference marker information (hereinafter referred to as 'first reference marker information').

In addition, when the near infrared camera is photographed in a state where the gyro is located at a distance different from the distance photographed for generating the first reference marker information, for example, a distance further apart, the image processor 3140 may determine the first reference marker information. Another reference marker information (hereinafter referred to as 'second reference marker information') is generated and set in the same manner as the generation.

FIG. 12A illustrates an example of an image photographed at a close distance from the fluorescent marker, and FIG. 12B illustrates an example image captured at a farther distance from the fluorescent marker. As shown in FIG. 12, the size of the fluorescent marker in the image photographed at a close distance is increased or the distance between the fluorescent markers is increased, and the size of the fluorescent marker photographed at a long distance is decreased or the distance between the fluorescent markers is closer.

When the information on the fluorescent marker and the size of the fluorescent image in the corresponding information are determined and registered as reference information, the fluorescent marker is extracted from the near-infrared fluorescent image taken by the near-infrared camera during the surgical procedure, and the size of the fluorescent marker When the distance between the fluorescent markers is compared with the first reference marker information and the second reference marker information, the size of the target image, that is, the fluorescent image, in the near infrared fluorescent image may be determined. For example, when the distance between the fluorescent markers extracted from the near infrared fluorescence image is located within the range between the distance between the fluorescent markers in the first reference marker information and the second reference marker information, the fluorescent image in the near infrared fluorescence image is determined according to the ratio of the corresponding distance. Size can be adjusted linearly.

That is, the image processor 3140 compares the change in the geometrical structure of the fluorescent marker extracted from the near infrared fluorescence image, for example, the change in the size of the fluorescent marker, the change in the distance between the fluorescent markers, and the reference marker information. The distance to the site is detected, and through this, the size of the near-infrared fluorescent image is converted in real time to generate a canal tube converted fluorescent image.

Hereinafter, a method for providing a surgery support image using a smart glass 1000 according to a second embodiment of the present invention will be described with reference to FIGS. 11 and 7.

In the state where the gynecologist wears the smart glass 1000, the registration process of the reference information as described above is performed. In this case, as shown in FIG. 12, three fluorescent markers are disposed around the surgical site of the patient.

First, when the near-infrared camera of the smart glass 1000 captures an image to include the fluorescent marker at a first distance, for example, a distance close to the fluorescent marker (S50), the captured near-infrared image (hereinafter, 'first near-infrared image'). 'Is transmitted to the image processing module 3100 through the glass communication unit 1400. Here, the first near-infrared image photographed by the near-infrared camera may be one image, may be a real-time image, and is not limited to one image by the expression of near-infrared image.

Next, the image processor 3140 of the image processing module 3100 extracts an image portion indicated by a fluorescent marker and a fluorescent image, that is, a near infrared fluorescent dye from the first near infrared image (S61). In addition, the image processor 3140 may adjust information about the fluorescent marker, for example, information about a size or a distance between the markers and the size of the fluorescent image displayed on the transparent display unit 1100 as described above. First reference marker information including information on the size of the determined fluorescent image is set (S62).

Similarly, when the near infrared camera of the smart glass 1000 captures an image to include the fluorescent marker at a second distance, for example, a distance further from the fluorescent marker (S51), the captured near infrared image (hereinafter, 'second' The NIR image is transmitted to the image processing module 3100 through the glass communication unit 1400. Here, the second near infrared image captured by the near infrared camera may be a single image, may be a real time image, and is not limited to a single image by the expression of near infrared image.

Next, the image processor 3140 of the image processing module 3100 extracts an image portion indicated by a fluorescent marker and a fluorescent image, that is, a near infrared fluorescent dye, from the second near infrared image (S63). In addition, the image processor 3140 may adjust information about the fluorescent marker, for example, information about a size or a distance between the markers and the size of the fluorescent image displayed on the transparent display unit 1100 as described above. By setting second reference marker information including information on the size of the fluorescent image determined as (S64), the first reference marker information and the second reference marker information are registered as the reference information.

When the registration process of the above-mentioned reference information is completed, the near-infrared camera captures the near-infrared fluorescence image in real time in the process of performing the actual concentrating (S52), and the captured near-infrared fluorescence image is real-time image processing module 3100. Is sent to.

The image processor 3140 of the image processing module 3100 extracts a fluorescent marker and a fluorescence image displayed by a near infrared fluorescence image, that is, a near infrared fluorescence dye from the near infrared fluorescence image (S65), and the first reference marker information and By comparing the second reference marker information and the extracted fluorescent marker, the size of the fluorescent image is adjusted to convert the near-infrared fluorescent image (S66), thereby generating a real-time converted fluorescent image (S67).

As described above, the real-time conversion fluorescence image generated in real time is transmitted to the smart glass 1000, the real-time conversion fluorescence image is displayed on the transparent display unit 1100 of the smart glass 1000 (S53), so that the collection of the transparent display By looking at the real-time conversion fluorescence image displayed on the unit 1100, the real patient's surgery area beyond the transparent display unit 1100 is seen, whereby the real-time conversion fluorescence image displayed on the transparent display unit 1100 overlaps the actual operation site of the patient. By visual recognition, the same effect can be obtained as the colored fluorescent substance is labeled on the actual surgical site of the patient.

In addition, even if the surgeon moves during operation, even if the size of the fluorescent image varies depending on the distance between the surgeon and the surgical site, which is changed according to the movement of the surgeon, the near-infrared fluorescence image is corrected on the basis of the fluorescent marker, thereby more accurately displaying the near-infrared fluorescent image Becomes possible.

Although some embodiments of the invention have been shown and described, it will be apparent to those skilled in the art that modifications may be made to the embodiment without departing from the spirit or spirit of the invention. . It is intended that the scope of the invention be defined by the claims appended hereto and their equivalents.

[Description of the code]

100,1000: smart glass 110,1100: transparent display unit

120: visible light camera 130,1300: glass control unit

140,1400: glass communication unit 150,1500: glass frame

160,1600: Lens 200: NIR module

210: near infrared camera 220: near infrared light source

230: module communication unit 300, 3000: information processing device

310,3100: image processing module 311: first communication unit

312: second communication unit 3110: communication unit

313,3130: main control unit 3140: image processing unit

314: image matching unit 320, 3200: field display unit

The surgeon is applied to the field of providing surgical support images that can more accurately identify the surgical site through fluorescence images.

Claims

In the smart glass system that provides a surgical support image,

An image processing module,

A smart glass having a transparent display unit for displaying an image, a visible light camera, a glass communication unit for transmitting a visible light image captured by the visible light camera to the image processing module through wireless communication;

A near infrared imaging module having a near infrared camera and a module communication unit for transmitting a near infrared fluorescence image photographed by the near infrared camera module to the image processing module;

The image processing module

Converting the near-infrared fluorescent image into at least one of a direction and a size of the visible light image based on the visible light image in real time to generate a real-time converted fluorescent image, and transmitting the real-time converted fluorescent image to the smart glass;

The real-time conversion fluorescence image received through the glass communication unit is displayed on the transparent display unit of the smart glass, the operation support image, characterized in that the real-time conversion fluorescence image overlaps the field of view of the house wearing the smart glass. Providing smart glass system.
The method of claim 1,

At least three color fluorescent markers are arranged in a geometric structure at a position photographable by the near infrared camera and the visible light camera around a surgical site of a patient;

The image processing module

Extracting the color fluorescent markers from the visible light image and the near infrared fluorescence image, respectively;

And generating the real-time converted fluorescence image by converting the near infrared fluorescence image so that the position of the color fluorescent marker extracted from the near infrared fluorescence image overlaps with the position of the color fluorescent marker extracted from the visible light image. Smart glass system that provides video.
The method of claim 2,

The near-infrared imaging module is a smart glass system for providing a surgical support image, characterized in that the gynecologist is provided in the form of a head mount that can be worn on the head.
The method of claim 3,

The near-infrared photographing module further includes a near-infrared light source for irradiating near-infrared light to enable the near-infrared fluorescence image to be captured by the near-infrared camera.
The method of claim 1,

The visible light camera is a smart glass system for providing a surgical support image, characterized in that installed in the smart glass to be photographed the position corresponding to the gaze of the house when wearing the smart glass.
In the surgical support image providing method using a smart glass having a transparent display unit, an visible light camera and a glass communication unit, the image is displayed,

(a) photographing a visible light image in real time by the visible light camera;

(b) transmitting the visible light image to the image processing module through the glass communication unit;

(c) photographing a near infrared fluorescence image by a near infrared camera;

(d) transmitting the near infrared acid fluorescence image to the image processing module;

(e) converting the near-infrared fluorescent image in real time to at least one of a direction and a size of the visible light image based on the visible light image in the image processing module to generate a real-time converted fluorescent image;

(f) transmitting the real-time converted fluorescence image from the image processing module to the smart glass;

(g) displaying the real-time conversion fluorescence image received through the glass communication unit on the transparent display unit of the smart glass.
The method of claim 6,

At least three color fluorescent markers are arranged in a geometric structure at a position photographable by the near infrared camera and the visible light camera around a surgical site of a patient;

Step (e) is

(e1) extracting the color fluorescent markers from the visible light image and the near infrared fluorescence image, respectively;

(e2) generating the real-time converted fluorescence image by converting the near infrared fluorescence image so that the position of the color fluorescent marker extracted from the near infrared fluorescence image overlaps with the position of the color fluorescent marker extracted from the visible light image Surgical support image providing method using a smart glass, characterized in that.
The method of claim 7, wherein

The near-infrared camera is a surgical support image providing method using a smart glass, characterized in that the gynecologist is provided in the form of a head mount that can be worn on the head.
The method of claim 6,

The visible light camera is a surgical support image providing method using a smart glass, characterized in that installed in the smart glass to be photographed to the position corresponding to the gaze of the house when wearing the smart glass.
In the smart glass system that provides a surgical support image,

An image processing module,

A smart glass having a transparent display unit for displaying an image, a near infrared camera, and a glass communication unit configured to transmit a near infrared fluorescent image photographed by the near-infrared camera module to the image processing module;

The image processing module

Generating a real-time converted fluorescent image by real-time converting the size of the near-infrared fluorescent image transmitted from the smart glass based on pre-registered reference information, and transmitting the real-time converted fluorescent image to the smart glass;

The real-time conversion fluorescence image transmitted from the glass communication unit is displayed on the transparent display unit of the smart glass, and the real-time conversion fluorescence image overlaps the field of view of the house wearing the smart glass. Providing smart glass system.
The method of claim 1,

A fluorescent marker that can be photographed by the near infrared camera is disposed at a position that can be photographed by the near infrared camera around a surgical site of a patient;

The image processing module

Extracting the fluorescent marker from the near infrared fluorescence image,

And comparing the extracted fluorescent marker with the reference information to convert the size of the near-infrared fluorescence image in real time to generate the real-time converted fluorescence image.
The method of claim 2,

The fluorescent markers are arranged in at least three geometric structures;

The reference information includes at least two reference marker information for the fluorescent markers respectively extracted from an image photographed at different distances from the fluorescent markers;

The image processing module

Comparing the reference marker information with a change in the geometry of the fluorescent marker extracted from the near infrared fluorescence image to detect a distance from the near infrared camera,

Smart glass system for providing a surgical support image, characterized in that for converting the size of the near-infrared fluorescent image in real time based on the distance to the near-infrared camera.
The method of claim 3,

The smart glass further includes a near-infrared light source for irradiating near-infrared to enable the imaging of the near-infrared fluorescent image by the near-infrared camera.
The method of claim 1,

The near-infrared camera is a smart glass system for providing a surgical support image, characterized in that installed in the smart glass to be photographed at a position corresponding to the gaze of the house when wearing the smart glass.
In the surgical support image providing method using a smart glass having a transparent display unit, an near-infrared camera and a glass communication unit for displaying an image,

(a) registering the reference information with the image processing module;

(b) photographing a near infrared fluorescence image by the near infrared camera;

(c) transmitting the near-infrared fluorescent image from the smart glass to the image processing module;

(d) generating, by the image processing module, a real-time converted fluorescence image by converting the size of the near infrared fluorescence image based on the reference information;

(e) transmitting the real-time converted fluorescence image from the image processing module to the smart glass;

and (f) displaying the real-time converted fluorescence image received through the glass communication unit on the transparent display unit of the smart glass.
The method of claim 6,

A fluorescent marker that can be photographed by the near infrared camera is disposed at a position that can be photographed by the near infrared camera around a surgical site of a patient;

Step (d)

(d1) extracting the fluorescent marker from the near infrared fluorescence image;

(d2) comparing the fluorescent marker extracted in the step (d1) with the reference information to convert the size of the near-infrared fluorescent image in real time to generate the real-time converted fluorescent image. How to provide surgery support image.
The method of claim 7, wherein

The fluorescent markers are arranged in at least three geometric structures;

Step (a) is

(a1) photographing the at least two or more fluorescent markers at different distances from the fluorescent marker by the near infrared camera;

(a2) registering at least two reference marker information for the fluorescent marker, respectively, extracted from the two or more images captured in the step (a1) as the reference information;

Step (d2) is

(d21) comparing the change of the geometry of the fluorescent marker extracted in step (d1) with the reference marker information to detect a distance from the near infrared camera;

and (d22) real-time converting the size of the near-infrared fluorescent image based on the distance from the near-infrared camera detected in step (d22) to generate the real-time converted fluorescent image. How to provide surgery support video.
The method of claim 6,

The near-infrared camera is a surgical support image providing method using a smart glass, characterized in that is installed in the smart glass so that when photographing the position corresponding to the gaze of the house when wearing the smart glass.