KR102582154B1 - User terminal for providing augmented reality medical image using depth camera and method for providing augmented reality medical image - Google Patents

Abstract

The present invention relates to a user terminal for providing augmented reality medical images and a method for providing augmented reality medical images.
According to the present invention, in the augmented reality medical image providing method, the steps include receiving a medical image captured of a body part of a measurement subject from a medical image server, reconstructing a 3D image, and extracting feature points from the reconstructed 3D image. , Photographing a part of the body of the measurement subject using a depth camera mounted on the user terminal, obtaining 3D data of the measurement subject from the captured image, and extracting feature points from the 3D data, 3D images and 3D A step of matching and registering feature points extracted from the data; setting a point spaced a certain distance in the vertical direction from the user terminal as a virtual reference point, and 3D coordinates corresponding to the reference point are located within a part of the body of the measurement subject , which includes the step of providing an augmented reality image of the cross section corresponding to the corresponding reference point on the screen of the user terminal by overlapping the image captured by the depth camera, and the step of providing the augmented reality image of the cross section corresponding to the corresponding reference point on the screen is a step between the user terminal and the measurement subject. As the distance changes, it includes continuously displaying and providing an augmented reality image corresponding to a cross-section of a part of the body on the screen.

Description

User terminal for providing augmented reality medical images using a depth camera and method for providing augmented reality medical images {USER TERMINAL FOR PROVIDING AUGMENTED REALITY MEDICAL IMAGE USING DEPTH CAMERA AND METHOD FOR PROVIDING AUGMENTED REALITY MEDICAL IMAGE}

The present invention relates to a user terminal for providing augmented reality medical images using a depth camera and a method for providing augmented reality medical images, which provides a three-dimensional image of each measurement subject together with the image of the measurement subject using augmented reality. This relates to a user terminal for providing augmented reality medical images and a method for providing augmented reality medical images.

Augmented reality technology is a technology that superimposes virtual objects on the real world that the user sees by projecting a virtual image onto the actual image that the user is viewing. According to Ronald Azuma, augmented reality is a technology that combines real images and virtual images. It is defined as something that can interact in real time and is located in a three-dimensional space.

This is different from virtual reality technology, in which users are immersed in a virtual image and cannot see the real surroundings, and provides a better sense of reality and additional information through a mixture of the real environment and virtual objects.

This augmented reality technology has been studied mainly in the United States since the late 1990s, and with the recent spread of smartphones, full-scale commercialization is being attempted, and various products are being released in the game and education fields.

A method of showing augmented reality to the wearer is a computer screen device in the form of a wearable computer worn on the head, a so-called head mounted display (HMD). This HMD superimposes virtual images on the real environment to the user and displays them in real time. It shows. Recently, development of HMDs is actively taking place, with Microsoft releasing Hololens.

In addition, with the development of augmented reality technology, research is being conducted around the world to apply it to medical care. In particular, attempts are being made to use it as an educational program that shows anatomical structures or as an aid to surgery. However, existing medical augmented reality is an augmented reality implementation based on image tracking using simple visual markers.

In the case of a method based on image tracking, image matching is performed by shooting an image and analyzing the captured image. However, in the case of this image tracking-based method, in medical procedures or surgeries based on the principle of sterility and no contact, draping is preceded, and most of the body is covered by a sterile sheet, making it impossible to capture images to match, making it difficult to apply augmented reality. There are limits.

Therefore, there is a need for a system that provides augmented reality without the use of visual markers.

The technology behind the present invention is disclosed in Republic of Korea Patent No. 10-1831514 (announced on February 26, 2018).

The technical problem to be achieved by the present invention is to provide a user terminal and augmented reality medical images for providing augmented reality medical images using a depth camera that provides a 3D image of each measurement subject together with the image of the measurement subject using augmented reality. It is intended to provide a method.

According to an embodiment of the present invention to achieve this technical problem, in a method of providing augmented reality medical images using a user terminal, medical images captured of a body part of a measurement subject are received from a medical image server and a 3D image is reconstructed. Extracting feature points from the reconstructed 3D image, photographing a body part of the measurement subject using a depth camera mounted on the user terminal, and obtaining 3D data of the measurement subject from the captured image. , extracting feature points from the 3D data, matching feature points extracted from the 3D image with feature points extracted from the 3D data, and selecting a point spaced a certain distance in the vertical direction from the user terminal. If it is set as a virtual reference point, and the three-dimensional coordinates corresponding to the reference point are located within a body part of the measurement subject, the augmented reality image of the cross section corresponding to the reference point is overlaid on the image captured by the depth camera. A step of providing on the screen of the user terminal, wherein providing an augmented reality image of a cross-section corresponding to the corresponding reference point overlaid on an image captured by the depth camera, the user terminal and the measurement subject As the distance between them changes, augmented reality images corresponding to cross-sections of parts of the body are continuously displayed on the screen.

The user terminal includes a 3-axis geomagnetic sensor, and in response to the current position and 3-axis angle of the user terminal, displays an augmented reality image of a cross section corresponding to a plane parallel to the user terminal centered on the reference point. You can.

According to another embodiment of the present invention, in a user terminal for providing augmented reality medical images, a medical image captured of a body part of a measurement subject is received from a medical image server, a three-dimensional image is reconstructed, and the reconstructed A first feature point extraction unit for extracting feature points from a 3D image, an imaging unit for photographing a body part of the measurement subject using a depth camera, obtaining 3D data of the measurement subject from the captured image, and the 3 A second feature point extraction unit that extracts feature points from dimensional data, an image registration section that matches and registers feature points extracted from the 3D image and feature points extracted from the 3D data, and a predetermined distance in the vertical direction from the user terminal. When a separated point is set as a virtual reference point and the 3D coordinates corresponding to the reference point are located within a body part of the measurement subject, an augmented reality image of the cross section corresponding to the reference point is captured by the user terminal. and a control unit that provides an overlapping image, wherein as the distance between the user terminal and the measurement subject changes, the control unit continuously displays an augmented reality image corresponding to a cross section of a part of the body on the screen. to provide.

In this way, according to the present invention, augmented reality images of the inside of the body of the measurement subject are provided in real time according to the distance between the user terminal and the measurement subject, and can be used for pre-surgical simulation and medical education.

In addition, because it uses medical images and captured depth images, production costs and computational volume can be reduced compared to methods that provide augmented reality using existing markers.

1 is a configuration diagram illustrating an augmented reality medical image providing system according to an embodiment of the present invention.
Figure 2 is a flowchart illustrating a method for providing augmented reality medical images according to an embodiment of the present invention.
Figure 3 is a diagram for explaining step S260 of Figure 2.
Figure 4 is a diagram showing the augmented reality image provided by Figure 3.
Figure 5 is a diagram for explaining step S270 of Figure 2.
Figure 6a is a screen provided when the front of a measurement subject is photographed according to an embodiment of the present invention.
Figure 6b is a screen provided when the left side of a measurement subject is photographed according to an embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification, when a part is said to “include” a certain element, this means that it may further include other elements rather than excluding other elements, unless specifically stated to the contrary.

Then, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention.

1 is a configuration diagram illustrating an augmented reality medical image providing system according to an embodiment of the present invention.

As shown in FIG. 1, the system for providing augmented reality for medical images according to an embodiment of the present invention includes a medical image server 100 and a user terminal 200.

First, the medical image server 100 individually stores previously captured medical images for each patient.

At this time, the medical image includes at least one of CT, MRI, X-ray, or ultrasound images.

Here, the method of using X-rays passing through the human body is to use A method of imaging structures and using CT (Computed Tomography) is used to image internal structures by projecting Method may be included.

Additionally, a method using gamma rays injected into the human body may be a PET (Positron Emission Tomography) method. The third method of using ultrasound reflected within the human body is to use Ultrasound (ultrasound imaging technology) to receive ultrasound reflected from biological tissue for the transmitted ultrasound, amplify and detect the reflected ultrasound, and image it on a monitor. It can be included.

In addition, the method of using a magnetic field injected into the human body may include a method of acquiring a tomography image of any living body using a magnetic field generated by magnetism using MRI (Magnetic Resonance Imaging). .

The user terminal 200 includes a first feature point extraction unit 210, a photographing unit 220, a second feature point extraction unit 230, an image matching unit 240, and a control unit 250.

First, the first feature point extractor 210 receives a medical image captured of a body part of a measurement subject from the medical image server 100, reconstructs a 3D image from the received medical image, and creates the reconstructed 3D image. Extract feature points from.

Additionally, the medical image server 100 may reconstruct a 3D image using a pre-stored medical image and provide the image to the first feature point extractor 210.

Then, the first feature point extraction unit 210 extracts feature points from the reconstructed 3D image.

Here, the feature point extracted by the first feature point extractor 210 includes at least one of the eyes, nose, and mouth of the measurement subject.

Next, the imaging unit 220 is equipped with a depth camera, and a part of the body of the measurement subject is photographed using the mounted depth camera.

Here, the depth camera is a camera that can measure the distance between the camera and an object using images captured through a plurality of lenses, and according to an embodiment of the present invention, the measurement subject is the lens of the user terminal 200. In a state in which the user terminal 200 is held facing , the user terminal 200 can be photographed by moving it forward, backward, left, and right, and a third party can also photograph a part of the body of the measurement subject through the user terminal 200.

Then, the depth camera measures the distance between the user terminal 200 and the measurement subject and provides an image of the corresponding point.

Additionally, the user terminal 200 may further include a 3-axis geomagnetic sensor (not shown), and determines the current 3-axis angle of the user terminal using the 3-axis geomagnetic sensor.

The second feature point extraction unit 230 obtains 3D data of the measurement subject from an image captured by a depth camera and extracts feature points from the 3D data.

Likewise, the feature point extracted by the second feature point extractor 230 includes at least one of the eyes, nose, and mouth of the measurement subject.

Next, the image matching unit 240 produces an augmented reality image by matching the feature points extracted by the first feature point extractor 210 and the feature points extracted by the second feature point extractor 230.

Here, the augmented reality image is an image that can be modified or interfered with in real time by combining a real image and a virtual image created in a three-dimensional space. According to an embodiment of the present invention, the augmented reality image is an image of the user terminal 200. It changes in real time depending on the 3-axis angle and the distance to the measurement subject.

Next, the control unit 250 sets a point spaced a certain distance away from the front of the user terminal 200 in the vertical direction facing forward as a virtual reference point.

In addition, when the three-dimensional coordinates corresponding to the reference point are located within a part of the body of the measurement subject, the control unit 250 superimposes the augmented reality image of the cross section corresponding to the reference point on the image captured by the photographing unit 220. Provided on screen.

Additionally, as the distance between the user terminal 200 and the measurement subject changes, the control unit 250 continuously displays an augmented reality image corresponding to a cross section of a part of the body on the screen of the user terminal 200.

Hereinafter, a method for providing augmented reality medical images according to an embodiment of the present invention will be described using FIGS. 2 to 6B.

Figure 2 is a flowchart illustrating a method for providing augmented reality medical images according to an embodiment of the present invention.

First, the first feature point extraction unit 210 receives a medical image captured of a body part of a measurement subject from the medical image server 100, reconstructs a 3D image, and extracts feature points from the reconstructed 3D image ( S210).

At this time, the medical image is an image captured using X-rays, gamma rays, ultrasound, and magnetic fields, as described in FIG. 1.

Here, the body part of the person being measured refers to a human body part such as the face, arms, legs, or upper body, and each body part has different characteristic points.

For example, feature points for the face may be eyes, nose, mouth, or ears, and feature points for the legs may be ankle bones or toes.

If it is difficult to identify a feature point in three dimensions, the first feature point extractor 210 can directly specify the feature point from an image projected on the terminal screen through a camera.

For example, if you want to observe the abdomen while wearing clothes, you can designate the solar plexus area, nipples, or navel by tapping directly on the user terminal screen or pointing to the observation object with an indicator such as a finger.

In addition, the medical image server 100 can match medical images previously stored for each measurement subject, reconstruct them into 3D images, and transmit the reconstructed 3D images to the user terminal 200.

At this time, the reconstructed 3D image includes each coordinate and location information.

Next, a part of the body of the measurement subject is photographed using the imaging unit 220, and the second feature point extraction unit 230 obtains 3D data of the measurement subject from the captured image and extracts feature points from the 3D data. Do it (S220).

Here, the imaging unit 220 uses a dot projector to acquire 3D data of the measurement subject.

At this time, according to an embodiment of the present invention, in order to obtain 3D data of the measurement subject, the dot projector disperses one line of laser into more than 30,000 lasers to obtain 3D data.

In other words, 3D data corresponding to feature points each include corresponding location information.

Next, the image matching unit 240 matches the feature points extracted from the 3D image by the first feature point extractor 210 and the feature points extracted from the 3D data by the second feature point extractor 230 to create the image. Match (S230).

That is, the image registration unit 240 matches the position coordinates of the feature point extracted from the 3D image in step S210 with the coordinates corresponding to the feature point extracted from the 3D data in step S220.

For example, if the position coordinates of a feature point extracted from a 3D image are (242, 546, 254), the image matching unit 240 matches the image by matching 3D data having the same position coordinates as the corresponding position coordinates. .

Additionally, the image registration unit 240 may match the image by matching the feature points extracted from the 3D image and the feature points extracted from the 3D data using a deep learning method.

Then, the image matching unit 240 generates an augmented reality image using the matched image.

Next, the control unit 250 sets a point spaced a certain distance in the vertical direction corresponding to the front of the user terminal 200 as a virtual reference point (S240).

That is, the virtual reference point refers to a point spaced a certain distance away from the front of the user terminal 200.

At this time, the virtual reference point may be located inside or outside the measurement subject depending on the distance between the user terminal 200 and the measurement subject.

Next, the control unit 250 determines whether the virtual reference point exists inside the body of the measurement subject (S250). That is, the control unit 250 compares the distance (R) between the user terminal 200 and the virtual reference point and the distance (D) between the user terminal 200 and the measurement subject.

Here, the distance (R) between the user terminal 200 and the virtual reference point represents a preset value in step S240.

At this time, if the distance (R) between the user terminal 200 and the virtual reference point is longer than the distance (D) between the user terminal 200 and the measurement subject, the control unit 250 determines that the virtual reference point is inside the measurement subject. Upon determining that it is located, the control unit 250 provides an augmented reality image of the corresponding virtual cross-section overlaid on the image captured by the user terminal 200 (S260).

Figure 3 is a diagram for explaining step S260 of Figure 2, and Figure 4 is a diagram showing the augmented reality image provided by Figure 3.

That is, as shown in FIG. 3, when the distance (R) between the user terminal 200 and the virtual reference point is longer than the distance (D) between the user terminal 200 and the measurement subject, the control unit 250 detects the virtual reference point It is judged to exist inside the body of the person being measured.

Then, as shown in FIG. 4, the control unit 250 provides a cross-section of the brain of the measurement subject corresponding to a virtual plane centered on a virtual reference point in augmented reality.

To this end, the control unit 250 provides a virtual reference point and 3-axis angle information of the user terminal 200 to the medical image providing server 100.

Then, the medical image providing server 100 centers the virtual reference point (0,0,0) and sends the augmented reality image set at the same 3-axis angle as the 3-axis angle of the user terminal 200 to the control unit 250. Deliver.

That is, the medical image providing server 100 provides the user terminal 200 with a cross-sectional image corresponding to the virtual reference point and 3-axis angle information received from the control unit 250.

For example, assuming that the user terminal 200 is tilted 40 degrees based on the x-axis, the medical image providing server 100 is tilted 40 degrees in the x-axis direction with the reference point as the center (0,0,0). An augmented reality image corresponding to the true plane is provided to the control unit 250.

Accordingly, the user terminal 200 provides the augmented reality image corresponding to the cross-section received from the medical image providing server 100 by overlapping it with the image captured by the photographing unit 220.

At this time, if the position of the reference point is the same and the posture of the user terminal 200 changes, the control unit 250 corresponds to a cross section having the same three-axis angle as the user terminal 200 according to the posture of the user terminal 200. Provides augmented reality video.

On the contrary, when the distance (R) between the user terminal 200 and the virtual reference point is shorter than or equal to the distance (D) between the user terminal 200 and the measurement subject, the control unit 250 determines that the virtual reference point is the measurement subject. It is determined to be located outside of , and an augmented reality image corresponding to the brain surface or organ surface of the measurement subject is provided overlaid on the image captured by the imaging unit 220 (S270).

Figure 5 is a diagram for explaining step S270 of Figure 2, Figure 6a is a screen provided when the front of the measurement subject according to an embodiment of the present invention is photographed, and Figure 6b is a measurement subject according to an embodiment of the present invention. This is the screen provided when the left side of is taken.

That is, as shown in FIG. 5, when the distance (R) between the user terminal 200 and the virtual reference point is shorter than or equal to the distance (D) between the user terminal 200 and the measurement subject, the control unit 250 The virtual reference point is determined to exist outside the body of the person being measured.

If the current position of the user terminal 200 is facing the front of the measurement subject's face, as shown in FIG. 6A, the control unit 250 sends an augmented reality image corresponding to the front part of the measurement subject's brain to the user terminal 200. ) can be provided on the screen.

In addition, when the current location of the user terminal 200 is toward the left side of the measurement subject's face, as shown in FIG. 6B, the control unit 250 sends an augmented reality image corresponding to the left side of the measurement subject to the user terminal 200. ) can be provided on the screen.

At this time, the control unit 250 may reduce or enlarge the augmented reality image according to the size of the captured measurement subject on the screen and provide it to the user terminal 200.

And, if the measurement subject does not exist in the image captured through the user terminal 200, the control unit 250 receives an arbitrary virtual reference point through the user terminal 200, and determines the virtual reference point, the terminal, and the virtual reference point. It can provide cross-sections of medical images according to the distance from the reference point, virtual plane, and 3-axis angle.

Additionally, when the image captured by the user terminal 200 is an anatomical model rather than a measurement subject, the control unit 250 may provide an arbitrary augmented reality image on the screen of the user terminal 200.

At this time, the control unit 250 may provide an augmented reality image of a random measurement subject or an augmented reality image of a measurement subject requiring confirmation on the screen of the user terminal 200.

In this way, according to an embodiment of the present invention, augmented reality images of the inside of the body of the measurement subject are provided in real time according to the distance between the user terminal and the measurement subject, and can be used for pre-surgical simulation and medical education.

In addition, because it uses medical images and captured depth images, production costs and computational volume can be reduced compared to methods that provide augmented reality using existing markers.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely illustrative, and those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Therefore, the true scope of technical protection of the present invention should be determined by the technical spirit of the attached patent claims.

100: medical image provision server, 200: user terminal,
210: first feature point extraction unit, 220: photography unit,
230: second feature point extraction unit, 240: image registration unit,
250: control unit

Claims (4)

In a method of providing augmented reality medical images using a user terminal,
Receiving a medical image taken of a body part of a measurement subject from a medical image server, reconstructing a 3D image, and extracting feature points from the reconstructed 3D image,
Photographing a body part of the measurement subject using a depth camera mounted on a user terminal, obtaining 3D data of the measurement subject from the captured image, and extracting feature points from the 3D data,
A step of matching and registering feature points extracted from the 3D image and feature points extracted from the 3D data,
A point spaced a certain distance in the vertical direction from the user terminal is set as a virtual reference point, and when the three-dimensional coordinates corresponding to the reference point are located within a part of the body of the measurement subject, the cross section corresponding to the reference point is set as a virtual reference point. It includes providing the augmented reality image on the screen of the user terminal by overlapping it with the image captured by the depth camera,
The step of providing an augmented reality image of a cross section corresponding to the corresponding reference point by overlapping it with an image captured by the depth camera,
A method of providing an augmented reality medical image that continuously displays and provides an augmented reality image corresponding to a cross section of a part of the body on a screen as the distance between the user terminal and the measurement subject changes. According to paragraph 1,
The user terminal includes a 3-axis geomagnetic sensor,
A method of providing an augmented reality medical image that displays an augmented reality image of a cross-section corresponding to a plane parallel to the user terminal centered on the reference point, in response to the current position and three-axis angle of the user terminal. In a user terminal for providing augmented reality medical images,
A first feature point extraction unit that receives medical images taken of a body part of a measurement subject from a medical image server, reconstructs a 3D image, and extracts feature points from the reconstructed 3D image;
A photographing unit for photographing a body part of the measurement subject using a depth camera,
a second feature point extraction unit that acquires 3D data of the measurement subject from the captured image and extracts feature points from the 3D data;
an image registration unit that matches and registers feature points extracted from the 3D image and feature points extracted from the 3D data, and
A point spaced a certain distance in the vertical direction from the user terminal is set as a virtual reference point, and when the three-dimensional coordinates corresponding to the reference point are located within a part of the body of the measurement subject, the cross section corresponding to the reference point is set as a virtual reference point. It includes a control unit that provides an augmented reality image by overlapping it with an image captured by the user terminal,
The control unit,
A user terminal that continuously displays and provides an augmented reality image corresponding to a cross-section of a part of the body on a screen as the distance between the user terminal and the measurement subject changes. According to paragraph 3,
It further includes a 3-axis geomagnetic sensor,
The control unit,
A user terminal that displays an augmented reality image of a cross-section corresponding to a plane parallel to the user terminal centered on the reference point, in response to the current position and three-axis angle of the user terminal.