KR102366326B1 - Medical Treatment System Using AI Big Data Based on Mixed Reality and Lesion information providing Method Using the Same - Google Patents

Abstract

The present invention relates to a medical treatment system using artificial intelligence (AI) big data based on mixed reality, which enables a patient to receive intuitive and convenient medical services. According to the present invention, the medical treatment system comprises: an imaging unit photographing the internal structure of a pats body organ to be treated to acquire image information; an image implementation unit using the image information acquired through the imaging unit to model and implement a 3D image; a 3D image database storing the 3D images modeled and implemented by the image implementation unit; a mannequin image database storing mannequin images made by modeling the head of the human body; an image synthesis unit selecting and loading an arbitrary mannequin image from the mannequin image database and loading the 3D images of the patient from the 3D image database to synthesize the loaded images and build a synthesized image; and a first head-mounted display device provided to be worn on the patient's head and including a first display unit transmitting the synthesized image synthesized by the image synthesis unit to the patient's field of view.

Description

Medical Treatment System Using AI Big Data Based on Mixed Reality and Lesion information providing Method Using the Same}

The present invention relates to a medical treatment system using artificial intelligence big data based on mixed reality and a medical treatment method using the same. It relates to an otolaryngology treatment system that can provide

Otolaryngology is a specialized department that provides medical and surgical treatment for diseases related to the ear, nose, and throat (pharynx, larynx).

Since it is difficult to visually check the diseased area from the outside due to its characteristics, such an otolaryngology department generally inserts a separate imaging device such as an endoscope into the inside of a body organ for treatment to acquire image information and monitor it The treatment is carried out in a way that detects and observes the disease site by sending it through the

However, in the process of the medical practitioner showing the image to the patient through the flat monitor and explaining the contents of the treatment, the patient is watching the 2D screen, so it is difficult to intuitively grasp the structure of his or her body organ where the disease has occurred. there is a problem.

As a result, it is not easy for medical practitioners to convey the information to patients sufficiently, and in some cases, miscommunication may occur due to miscommunication.

Therefore, a method for solving these problems is required.

Korean Patent No. 10-0968280

The present invention is an invention devised to solve the problems of the prior art, and assists the medical practitioner's medical practice through a mixed reality technology that combines virtual reality and augmented reality, and provides a more intuitive and convenient medical service to the target patient It has a purpose to provide a medical treatment system using artificial intelligence big data based on mixed reality and a medical treatment method using the same.

The problems of the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

The medical treatment system using the mixed reality-based artificial intelligence big data of the present invention for achieving the above object includes an imaging unit that acquires image information by photographing the internal structure of a body organ to be treated of a target patient, the imaging unit An image implementation unit that models and implements a 3D image using the image information obtained through a 3D image database that stores the 3D image modeled and implemented by the image implementation unit, and a pre-made mannequin image by modeling the head of the human body. A mannequin image database to store and store, an image synthesizing unit that selects and loads an arbitrary mannequin image from the mannequin image database, loads a 3D image of a target patient from the 3D image database, and builds a composite image image by synthesizing each other; It is provided to be worn on the patient's head and includes a first head-mounted display device including a first display unit for transmitting the synthesized image image synthesized by the image synthesis unit to the patient's field of view.

In addition, a plurality of mannequin images having different body conditions are stored in the mannequin image database, and the image synthesizing unit receives the patient's personal information and selects a mannequin image based on a preset selection criterion based on the patient's personal information may be configured to automatically load from the mannequin image database.

In addition, the image synthesizing unit, in a process of synthesizing the loaded mannequin image and the 3D image, compares the scales of the mannequin image and the 3D image and corrects them so that the scales match each other.

On the other hand, the present invention is provided to be worn on the head of the medical practitioner, the second head including a second display unit for transmitting the synthesized image image synthesized by the image synthesizing unit in conjunction with the first display unit to the medical practitioner's field of vision It may further include a mount display device.

In this case, the present invention may further include an auxiliary projection image generating unit for generating an auxiliary projection image for auxiliary projection of an auxiliary image on the composite image image transmitted to at least one of the first display unit and the second display unit. .

In addition, the present invention further includes a lesion region identification unit for generating lesion identification information by identifying a lesion site in the 3D image. In this case, the auxiliary projection image generating unit receives the lesion identification information generated by the lesion region identification unit Based on the composite image image, an auxiliary projection image indicating the lesion area may be generated.

In addition, the imaging unit, the image realizing unit, the 3D image database, the mannequin image database, and the image synthesizing unit may be provided in the ENT equipment.

In addition, the medical treatment method using artificial intelligence big data based on mixed reality to achieve the above object is a step (a) of acquiring image information by photographing the internal structure of the body organ of the target patient through an imaging unit, (b) of an image implementation unit modeling and implementing a 3D image using the image information acquired by the imaging unit, (c) of storing the 3D image implemented by the step (b) in a 3D image database; (d) selecting and loading an arbitrary mannequin image from a mannequin image database that stores and stores a mannequin image made by the image synthesizing unit modeling the human head, the image synthesizing unit 3D image of the target patient from the 3D image database (e) step of loading, (f) step of constructing a composite image image by synthesizing the 3D image loaded in step (e) with the mannequin image loaded in step (d) by the image synthesizing unit, and the first head mount and (g) transmitting the composite image image constructed in step (f) through the first display unit of the display device.

In this case, the step (d) is a step (d-1) of receiving the patient's personal information, and the image synthesizing unit searches the mannequin image according to a preset selection criterion from the mannequin image database based on the patient's personal information. Step (d-2) and the image synthesizing unit may include a step (d-3) of selecting and loading a mannequin image that meets a preset selection criterion.

And the step (f) is a (f-1) step of analyzing the scale of the mannequin image loaded in step (d) by the image synthesizing unit, and the scale of the 3D image loaded by the image synthesizing unit in step (e) (f-2) step of analyzing, (f-3) step in which the image synthesizing unit corrects the mannequin image loaded in step (d) and the 3D image loaded in step (e) to match each other's scales and (f-4) of the image synthesizing unit synthesizing the 3D image with the mannequin image whose scale is matched by the (f-3) step to construct a composite image image.

On the other hand, the medical treatment system using the mixed reality-based artificial intelligence big data is provided to be worn on the head of the medical practitioner, and the composite image image synthesized by the image synthesis unit in conjunction with the first display unit is displayed by the medical practitioner. Further comprising a second head-mounted display device comprising a second display unit for transmitting to the field of view of, in this case, the step (h) of transmitting the composite image image constructed in step (f) through the second display unit may include more.

In addition, the medical treatment system using the mixed reality-based artificial intelligence big data is an auxiliary projection image for auxiliary projection of an auxiliary image on the composite image transmitted to at least one of the first display unit and the second display unit and a lesion region identification unit generating lesion identification information by identifying a lesion site in the 3D image and generating an auxiliary projection image generating unit. In this case, after steps (g) and (h), the Step (i-1) in which the lesion region identification unit identifies the lesion region in the 3D image, and the auxiliary projection image generation unit displays at least one of the first display unit and the second display unit based on the lesion identification information. Step (i) including the step (i-2) of generating an auxiliary projection image indicating the lesion area on the transmitted composite image may be further performed.

The medical treatment system using the mixed reality-based artificial intelligence big data of the present invention for solving the above problems and the medical treatment method using the same, select and load an arbitrary mannequin image from a mannequin image database, and 3D the target patient By providing a composite image image constructed by loading images from the 3D image database and synthesizing each other to the first head-mounted display device worn by the target patient, the target patient can receive more intuitive and convenient medical services, and It has the advantage of being able to carry out a series of medical procedures.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a diagram conceptually showing each component constituting a medical treatment system utilizing artificial intelligence big data based on mixed reality according to an embodiment of the present invention;
2 is a view showing each process of a medical treatment method using a medical treatment system utilizing artificial intelligence big data based on mixed reality according to an embodiment of the present invention;
3 is a view showing a detailed process of step (d) in a medical treatment method using a medical treatment system utilizing artificial intelligence big data based on mixed reality according to an embodiment of the present invention;
4 is a view showing a detailed process of step (f) in a medical treatment method using a medical treatment system using artificial intelligence big data based on mixed reality according to an embodiment of the present invention; and
5 is a view showing each process performed after step (f) in a medical treatment method using a medical treatment system using artificial intelligence big data based on mixed reality according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention in which the object of the present invention can be specifically realized will be described with reference to the accompanying drawings. In describing the present embodiment, the same names and the same reference numerals are used for the same components, and an additional description thereof will be omitted.

1 is a diagram conceptually illustrating each component constituting a medical treatment system utilizing artificial intelligence big data based on mixed reality according to an embodiment of the present invention.

As shown in FIG. 1 , the medical treatment system using AI big data based on mixed reality according to an embodiment of the present invention includes a first head-mounted display device 100 and a second head-mounted display device 200 . ) and ENT equipment 300 may be included.

Here, the ENT equipment 300 is a medical equipment used in the otolaryngology department, and may include various configurations that are basically provided in the general ENT equipment 300, which is obvious to those skilled in the art, so a detailed description of the general configuration will be omitted. do.

At this time, in this embodiment, assuming that the target patient has an otolaryngology disease, the medical treatment system using the mixed reality-based artificial intelligence big data of the present invention is exemplified as being applied to the otolaryngology, but this is not limited only to this embodiment. Of course, the present invention can be applied throughout the medical field.

And in this embodiment, the ENT equipment 300 may further include detailed components provided to perform mixed reality-based medical treatment.

Specifically, in the case of this embodiment, the ENT equipment 300 uses the imaging unit 310 to acquire image information by photographing the internal structure of the body organ to be treated, and the image information acquired through the imaging unit 310. An image implementation unit 320 that is modeled and implemented as a 3D image, a 3D image database 330 that stores a 3D image modeled and implemented by the image implementation unit 320, and a previously manufactured head by modeling the human head The mannequin image database 340 that stores and stores the mannequin image, selects and loads an arbitrary mannequin image from the mannequin image database 340, and loads the 3D image of the target patient from the 3D image database 330 to synthesize them. It includes an image synthesizing unit 350 for constructing a composite image according to the image.

In addition, the first head-mounted display device 100 is provided to be worn on the patient's head, and includes a first display unit 110 that transmits the synthesized image image synthesized by the image synthesis unit 350 to the patient's field of view. .

And the second head-mounted display device 200 is provided to be worn on the head of the medical practitioner, and the combined image image synthesized by the image synthesizing unit 350 in conjunction with the first display unit 110 is displayed in the view of the medical practitioner. It includes a second display unit 210 for transmitting.

On the other hand, in this embodiment, the ENT equipment 300 supports the auxiliary image on the composite image image transmitted to at least one of the first display unit 110 and the second display unit 210 together with each of the above-described components. It may further include an auxiliary projection image generator 260 for generating an auxiliary projection image for projection, and a lesion region identification unit 370 for generating lesion identification information by identifying a lesion site in the 3D image.

At this time, the auxiliary projection image generating unit 260 generates an auxiliary projection image indicating the lesion area on the composite image image based on the lesion identification information generated by the lesion area identification unit 370 .

That is, the auxiliary projection image generator 260 may additionally display an arbitrary auxiliary image in the composite image image in which the mannequin image and the 3D image are synthesized with each other, and such auxiliary image may provide various medical information without limitation. In particular, in the present embodiment, the auxiliary projection image generator 260 provides an auxiliary projection image based on the lesion identification information generated by the lesion region identification unit 370 .

In addition, in the present embodiment, a plurality of mannequin images having various different body conditions are stored in the mannequin image database 340, and the image synthesizing unit 350 receives the patient's personal information, and based on the patient's personal information, Accordingly, the selected mannequin image may be automatically loaded from the mannequin image database 340 according to a preset selection criterion.

This is to load a mannequin image having a body condition most similar to the current patient's personal information from the big data built through the mannequin image database 340 through the image synthesizing unit 350 .

In this case, in the process of synthesizing the loaded mannequin image and the 3D image, the image synthesizing unit 350 may compare the scales of the mannequin image and the 3D image and correct the scales to match each other. The reason for doing this is to obtain a natural synthetic image image by matching the scale between the mannequin image and the 3D image.

As described above, the medical treatment system using the mixed reality-based artificial intelligence big data of the present invention selects and loads an arbitrary mannequin image from the mannequin image database 340, and stores the 3D image of the target patient in the 3D image database 330. By providing the composite image image constructed by loading from and synthesizing with each other to the first head-mounted display device 100 worn by the target patient, the target patient can receive more intuitive and convenient medical services through 3D graphics. It has the advantage of being able to smoothly perform a series of treatment processes.

Hereinafter, a process of performing medical treatment using a medical treatment system using such mixed reality-based artificial intelligence big data will be described in detail.

2 is a view showing each process of a medical treatment method using a medical treatment system using artificial intelligence big data based on mixed reality according to an embodiment of the present invention.

As shown in FIG. 2 , a medical treatment method using a medical treatment system using artificial intelligence big data based on mixed reality according to an embodiment of the present invention includes steps (a) to (g).

First, step (a) is a process of acquiring image information by photographing the internal structure of a body organ to be treated through the imaging unit 310 .

And in step (b), the image implementation unit 320 models and implements a 3D image using the image information obtained by the imaging unit 310, and in step (c), it is stored in the 3D image database 330 (b) The 3D image implemented by the steps will be saved.

That is, through steps (a) to (c), the internal structure of the body organ to be treated of the target patient may be implemented in 3D graphics.

Next, in step (d), the image synthesizing unit 350 selects and loads an arbitrary mannequin image from the mannequin image database 340 that stores and stores a pre-fabricated mannequin image by modeling the head of the human body, (e) In the step, the image synthesizing unit 350 loads the 3D image of the target patient from the 3D image database 330 .

Then, in step (f), the image synthesis unit 350 synthesizes the mannequin image loaded in step (d) and the 3D image loaded in step (e) to construct a composite image image.

The composite image image synthesized by the above process is transmitted through the first display unit 110 of the first head-mounted display device 100 by the step (g).

Meanwhile, FIG. 3 is a diagram illustrating a detailed process of step (d) in a medical treatment method using a medical treatment system using artificial intelligence big data based on mixed reality according to an embodiment of the present invention.

As shown in FIG. 3, the image synthesis unit 350 selects and loads an arbitrary mannequin image from the mannequin image database 340, which stores and stores the mannequin image pre-fabricated by modeling the head of the human body ( Step d) includes multiple detailed steps.

Specifically, step (d) in this embodiment includes a step (d-1) of receiving the patient's personal information, and the image synthesizing unit 350 is generated from the mannequin image database 340 based on the patient's personal information. It includes a step (d-2) of searching for a mannequin image according to a set selection criterion, and a step (d-3) of the image synthesizing unit 350 selecting and loading a mannequin image that meets the preset selection criterion.

At this time, the preset selection criteria can be set in various ways, and in this embodiment, the image synthesizing unit 350 uses the mannequin image having the most similar physical condition to the current patient's personal information from the big data built through the mannequin image database 340 . ) to load it through.

And FIG. 4 is a view showing a detailed process of step (f) in a medical treatment method using a medical treatment system using artificial intelligence big data based on mixed reality according to an embodiment of the present invention.

As shown in FIG. 4 , step (f) of constructing a composite image image by synthesizing the 3D image loaded in step (e) with the mannequin image loaded in step (d) by the image synthesizing unit 350 is also detailed. It may include multiple steps.

Specifically, step (f) in this embodiment includes a step (f-1) in which the image synthesis unit 350 analyzes the scale of the mannequin image loaded in step (d), and a step (f-1) in which the image synthesis unit 350 performs (e). Step (f-2) of analyzing the scale of the 3D image loaded in step ), and the image synthesizing unit 350 correcting the mannequin image loaded in step (d) and the 3D image loaded in step (e) to each other Step (f-3) of matching the scale of , and the image synthesizing unit 350 constructs a composite image image by synthesizing the 3D image with the mannequin image whose scale is matched by the step (f-3) (f-4) ) step.

In the detailed process of step (f), it is possible to obtain a natural synthetic image image by matching the scale between the mannequin image and the 3D image.

Meanwhile, FIG. 5 is a view showing each process performed after step (f) in a medical treatment method using a medical treatment system using artificial intelligence big data based on mixed reality according to an embodiment of the present invention.

5, after step (f) in this embodiment, along with the step (g) of transmitting the composite image image through the first display unit 110 of the first head-mounted display device 100 , step (h) of transmitting the composite image image constructed in step (f) through the second display unit 210 of the second head-mounted display device 200 may be performed. Accordingly, the medical practitioner can check the screen the target patient is currently watching.

And after steps (g) and (h), the lesion region identification unit 370 identifies the lesion region in the 3D image (i-1), and the auxiliary projection image generator 360 generates the lesion identification information. (i-2) generating an auxiliary projection image indicating the lesion area on the composite image transmitted to at least one of the first display unit 110 and the second display unit 210 based on (i-2) ( Step i) may be further performed.

Through this process, the target patient can intuitively identify and understand their lesion area, and the medical practitioner can provide a higher level of medical service to the target patient.

Meanwhile, the present invention can be extended and applied to various fields in addition to the application method shown in the above-described embodiment. For example, the medical treatment system using the mixed reality-based artificial intelligence big data according to the present invention and the medical treatment method using the same can be usefully applied to the health care field, including dizziness treatment, rehabilitation treatment, and psychological treatment. In the process, a high treatment effect can be obtained by using the technology of artificial intelligence and the information power of big data.

As described above, preferred embodiments according to the present invention have been reviewed, and the fact that the present invention can be embodied in other specific forms without departing from the spirit or scope of the present invention other than the above-described embodiments is a fact having ordinary skill in the art. It is obvious to them. Therefore, the above-described embodiments are to be regarded as illustrative rather than restrictive, and accordingly, the present invention is not limited to the above description, but may be modified within the scope of the appended claims and their equivalents.

100: first head-mounted display device
110: first display unit
200: second head-mounted display device
210: second display unit
300: ENT equipment
310: imaging unit
320: image implementation unit
330: 3D image database
340: mannequin image database
350: image synthesizing unit
360: auxiliary projection image generator
370: lesion region identification unit

Claims (12)

delete delete delete delete delete delete delete In a medical treatment method using artificial intelligence big data based on mixed reality using a medical treatment system using AI big data based on mixed reality,
(a) acquiring image information by photographing the internal structure of a body organ to be treated through an imaging unit;
(b) of an image implementation unit modeling and implementing a 3D image of an internal structure of a body organ to be treated using the image information acquired by the imaging unit;
(c) storing a 3D image of the internal structure of the body organ to be treated implemented by the step (b) in a 3D image database;
(d) selecting and loading an arbitrary mannequin image from a mannequin image database that stores and stores a pre-fabricated mannequin image by an image synthesizing unit modeling the head of the human body including the internal structure of the body to be treated;
(e) loading the 3D image of the target patient from the 3D image database by the image synthesizing unit;
The internal structure of the body to be treated based on the image of the internal structure of the body to be treated in the mannequin image loaded in step (d) by the image synthesizing unit and the image information obtained from the imaging unit loaded in step (e) (f) constructing a composite image image by synthesizing a 3D image for and
(g) transmitting the composite image image constructed in the step (f) through the first display unit of the first head-mounted display device;
including,
an auxiliary projection image generating unit for generating an auxiliary projection image for auxiliary projection of an auxiliary image on the composite image transmitted to the first display unit; and
a lesion region identification unit for generating lesion identification information by identifying a lesion region in a 3D image obtained based on image information acquired through the imaging unit;
further comprising,
After step (g),
(i-1) step of the lesion region identification unit identifying the lesion region in the 3D image; and
(i-2) generating, by the auxiliary projection image generating unit, an auxiliary projection image displaying a lesion area on the composite image transmitted to the first display unit based on the lesion identification information;
(i) further comprising a step is performed,
Is,
A method of providing lesion information using artificial intelligence big data based on mixed reality. 9. The method of claim 8,
Step (d) is,
(d-1) receiving the patient's personal information;
(d-2) of the image synthesizing unit searching for a mannequin image from the mannequin image database based on the patient's personal information according to a preset selection criterion; and
(d-3) step of loading by the image synthesizing unit selecting a mannequin image that meets a preset selection criterion;
containing,
A method of providing lesion information using artificial intelligence big data based on mixed reality. 9. The method of claim 8,
The step (f) is,
(f-1) step of analyzing the scale of the mannequin image loaded in step (d) by the image synthesizing unit;
(f-2) step of analyzing the scale of the 3D image loaded in the step (e) by the image synthesizing unit;
(f-3) step of the image synthesizing unit correcting the mannequin image loaded in step (d) and the 3D image loaded in step (e) to match each other's scales; and
(f-4) step of the image synthesizing unit synthesizing the 3D image with the mannequin image whose scale is matched by the (f-3) step to construct a composite image image;
containing,
A method of providing lesion information using artificial intelligence big data based on mixed reality. 9. The method of claim 8,
The medical treatment system using the artificial intelligence big data based on the mixed reality is,
A second head-mounted display device that is provided to be worn on the medical practitioner's head and includes a second display unit for transmitting the synthesized image image synthesized by the image synthesis unit to the medical practitioner's field of view in conjunction with the first display unit including more,
Further comprising the step (h) of transmitting the composite image image constructed in the step (f) through the second display unit,
A method of providing lesion information using artificial intelligence big data based on mixed reality. 12. The method of claim 11,
In step (i), a method for providing lesion information using mixed reality-based artificial intelligence big data to generate an auxiliary projection image for displaying a lesion area on the composite image transmitted to the second display unit.

Images