WO2022124705A1

WO2022124705A1 - Apparatus and method for providing medical image-based hologram

Info

Publication number: WO2022124705A1
Application number: PCT/KR2021/018211
Authority: WO
Inventors: 김광기; 백정흠; 김영재
Original assignee: 가천대학교 산학협력단; 의료법인 길의료재단
Priority date: 2020-12-11
Filing date: 2021-12-03
Publication date: 2022-06-16
Also published as: KR20220083477A; KR102476957B1

Abstract

Provided are an apparatus and a method for providing a medical image-based hologram according to an embodiment of the present invention. The apparatus is configured to: obtain a medical image generated by capturing a target site of an examinee; obtain a target site and suspected lesion site prediction result data from the medical image by using an artificial neural network model trained to predict a target site and a suspected lesion site suspected of a lesion in the target site on the basis of the obtained medical image; generate a three-dimensional image representing the target site and the suspected lesion site on the basis of the result data; generate a hologram image used for generating a hologram by using the generated three-dimensional image; and provide the generated hologram image to a hologram device so that the hologram device generates a hologram.

Description

Medical image-based hologram providing apparatus and method

The present invention relates to an apparatus and method for providing a hologram based on a medical image.

In general, hospital treatment is performed in such a way that a patient visits a hospital, sees a specialist, receives treatment, or receives an examination for diagnosing a disease, and receives the result from the specialist. For example, imaging medical examinations of the patient (such as X-rays, Computer Tomography (CT), Angiography, Positron Emission Tomography (PET-CT), or Magnetic Resonance Imaging (MRI)) ), the specialist shows the patient a medical image such as an X-ray, CT image, angiography image, PET-CT image, or MRI image of the patient's target area (eg, lung or colon) for diagnosis Or deliver test results, etc.

In this case, it is necessary for the specialist to efficiently deliver diagnosis or test results, explanations of diseases, and treatment plans to many patients at a set time.

Meanwhile, the medical image may be used by a specialist to operate on a patient in an operating room. As such, there is a problem in that a medical assistant for monitoring is required to check the medical image, or a specialist must move to a monitoring device that displays a medical image in order to check the medical image during an operation.

Recently, a method for checking a medical image using augmented reality (AR) has been used, but in order to use augmented reality, it is inconvenient to wear a separate device such as augmented reality glasses (or goggles).

Accordingly, there is a need for a medical image-based hologram providing apparatus and method for a specialist to conveniently check a patient's medical image and to more easily deliver a diagnosis or examination result to a patient.

The inventors of the present invention are aware of the fact that the medical image presented by the specialist is difficult for non-medical patients to understand, and there is no additional means for reference other than the medical image for the patient's understanding. recognized.

In addition, the inventors of the present invention recognized the fact that there is a problem in that an assistant manpower for monitoring a medical image in an operating room is required or that a specialist needs to move to a monitoring device during an operation.

In addition, the inventors of the present invention recognized the fact that even if augmented reality is used to check a medical image, there is an inconvenience in that a specialist needs to wear a separate device in the operating room.

Accordingly, an object of the present invention is to provide an apparatus and method for providing a hologram based on a medical image.

Specifically, an object of the present invention is to provide an apparatus and method for providing a hologram based on a medical image in order to easily transmit a diagnosis or examination result to a patient.

In addition, the problem to be solved by the present invention is a medical image-based hologram providing apparatus and method for conveniently checking a patient's medical image without requiring an assistant to check a medical image or a specialist moving to a monitoring device during surgery is to provide

Another object of the present invention is to provide a medical image-based hologram providing apparatus and method for easily checking a patient's medical image without the need for a specialist to wear a separate device.

The problems of the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

An apparatus and method for providing a hologram based on a medical image are provided in order to solve the above problems.

In an embodiment of the present invention, there is provided an apparatus for providing a medical image-based hologram, comprising: a communication unit configured to transmit and receive data; and a control unit configured to be connected to the communication unit, wherein the control unit obtains a medical image obtained by photographing a target site of the subject through the communication unit, and based on the obtained medical image, the target site and the lesion at the target site Obtaining result data of predicting the target region and the lesion suspected region from the medical image using an artificial neural network model trained to predict the suspected lesion region, and based on the result data, the target region and the lesion region A three-dimensional image representing the suspicious region is generated, a hologram image used to generate a hologram is generated using the generated three-dimensional image, and the generated hologram image is used so that the hologram device generates the hologram. configured to serve as a holographic device.

In a medical image-based hologram providing method performed by a controller of a medical image-based hologram providing apparatus according to an embodiment of the present invention, the method comprising: acquiring a medical image obtained by photographing a target part of a subject; Results of predicting the target site and the suspected lesion region from the medical image using an artificial neural network model trained to predict the target site and the lesion suspected area at the target site based on the obtained medical image acquiring data; generating a three-dimensional image representing the target region and the suspected lesion region based on the result data; generating an image for a hologram used to generate a hologram by using the generated 3D image; and providing the generated hologram image to the hologram device so that the hologram device generates the hologram.

Details of other embodiments are included in the detailed description and drawings.

The present invention provides a hologram based on a medical image, so that a specialist can more easily deliver a diagnosis or test result to a patient.

In addition, the present invention increases the patient's understanding of the diagnosis or examination results during a prescribed treatment time because the patient can listen to the expert's explanation of the diagnosis or examination result while the specialist and the patient view the hologram based on the medical image together. can

In addition, the present invention can improve patient satisfaction with outpatient treatment even when a specialist treats many patients during a prescribed treatment time.

In addition, according to the present invention, a specialist can efficiently deliver a diagnosis or examination result, a description of a disease, a treatment plan, and the like to a patient within a prescribed treatment time, thereby providing a higher medical service to the patient.

In addition, according to the present invention, a specialist can easily check a patient's medical image without adding an assistant for monitoring the medical image in the operating room.

In addition, according to the present invention, a medical professional can conveniently check a patient's medical image without wearing a separate device in the operating room or without unnecessary movement to check the medical image.

The effect according to the present invention is not limited by the contents exemplified above, and more various effects are included in the present specification.

1 is a schematic diagram for explaining a hologram providing system according to an embodiment of the present invention.

2 is a schematic block diagram of an electronic device according to an embodiment of the present invention.

3 is an exemplary diagram for explaining a method of generating an image for a hologram using an artificial neural network model according to an embodiment of the present invention.

4 is a diagram illustrating a hologram generated by a hologram device according to an embodiment of the present invention.

5 is an exemplary diagram for explaining a method of generating an image for a hologram using pre-stored standard modeling images and readouts for various parts according to an embodiment of the present invention.

6 is a diagram illustrating a hologram generated by a hologram device according to an embodiment of the present invention.

7 is a flowchart illustrating a method for providing a medical image-based hologram in an electronic device according to an embodiment of the present invention.

8 is an exemplary diagram illustrating an interface screen for providing a hologram image using an artificial neural network model according to an embodiment of the present invention.

9 is an exemplary diagram illustrating an interface screen for providing an image for a hologram using a pre-stored standard modeling image and readout according to an embodiment of the present invention.

It is provided for the purpose of the present invention, and the present invention is only defined by the scope of the claims. In connection with the description of the drawings, like reference numerals may be used for like components.

In this document, expressions such as "have," "may have," "includes," or "may include" refer to the presence of a corresponding characteristic (eg, a numerical value, function, operation, or component such as a part). and does not exclude the presence of additional features.

In this document, expressions such as “A or B,” “at least one of A or/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, "A or B," "at least one of A and B," or "at least one of A or B" means (1) includes at least one A, (2) includes at least one B; Or (3) it may refer to all cases including both at least one A and at least one B.

As used herein, expressions such as "first," "second," "first," or "second," may modify various elements, regardless of order and/or importance, and refer to one element. It is used only to distinguish it from other components, and does not limit the components. For example, the first user equipment and the second user equipment may represent different user equipment regardless of order or importance. For example, without departing from the scope of the rights described in this document, the first component may be named as the second component, and similarly, the second component may also be renamed as the first component.

A component (eg, a first component) is "coupled with/to (operatively or communicatively)" to another component (eg, a second component); When referring to "connected to", it will be understood that the certain element may be directly connected to the other element or may be connected through another element (eg, a third element). On the other hand, when it is said that a component (eg, a first component) is "directly connected" or "directly connected" to another component (eg, a second component), the component and the It may be understood that other components (eg, a third component) do not exist between other components.

The expression "configured to (or configured to)" as used in this document, depending on the context, for example, "suitable for," "having the capacity to ," "designed to," "adapted to," "made to," or "capable of." The term “configured (or configured to)” may not necessarily mean only “specifically designed to” in hardware. Instead, in some circumstances, the expression “a device configured to” may mean that the device is “capable of” with other devices or parts. For example, the phrase "a processor configured (or configured to perform) A, B, and C" refers to a dedicated processor (eg, an embedded processor) for performing the corresponding operations, or by executing one or more software programs stored in a memory device. , may mean a generic-purpose processor (eg, a CPU or an application processor) capable of performing corresponding operations.

Terms used in this document are only used to describe specific embodiments, and may not be intended to limit the scope of other embodiments. The singular expression may include the plural expression unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meanings as commonly understood by one of ordinary skill in the art described in this document. Among terms used in this document, terms defined in a general dictionary may be interpreted with the same or similar meaning as the meaning in the context of the related art, and unless explicitly defined in this document, ideal or excessively formal meanings is not interpreted as In some cases, even terms defined in this document cannot be construed to exclude embodiments of this document.

Each feature of the various embodiments of the present invention may be partially or wholly combined or combined with each other, and as those skilled in the art will fully understand, technically various interlocking and driving are possible, and each embodiment may be implemented independently of each other, It may be possible to implement together in a related relationship.

In the present specification, an image may be a still image and/or a video, but is not limited thereto.

In the present specification, a hologram may mean at least one of a similar hologram and a floating hologram.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Referring to FIG. 1 , a hologram providing system 100 is a system for providing a hologram using a medical image of a subject, and an electronic device that generates an image for a hologram used to implement a hologram using the medical image of a subject. and a hologram device 120 for generating a hologram by using the 110 and the hologram image. Here, the medical image is an image obtained by photographing a target region of a subject, and includes computer tomography (CT), x-ray, angiography, positron emission tomography (PET-CT), and/or magnetic resonance (MRI). Imaging) image, etc., but is not limited thereto. The target site is a specific body part of the subject whose condition is to be predicted, such as the presence or absence of a disease. Cervix, heart, hypopharyngeal gland, lung, bronchus, liver, skin, ureter, urethra, testis, vagina, anus, larynx, ovary, thyroid gland, esophagus, nasopharyngeal gland, pituitary gland, salivary gland, prostate, pancreas, adrenal gland, lymph node, It may be the spleen, brain, varicose veins, and/or the musculoskeletal system, but is not limited thereto, and various regions that can be acquired as images by a CT device, an X-ray device, an angiography device, a positron emission tomography device, and/or an MRI device. can be

First, the electronic device 110 may be at least one of a tablet PC, a notebook computer, and/or a PC that generates an image for a hologram using a medical image of a subject. In various embodiments, the electronic device 110 may provide a user interface for providing a medical image-based hologram.

The electronic device 110 acquires a medical image from a photographing device that captures a target part of the subject, and recognizes the target part and the suspected lesion region from the acquired medical image, and a holographic image including the recognized target region and the lesion suspect region. can create

In order to recognize the target region and the suspected lesion region from the medical image, the electronic device 110 may use at least one artificial neural network model trained to segment the target region and/or the suspected lesion region of the subject. The electronic device 110 may generate an image for a hologram by using the result data obtained through the artificial neural network model.

In various embodiments, in order to generate an image for a hologram, the electronic device 110 may use a standard modeling image pre-stored in relation to a target site, and a reading such as a diagnosis or opinion of a specialist. For example, the electronic device 110 determines the location of the suspected lesion region using a natural language processing algorithm (NLP) and generates a hologram image based on the standard modeling image and the location of the suspicious lesion region. can

The electronic device 110 may transmit the generated hologram image to the hologram realization device 120 so that the hologram device 120 implements the hologram.

Next, the hologram device 120 is a device for implementing a hologram using a hologram image, and includes a display unit 122 and a display unit that display various contents (eg, text, image, video, icon, banner or symbol, etc.) and a reflector 124 for generating a hologram by reflecting the content displayed through 122 .

Specifically, the display unit 122 may display an image for a hologram including multidirectional 3D objects indicating a target region and a suspected lesion region to implement a hologram.

The reflection unit 124 reflects or refracts the hologram image displayed through the display unit 122 and projects it to the user so that the image is formed. Optical glass made of a transparent material, such as acrylic, or the like, is formed in a polygonal pyramid shape. It may be made, but is not limited thereto, and may be formed in a cylindrical shape depending on the implementation method. The optical glass constituting the reflection unit 124 may be installed to be inclined at a specific angle with respect to the display unit 122 . Here, the specific angle may be an angle suitable for the optical glass to generate a hologram by reflecting each of the multidirectional three-dimensional objects included in the hologram image, for example, may be 45 degrees to 60 degrees, but is not limited thereto. does not

The hologram device 120 can form (or implement) a hologram by reflecting the multidirectional 3D objects of the hologram image displayed through the display unit 122 through the reflection unit 124 .

In various embodiments, the hologram providing system 100 may further include a separate controller (not shown) for controlling the hologram device 120 . Using such a controller, the user can zoom-in, zoom-out, or rotate the hologram generated by the hologram device 120 at various angles to be displayed. In this case, the hologram device 120 receives an instruction from the controller, and can zoom in, zoom out, and/or rotate the hologram at various angles according to the received instruction. You can request a video for use, but is not limited thereto.

In various embodiments, the hologram device 120 may further include a sensor (not shown) for controlling the operation of the hologram device 120 . Here, the sensor may be a motion detection sensor or a camera for detecting a user's movement, but is not limited thereto. For example, when a specific motion is detected through a sensor, the hologram device 120 may perform an operation corresponding to the detected motion (eg, zoom in, zoom out, and/or rotate to various angles).

Through this, according to the present invention, medical staff such as a specialist can easily and conveniently check a medical image using a hologram, and more easily deliver a diagnosis or examination result to a patient.

Hereinafter, the electronic device 110 will be described in more detail with reference to FIG. 2 .

Referring to FIG. 2 , the electronic device 200 includes a communication unit 210 , a storage unit 220 , a display unit 230 , and a control unit 240 . In various embodiments, the display unit 230 may be selectively provided. In the presented embodiment, the electronic device 200 may refer to the electronic device 110 of FIG. 1 .

The communication unit 210 connects the electronic device 200 to enable communication with an external device. The communication unit 210 may include the hologram generating device 120 or an imaging device (eg, a CT device, an X-ray device, an angiography device, a positron emission tomography device, and/or an MRI device for imaging a medical image) using wired/wireless communication. etc.) (not shown) to transmit and receive various data for providing a medical image-based hologram. Specifically, the communication unit 210 may receive a medical image from the imaging device, and transmit the hologram image to the hologram generating device 120 .

The storage 220 may store various data used to provide a medical image-based hologram. For example, the storage 220 may store standard modeling images for various target regions.

In various embodiments, the storage unit 220 may include a flash memory type, a hard disk type, a multimedia card micro type, and a card type memory (eg, SD or XD). memory, etc.), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM) , a magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium. The electronic device 200 may operate in relation to a web storage that performs a storage function of the storage unit 220 on the Internet.

The display unit 230 may display various contents to the user. For example, the display unit 230 may display various interface screens for providing a medical image-based hologram.

In various embodiments, the display unit 230 may include a touch screen, and for example, a touch, a gesture, a proximity, a drag, and a swipe using an electronic pen or a part of the user's body. A swipe or hovering input may be received.

The control unit 240 is operatively connected to the communication unit 210 , the storage unit 220 , and the display unit 230 , and may perform various commands for providing a medical image-based hologram.

The control unit 240 is at least one of a central processing unit (CPU), a graphic processing unit (GPU), an application processor (AP), a digital signal processing unit (DSP), an arithmetic logic unit (ALU), and an artificial neural network processor (NPU) It may be configured to include

Specifically, the controller 240 acquires a medical image from the imaging device through the communication unit 210 , and predicts (or recognizes) a target region of the subject and a region suspected of being a lesion in the target region based on the acquired medical image. , classification) may be used to obtain data as a result of predicting a target region and/or a suspected lesion region from a medical image using at least one image segmentation model. The controller 240 generates a three-dimensional image representing the target site and the suspected lesion region by using the obtained result data, and after generating a hologram image used to generate a hologram by using the generated three-dimensional image, the communication unit It may be transmitted to the hologram device 120 through 210 .

Hereinafter, an operation of the controller 240 for generating an image for a hologram using an artificial neural network model will be described in detail with reference to FIG. 3 .

3 is an exemplary diagram for explaining a method of generating an image for a hologram using an artificial neural network model according to an embodiment of the present invention. Operations to be described later in the presented embodiment may be performed by the controller 240 of FIG. 2 .

Referring to FIG. 3 , the controller 240 receives the medical image 300 as an input and uses the segmented model 310 trained to predict the target region and/or the suspected lesion region from the medical image 300 and the target region and the / Alternatively, segmented data 320 that is data as a result of predicting the lesion suspicious region may be acquired.

The segmentation model 310 includes a first artificial neural network model trained to predict a target site based on the medical image 300 and a second artificial neural network model trained to predict a suspected lesion region based on the medical image 300 . can do. In various embodiments, the segmentation model 310 may be a single artificial neural network model trained to predict a target site and a suspected lesion region based on the medical image 300 . In various embodiments, the segmentation model 310 includes a first artificial neural network model trained to predict a target site based on the medical image 300 and a second artificial neural network model trained to predict a suspected lesion region based on the result data of the first artificial neural network. 2 It may include an artificial neural network model. As such, the segmentation model 310 may be formed of a single artificial neural network or a combination of a plurality of artificial neural networks, but is not limited thereto.

Meanwhile, the segmentation model 310 may be an artificial neural network model configured to learn a plurality of reference images in advance and predict a region corresponding to a target region and/or a region suspected of a lesion from a newly input medical image. Here, the plurality of reference images may include, but is not limited to, medical images obtained by photographing target regions of various subjects and/or medical images obtained by photographing regions suspected of lesions of various subjects.

The segmentation model 310 may be a pre-trained convolutional neural network (CNN), but is not limited thereto. The pre-trained convolutional neural network may be composed of one or more layers that perform convolution operations on an input value, and may infer an output value by performing a convolution operation on the input value. For example, a pre-trained convolutional neural network performs an image segmentation operation for segmenting (or classifying) a predicted region into an object (ie, a target region and/or a region suspected of a lesion) in a plurality of artificial neural network stages. FCN (Fully Convolutional Network), U-net, DeepLab, and Mask R-CNN (Regions with Convolutional Neural Network) that perform In order to improve the performance of such an artificial neural network model, performance evaluation using a dice coefficient may be performed.

The segmented data 320 output through the segmentation model 310 includes a first segmented region that masks a pixel region predicted as a target region and a second segmented region that masks a pixel region predicted as a lesion suspect region. may be an image, but is not limited thereto. Here, the first divided area and the second divided area may be expressed to be distinguished from each other. According to various embodiments, the segmented data 320 may further include data indicating prediction accuracy for each of the first segmented region and the second segmented region.

The controller 240 generates a three-dimensional image 330 indicating a target site and/or a suspected lesion region based on the divided data 320 , and based on the generated three-dimensional image 330 , a hologram image 340 . ) can be created.

Specifically, the controller 240 extracts a target site and/or a suspected lesion region from the medical image 300 based on the segmentation data 320, and a three-dimensional image ( 330) can be created. For example, a rendering technique for expressing a surface texture may be used to generate a 3D image based on segmentation data. Also, in the 3D image, the target region and the region suspected of a lesion may be expressed to be distinguished from each other, for example, may be expressed to be distinguished from each other in different colors.

The controller 240 uses the 3D image 330 to display the 3D objects 342 , 344 , 346 , 348 arranged in multiple directions to correspond to the reflector 124 of the hologram device 120 . (340) can be created. For example, the hologram image 340 is a 3D modeled object (ie, a 3D object) (eg, lung) rotated by 0°, 90°, 180°, and 270° with respect to the central axis. It may be an image in which a pair of objects are disposed on top, bottom, left, and right to face each other, but is not limited thereto. In other words, the three-dimensional object is disposed on the left and right with respect to the center line that bisects one direction of the hologram image 340, and up and down based on the center line that bisects one direction and the other direction of the hologram image 340 can be placed. The 3D objects 342 and 346 disposed up and down may be symmetric with each other, and the 3D objects 344 and 348 disposed on the left and right may also form symmetry with each other. The number, arrangement direction, and position of these three-dimensional objects may be determined by the arrangement shape and number of optical glasses constituting the reflection unit 124 , but is not limited thereto.

The controller 240 may provide the generated hologram image 340 to the hologram device 120 . Hereinafter, a hologram generated by the hologram device 120 receiving the hologram image will be described with reference to FIG. 4 .

Referring to FIG. 4 , the hologram device 120 displays the hologram image 340 provided from the electronic device 200 through the display unit 122 , and the hologram image 340 is reflected through the reflector 124 . Thus, a hologram can be generated as indicated by reference numeral 400 .

On the other hand, the artificial neural network model may be difficult to accurately recognize (or predict) the target site and/or the suspected lesion area depending on the state, shape, and/or surrounding area of the target site shown in the medical image. For example, the lung among the target regions may be suitable for recognition using an artificial neural network model, but in the case of the large intestine, it may be difficult to recognize using the artificial neural network model due to the shape of the colon or other organs located nearby. .

As such, when it is difficult to recognize the target site and/or the suspected lesion region in the medical image obtained by photographing the subject's target site, the control unit 240 uses pre-stored standard modeling images for various target sites and readings such as a diagnosis or opinion of a specialist. Thus, it is possible to generate an image for a hologram.

Hereinafter, an operation of the controller 240 for generating an image for a hologram by using the standard modeling image and reading text stored in advance for various parts will be described in detail with reference to FIG. 5 .

5 is an exemplary diagram for explaining a method of generating an image for a hologram using pre-stored standard modeling images and readouts for various parts according to an embodiment of the present invention. Operations to be described later in the presented embodiment may be performed by the controller 240 of FIG. 2 .

Referring to FIG. 5 , the controller 240 determines whether the target site is a predictable site from the medical image 500 using the artificial neural network model, and the target site is predictable from the medical image 500 using the artificial neural network model. In the case of a region, the operation described in FIG. 3 may be performed.

If the target site is not a predictable site from the medical image 500 using the artificial neural network model, the controller 240 determines the pre-stored standard modeling image in relation to the target site and the expert in relation to the suspected lesion region of the target site. An image for a hologram can be generated using the written reading (eg, a medical certificate and/or a statement of opinion).

The operation of the controller 240 for determining whether the target region is a predictable region from the medical image using the artificial neural network model is performed after obtaining the medical image, or the result of predicting the target region from the medical image using the artificial neural network model This may be performed after data is output.

In one embodiment, when the medical image 500 is acquired, the controller 240 determines that the target part of the acquired medical image 500 is a first part predictable using an artificial neural network or difficult to predict using an artificial neural network. It can be determined whether it is the second part. For example, predetermined reference data for the first portion and the second portion may be pre-stored in the storage unit 220 . Here, the first part is a part (eg, lung, etc.) where the accuracy of data as a result of predicting the target part using an artificial neural network is determined to correspond to a preset threshold accuracy or higher, and the second part is a part where the accuracy of the result data is critical accuracy. It may be a site determined to be less than (eg, large intestine, etc.).

The controller 240 may determine whether the target region related to the medical image is the first region or the second region based on the reference data.

When the target region of the medical image 500 corresponds to the first region, the controller 240 performs the operation as described in FIG. 3 , and when the medical image 500 corresponds to the first region, the controller 240 Based on the medical image 500, a readout 520 written by a specialist in relation to a suspected lesion region is obtained, and a three-dimensional image using the standard modeling image 510 and the obtained readout 520 stored in advance for the target site. 530 may be created.

Specifically, the controller 240 may analyze at least one sentence constituting the read text and extract a keyword related to the suspected lesion region from the at least one sentence. To this end, the controller 240 may use a natural language processing algorithm, but is not limited thereto.

The controller 240 may determine a location of a suspected lesion region and a corresponding suspicious lesion region based on the extracted keyword, and may generate a lesion modeling image indicating the suspected lesion region based on the determined location.

The controller 240 may generate the 3D image 530 by combining the standard modeling image and the lesion modeling image. For coupling, the controller 240 may use the determined position of the suspected lesion region.

In another embodiment, as described with reference to FIG. 3 , the controller 240 determines that the prediction accuracy of the first segmented region corresponding to the target portion in segmented data 320 that is the result data of the segmentation model 310 is greater than or equal to a preset threshold. cognition can be determined.

When the prediction accuracy of the first divided region is equal to or greater than the threshold value, the controller 240 may perform the 3D image generation operation described with reference to FIG. 3 .

When the prediction accuracy of the first segmented region is less than the threshold value, the controller 240 may generate the 3D image 530 using the standard modeling image 510 of the target region and the reading text 520 prepared by a specialist.

In various embodiments, the controller 240 may evaluate the performance of the segmentation model and generate a 3D image according to the evaluation result. For example, when the evaluation result (eg, accuracy, precision, recall, Receiver Operating Characteristic (ROC) curve, and/or Area Under Curve (AUC) value, etc.) is greater than or equal to a preset threshold value, the controller 240 may The described 3D image generation operation may be performed. When the evaluation result is less than the threshold value, the controller 240 may generate the 3D image 530 using the standard modeling image 510 of the target site and the reading text 520 prepared by a specialist.

In various embodiments, when a request for generation of a 3D image using an artificial neural network model or a request for generation of a 3D image using a standard modeling image and readout is requested, the control unit 240 generates an artificial neural network model according to the request. An operation of generating a 3D image using a 3D image or generating a 3D image using a standard modeling image and a reading may be performed.

When the three-dimensional image 530 is generated in this way, the controller 240 uses the three-dimensional image 530 to generate the three-dimensional objects 542, An image 540 for a hologram representing 544 , 546 , and 548 may be generated. Since the operation of the controller 240 for generating the hologram image 540 is similar to the operation of the controller 240 described with reference to FIG. 3 , a detailed description thereof will be omitted.

The controller 240 may provide the generated hologram image 540 to the hologram device 120 . Hereinafter, a hologram generated by the hologram device 120 receiving the hologram image will be described with reference to FIG. 6 .

Referring to FIG. 6 , the hologram device 120 displays the hologram image 540 provided from the electronic device 200 through the display unit 122 , and the hologram image is reflected through the reflection unit 124 , and the reference numerals You can create a hologram like 600.

In various embodiments, the control unit 240 predicts a target site from a medical image using an artificial neural network model, determines the location of the suspected lesion area and the suspected lesion area using a readout such as a diagnosis or observation written by a specialist, and based on this It is also possible to create a 3D image.

In various embodiments, the controller 240 predicts a suspected lesion region from a medical image using an artificial neural network model, and generates a three-dimensional image based on a pre-stored standard modeling image and the predicted lesion suspicious region in relation to the target site. may be

In various embodiments, the controller 240 may display various interface screens for providing a medical image-based hologram through the display unit 230 . Various interface screens acquire a medical image, generate a 3D image using an artificial neural network model, create a 3D image using a pre-stored standard modeling image and a reading written by a specialist, and create and provide an image for a hologram It may include an interface screen for various operations such as

Through this, in the present invention, since the medical staff can explain the diagnosis or examination result to the patient using the hologram, it is possible to increase the patient's understanding of the diagnosis or examination result within the prescribed treatment time.

In addition, according to the present invention, a medical professional can easily and conveniently check a medical image using a hologram without unnecessary movement for checking a medical image during surgery or a separate device.

Hereinafter, a method for providing a medical image-based hologram in an electronic device will be described with reference to FIG. 7 .

7 is a flowchart illustrating a method for providing a medical image-based hologram in an electronic device according to an embodiment of the present invention. The operations described below may be performed by the controller 240 of FIG. 2 .

Referring to FIG. 7 , the controller 240 acquires a medical image obtained by photographing a target part of the subject ( S700 ). Here, the medical image may be at least one of an X-ray image, a CT image, an angiography image, a positron emission tomography image, and an MRI image.

The controller 240 recognizes the target region and the suspected lesion region from the medical image using the artificial neural network model trained to recognize the lesion suspected region in the target region and the target region based on the obtained medical image. is obtained (S710). Here, the artificial neural network model is a segmented model trained to predict a target site and/or a suspected lesion region, and uses the segmented model to output segmented data representing a predicted target site and/or a suspected lesion region from a medical image as result data. can do.

The controller 240 generates a three-dimensional image representing the target region and the suspected lesion region based on the result data (S720). For example, the controller 240 may generate a 3D image by extracting a target region and a suspected lesion region from the medical image, and 3D rendering a partial image corresponding to the extracted target region and the suspected lesion region. not limited

The controller 240 generates a hologram image used to generate a hologram by using the generated 3D image (S730), and uses the hologram image generated so that the hologram device 120 generates a hologram to the hologram device 120 ) is provided (S740).

Hereinafter, various interface screens for providing a medical image-based hologram will be described with reference to FIGS. 8 and 9 .

* FIG. 8 is an exemplary diagram illustrating an interface screen for providing a hologram image using an artificial neural network model according to an embodiment of the present invention. In the presented embodiment, the interface screen may be displayed through the display unit 230 of FIG. 2 .

Referring to FIG. 8 , the controller 240 may display an interface screen 800 for providing an image for a hologram through the display unit 230 using an artificial neural network model.

The interface screen 800 includes a first graphic object 805 for requesting display of at least one medical image, and a display area for displaying at least one medical image (ie, a first display area 810 , and a second Display area 815, third display area 820), a second graphic object 825 for requesting generation of a 3D image using an artificial neural network model, and a fourth display area 830 for displaying a 3D image ) and a third graphic object 835 for providing a hologram. The interface screen 800 may further include an input area 840 for inputting a reading and a fourth graphic object 845 for storing the inputted reading.

When the first graphic object 805 is selected, the controller 240 may display at least one medical image for providing a hologram on the

display areas

810 , 815 , and 820 . For example, when the first graphic object 805 is selected, the controller 240 requests a medical image from the imaging device, receives the medical image from the imaging device, and displays the received medical image on the

display areas

810 , 815 , and 820 . can be displayed in In various embodiments, when a medical image is pre-stored in the storage 220 , the controller 240 may display the stored medical image on the

display areas

810 , 815 , and 820 .

When the second graphic object 825 is selected, the control unit 240 obtains the result data of predicting the target region and the lesion suspected region using the artificial neural network model, and generates a 3D image based on the obtained result data. have. The generated 3D image may be displayed on the fourth display area 830 .

When the third graphic object 835 is selected to provide the hologram, the controller 240 may generate an image for the hologram by using the 3D image, and transmit the generated image for the hologram to the hologram device 120 . The hologram device 120 receiving the hologram image may generate a hologram as described with reference to FIG. 4 .

In various embodiments, when a reading is input through the input area 840 and a fourth graphic object 845 for requesting storage of the inputted reading is selected, the controller 240 stores the inputted reading in the storage unit 220 . can be stored in

In various embodiments, when a suspected lesion region is predicted, the controller 240 may display a graphic object indicating the predicted area in the form of a notification window, but is not limited thereto.

In various embodiments, when the 3D image is updated by the user's manipulation of an input device (not shown) (or a controller), the controller 240 generates an image for a hologram using the updated 3D image, By transmitting the generated hologram image to the hologram device 120 , the hologram device 120 may reflect the updated information on the hologram in real time.

9 is an exemplary diagram illustrating an interface screen for providing an image for a hologram using a pre-stored standard modeling image and readout according to an embodiment of the present invention. In the presented embodiment, the interface screen may be displayed through the display unit 230 of FIG. 2 .

Referring to FIG. 9 , the controller 240 may display an interface screen 900 for providing an image for a hologram through the display unit 230 using a standard modeling image and a readout.

The interface screen 900 includes a first graphic object 905 for requesting display of at least one medical image, a display area for displaying at least one medical image (ie, a first display area 910 , and a second A display area 915, a third display area 920), a second graphic object 925 for requesting display for a standard modeling image, an input area 930 for inputting a readout, and storing the inputted readout It may include a third graphic object 935 for displaying a 3D image, a fourth display area 940 for displaying a 3D image, and a fourth graphic object 945 for providing a hologram.

When the first graphic object 905 is selected, the controller 240 may display at least one medical image for providing a hologram on the

display areas

910 , 915 , and 920 . For example, when the displayed medical image is a medical image of the large intestine, the user may select the second graphic object 925 for generating a 3D image using a standard modeling image and a readout.

In this way, in order to generate a three-dimensional image using the standard modeling image and the reading, a reading is input from the user through the input area 930, and a third graphic object 935 for requesting storage of the inputted reading can be selected. have. When the third graphic object is selected, the control unit 240 may store the input reading text in the storage unit 220 .

When the second graphic object 925 is selected, the control unit 240 analyzes at least one sentence constituting the read text, extracts a keyword related to the suspicious lesion region from the at least one sentence, and based on the extracted keyword, the suspected lesion region and a location of a suspected lesion region may be determined. The control unit 240 generates a lesion modeling image representing the suspected lesion region based on the position of the suspected lesion region and the lesion suspect region, and combines the standard modeling image and the lesion modeling image of the lung pre-stored in the storage 220 to A 3D image can be created. For example, the controller 240 may combine the lesion modeling images to correspond to positions determined from the standard modeling images. The generated 3D image may be displayed on the fourth display area 940 .

In various embodiments, when the location of the suspected lesion area and the location of the suspected lesion area is determined, the controller 240 may display a graphic object indicating the same in the form of a notification window, but is not limited thereto.

When the fourth graphic object 945 is selected to provide the hologram, the controller 240 may generate an image for the hologram by using the 3D image, and transmit the generated image for the hologram to the hologram device 120 . The hologram device 120 receiving the hologram image may generate a hologram as described with reference to FIG. 6 .

In various embodiments, an interface screen for providing an image for a hologram using an artificial neural network model and an interface screen for providing an image for a hologram using a standard modeling image and readout may be provided as one interface screen, but is not limited thereto. does not

In the presented embodiment, the configuration of various interface screens and graphic objects related to providing a medical image-based hologram is not limited to the above description, and each interface screen and graphic objects may be configured in various ways.

Through this, according to the present invention, the medical staff can provide a high service for outpatient treatment, and even if the medical staff treats many patients for a predetermined time, the patient's satisfaction can be increased.

In addition, according to the present invention, a specialist can focus on surgery in an operating room, and can easily and conveniently check a patient's medical image.

The apparatus and method according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination.

The program instructions recorded on the computer readable medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the computer software field. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - Includes magneto-optical media and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Although the embodiments of the present invention have been described in more detail with reference to the accompanying drawings, the present invention is not necessarily limited to these embodiments, and various modifications may be made within the scope without departing from the technical spirit of the present invention. . Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The protection scope of the present invention should be construed by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

[National R&D project supporting this invention]

[Project unique number] 1711102921

[task number] 2017-0-01630-004

[Name of Ministry] Ministry of Science and Technology Information and Communication

[Name of project management (specialized) institution] Information and Communication Planning and Evaluation Institute

[Research project name] University ICT Research Center Fostering Support Project

[Research project name] Development and development of brain disease prediction and prevention technology using medical big data

literacy training

[Contribution rate] 1/2

[Name of project execution institution] Gachon University Industry-Academic Cooperation Foundation

[Research Period] 2020.01.01 ~ 2020.12.31

[National R&D project supporting this invention]

[Project unique number] 1425142088

[task number] S2797147

[Ministry name] Ministry of SMEs and Startups

[Name of task management (specialized) institution] Small and Medium Business Technology Information Promotion Agency

[Research project name] Startup growth technology development project

[Research project name] Deep classification of lesions using cervical color image database

Development of learning-based artificial intelligence model and establishment and commercialization of collective intelligence cooperation system

[Contribution rate] 1/2

[Name of the organization performing the task] Bethesda Soft Co., Ltd.

[Research period] 2019.11.25 ~ 2021.11.24

Claims

a communication unit configured to transmit and receive data; and

a control unit configured to connect with the communication unit;

The control unit is

Obtaining a medical image of the target part of the subject through the communication unit,

The result of predicting the target site and the suspected lesion area from the medical image using an artificial neural network model trained to predict the target site and the lesion suspected area at the target site based on the obtained medical image acquire data,

generating a three-dimensional image representing the target site and the suspected lesion region based on the result data;

generating an image for a hologram used to generate a hologram by using the generated three-dimensional image;

The hologram device is configured to provide the generated hologram image to the hologram device so that the hologram device generates the hologram.
According to claim 1, wherein the artificial neural network model,

Medical image based, comprising a first artificial neural network model trained in advance to predict the target site based on the medical image, and a second artificial neural network model trained in advance to predict the lesion suspicious region based on the medical image Hologram providing device.
According to claim 1, wherein the result data,

A medical image-based hologram providing apparatus representing a first segmented area masking the area predicted as the target region and a second segmented area masking the area predicted as the suspected lesion area.
According to claim 3, wherein the control unit,

extracting a partial image corresponding to the target region and the suspected lesion region from the medical image based on the result data;

3D rendering the extracted partial image to generate the 3D image, a medical image-based hologram providing apparatus.
According to claim 1, wherein the control unit,

Determining whether the target site is a predictable site using the artificial neural network model,

When the target region is a region predictable using the artificial neural network model, the medical image-based hologram providing apparatus is configured to predict the target region using the artificial neural network model.
According to claim 5, wherein the control unit,

To determine whether the target site is a first site in which the accuracy of result data predicted using the artificial neural network is equal to or greater than a preset threshold accuracy or a second site in which the accuracy of result data predicted using the artificial neural network is less than the threshold accuracy Consisting of, a medical image-based hologram providing device.
According to claim 5, further comprising a storage unit configured to store the standard modeling image pre-stored in relation to the target site,

The control unit is

When the target site is not a predictable site using the artificial neural network model, a reading written by a specialist in relation to the suspected lesion area is obtained,

and generating a three-dimensional image representing the target region and the suspected lesion region by using the standard modeling image and the obtained readout.
According to claim 7, wherein the control unit,

extracting a keyword related to the suspicious lesion region from the at least one sentence by analyzing at least one sentence constituting the read sentence;

determining the positions of the suspected lesion region and the suspected lesion region based on the extracted keyword;

generating a lesion modeling image representing the suspected lesion region based on the position of the suspected lesion region and the suspected lesion region;

A medical image-based hologram providing apparatus, configured to generate the 3D image by combining the standard modeling image and the lesion modeling image.
According to claim 1, wherein the control unit,

Determining whether the prediction accuracy of the data as a result of predicting the target region from the medical image is greater than or equal to a preset threshold,

When the prediction accuracy is greater than or equal to the threshold, the 3D image is generated,

If the prediction accuracy is less than the threshold, obtaining a reading prepared by a specialist in relation to the suspected lesion region;

A medical image-based hologram providing apparatus, configured to generate a three-dimensional image representing the target site and the suspected lesion region by using a pre-stored standard modeling image and the obtained readout in relation to the target site.
In the medical image-based hologram providing method performed by a controller of a medical image-based hologram providing apparatus,

acquiring a medical image of a target part of the subject;

The result of predicting the target site and the suspected lesion area from the medical image using an artificial neural network model trained to predict the target site and the lesion suspected area at the target site based on the obtained medical image acquiring data;

generating a three-dimensional image representing the target region and the suspected lesion region based on the result data;

generating an image for a hologram used to generate a hologram by using the generated 3D image; and

A hologram device comprising the step of providing the generated hologram image to the hologram device to generate the hologram, a medical image-based hologram providing device.
11. The method of claim 10, wherein the artificial neural network model,

Medical image based, comprising a first artificial neural network model trained in advance to predict the target site based on the medical image, and a second artificial neural network model trained in advance to predict the lesion suspicious region based on the medical image How to provide a hologram.
The method of claim 10, wherein the result data,

A method for providing a medical image-based hologram, indicating a first divided region masking the region predicted as the target region and a second divided region masking the region predicted as the suspected lesion region.
The method of claim 12, wherein the generating of the 3D image comprises:

extracting a partial image corresponding to the target region and the suspected lesion region from the medical image based on the result data; and

and generating the 3D image by 3D rendering the extracted partial image.
11. The method of claim 10,

determining whether the target site is a predictable site using the artificial neural network model when the medical image is obtained; and

When the target site is a predictable site using the artificial neural network model, the method further comprising predicting the target site using the artificial neural network model.
The method of claim 14, wherein the step of determining whether the target site is a predictable site using the artificial neural network model comprises:

Whether the target site is a first site in which the accuracy of result data predicted using the artificial neural network is greater than or equal to a preset threshold accuracy, or a second site in which the accuracy of result data predicted using the artificial neural network is less than the threshold accuracy The determining step, a method of providing a hologram based on a medical image.
15. The method of claim 14,

obtaining a reading prepared by a specialist in relation to the suspected lesion region when the target region is not a predictable region using the artificial neural network model; and

and generating a three-dimensional image representing the target region and the suspected lesion region by using a standard modeling image and the acquired readout previously stored in relation to the target region.
The method of claim 16, wherein the generating of the three-dimensional image using the standard modeling image and the obtained readout comprises:

extracting a keyword related to the suspected lesion region from the at least one sentence by analyzing at least one sentence constituting the read sentence;

determining positions of the suspected lesion region and the suspected lesion region based on the extracted keyword;

generating a lesion modeling image representing the suspected lesion region based on the position of the suspected lesion region and the suspected lesion region; and

and generating the 3D image by combining the standard modeling image and the lesion modeling image.
The method of claim 10, wherein the generating of the 3D image based on the result data comprises:

determining whether prediction accuracy of data as a result of predicting the target region from the medical image is equal to or greater than a preset threshold;

generating the 3D image when the prediction accuracy is greater than or equal to the threshold value;

if the prediction accuracy is less than the threshold, obtaining a reading prepared by a specialist in relation to the suspected lesion region; and

and generating a three-dimensional image representing the target region and the suspected lesion region by using a standard modeling image and the acquired readout previously stored in relation to the target region.