WO2022124705A1 - Appareil et procédé pour fournir un hologramme basé sur une image médicale - Google Patents

Appareil et procédé pour fournir un hologramme basé sur une image médicale Download PDF

Info

Publication number
WO2022124705A1
WO2022124705A1 PCT/KR2021/018211 KR2021018211W WO2022124705A1 WO 2022124705 A1 WO2022124705 A1 WO 2022124705A1 KR 2021018211 W KR2021018211 W KR 2021018211W WO 2022124705 A1 WO2022124705 A1 WO 2022124705A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
hologram
region
medical image
neural network
Prior art date
Application number
PCT/KR2021/018211
Other languages
English (en)
Korean (ko)
Inventor
김광기
백정흠
김영재
Original Assignee
가천대학교 산학협력단
의료법인 길의료재단
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 가천대학교 산학협력단, 의료법인 길의료재단 filed Critical 가천대학교 산학협력단
Publication of WO2022124705A1 publication Critical patent/WO2022124705A1/fr

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/74—Details of notification to user or communication with user or patient ; user input means
    • A61B5/742—Details of notification to user or communication with user or patient ; user input means using visual displays
    • A61B5/745—Details of notification to user or communication with user or patient ; user input means using visual displays using a holographic display
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/0033—Features or image-related aspects of imaging apparatus classified in A61B5/00, e.g. for MRI, optical tomography or impedance tomography apparatus; arrangements of imaging apparatus in a room
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B6/00—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B6/00—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
    • A61B6/46—Arrangements for interfacing with the operator or the patient
    • A61B6/461—Displaying means of special interest
    • A61B6/466—Displaying means of special interest adapted to display 3D data
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B6/00—Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment
    • A61B6/52—Devices using data or image processing specially adapted for radiation diagnosis
    • A61B6/5211—Devices using data or image processing specially adapted for radiation diagnosis involving processing of medical diagnostic data
    • A61B6/5217—Devices using data or image processing specially adapted for radiation diagnosis involving processing of medical diagnostic data extracting a diagnostic or physiological parameter from medical diagnostic data
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems

Definitions

  • the present invention relates to an apparatus and method for providing a hologram based on a medical image.
  • hospital treatment is performed in such a way that a patient visits a hospital, sees a specialist, receives treatment, or receives an examination for diagnosing a disease, and receives the result from the specialist.
  • imaging medical examinations of the patient such as X-rays, Computer Tomography (CT), Angiography, Positron Emission Tomography (PET-CT), or Magnetic Resonance Imaging (MRI)
  • CT Computer Tomography
  • PET-CT Positron Emission Tomography
  • MRI Magnetic Resonance Imaging
  • the specialist shows the patient a medical image such as an X-ray, CT image, angiography image, PET-CT image, or MRI image of the patient's target area (eg, lung or colon) for diagnosis Or deliver test results, etc.
  • target area eg, lung or colon
  • the medical image may be used by a specialist to operate on a patient in an operating room.
  • a medical assistant for monitoring is required to check the medical image, or a specialist must move to a monitoring device that displays a medical image in order to check the medical image during an operation.
  • AR augmented reality
  • the inventors of the present invention are aware of the fact that the medical image presented by the specialist is difficult for non-medical patients to understand, and there is no additional means for reference other than the medical image for the patient's understanding. recognized.
  • the inventors of the present invention recognized the fact that there is a problem in that an assistant manpower for monitoring a medical image in an operating room is required or that a specialist needs to move to a monitoring device during an operation.
  • the inventors of the present invention recognized the fact that even if augmented reality is used to check a medical image, there is an inconvenience in that a specialist needs to wear a separate device in the operating room.
  • an object of the present invention is to provide an apparatus and method for providing a hologram based on a medical image.
  • an object of the present invention is to provide an apparatus and method for providing a hologram based on a medical image in order to easily transmit a diagnosis or examination result to a patient.
  • the problem to be solved by the present invention is a medical image-based hologram providing apparatus and method for conveniently checking a patient's medical image without requiring an assistant to check a medical image or a specialist moving to a monitoring device during surgery is to provide
  • Another object of the present invention is to provide a medical image-based hologram providing apparatus and method for easily checking a patient's medical image without the need for a specialist to wear a separate device.
  • An apparatus and method for providing a hologram based on a medical image are provided in order to solve the above problems.
  • an apparatus for providing a medical image-based hologram comprising: a communication unit configured to transmit and receive data; and a control unit configured to be connected to the communication unit, wherein the control unit obtains a medical image obtained by photographing a target site of the subject through the communication unit, and based on the obtained medical image, the target site and the lesion at the target site Obtaining result data of predicting the target region and the lesion suspected region from the medical image using an artificial neural network model trained to predict the suspected lesion region, and based on the result data, the target region and the lesion region A three-dimensional image representing the suspicious region is generated, a hologram image used to generate a hologram is generated using the generated three-dimensional image, and the generated hologram image is used so that the hologram device generates the hologram. configured to serve as a holographic device.
  • a medical image-based hologram providing method performed by a controller of a medical image-based hologram providing apparatus according to an embodiment of the present invention, the method comprising: acquiring a medical image obtained by photographing a target part of a subject; Results of predicting the target site and the suspected lesion region from the medical image using an artificial neural network model trained to predict the target site and the lesion suspected area at the target site based on the obtained medical image acquiring data; generating a three-dimensional image representing the target region and the suspected lesion region based on the result data; generating an image for a hologram used to generate a hologram by using the generated 3D image; and providing the generated hologram image to the hologram device so that the hologram device generates the hologram.
  • the present invention provides a hologram based on a medical image, so that a specialist can more easily deliver a diagnosis or test result to a patient.
  • the present invention increases the patient's understanding of the diagnosis or examination results during a prescribed treatment time because the patient can listen to the expert's explanation of the diagnosis or examination result while the specialist and the patient view the hologram based on the medical image together.
  • the present invention can improve patient satisfaction with outpatient treatment even when a specialist treats many patients during a prescribed treatment time.
  • a specialist can efficiently deliver a diagnosis or examination result, a description of a disease, a treatment plan, and the like to a patient within a prescribed treatment time, thereby providing a higher medical service to the patient.
  • a specialist can easily check a patient's medical image without adding an assistant for monitoring the medical image in the operating room.
  • a medical professional can conveniently check a patient's medical image without wearing a separate device in the operating room or without unnecessary movement to check the medical image.
  • the effect according to the present invention is not limited by the contents exemplified above, and more various effects are included in the present specification.
  • FIG. 1 is a schematic diagram for explaining a hologram providing system according to an embodiment of the present invention.
  • FIG. 2 is a schematic block diagram of an electronic device according to an embodiment of the present invention.
  • FIG. 3 is an exemplary diagram for explaining a method of generating an image for a hologram using an artificial neural network model according to an embodiment of the present invention.
  • FIG. 4 is a diagram illustrating a hologram generated by a hologram device according to an embodiment of the present invention.
  • FIG. 5 is an exemplary diagram for explaining a method of generating an image for a hologram using pre-stored standard modeling images and readouts for various parts according to an embodiment of the present invention.
  • FIG. 6 is a diagram illustrating a hologram generated by a hologram device according to an embodiment of the present invention.
  • FIG. 7 is a flowchart illustrating a method for providing a medical image-based hologram in an electronic device according to an embodiment of the present invention.
  • FIG. 8 is an exemplary diagram illustrating an interface screen for providing a hologram image using an artificial neural network model according to an embodiment of the present invention.
  • FIG. 9 is an exemplary diagram illustrating an interface screen for providing an image for a hologram using a pre-stored standard modeling image and readout according to an embodiment of the present invention.
  • expressions such as “have,” “may have,” “includes,” or “may include” refer to the presence of a corresponding characteristic (eg, a numerical value, function, operation, or component such as a part). and does not exclude the presence of additional features.
  • expressions such as “A or B,” “at least one of A or/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together.
  • “A or B,” “at least one of A and B,” or “at least one of A or B” means (1) includes at least one A, (2) includes at least one B; Or (3) it may refer to all cases including both at least one A and at least one B.
  • first may modify various elements, regardless of order and/or importance, and refer to one element. It is used only to distinguish it from other components, and does not limit the components.
  • first user equipment and the second user equipment may represent different user equipment regardless of order or importance.
  • the first component may be named as the second component, and similarly, the second component may also be renamed as the first component.
  • a component eg, a first component is "coupled with/to (operatively or communicatively)" to another component (eg, a second component);
  • another component eg, a second component
  • the certain element may be directly connected to the other element or may be connected through another element (eg, a third element).
  • a component eg, a first component
  • another component eg, a second component
  • a device configured to may mean that the device is “capable of” with other devices or parts.
  • a processor configured (or configured to perform) A, B, and C refers to a dedicated processor (eg, an embedded processor) for performing the corresponding operations, or by executing one or more software programs stored in a memory device.
  • a generic-purpose processor eg, a CPU or an application processor
  • an image may be a still image and/or a video, but is not limited thereto.
  • a hologram may mean at least one of a similar hologram and a floating hologram.
  • FIG. 1 is a schematic diagram for explaining a hologram providing system according to an embodiment of the present invention.
  • a hologram providing system 100 is a system for providing a hologram using a medical image of a subject, and an electronic device that generates an image for a hologram used to implement a hologram using the medical image of a subject. and a hologram device 120 for generating a hologram by using the 110 and the hologram image.
  • the medical image is an image obtained by photographing a target region of a subject, and includes computer tomography (CT), x-ray, angiography, positron emission tomography (PET-CT), and/or magnetic resonance (MRI). Imaging) image, etc., but is not limited thereto.
  • the target site is a specific body part of the subject whose condition is to be predicted, such as the presence or absence of a disease. Cervix, heart, hypopharyngeal gland, lung, bronchus, liver, skin, ureter, urethra, testis, vagina, anus, larynx, ovary, thyroid gland, esophagus, nasopharyngeal gland, pituitary gland, salivary gland, prostate, pancreas, adrenal gland, lymph node, It may be the spleen, brain, varicose veins, and/or the musculoskeletal system, but is not limited thereto, and various regions that can be acquired as images by a CT device, an X-ray device, an angiography device, a positron emission tomography device, and/or an MRI device. can be
  • the electronic device 110 may be at least one of a tablet PC, a notebook computer, and/or a PC that generates an image for a hologram using a medical image of a subject.
  • the electronic device 110 may provide a user interface for providing a medical image-based hologram.
  • the electronic device 110 acquires a medical image from a photographing device that captures a target part of the subject, and recognizes the target part and the suspected lesion region from the acquired medical image, and a holographic image including the recognized target region and the lesion suspect region. can create
  • the electronic device 110 may use at least one artificial neural network model trained to segment the target region and/or the suspected lesion region of the subject.
  • the electronic device 110 may generate an image for a hologram by using the result data obtained through the artificial neural network model.
  • the electronic device 110 may use a standard modeling image pre-stored in relation to a target site, and a reading such as a diagnosis or opinion of a specialist. For example, the electronic device 110 determines the location of the suspected lesion region using a natural language processing algorithm (NLP) and generates a hologram image based on the standard modeling image and the location of the suspicious lesion region.
  • NLP natural language processing algorithm
  • the electronic device 110 may transmit the generated hologram image to the hologram realization device 120 so that the hologram device 120 implements the hologram.
  • the hologram device 120 is a device for implementing a hologram using a hologram image, and includes a display unit 122 and a display unit that display various contents (eg, text, image, video, icon, banner or symbol, etc.) and a reflector 124 for generating a hologram by reflecting the content displayed through 122 .
  • contents eg, text, image, video, icon, banner or symbol, etc.
  • the display unit 122 may display an image for a hologram including multidirectional 3D objects indicating a target region and a suspected lesion region to implement a hologram.
  • the reflection unit 124 reflects or refracts the hologram image displayed through the display unit 122 and projects it to the user so that the image is formed.
  • Optical glass made of a transparent material, such as acrylic, or the like, is formed in a polygonal pyramid shape. It may be made, but is not limited thereto, and may be formed in a cylindrical shape depending on the implementation method.
  • the optical glass constituting the reflection unit 124 may be installed to be inclined at a specific angle with respect to the display unit 122 .
  • the specific angle may be an angle suitable for the optical glass to generate a hologram by reflecting each of the multidirectional three-dimensional objects included in the hologram image, for example, may be 45 degrees to 60 degrees, but is not limited thereto. does not
  • the hologram device 120 can form (or implement) a hologram by reflecting the multidirectional 3D objects of the hologram image displayed through the display unit 122 through the reflection unit 124 .
  • the hologram providing system 100 may further include a separate controller (not shown) for controlling the hologram device 120 .
  • a separate controller for controlling the hologram device 120 .
  • the user can zoom-in, zoom-out, or rotate the hologram generated by the hologram device 120 at various angles to be displayed.
  • the hologram device 120 receives an instruction from the controller, and can zoom in, zoom out, and/or rotate the hologram at various angles according to the received instruction. You can request a video for use, but is not limited thereto.
  • the hologram device 120 may further include a sensor (not shown) for controlling the operation of the hologram device 120 .
  • the sensor may be a motion detection sensor or a camera for detecting a user's movement, but is not limited thereto.
  • the hologram device 120 may perform an operation corresponding to the detected motion (eg, zoom in, zoom out, and/or rotate to various angles).
  • medical staff such as a specialist can easily and conveniently check a medical image using a hologram, and more easily deliver a diagnosis or examination result to a patient.
  • FIG. 2 is a schematic block diagram of an electronic device according to an embodiment of the present invention.
  • the electronic device 200 includes a communication unit 210 , a storage unit 220 , a display unit 230 , and a control unit 240 .
  • the display unit 230 may be selectively provided.
  • the electronic device 200 may refer to the electronic device 110 of FIG. 1 .
  • the communication unit 210 connects the electronic device 200 to enable communication with an external device.
  • the communication unit 210 may include the hologram generating device 120 or an imaging device (eg, a CT device, an X-ray device, an angiography device, a positron emission tomography device, and/or an MRI device for imaging a medical image) using wired/wireless communication. etc.) (not shown) to transmit and receive various data for providing a medical image-based hologram.
  • the communication unit 210 may receive a medical image from the imaging device, and transmit the hologram image to the hologram generating device 120 .
  • the storage 220 may store various data used to provide a medical image-based hologram.
  • the storage 220 may store standard modeling images for various target regions.
  • the storage unit 220 may include a flash memory type, a hard disk type, a multimedia card micro type, and a card type memory (eg, SD or XD). memory, etc.), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM) , a magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium.
  • the electronic device 200 may operate in relation to a web storage that performs a storage function of the storage unit 220 on the Internet.
  • the display unit 230 may display various contents to the user.
  • the display unit 230 may display various interface screens for providing a medical image-based hologram.
  • the display unit 230 may include a touch screen, and for example, a touch, a gesture, a proximity, a drag, and a swipe using an electronic pen or a part of the user's body. A swipe or hovering input may be received.
  • the control unit 240 is operatively connected to the communication unit 210 , the storage unit 220 , and the display unit 230 , and may perform various commands for providing a medical image-based hologram.
  • the control unit 240 is at least one of a central processing unit (CPU), a graphic processing unit (GPU), an application processor (AP), a digital signal processing unit (DSP), an arithmetic logic unit (ALU), and an artificial neural network processor (NPU) It may be configured to include
  • the controller 240 acquires a medical image from the imaging device through the communication unit 210 , and predicts (or recognizes) a target region of the subject and a region suspected of being a lesion in the target region based on the acquired medical image. , classification) may be used to obtain data as a result of predicting a target region and/or a suspected lesion region from a medical image using at least one image segmentation model.
  • the controller 240 generates a three-dimensional image representing the target site and the suspected lesion region by using the obtained result data, and after generating a hologram image used to generate a hologram by using the generated three-dimensional image, the communication unit It may be transmitted to the hologram device 120 through 210 .
  • controller 240 for generating an image for a hologram using an artificial neural network model will be described in detail with reference to FIG. 3 .
  • FIG. 3 is an exemplary diagram for explaining a method of generating an image for a hologram using an artificial neural network model according to an embodiment of the present invention. Operations to be described later in the presented embodiment may be performed by the controller 240 of FIG. 2 .
  • the controller 240 receives the medical image 300 as an input and uses the segmented model 310 trained to predict the target region and/or the suspected lesion region from the medical image 300 and the target region and the / Alternatively, segmented data 320 that is data as a result of predicting the lesion suspicious region may be acquired.
  • the segmentation model 310 includes a first artificial neural network model trained to predict a target site based on the medical image 300 and a second artificial neural network model trained to predict a suspected lesion region based on the medical image 300 . can do.
  • the segmentation model 310 may be a single artificial neural network model trained to predict a target site and a suspected lesion region based on the medical image 300 .
  • the segmentation model 310 includes a first artificial neural network model trained to predict a target site based on the medical image 300 and a second artificial neural network model trained to predict a suspected lesion region based on the result data of the first artificial neural network. 2 It may include an artificial neural network model.
  • the segmentation model 310 may be formed of a single artificial neural network or a combination of a plurality of artificial neural networks, but is not limited thereto.
  • the segmentation model 310 may be an artificial neural network model configured to learn a plurality of reference images in advance and predict a region corresponding to a target region and/or a region suspected of a lesion from a newly input medical image.
  • the plurality of reference images may include, but is not limited to, medical images obtained by photographing target regions of various subjects and/or medical images obtained by photographing regions suspected of lesions of various subjects.
  • the segmentation model 310 may be a pre-trained convolutional neural network (CNN), but is not limited thereto.
  • the pre-trained convolutional neural network may be composed of one or more layers that perform convolution operations on an input value, and may infer an output value by performing a convolution operation on the input value.
  • a pre-trained convolutional neural network performs an image segmentation operation for segmenting (or classifying) a predicted region into an object (ie, a target region and/or a region suspected of a lesion) in a plurality of artificial neural network stages.
  • FCN Full Convolutional Network
  • U-net U-net
  • DeepLab DeepLab
  • Mask R-CNN Registered with Convolutional Neural Network
  • the segmented data 320 output through the segmentation model 310 includes a first segmented region that masks a pixel region predicted as a target region and a second segmented region that masks a pixel region predicted as a lesion suspect region. may be an image, but is not limited thereto.
  • the first divided area and the second divided area may be expressed to be distinguished from each other.
  • the segmented data 320 may further include data indicating prediction accuracy for each of the first segmented region and the second segmented region.
  • the controller 240 generates a three-dimensional image 330 indicating a target site and/or a suspected lesion region based on the divided data 320 , and based on the generated three-dimensional image 330 , a hologram image 340 . ) can be created.
  • the controller 240 extracts a target site and/or a suspected lesion region from the medical image 300 based on the segmentation data 320, and a three-dimensional image ( 330) can be created.
  • a rendering technique for expressing a surface texture may be used to generate a 3D image based on segmentation data.
  • the target region and the region suspected of a lesion may be expressed to be distinguished from each other, for example, may be expressed to be distinguished from each other in different colors.
  • the controller 240 uses the 3D image 330 to display the 3D objects 342 , 344 , 346 , 348 arranged in multiple directions to correspond to the reflector 124 of the hologram device 120 .
  • (340) can be created.
  • the hologram image 340 is a 3D modeled object (ie, a 3D object) (eg, lung) rotated by 0°, 90°, 180°, and 270° with respect to the central axis. It may be an image in which a pair of objects are disposed on top, bottom, left, and right to face each other, but is not limited thereto.
  • the three-dimensional object is disposed on the left and right with respect to the center line that bisects one direction of the hologram image 340, and up and down based on the center line that bisects one direction and the other direction of the hologram image 340 can be placed.
  • the 3D objects 342 and 346 disposed up and down may be symmetric with each other, and the 3D objects 344 and 348 disposed on the left and right may also form symmetry with each other.
  • the number, arrangement direction, and position of these three-dimensional objects may be determined by the arrangement shape and number of optical glasses constituting the reflection unit 124 , but is not limited thereto.
  • the controller 240 may provide the generated hologram image 340 to the hologram device 120 .
  • a hologram generated by the hologram device 120 receiving the hologram image will be described with reference to FIG. 4 .
  • FIG. 4 is a diagram illustrating a hologram generated by a hologram device according to an embodiment of the present invention.
  • the hologram device 120 displays the hologram image 340 provided from the electronic device 200 through the display unit 122 , and the hologram image 340 is reflected through the reflector 124 .
  • a hologram can be generated as indicated by reference numeral 400 .
  • the artificial neural network model may be difficult to accurately recognize (or predict) the target site and/or the suspected lesion area depending on the state, shape, and/or surrounding area of the target site shown in the medical image.
  • the lung among the target regions may be suitable for recognition using an artificial neural network model, but in the case of the large intestine, it may be difficult to recognize using the artificial neural network model due to the shape of the colon or other organs located nearby. .
  • control unit 240 uses pre-stored standard modeling images for various target sites and readings such as a diagnosis or opinion of a specialist. Thus, it is possible to generate an image for a hologram.
  • controller 240 for generating an image for a hologram by using the standard modeling image and reading text stored in advance for various parts will be described in detail with reference to FIG. 5 .
  • FIG. 5 is an exemplary diagram for explaining a method of generating an image for a hologram using pre-stored standard modeling images and readouts for various parts according to an embodiment of the present invention. Operations to be described later in the presented embodiment may be performed by the controller 240 of FIG. 2 .
  • the controller 240 determines whether the target site is a predictable site from the medical image 500 using the artificial neural network model, and the target site is predictable from the medical image 500 using the artificial neural network model. In the case of a region, the operation described in FIG. 3 may be performed.
  • the controller 240 determines the pre-stored standard modeling image in relation to the target site and the expert in relation to the suspected lesion region of the target site.
  • An image for a hologram can be generated using the written reading (eg, a medical certificate and/or a statement of opinion).
  • the operation of the controller 240 for determining whether the target region is a predictable region from the medical image using the artificial neural network model is performed after obtaining the medical image, or the result of predicting the target region from the medical image using the artificial neural network model This may be performed after data is output.
  • the controller 240 determines that the target part of the acquired medical image 500 is a first part predictable using an artificial neural network or difficult to predict using an artificial neural network. It can be determined whether it is the second part.
  • predetermined reference data for the first portion and the second portion may be pre-stored in the storage unit 220 .
  • the first part is a part (eg, lung, etc.) where the accuracy of data as a result of predicting the target part using an artificial neural network is determined to correspond to a preset threshold accuracy or higher
  • the second part is a part where the accuracy of the result data is critical accuracy. It may be a site determined to be less than (eg, large intestine, etc.).
  • the controller 240 may determine whether the target region related to the medical image is the first region or the second region based on the reference data.
  • the controller 240 When the target region of the medical image 500 corresponds to the first region, the controller 240 performs the operation as described in FIG. 3 , and when the medical image 500 corresponds to the first region, the controller 240 Based on the medical image 500, a readout 520 written by a specialist in relation to a suspected lesion region is obtained, and a three-dimensional image using the standard modeling image 510 and the obtained readout 520 stored in advance for the target site. 530 may be created.
  • the controller 240 may analyze at least one sentence constituting the read text and extract a keyword related to the suspected lesion region from the at least one sentence. To this end, the controller 240 may use a natural language processing algorithm, but is not limited thereto.
  • the controller 240 may determine a location of a suspected lesion region and a corresponding suspicious lesion region based on the extracted keyword, and may generate a lesion modeling image indicating the suspected lesion region based on the determined location.
  • the controller 240 may generate the 3D image 530 by combining the standard modeling image and the lesion modeling image. For coupling, the controller 240 may use the determined position of the suspected lesion region.
  • the controller 240 determines that the prediction accuracy of the first segmented region corresponding to the target portion in segmented data 320 that is the result data of the segmentation model 310 is greater than or equal to a preset threshold. cognition can be determined.
  • the controller 240 may perform the 3D image generation operation described with reference to FIG. 3 .
  • the controller 240 may generate the 3D image 530 using the standard modeling image 510 of the target region and the reading text 520 prepared by a specialist.
  • the controller 240 may evaluate the performance of the segmentation model and generate a 3D image according to the evaluation result. For example, when the evaluation result (eg, accuracy, precision, recall, Receiver Operating Characteristic (ROC) curve, and/or Area Under Curve (AUC) value, etc.) is greater than or equal to a preset threshold value, the controller 240 may The described 3D image generation operation may be performed. When the evaluation result is less than the threshold value, the controller 240 may generate the 3D image 530 using the standard modeling image 510 of the target site and the reading text 520 prepared by a specialist.
  • the evaluation result eg, accuracy, precision, recall, Receiver Operating Characteristic (ROC) curve, and/or Area Under Curve (AUC) value, etc.
  • ROC Receiver Operating Characteristic
  • AUC Area Under Curve
  • control unit 240 when a request for generation of a 3D image using an artificial neural network model or a request for generation of a 3D image using a standard modeling image and readout is requested, the control unit 240 generates an artificial neural network model according to the request. An operation of generating a 3D image using a 3D image or generating a 3D image using a standard modeling image and a reading may be performed.
  • the controller 240 uses the three-dimensional image 530 to generate the three-dimensional objects 542,
  • An image 540 for a hologram representing 544 , 546 , and 548 may be generated. Since the operation of the controller 240 for generating the hologram image 540 is similar to the operation of the controller 240 described with reference to FIG. 3 , a detailed description thereof will be omitted.
  • the controller 240 may provide the generated hologram image 540 to the hologram device 120 .
  • a hologram generated by the hologram device 120 receiving the hologram image will be described with reference to FIG. 6 .
  • FIG. 6 is a diagram illustrating a hologram generated by a hologram device according to an embodiment of the present invention.
  • the hologram device 120 displays the hologram image 540 provided from the electronic device 200 through the display unit 122 , and the hologram image is reflected through the reflection unit 124 , and the reference numerals You can create a hologram like 600.
  • control unit 240 predicts a target site from a medical image using an artificial neural network model, determines the location of the suspected lesion area and the suspected lesion area using a readout such as a diagnosis or observation written by a specialist, and based on this It is also possible to create a 3D image.
  • the controller 240 predicts a suspected lesion region from a medical image using an artificial neural network model, and generates a three-dimensional image based on a pre-stored standard modeling image and the predicted lesion suspicious region in relation to the target site.
  • the controller 240 may display various interface screens for providing a medical image-based hologram through the display unit 230 .
  • Various interface screens acquire a medical image, generate a 3D image using an artificial neural network model, create a 3D image using a pre-stored standard modeling image and a reading written by a specialist, and create and provide an image for a hologram It may include an interface screen for various operations such as
  • the medical staff can explain the diagnosis or examination result to the patient using the hologram, it is possible to increase the patient's understanding of the diagnosis or examination result within the prescribed treatment time.
  • a medical professional can easily and conveniently check a medical image using a hologram without unnecessary movement for checking a medical image during surgery or a separate device.
  • FIG. 7 is a flowchart illustrating a method for providing a medical image-based hologram in an electronic device according to an embodiment of the present invention. The operations described below may be performed by the controller 240 of FIG. 2 .
  • the controller 240 acquires a medical image obtained by photographing a target part of the subject ( S700 ).
  • the medical image may be at least one of an X-ray image, a CT image, an angiography image, a positron emission tomography image, and an MRI image.
  • the controller 240 recognizes the target region and the suspected lesion region from the medical image using the artificial neural network model trained to recognize the lesion suspected region in the target region and the target region based on the obtained medical image. is obtained (S710).
  • the artificial neural network model is a segmented model trained to predict a target site and/or a suspected lesion region, and uses the segmented model to output segmented data representing a predicted target site and/or a suspected lesion region from a medical image as result data. can do.
  • the controller 240 generates a three-dimensional image representing the target region and the suspected lesion region based on the result data (S720). For example, the controller 240 may generate a 3D image by extracting a target region and a suspected lesion region from the medical image, and 3D rendering a partial image corresponding to the extracted target region and the suspected lesion region. not limited
  • the controller 240 generates a hologram image used to generate a hologram by using the generated 3D image (S730), and uses the hologram image generated so that the hologram device 120 generates a hologram to the hologram device 120 ) is provided (S740).
  • FIG. 8 is an exemplary diagram illustrating an interface screen for providing a hologram image using an artificial neural network model according to an embodiment of the present invention.
  • the interface screen may be displayed through the display unit 230 of FIG. 2 .
  • the controller 240 may display an interface screen 800 for providing an image for a hologram through the display unit 230 using an artificial neural network model.
  • the interface screen 800 includes a first graphic object 805 for requesting display of at least one medical image, and a display area for displaying at least one medical image (ie, a first display area 810 , and a second Display area 815, third display area 820), a second graphic object 825 for requesting generation of a 3D image using an artificial neural network model, and a fourth display area 830 for displaying a 3D image ) and a third graphic object 835 for providing a hologram.
  • the interface screen 800 may further include an input area 840 for inputting a reading and a fourth graphic object 845 for storing the inputted reading.
  • the controller 240 may display at least one medical image for providing a hologram on the display areas 810 , 815 , and 820 .
  • the controller 240 requests a medical image from the imaging device, receives the medical image from the imaging device, and displays the received medical image on the display areas 810 , 815 , and 820 .
  • the controller 240 may display the stored medical image on the display areas 810 , 815 , and 820 .
  • control unit 240 obtains the result data of predicting the target region and the lesion suspected region using the artificial neural network model, and generates a 3D image based on the obtained result data. have.
  • the generated 3D image may be displayed on the fourth display area 830 .
  • the controller 240 may generate an image for the hologram by using the 3D image, and transmit the generated image for the hologram to the hologram device 120 .
  • the hologram device 120 receiving the hologram image may generate a hologram as described with reference to FIG. 4 .
  • the controller 240 when a reading is input through the input area 840 and a fourth graphic object 845 for requesting storage of the inputted reading is selected, the controller 240 stores the inputted reading in the storage unit 220 . can be stored in
  • the controller 240 may display a graphic object indicating the predicted area in the form of a notification window, but is not limited thereto.
  • the controller 240 when the 3D image is updated by the user's manipulation of an input device (not shown) (or a controller), the controller 240 generates an image for a hologram using the updated 3D image, By transmitting the generated hologram image to the hologram device 120 , the hologram device 120 may reflect the updated information on the hologram in real time.
  • FIG. 9 is an exemplary diagram illustrating an interface screen for providing an image for a hologram using a pre-stored standard modeling image and readout according to an embodiment of the present invention.
  • the interface screen may be displayed through the display unit 230 of FIG. 2 .
  • the controller 240 may display an interface screen 900 for providing an image for a hologram through the display unit 230 using a standard modeling image and a readout.
  • the interface screen 900 includes a first graphic object 905 for requesting display of at least one medical image, a display area for displaying at least one medical image (ie, a first display area 910 , and a second A display area 915, a third display area 920), a second graphic object 925 for requesting display for a standard modeling image, an input area 930 for inputting a readout, and storing the inputted readout It may include a third graphic object 935 for displaying a 3D image, a fourth display area 940 for displaying a 3D image, and a fourth graphic object 945 for providing a hologram.
  • the controller 240 may display at least one medical image for providing a hologram on the display areas 910 , 915 , and 920 .
  • the displayed medical image is a medical image of the large intestine
  • the user may select the second graphic object 925 for generating a 3D image using a standard modeling image and a readout.
  • a reading is input from the user through the input area 930, and a third graphic object 935 for requesting storage of the inputted reading can be selected.
  • the control unit 240 may store the input reading text in the storage unit 220 .
  • the control unit 240 analyzes at least one sentence constituting the read text, extracts a keyword related to the suspicious lesion region from the at least one sentence, and based on the extracted keyword, the suspected lesion region and a location of a suspected lesion region may be determined.
  • the control unit 240 generates a lesion modeling image representing the suspected lesion region based on the position of the suspected lesion region and the lesion suspect region, and combines the standard modeling image and the lesion modeling image of the lung pre-stored in the storage 220 to A 3D image can be created.
  • the controller 240 may combine the lesion modeling images to correspond to positions determined from the standard modeling images.
  • the generated 3D image may be displayed on the fourth display area 940 .
  • the controller 240 may display a graphic object indicating the same in the form of a notification window, but is not limited thereto.
  • the controller 240 may generate an image for the hologram by using the 3D image, and transmit the generated image for the hologram to the hologram device 120 .
  • the hologram device 120 receiving the hologram image may generate a hologram as described with reference to FIG. 6 .
  • an interface screen for providing an image for a hologram using an artificial neural network model and an interface screen for providing an image for a hologram using a standard modeling image and readout may be provided as one interface screen, but is not limited thereto. does not
  • each interface screen and graphic objects related to providing a medical image-based hologram is not limited to the above description, and each interface screen and graphic objects may be configured in various ways.
  • the medical staff can provide a high service for outpatient treatment, and even if the medical staff treats many patients for a predetermined time, the patient's satisfaction can be increased.
  • a specialist can focus on surgery in an operating room, and can easily and conveniently check a patient's medical image.
  • the apparatus and method according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium.
  • the computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination.
  • the program instructions recorded on the computer readable medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the computer software field.
  • Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks.
  • Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.
  • the hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Landscapes

  • Health & Medical Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Medical Informatics (AREA)
  • Public Health (AREA)
  • Biomedical Technology (AREA)
  • General Health & Medical Sciences (AREA)
  • Pathology (AREA)
  • Veterinary Medicine (AREA)
  • Physics & Mathematics (AREA)
  • Radiology & Medical Imaging (AREA)
  • Biophysics (AREA)
  • Animal Behavior & Ethology (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Molecular Biology (AREA)
  • Surgery (AREA)
  • Primary Health Care (AREA)
  • Epidemiology (AREA)
  • High Energy & Nuclear Physics (AREA)
  • Optics & Photonics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Human Computer Interaction (AREA)
  • Physiology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Measuring And Recording Apparatus For Diagnosis (AREA)
  • Image Analysis (AREA)

Abstract

La présente invention concerne, selon un mode de réalisation, un appareil et un procédé qui permettent de fournir un hologramme basé sur une image médicale. L'appareil est configuré pour : obtenir une image médicale générée par la capture d'un site cible d'un patient ; obtenir des données de résultat de prédiction de site cible et de site de lésion suspecté à partir de l'image médicale à l'aide d'un modèle de réseau neuronal artificiel entraîné à prédire un site cible et un site de lésion suspecté d'une lésion dans le site cible sur la base de l'image médicale obtenue ; générer une image tridimensionnelle représentant le site cible et le site de lésion suspecté sur la base des données de résultat ; générer une image holographique utilisée pour générer un hologramme à l'aide de l'image tridimensionnelle générée ; et fournir l'image holographique générée à un dispositif holographique de sorte que le dispositif holographique génère un hologramme.
PCT/KR2021/018211 2020-12-11 2021-12-03 Appareil et procédé pour fournir un hologramme basé sur une image médicale WO2022124705A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020200173764A KR102476957B1 (ko) 2020-12-11 2020-12-11 의료 영상 기반 홀로그램 제공 장치 및 방법
KR10-2020-0173764 2020-12-11

Publications (1)

Publication Number Publication Date
WO2022124705A1 true WO2022124705A1 (fr) 2022-06-16

Family

ID=81974735

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2021/018211 WO2022124705A1 (fr) 2020-12-11 2021-12-03 Appareil et procédé pour fournir un hologramme basé sur une image médicale

Country Status (2)

Country Link
KR (1) KR102476957B1 (fr)
WO (1) WO2022124705A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116309591A (zh) * 2023-05-19 2023-06-23 杭州健培科技有限公司 一种医学影像3d关键点检测方法、模型训练方法及装置

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20240020388A (ko) * 2022-08-08 2024-02-15 주식회사 빔웍스 초음파 영상을 기반으로 병변을 예측하는 방법

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009087038A (ja) * 2007-09-28 2009-04-23 Canon Inc 画像処理装置および画像処理方法
KR20180128215A (ko) * 2017-05-23 2018-12-03 아주대학교산학협력단 캡슐내시경의 촬영 제어 방법 및 시스템
KR20190102399A (ko) * 2018-02-26 2019-09-04 (주)헬스허브 정제된 인공지능 강화학습 데이터 생성을 통한 의료영상 판독 시스템 및 그 방법
JP2019537481A (ja) * 2016-11-16 2019-12-26 テラリコン インコーポレイテッド 医用画像処理からの三次元プリンティング、ホログラフィ及び仮想現実レンダリングのためのシステム及び方法
KR20200109093A (ko) * 2019-03-12 2020-09-22 삼성메디슨 주식회사 초음파 영상 표시 방법, 초음파 진단 장치 및 컴퓨터 프로그램 제품

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102008015312A1 (de) * 2008-03-20 2009-10-01 Siemens Aktiengesellschaft Displaysystem zur Wiedergabe medizinischer Hologramme
CN102150072B (zh) * 2008-07-10 2013-08-21 实景成像有限公司 宽视角显示和用户接口
DE102008034686A1 (de) * 2008-07-25 2010-02-04 Siemens Aktiengesellschaft Verfahren zur Darstellung von interventionellen Instrumenten in einem 3 D-Datensatz einer zu behandelnden Anatomie sowie Wiedergabesystem zur Durchführung des Verfahrens
US10010379B1 (en) * 2017-02-21 2018-07-03 Novarad Corporation Augmented reality viewing and tagging for medical procedures
KR102155271B1 (ko) * 2018-02-27 2020-09-11 한국광기술원 홀로그래픽 머신비전 장치를 이용한 비 절제 생검 시스템
KR102245693B1 (ko) 2018-09-18 2021-04-29 서울대학교산학협력단 3차원 영상 재구성 장치 및 그 방법

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009087038A (ja) * 2007-09-28 2009-04-23 Canon Inc 画像処理装置および画像処理方法
JP2019537481A (ja) * 2016-11-16 2019-12-26 テラリコン インコーポレイテッド 医用画像処理からの三次元プリンティング、ホログラフィ及び仮想現実レンダリングのためのシステム及び方法
KR20180128215A (ko) * 2017-05-23 2018-12-03 아주대학교산학협력단 캡슐내시경의 촬영 제어 방법 및 시스템
KR20190102399A (ko) * 2018-02-26 2019-09-04 (주)헬스허브 정제된 인공지능 강화학습 데이터 생성을 통한 의료영상 판독 시스템 및 그 방법
KR20200109093A (ko) * 2019-03-12 2020-09-22 삼성메디슨 주식회사 초음파 영상 표시 방법, 초음파 진단 장치 및 컴퓨터 프로그램 제품

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116309591A (zh) * 2023-05-19 2023-06-23 杭州健培科技有限公司 一种医学影像3d关键点检测方法、模型训练方法及装置
CN116309591B (zh) * 2023-05-19 2023-08-25 杭州健培科技有限公司 一种医学影像3d关键点检测方法、模型训练方法及装置

Also Published As

Publication number Publication date
KR20220083477A (ko) 2022-06-20
KR102476957B1 (ko) 2022-12-12

Similar Documents

Publication Publication Date Title
WO2022124705A1 (fr) Appareil et procédé pour fournir un hologramme basé sur une image médicale
WO2018012945A1 (fr) Procédé et dispositif d'obtention d'image, et support d'enregistrement associé
WO2015099403A1 (fr) Procédé de commande d'appareil médical et appareil mobile associé
WO2021060899A1 (fr) Procédé d'apprentissage pour spécialiser un modèle d'intelligence artificielle dans une institution pour un déploiement et appareil pour l'apprentissage d'un modèle d'intelligence artificielle
WO2016072586A1 (fr) Appareil et procédé de traitement d'images médicales
WO2022131642A1 (fr) Appareil et procédé pour déterminer la gravité d'une maladie sur la base d'images médicales
WO2022139246A1 (fr) Procédé de détection de fracture et dispositif l'utilisant
WO2019117563A1 (fr) Appareil d'analyse prédictive intégrée pour télésanté interactive et procédé de fonctionnement associé
WO2021230534A1 (fr) Appareil de prédiction de lésion orbitaire et périorbitaire et procédé de prédiction associé
WO2017142223A1 (fr) Système de transmission d'image à distance, appareil d'affichage et son procédé d'affichage de guidage
WO2021246770A1 (fr) Procédé et système de lecture automatique d'images radiographiques en temps réel sur la base d'intelligence artificielle
WO2022231329A1 (fr) Procédé et dispositif d'affichage de tissu d'image biologique
WO2014208950A1 (fr) Procédé et appareil de gestion de données médicales
WO2021233086A1 (fr) Procédé de traitement d'informations, dispositif électronique et support de stockage informatique
WO2021251652A1 (fr) Appareil et procédé d'analyse vidéo
JP2008217294A (ja) 携帯端末を利用した医療用通信装置
JP2019509798A (ja) 乳癌検診用ソフトウェアツール
WO2023128284A1 (fr) Procédé pour fournir des informations concernant le diagnostic du cancer du col de l'utérus et dispositif pour fournir des informations concernant le diagnostic du cancer du col de l'utérus à l'aide de celui-ci
JP7172376B2 (ja) 情報提供装置、情報提供システム、情報提供方法、及びプログラム
WO2022050713A1 (fr) Procédé de lecture d'image de poitrine
WO2023058942A1 (fr) Dispositif et procédé pour fournir un service d'analyse de santé buccale
WO2022231200A1 (fr) Procédé d'entraînement pour l'entraînement d'un réseau de neurones artificiels pour déterminer une zone de lésion du cancer du sein, et système informatique le réalisant
WO2019164273A1 (fr) Méthode et dispositif de prédiction de temps de chirurgie sur la base d'une image chirurgicale
WO2020212962A2 (fr) Procédé d'évaluation de type de pied et dispositif d'évaluation de type de pied l'utilisant
WO2023140475A1 (fr) Appareil et procédé de liaison et de fourniture d'emplacement de lésion entre une image de guidage et une image de tomosynthèse 3d comprenant une pluralité de tranches d'image 3d

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21903751

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21903751

Country of ref document: EP

Kind code of ref document: A1