WO2023188903A1 - 画像処理装置、医療診断装置、超音波内視鏡装置、画像処理方法、及びプログラム - Google Patents

画像処理装置、医療診断装置、超音波内視鏡装置、画像処理方法、及びプログラム Download PDF

Info

Publication number
WO2023188903A1
WO2023188903A1 PCT/JP2023/004985 JP2023004985W WO2023188903A1 WO 2023188903 A1 WO2023188903 A1 WO 2023188903A1 JP 2023004985 W JP2023004985 W JP 2023004985W WO 2023188903 A1 WO2023188903 A1 WO 2023188903A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
area
lesion
image area
region
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2023/004985
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
理都 村瀬
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fujifilm Corp
Original Assignee
Fujifilm Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fujifilm Corp filed Critical Fujifilm Corp
Priority to JP2024511381A priority Critical patent/JPWO2023188903A1/ja
Priority to CN202380030763.8A priority patent/CN118946315A/zh
Publication of WO2023188903A1 publication Critical patent/WO2023188903A1/ja
Priority to US18/890,704 priority patent/US20250009329A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/12—Diagnosis using ultrasonic, sonic or infrasonic waves in body cavities or body tracts, e.g. by using catheters
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/08—Clinical applications
    • A61B8/0833—Clinical applications involving detecting or locating foreign bodies or organic structures
    • A61B8/085—Clinical applications involving detecting or locating foreign bodies or organic structures for locating body or organic structures, e.g. tumours, calculi, blood vessels, nodules
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/52—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/5207—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of raw data to produce diagnostic data, e.g. for generating an image
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/52—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/5215—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of medical diagnostic data
    • A61B8/5223—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of medical diagnostic data for extracting a diagnostic or physiological parameter from medical diagnostic data
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/46—Ultrasonic, sonic or infrasonic diagnostic devices with special arrangements for interfacing with the operator or the patient
    • A61B8/461—Displaying means of special interest
    • A61B8/463—Displaying means of special interest characterised by displaying multiple images or images and diagnostic data on one display

Definitions

  • the technology of the present disclosure relates to an image processing device, a medical diagnostic device, an ultrasound endoscope device, an image processing method, and a program.
  • Japanese Unexamined Patent Publication No. 2021-100555 discloses a medical image processing device having at least one processor.
  • at least one processor acquires a medical image, acquires part information indicating a part of the human body of a subject in the medical image, and extracts information from the medical image.
  • a lesion is detected and lesion type information indicating the type of the lesion is acquired, the presence or absence of a contradiction between the part information and the lesion type information is determined, and the reporting mode of the part information and the lesion type information is determined based on the result of the determination.
  • One embodiment of the technology of the present disclosure provides an image processing device, a medical diagnostic device, an ultrasound endoscope device, an image processing method, and a program that allow a user etc. to accurately understand a lesion. .
  • a first aspect of the technology of the present disclosure includes a processor, and the processor selects a first image area indicating a part and a lesion from a medical image obtained by imaging an observation target area including a part of a human body and a lesion. , and displays the results of detecting the first image area and the second image area on a display device in a display mode according to the positional relationship between the first image area and the second image area.
  • This is an image processing device for displaying images.
  • a second aspect according to the technology of the present disclosure is an image processing device according to the first aspect, in which a display mode is determined according to a site, a lesion, and a positional relationship.
  • a third aspect according to the technology of the present disclosure is the image processing device according to the first aspect or the second aspect, in which the display aspect is determined depending on the positional relationship and the consistency between the site and the lesion.
  • a fourth aspect of the technology of the present disclosure is that the display mode for the first image area varies depending on the site, lesion, and positional relationship, and the display mode for the second image area differs depending on the site, lesion, and positional relationship.
  • This is an image processing device according to any one of the first to third aspects that are displayed on the device.
  • a fifth aspect of the technology of the present disclosure is that when the site and the lesion do not match, the display mode for the first image area is such that the first image area is not displayed on the display device, and the display mode for the second image area is such that the first image area is not displayed on the display device.
  • the image processing apparatus according to the fourth aspect is a display mode in which the second image area is displayed on the display device.
  • a sixth aspect of the technology of the present disclosure is that when the site and the lesion match, the display mode for the first image area is determined depending on the first image area being displayed on the display device and the positional relationship.
  • the image processing apparatus according to the fourth aspect or the fifth aspect, wherein the display aspect of the second image area is such that the second image area is displayed on the display device.
  • a seventh aspect according to the technology of the present disclosure is any one of the first to sixth aspects, wherein the positional relationship is defined by the degree of overlap or distance between the first image area and the second image area.
  • 1 is an image processing device according to one embodiment.
  • the positional relationship is defined by the degree of overlap, and when the degree of overlap is equal to or higher than the first degree, the display mode is such that the second image region is displayed in a manner that allows identification of the second image region within the medical image.
  • the positional relationship is defined by the degree of overlap, and when the degree of overlap is equal to or higher than the first degree, the display mode is such that the second image region is displayed in a manner that allows identification of the second image region within the medical image.
  • This is an image processing device according to a seventh aspect, in which the first image area is displayed in such a manner that the second image area and the second image area can be compared with each other.
  • a tenth aspect of the technology of the present disclosure is that the processor detects a first confidence level that is a confidence level for the result of detecting the first image area and a second confidence level that is a confidence level for the result of detecting the second image area.
  • the image processing apparatus according to any one of the first to ninth aspects, wherein the display mode is determined according to the first certainty factor, the second certainty factor, and the positional relationship.
  • An eleventh aspect according to the technology of the present disclosure is the image processing device according to the tenth aspect, in which the display mode is determined according to the magnitude relationship and positional relationship between the first certainty factor and the second certainty factor.
  • the display mode is determined according to a plurality of positional relationships, and the plurality of positional relationships are between a plurality of first image areas and a second image area for a plurality of types of body parts.
  • An image processing apparatus according to any one of the first to eleventh aspects regarding the positional relationship.
  • a thirteenth aspect according to the technology of the present disclosure is the image processing device according to the twelfth aspect, in which the display mode for each of the plurality of first image regions differs depending on each of the plurality of positional relationships.
  • a fourteenth aspect according to the technology of the present disclosure is the twelfth aspect or This is an image processing device according to a thirteenth aspect.
  • the medical image is an image defined by a plurality of frames
  • the processor detects the first image area and the second image area for each frame
  • the display mode is An image processing device according to any one of the first to fourteenth aspects, which is determined for each frame.
  • a sixteenth aspect of the technology of the present disclosure is that the processor determines the combination of the first image region and the second image region for each frame based on the correspondence between the plurality of types of regions and the lesions corresponding to each region.
  • the display mode corresponding to the frame used as the judgment target is determined by determining the combination of the first image area and the second image area.
  • the image processing apparatus performs correction based on a display mode corresponding to a frame used as a determination target when it is determined that the frame is correct.
  • a seventeenth aspect according to the technology of the present disclosure is a medical diagnostic device comprising the image processing device according to any one of the first to sixteenth aspects, and an imaging device that images an observation target area. be.
  • An 18th aspect according to the technology of the present disclosure includes the image processing device according to any one of the 1st to 16th aspects, and an ultrasound device that acquires an ultrasound image as a medical image.
  • This is an ultrasound endoscope device.
  • a nineteenth aspect of the technology of the present disclosure provides a first image area showing the part and a second image area showing the lesion from a medical image obtained by imaging a region to be observed including a part of the human body and a lesion. and displaying the results of detecting the first image area and the second image area on a display device in a display mode according to the positional relationship between the first image area and the second image area.
  • a twenty-fifth aspect according to the technology of the present disclosure is an image processing device according to any one of the twenty-first to twenty-third aspects, in which the processor determines the certainty of the first image region.
  • a twenty-seventh aspect according to the technology of the present disclosure is an image processing device according to the twenty-sixth aspect.
  • the position where the information indicating that a lesion has been detected is displayed in an area corresponding to the second image area of the display area where the medical image is displayed.
  • FIG. 1 is a conceptual diagram showing an example of a mode in which an ultrasound endoscope system is used.
  • 1 is a conceptual diagram showing an example of the overall configuration of an ultrasound endoscope system.
  • FIG. 2 is a conceptual diagram showing an example of a mode in which an insertion section of an ultrasound endoscope is inserted into the stomach of a subject.
  • FIG. 2 is a block diagram showing an example of the hardware configuration of an endoscope processing device.
  • FIG. 2 is a block diagram showing an example of the hardware configuration of an ultrasonic processing device.
  • FIG. 2 is a block diagram showing an example of the hardware configuration of a display control device.
  • FIG. 2 is a block diagram illustrating an example of main functions of a processor of the display control device.
  • FIG. 3 is a conceptual diagram illustrating an example of processing contents of an acquisition unit.
  • FIG. 2 is a conceptual diagram showing an example of processing contents of an acquisition unit, a detection unit, and a determination unit.
  • FIG. 2 is a conceptual diagram illustrating an example of processing contents of an acquisition unit, a determination unit, and a control unit.
  • FIG. 3 is a conceptual diagram illustrating an example of processing contents of an acquisition unit, a determination unit, and a positional relationship identification unit.
  • FIG. 7 is a conceptual diagram illustrating an example of processing contents of an acquisition unit, a detection unit, a positional relationship specifying unit, and a control unit when the degree of overlap is less than a predetermined degree of overlap.
  • FIG. 1 is a conceptual diagram showing an example of processing contents of an acquisition unit, a detection unit, and a determination unit.
  • FIG. 2 is a conceptual diagram illustrating an example of processing contents of an acquisition unit, a determination unit, and a control unit.
  • FIG. 3 is a conceptual diagram
  • FIG. 7 is a conceptual diagram illustrating an example of processing contents of an acquisition unit, a detection unit, a positional relationship specifying unit, and a control unit when the degree of duplication is equal to or higher than a predetermined degree of duplication.
  • 3 is a flowchart illustrating an example of the flow of display control processing. It is a conceptual diagram which shows an example of the processing content of a 1st modification. It is a conceptual diagram which shows an example of the processing content of a 2nd modification. It is a conceptual diagram which shows an example of the processing content of a 3rd modification. It is a conceptual diagram which shows an example of the processing content based on the 4th modification of a detection part and a determination part.
  • FIG. 7 is a conceptual diagram illustrating an example of processing contents of an acquisition unit, a determination unit, a positional relationship specifying unit, and a control unit when the combination of a part region and a lesion area is incorrect and the degree of overlap is less than a predetermined degree of overlap.
  • FIG. 7 is a conceptual diagram illustrating an example of processing contents of an acquisition unit, a determination unit, a positional relationship specifying unit, and a control unit when the combination of a part region and a lesion area is incorrect and the degree of overlap is equal to or higher than a predetermined degree of overlap.
  • FIG. 7 is a conceptual diagram illustrating an example of processing contents of an acquisition unit, a determination unit, a positional relationship specifying unit, and a control unit when the combination of a part region and a lesion area is correct and the degree of overlap is equal to or higher than a predetermined degree of overlap.
  • 1 is a conceptual diagram illustrating an example of processing contents of an acquisition unit, a determination unit, a positional relationship identification unit, and a control unit when the combination of a part area and a lesion area is correct and the second certainty is less than or equal to the first certainty.
  • It is a flowchart which shows an example of the flow of display control processing concerning a 4th modification. This is a continuation of the flowchart shown in FIG. 25A.
  • FIG. 27A It is a conceptual diagram which shows an example of the ultrasound image of a conventional example, and an example of the ultrasound image on which the display control process based on the 5th modification was performed. It is a conceptual diagram which shows an example of the ultrasound image of a conventional example, and an example of the ultrasound image on which the display control process based on the 6th modification was performed. It is a flowchart which shows an example of the flow of display control processing concerning a 7th modification.
  • FIG. 30A This is a continuation of the flowchart shown in FIG. 30A. It is a flowchart which shows an example of the flow of display control processing concerning the 8th modification. This is a continuation of the flowchart shown in FIG. 31A. It is a conceptual diagram which shows an example of the processing content based on the 9th modification of a control part. It is a flowchart which shows an example of the flow of display control processing concerning the 10th modification. This is a continuation of the flowchart shown in FIG. 33A.
  • CPU is an abbreviation for "Central Processing Unit”.
  • GPU is an abbreviation for “Graphics Processing Unit.”
  • RAM is an abbreviation for “Random Access Memory.”
  • NVM is an abbreviation for “Non-volatile memory.”
  • EEPROM is an abbreviation for “Electrically Erasable Programmable Read-Only Memory.”
  • ASIC is an abbreviation for “Application Specific Integrated Circuit.”
  • PLD is an abbreviation for “Programmable Logic Device”.
  • AI is an abbreviation for “Artificial Intelligence.”
  • FIFO is an abbreviation for “First In First Out.”
  • FPC is an abbreviation for “Flexible Printed Circuit.”
  • IoU is an abbreviation for "Intersection over Union.”
  • an ultrasound endoscope system 10 includes an ultrasound endoscope device 12 and a display device 14.
  • the ultrasound endoscope device 12 is used by medical personnel (hereinafter referred to as "users") such as a doctor 16, a nurse, and/or a technician.
  • the ultrasound endoscope device 12 includes an ultrasound endoscope 18 and is a device for performing medical treatment on the inside of the body of a subject 20 (for example, a patient) via the ultrasound endoscope 18.
  • the ultrasound endoscope device 12 is an example of a “medical diagnostic device” and an “ultrasound endoscope device” according to the technology of the present disclosure.
  • the ultrasound endoscope 18 is an example of an "imaging device” according to the technology of the present disclosure.
  • the ultrasound endoscope 18 acquires and outputs an image showing the inside of the body by imaging the subject 20 by the doctor 16.
  • an ultrasonic endoscope 18 is inserted into a body cavity through the mouth of a subject 20.
  • the ultrasound endoscope 18 is inserted into the body cavity through the mouth of the subject 20, but this is just an example, and the ultrasound endoscope 18 is inserted into the body cavity through the mouth of the subject 20. It may be inserted into the body cavity of the person 20 through the nostril, anus, or perforation.
  • the display device 14 displays various information including images.
  • An example of the display device 14 is a liquid crystal display, an EL display, or the like.
  • a plurality of screens are displayed side by side on the display device 14.
  • a first screen 22 and a second screen 24 are shown as an example of a plurality of screens.
  • the ultrasound moving image 26 is displayed on the first screen 22.
  • the ultrasound moving image 26 is a moving image generated based on reflected waves obtained by irradiating ultrasound towards the observation target area within the body of the subject 20 and reflecting the ultrasound at the observation target area. It is.
  • the ultrasound moving image 26 is displayed on the first screen 22 using a live view method. Although a live view method is illustrated here, this is just an example, and other display methods such as a post view method may be used.
  • An example of an observation target area to which ultrasound is irradiated is an area including organs and lesions of the subject 20.
  • the observation target area to which ultrasound is irradiated is an example of the "observation target area” according to the technology of the present disclosure.
  • the organs and lesions of the subject are examples of "human body parts and lesions” according to the technology of the present disclosure.
  • the ultrasound moving image 26 (that is, the moving image generated based on the reflected waves obtained when the ultrasound is reflected by the observation target area) is a "video image of the observation target area" according to the technology of the present disclosure. This is an example of a medical image obtained by
  • An endoscopic moving image 28 is displayed on the second screen 24.
  • An example of the endoscopic moving image 28 is a moving image generated by capturing visible light, near-infrared light, or the like.
  • the endoscopic moving image 28 is displayed on the second screen 24 using a live view method. Note that in this embodiment, the endoscopic moving image 28 is also illustrated along with the ultrasound moving image 26, but this is just an example, and the technology of the present disclosure is applicable even without the endoscopic moving image 28. also holds true.
  • the ultrasound endoscope 18 includes an operating section 30 and an insertion section 32.
  • the insertion portion 32 is formed into a tubular shape.
  • the insertion portion 32 has a distal end portion 34, a curved portion 36, and a flexible portion 37.
  • the distal end portion 34, the curved portion 36, and the soft portion 37 are arranged in this order from the distal end side to the proximal end side of the insertion portion 32.
  • the flexible portion 37 is made of a long, flexible material, and connects the operating portion 30 and the curved portion 36 .
  • the curved portion 36 partially curves or rotates around the axis of the insertion portion 32 when the operating portion 30 is operated.
  • the insertion portion 32 may curve or bend depending on the shape of the body cavity (for example, the shape of the digestive tract such as the esophagus, stomach, duodenum, small intestine, and large intestine, or the shape of the bronchus). It is sent deep into the body cavity while rotating around its axis.
  • the shape of the body cavity for example, the shape of the digestive tract such as the esophagus, stomach, duodenum, small intestine, and large intestine, or the shape of the bronchus.
  • the distal end portion 34 is provided with an illumination device 38, an endoscope 40, an ultrasound probe 42, and a treatment tool opening 44.
  • the lighting device 38 has a lighting window 38A and a lighting window 38B.
  • the illumination device 38 irradiates light (for example, white light made of three primary colors, near-infrared light, etc.) through the illumination window 38A and the illumination window 38B.
  • the endoscope 40 images the inside of the body using an optical method.
  • An example of the endoscope 40 is a CMOS camera.
  • the CMOS camera is just an example, and other types of cameras such as a CCD camera may be used.
  • the ultrasonic probe 42 is provided on the distal end side of the distal end portion 34.
  • the outer surface 42A of the ultrasonic probe 42 is curved outward in a convex shape from the proximal end side to the distal end side of the ultrasonic probe 42.
  • the ultrasonic probe 42 transmits ultrasonic waves via the outer surface 42A, and receives reflected waves obtained by reflecting the transmitted ultrasonic waves at the observation target area via the outer surface 42A.
  • the treatment instrument opening 44 is formed closer to the proximal end of the distal end portion 34 than the ultrasound probe 42 is. This is an opening for allowing the treatment instrument 46 to protrude from the distal end portion 34.
  • a treatment instrument insertion port 48 is formed in the operation section 30, and the treatment instrument 46 is inserted into the insertion section 32 through the treatment instrument insertion port 48. The treatment instrument 46 passes through the insertion portion 32 and protrudes into the body from the treatment instrument opening 44.
  • a puncture needle 50 with a guide sheath protrudes from the treatment tool opening 44 as the treatment tool 46.
  • the puncture needle with guide sheath 50 has a puncture needle 50A and a guide sheath 50B.
  • Puncture needle 50A passes through guide sheath 50B and protrudes from guide sheath 50B.
  • the puncture needle 50 with a guide sheath is illustrated as the treatment tool 46, but this is just one example, and the treatment tool 46 may be a grasping forceps, a scalpel, a snare, or the like.
  • the treatment tool opening 44 also functions as a suction port for sucking blood, body waste, and the like.
  • a reception device 62 is connected to the ultrasonic processing device 58.
  • the ultrasound processing device 58 sends and receives various signals to and from the ultrasound probe 42 in accordance with instructions received by the receiving device 62.
  • the ultrasound processing device 58 causes the ultrasound probe 42 to transmit ultrasound, generates and outputs an ultrasound moving image 26 based on the reflected waves received by the ultrasound probe 42 .
  • the insertion section 32 of the ultrasound endoscope 18 is inserted into the stomach 64 of the subject 20.
  • the endoscope 40 images the inside of the stomach 64 at a predetermined frame rate (for example, 30 frames/second or 60 frames/second, etc.), thereby converting a live view image showing the inside of the stomach 64 into an endoscopic video. It is generated as an image 28.
  • the distal end portion 34 is located at a target position within the stomach 64.
  • the ultrasound probe 42 When reaching , the outer surface 42A of the ultrasound probe 42 contacts the inner wall 64A of the stomach 64. With the outer surface 42A in contact with the inner wall 64A, the ultrasound probe 42 transmits ultrasound through the outer surface 42A.
  • the ultrasound waves are applied to an observation target area including organs and lesions such as the pancreas 65 and kidney 66 through the inner wall 64A. Reflected waves obtained by reflecting the ultrasound waves at the observation target area are received by the ultrasound probe 42 via the outer surface 42A. Then, the ultrasound moving image 26 is obtained by converting the reflected waves received by the ultrasound probe 42 into a live view image showing the aspect of the observation target area according to a predetermined frame rate.
  • ultrasonic waves are irradiated from inside the stomach 64 toward organs such as the pancreas 65 and kidney 66.
  • ultrasound may be irradiated from within the duodenum to organs such as the pancreas 65 and kidney 66.
  • the ultrasound endoscope 18 is inserted into the stomach 64, but this is just an example, and the ultrasound endoscope 18 is inserted into the stomach 64.
  • An ultrasound endoscope 18 may be inserted.
  • the endoscope processing device 54 includes a computer 67 and an input/output interface 68.
  • Computer 67 includes a processor 70, RAM 72, and NVM 74.
  • Input/output interface 68, processor 70, RAM 72, and NVM 74 are connected to bus 76.
  • the processor 70 includes a CPU and a GPU, and controls the entire endoscope processing device 54.
  • the GPU operates under the control of the CPU and is responsible for executing various graphics-related processes.
  • the processor 70 may be one or more CPUs with an integrated GPU function, or may be one or more CPUs without an integrated GPU function.
  • the acoustic lens is layered on the acoustic matching layer and is a lens that converges the ultrasonic waves emitted from the ultrasonic transducer unit toward the observation target area.
  • the acoustic lens is made of silicone resin, liquid silicone rubber, butadiene resin, and/or polyurethane resin, and if necessary, powder of titanium oxide, alumina, silica, etc. is mixed therein.
  • Each of the plurality of ultrasonic transducers 98 is formed by placing electrodes on both sides of a piezoelectric element.
  • piezoelectric elements include barium titanate, lead zirconate titanate, potassium niobate, and the like.
  • the electrodes include individual electrodes provided individually for the plurality of ultrasonic transducers 98 and a transducer ground common to the plurality of ultrasonic transducers 98 .
  • the electrodes are electrically connected to an ultrasonic processing device 58 via an FPC and a coaxial cable.
  • the transmitting circuit 92 and the receiving circuit 94 are electrically connected to each of the plurality of ultrasound transducers 98 via the multiplexer 90.
  • the multiplexer 90 selects at least one of the plurality of ultrasonic transducers 98 and opens a channel of the selected ultrasonic transducer, which is the selected ultrasonic transducer 98 .
  • the transmitting circuit 92 is controlled by the processor 82 via the input/output interface 80. Under the control of the processor 82, the transmission circuit 92 supplies a drive signal (for example, a plurality of pulsed signals) for ultrasound transmission to the selected ultrasound transducer.
  • the drive signal is generated according to transmission parameters set by processor 82.
  • the transmission parameters include the number of drive signals to be supplied to the selected ultrasonic transducer, the supply time of the drive signals, and the amplitude of the drive vibration.
  • the reception sensitivity of the ultrasonic transducer 98 is defined as the ratio of the amplitude of the electrical signal output by the ultrasonic transducer 48 receiving the ultrasonic wave to the amplitude of the ultrasonic wave transmitted by the ultrasonic transducer 98 .
  • the receiving circuit 94 receives an electrical signal from the ultrasonic transducer 98, amplifies the received electrical signal, and outputs it to the ADC 96.
  • the ADC 96 digitizes the electrical signal input from the receiving circuit 94.
  • the processor 82 acquires the ultrasound moving image 26 by acquiring the electrical signal digitized by the ADC 96 and generating the ultrasound moving image 26 (see FIGS. 1 and 3) based on the acquired electrical signal.
  • the combination of the ultrasound probe 42 and the ultrasound processing device 58 is an example of an "imaging device” according to the technology of the present disclosure. Furthermore, in this embodiment, the combination of the ultrasound probe 42 and the ultrasound processing device 58 is an example of an “ultrasonic device” according to the technology of the present disclosure.
  • the display control device 60 includes a computer 100 and an input/output interface 102.
  • Computer 100 includes a processor 104, RAM 106, and NVM 108.
  • Input/output interface 102, processor 104, RAM 106, and NVM 108 are connected to bus 110.
  • the display control device 60 is an example of an "image processing device” according to the technology of the present disclosure.
  • the computer 100 is an example of a "computer” according to the technology of the present disclosure.
  • the processor 104 is an example of a "processor” according to the technology of the present disclosure.
  • the processor 104 controls the entire display control device 60.
  • the plurality of hardware resources (processor 104, RAM 106, and NVM 108) included in the computer 100 shown in FIG. 6 are of the same type as the plurality of hardware resources included in the computer 67 shown in FIG. Explanation will be omitted.
  • a reception device 62 is connected to the input/output interface 102, and the processor 104 acquires instructions accepted by the reception device 62 via the input/output interface 102, and executes processing according to the acquired instructions. .
  • a display device 14 is connected to the input/output interface 102.
  • an endoscope processing device 54 is connected to the input/output interface 102, and the processor 104 exchanges various signals with the processor 70 of the endoscope processing device 54 via the input/output interface 102. conduct.
  • the input/output interface 102 is also connected to the ultrasound processing device 58 , and the processor 104 sends and receives various signals to and from the processor 82 of the ultrasound processing device 58 via the input/output interface 102 .
  • a display device 14 is connected to the input/output interface 102, and the processor 104 causes the display device 14 to display various information by controlling the display device 14 via the input/output interface 102.
  • the processor 104 acquires the endoscopic moving image 28 (see FIGS. 1 and 3) from the endoscope processing device 54, and the ultrasound moving image 26 (see FIGS. 1 and 3) from the ultrasound processing device 58. (see) and display the ultrasound moving image 26 and endoscopic moving image 28 on the display device 14.
  • display control processing is performed by the processor 104 in the display control device 60.
  • a display control processing program 112 is stored in the NVM 108.
  • the display control processing program 112 is an example of a "program" according to the technology of the present disclosure.
  • the processor 104 reads the display control processing program 112 from the NVM 108 and executes the read display control processing program 112 on the RAM 106 to perform display control processing.
  • the display control processing is realized by the processor 104 operating as an acquisition section 104A, a detection section 104B, a determination section 104C, a positional relationship identification section 104D, and a control section 104E according to the display control processing program 112.
  • the acquisition unit 104A acquires the endoscopic moving image 28 from the endoscope processing device 54.
  • the endoscopic moving image 28 is an image defined by a plurality of frames. That is, the endoscopic moving image 28 includes a plurality of endoscopic images 114 obtained as a plurality of time-series frames by the endoscopic processing device 54 according to a predetermined frame rate.
  • the acquisition unit 104A also acquires the ultrasound moving image 26 from the ultrasound processing device 58.
  • the ultrasound moving image 26 is an image defined by a plurality of frames. That is, the ultrasound moving image 26 is configured to include a plurality of ultrasound images 116 obtained as a plurality of time-series frames by the ultrasound processing device 58 according to a predetermined frame rate.
  • the detection unit 104B detects the part region 116A and the lesion region 116B from the ultrasound image 116 by performing AI-based image recognition processing on the ultrasound image 116.
  • AI-based image recognition processing is illustrated here, this is just an example, and instead of the AI-based image recognition processing, template matching-based image recognition processing is performed on the part area 116A and the lesion area 116B. may be detected. Further, the detection unit 104B may perform both AI-based image recognition processing and template matching-based image recognition processing.
  • Part region information 118 is information regarding part region 116A detected by detection unit 104B.
  • Part region information 118 includes coordinate information 118A and part name information 118B.
  • the coordinate information 118A is information including coordinates (for example, two-dimensional coordinates) that can specify the position of the part region 116A within the ultrasound image 116 (for example, the position of the outline of the part region 116A).
  • the part name information 118B is information that can identify the name of the part (i.e., the type of part) indicated by the part area 116A detected by the detection unit 104B (for example, information indicating the name of the organ itself or information that uniquely identifies the type of organ). (e.g., an identifier that can be identified by the person).
  • the lesion area information 120 is information regarding the lesion area 116B detected by the detection unit 104B.
  • the lesion area information 120 includes coordinate information 120A and lesion name information 120B.
  • the coordinate information 120A is information including coordinates (for example, two-dimensional coordinates) that can specify the position of the lesion area 116B within the ultrasound image 116 (for example, the position of the outline of the lesion area 116B).
  • the lesion name information 120B is information that can identify the name of the lesion (that is, the type of lesion) indicated by the lesion area 116B detected by the detection unit 104B (for example, information that uniquely identifies the name of the lesion or the type of lesion). (e.g., an identifiable identifier).
  • a consistency determination table 122 is stored in the NVM 112.
  • the consistency determination table 122 a plurality of site name information 118B and a plurality of lesion name information 118B are associated with each other on a one-to-one basis. That is, the consistency determination table 122 defines the name of the part specified from the part name information 120B and the name of the lesion specified from the lesion name information 120B. In other words, the consistency determination table 122 defines the correct combination of site and lesion. In the example shown in FIG. 9, a combination of pancreas and pancreatic cancer and a combination of kidney and kidney cancer are shown as some examples of correct combinations of sites and lesions.
  • the consistency determination table is an example of "correspondence" according to the technology of the present disclosure.
  • the determination unit 104C obtains part name information 118B and lesion name information 120B from the part area information 118. Then, the determination unit 104C refers to the consistency determination table 122 stored in the NVM 112 and determines the consistency of the combination of the site name information 118B and the lesion name information 120B, thereby determining whether the site area 116A and the lesion area are the same. 116B (in other words, whether the combination of site and lesion is correct or not) is determined. That is, the determination unit 104C refers to the consistency determination table 122 and determines whether the combination of the site name specified from the site name information 118B and the lesion name specified from the lesion name information 120B is correct.
  • the determination unit 104C determines whether the combination of the part area 116A and the lesion area 116B detected by the detection unit 104B is correct or not (that is, whether the combination of the part area 116A and the lesion area 116B matches or does not match). It is judged by.
  • the control unit 104E acquires an endoscopic image 114 and an ultrasound image 116 that is a determination target of the determination unit 104C, and displays the acquired ultrasound image 116 on the first screen 22.
  • the acquired endoscopic image 114 is displayed on the second screen 24.
  • the control unit 104E displays the ultrasound image 116 displayed on the first screen 22 in the first display mode. do.
  • the first display mode refers to a display mode in which the site area 116A is hidden and the lesion area 116B is displayed. In the example shown in FIG. 10, in the ultrasound image 116 in the first screen 22, the lesion area 116B is displayed without the part area 116A being displayed.
  • the positional relationship specifying unit 104D determines the positional relationship between the part area 116A and the lesion area 116B. get.
  • the positional relationship between the part area 116A and the lesion area 116B is defined by the degree of overlap, which is the degree to which the part area 116A and the lesion area 116B overlap.
  • the positional relationship specifying unit 104D acquires coordinate information 118A from the body part area information 118, acquires coordinate information 120A from the lesion area information 120, and determines the degree of overlap between the body part area 116A and the lesion area 116B using the coordinate information 118A and 120A. 124 is calculated.
  • An example of the index of the degree of overlap 124 is IoU.
  • the degree of overlap 124 is the ratio of the area of the area where the lesion area 116B and the site area 116A overlap to the area of the area where the lesion area 116B and the site area 116A are combined. In the example shown in FIG.
  • the degree of overlap 124 is an area where the lesion area 116B and the part area 116A overlap with respect to the lesion area 116B. It may be the ratio of the area of .
  • control unit 104E uses the results of detection of the body part area 116A and the lesion area 116B by the detection unit 104B to determine the positional relationship between the body part area 116A and the lesion area 116B (here, As an example, it is displayed on the display device 14 in a display mode according to the degree of overlap 124).
  • the display mode in which the display device 14 displays the results of the detection of the part region 116A and the lesion region 116B by the detection unit 104B is such that the part and the type of lesion (hereinafter simply referred to as (also referred to as a "lesion") and the positional relationship between the site area 116A and the lesion area 116B (for example, the consistency and overlap degree 124 between the site shown in the site area 116A and the lesion).
  • the part and the type of lesion hereinafter simply referred to as (also referred to as a "lesion”
  • the positional relationship between the site area 116A and the lesion area 116B for example, the consistency and overlap degree 124 between the site shown in the site area 116A and the lesion.
  • the control unit 104E acquires the endoscopic image 114 and the ultrasound image 116 that is the determination target of the determination unit 104C, displays the acquired ultrasound image 116 on the first screen 22, and displays the acquired endoscopic image 116 on the first screen 22.
  • a mirror image 114 is displayed on the second screen 24.
  • the control unit 104E displays the ultrasound image 116 displayed on the first screen 22 in the first display mode. .
  • the second display mode refers to a mode in which the lesion area 116B is displayed in an identifiable manner within the ultrasound image 116, and the part area 116A is displayed in a manner that can be compared with the lesion area 116B.
  • the outline of the body region 116A and the outline of the lesion region 116B are bordered by a curved line, and the outline of the lesion region 116B is bordered thicker than the outline of the body region 116A.
  • the aspect shown is shown below.
  • the position of the part area 116A and the position of the lesion area 116B in the ultrasound image 116 are displayed so as to be identifiable, and the lesion area 116B is displayed more highlighted than the part area 116A, so that the part area 116A and the lesion area 116B are displayed in a distinguishable state.
  • the lesion area 116B is displayed more highlighted than the part area 116A, it means that the lesion area 116B is displayed more prominently than the part area 116A.
  • Part region 116A may be made more conspicuous than lesion region 116B by differentiating the line types of the curves that frame the outline of region 116A and the outline of lesion region 116B. In this way, any display mode may be used as long as the site area 116A and the lesion area 116B are displayed in a manner that allows them to be specified and compared (that is, distinguishable).
  • step ST12 the detection unit 104B detects the part region 116A and the lesion region 116B from the ultrasound image 116 by performing AI-based image recognition processing on the ultrasound image 116 acquired in step ST10 (Fig. 9 reference). After the process of step ST12 is executed, the display control process moves to step ST14.
  • step ST20 the positional relationship specifying unit 104D determines the degree of overlap calculated in step ST18.
  • step ST16 determines whether the lesion area 116 is positive. If so, the lesion area 116 is determined to be positive.
  • step ST22 the control unit 104E displays the ultrasound image 116 acquired in step ST10 on the first screen 22, and displays the endoscopic image 114 acquired in step ST10 on the second screen 24.
  • the control unit 104E displays the ultrasound image 116 in the first display mode. That is, the control unit 104E hides the site area 116A in the ultrasound image 116 and displays the lesion area 116B (see FIGS. 10 and 12).
  • step ST26 the display control process moves to step ST26.
  • the results of detection of the part area 116A and the lesion area 116B are always displayed in a fixed display format. This allows the user to grasp the lesion more accurately than in the case of the conventional method.
  • the difference between the part and the lesion is shown to the user etc. It can be made easier to recognize.
  • the site area 116A is not displayed on the first screen 22, and the lesion area 116B is displayed on the first screen 22 (see FIG. 10). . Therefore, it is possible to prevent a region from being erroneously recognized as a lesion or a lesion from being erroneously recognized as a region. For example, compared to a case where both the site area 116A and the lesion area 116B are displayed even though the site and the lesion do not match, the site may be incorrectly recognized as a lesion, or the lesion may be incorrectly recognized as a site. can be restrained from doing so.
  • the positional relationship between the site region 116A and the lesion region 116B is defined by the degree of overlap 124. Therefore, the results of the detection of the body part region 116A and the lesion region 116B from the ultrasound image 116 by the detection unit 104B can be displayed on the first screen 22 in a display mode according to the degree of overlap 124 (see FIGS. 12 and 13). ).
  • the positional relationship between the part region 116A and the lesion region 116B is defined by the degree of overlap 124, and when the degree of overlap 124 is equal to or higher than the predetermined degree of overlap, The lesion area 116B is displayed in an identifiable manner (see FIG. 13). Therefore, it is possible for the user or the like to understand the lesions that are highly related to the region shown in the ultrasound image 116.
  • the positional relationship between the part region 116A and the lesion region 116B is defined by the degree of overlap 124, and when the degree of overlap 124 is equal to or higher than the predetermined degree of overlap, The lesion area 116B is displayed so that it can be identified, and the part area 116A is displayed so that it can be compared with the lesion area 116B. Therefore, it is possible for the user or the like to grasp the positional relationship between the region and lesions that are highly related to the region.
  • the certainty of the lesion 116 is determined by performing the processes of steps ST16 to ST20. For example, if the part area 116A and the lesion area 116B match and have a known positional relationship (for example, step ST20: Y), it is determined that the lesion area 116 is certain. Ru. Then, the part region 116A and the lesion region 116B in the ultrasound image 116 are displayed so as to be comparable and distinguishable (see FIG. 13). Thereby, it is possible for the user etc. to understand the parts and lesions that are highly related.
  • the control unit 104E displays the ultrasound image 116 in the second display mode. It is displayed on one screen 22, and the strength of the outline of the part region 116A is set to be the strength according to the degree of overlap 124. For example, the larger the degree of overlap 124 is, the more the outline is made to stand out.
  • Examples of methods to make the outline stand out include increasing the brightness of the outline or increasing the thickness of the outline.
  • the outline of body part region 116A displayed on the first screen 22 when the degree of overlap is "1.0" is displayed on the first screen 22 when the degree of overlap is "0.6". Since the outline of the body part area 116A is thicker than the contour of the body part area 116A, the body part area 116A that is displayed on the first screen 22 when the overlap degree is "1.0" is displayed on the first screen 22 when the overlap degree is "0.6". It is more conspicuous than the part area 116A displayed in 22. This allows the user or the like to recognize the positional relationship (for example, degree of overlap 124) between the matched region and the lesion.
  • the name of the part indicated by the part area 116A is not displayed on the first screen 22, but the technology of the present disclosure is not limited to this.
  • Information indicating the name of the indicated part may be displayed on the first screen 22.
  • the control unit 104E obtains the part name information 118B from the part area information 118, and displays information indicating the name of the part specified from the part name information 118B in a superimposed manner on the part area 116A of the first screen 22.
  • the characters "pancreas" are displayed superimposed on the part area 116A as information indicating the name of the part.
  • information indicating the name of the region in the example shown in FIG. 16, the characters "pancreas" may be displayed as a pop-up from the region region 116A.
  • control unit 104E may switch between displaying and non-displaying information indicating the name of a region in accordance with an instruction received by the reception device 62. In this way, the information indicating the name of the body part is displayed in association with the body part area 116A, thereby allowing the user etc. to understand the name of the body part indicated by the body part area 116A displayed on the first screen 22. be able to.
  • control unit 104E acquires the lesion name information 120B from the lesion area information 120, and displays information indicating the name of the lesion identified from the lesion name information 120B in a superimposed manner on the lesion area 116B of the first screen 22. Good too.
  • the display is not limited to superimposed display, and information indicating the name of the lesion may be displayed as a pop-up from the lesion area 116B.
  • the control unit 104E may switch between displaying and non-displaying information indicating the name of the lesion in accordance with an instruction received by the reception device 62. In this way, the information indicating the name of the lesion is displayed in association with the lesion area 116B, thereby allowing the user etc. to understand the name of the lesion indicated by the lesion area 116B displayed on the first screen 22. be able to.
  • the overlap degree 124 has been described as an example, but this is just an example. For example, as shown in FIG. It may be calculated. Similarly to the degree of overlap 124, the distance 126 is also calculated using the coordinate information 118A and 120A. When the degree of overlap 124 is "1.0", that is, when the entire lesion area 116B overlaps the site area 116A, the distance 126 is 0 mm. If a non-overlapping region exists between site region 116A and lesion region 116B, distance 126 is greater than 0 millimeters.
  • An example of the distance 126 is the distance between the part area 116A and a part of the outline of a region of the lesion area 116B that does not overlap with the part area 116A.
  • the part of the outline of the area that does not overlap with the part area 116A refers to the position 116B1 farthest from the part area 116A among the outlines of the area that does not overlap with the part area 116A. In this way, even if the distance 126 is used instead of the multiplicity 124, the same effects as in the above embodiment can be obtained.
  • the display mode of the ultrasound image 116 is determined according to the site, the lesion, and the positional relationship between the site region and the lesion region 116B, but the technology of the present disclosure is limited to this. Not done. For example, the confidence level for the result that body region 116A was detected by AI-based image recognition processing, the confidence level for the result that lesion region 116B was detected by AI-based image recognition process, and the confidence level for the result that body region 116A and lesion region 116B were detected.
  • the display mode of the ultrasound image 116 may be determined depending on the positional relationship.
  • the learning obtained by having a neural network perform machine learning for detecting body part area 116A is listed.
  • the learned result obtained by having a neural network perform machine learning to detect the lesion area 116B is A value corresponding to the maximum score among multiple scores obtained from the model is listed.
  • An example of a value corresponding to a score is a value obtained by converting a score by an activation function used as an output layer of a neural network (that is, a probability expressed as a value in the range of 0 to 1). .
  • An example of an activation function is a softmax function used as an output layer for multi-class classification.
  • the detection unit 104B acquires a first certainty factor 118C and a second certainty factor 120C used in the AI-based image recognition process for the ultrasound image 116.
  • the first confidence level 118C is the confidence level for the result of detecting the body part region 116A by the AI-based image recognition process.
  • the second confidence level 120C is the confidence level for the result of detecting the lesion area 116B by AI-based image recognition processing.
  • the detection unit 104B generates information including coordinate information 118A, part name information 118B, and first certainty factor 118C as part area information 118. Further, the detection unit 104B generates information including coordinate information 120A, lesion name information 120B, and second certainty factor 120C as lesion area information 120.
  • the determining unit 104C determines the consistency between the body part region 116A and the lesion region 116B in the same manner as in the above embodiment.
  • the display mode of the ultrasound image 116 depends on the magnitude relationship between the first certainty factor 118C and the second certainty factor 1120C, and the positional relationship between the body region 116A and the lesion region 116B. determined accordingly.
  • the positional relationship specifying unit 104D determines whether or not the second certainty factor 120C is greater than the first certainty factor 118C included in the body part region information 118.
  • the positional relationship specifying unit 104D calculates the degree of overlap 124 in the same manner as in the above embodiment, and determines whether the degree of overlap 124 is greater than or equal to the predetermined degree of overlap. Determine whether
  • the control unit 104E displays the endoscopic image 114 on the second screen 24. and display the ultrasound image 116 on the first screen 22 in the first display mode.
  • the control unit 104E displays the endoscopic image 114 on the second screen 24. , and the ultrasound image 116 is displayed on the first screen 22 in the third display mode.
  • the third display mode refers to a mode in which the lesion area 116B is highlighted.
  • Examples of methods for highlighting the lesion area 116B include a method of increasing the brightness of the outline of the lesion area 116B, a method of adding color or a pattern to the lesion area 116B, or a method of highlighting the lesion area 116B in the ultrasound image 116. Examples include a method of hiding areas other than the above. In this way, the highlighted display of the lesion area 116B is achieved by displaying it in a manner that allows it to be distinguished from other areas within the ultrasound image 116.
  • the positional relationship identification unit 104D determines the second certainty factor included in the lesion area information 120. 120C is larger than the first certainty factor 118C included in the part area information 118.
  • the control unit 104E displays the endoscopic image 114 acquired by the acquisition unit 104A on the second screen 24. and displays the ultrasound image 116 acquired by the acquisition unit 104A on the first screen 22.
  • the positional relationship specifying unit 104D determines whether or not the second certainty factor 120C is greater than the first certainty factor 118C included in the body part region information 118.
  • the positional relationship specifying unit 104D calculates the overlap degree 124 in the same manner as in the above embodiment, and determines whether the overlap degree 124 is greater than or equal to the predetermined overlap degree. Determine.
  • the control unit 104E displays the endoscopic image 114 on the second screen 24. and display the ultrasound image 116 on the first screen 22 in the first display mode.
  • the ultrasound image 116 is displayed on the first screen 22 in the second display mode. indicate.
  • the positional relationship specifying unit 104D determines the second certainty factor included in the lesion area information 120. 120C is larger than the first certainty factor 118C included in the part area information 118.
  • the control unit 104E displays the endoscopic image 114 acquired by the acquisition unit 104A on the second screen 24. At the same time, the ultrasound image 116 acquired by the acquisition unit 104A is displayed on the first screen 22.
  • FIG. 25A and FIG. This will be explained with reference to FIG. 25B.
  • FIGS. 25A and 25B differ from the flowchart shown in FIG. 14 in that steps ST50 to ST64 are applied instead of steps ST14 and ST16. Note that, here, the same steps as in the flowchart shown in FIG. 14 are given the same step numbers, and the description thereof will be omitted.
  • step ST50 the detection unit 104B generates information including coordinate information 118A, part name information 118B, and first certainty factor 118C as part area information 118. Further, the detection unit 104B generates information including coordinate information 120A, lesion name information 120B, and second certainty factor 120C as lesion area information 120.
  • step ST50 the display control process moves to step ST52.
  • step ST52 the determination unit 104C refers to the consistency determination table 122 and determines whether or not the part area 116A and the lesion area 116B are consistent based on the part area information 118 and the lesion area information 120 generated in step ST50. (See FIG. 18). In step ST52, if the part area 116A and the lesion area 116B do not match, the determination is negative and the display control process moves to step ST56 shown in FIG. 25B. In step ST52, if the part area 116A and the lesion area 116B match, the determination is affirmative and the display control process moves to step ST54.
  • step ST54 the positional relationship specifying unit 104D acquires the first certainty factor 118C from the part area information 118 generated in step ST50, and obtains the second certainty factor 120C from the lesion area information 120 generated in step ST50. Then, the positional relationship specifying unit 104D determines whether the second certainty factor 120C is greater than the first certainty factor 118C. In step ST54, if the second certainty factor 120C is less than or equal to the first certainty factor 118C, the determination is negative and the process moves to step ST64 shown in FIG. 25B. In step ST54, if the second certainty factor 120C is larger than the first certainty factor 118C, the determination is affirmative and the display control process moves to step ST18.
  • step ST56 shown in FIG. 25B the positional relationship specifying unit 104D obtains a first certainty factor 118C from the part area information 118 generated in step ST50, and obtains a second certainty factor 120C from the lesion area information 120 generated in step ST50. get. Then, the positional relationship specifying unit 104D determines whether the second certainty factor 120C is greater than the first certainty factor 118C. In step ST56, if the second certainty factor 120C is less than or equal to the first certainty factor 118C, the determination is negative and the process moves to step ST64. In step ST56, if the second certainty factor 120C is larger than the first certainty factor 118C, the determination is affirmative and the display control process moves to step ST58.
  • step ST58 the positional relationship specifying unit 104D acquires coordinate information 118A from the body part area information 118 generated in step ST50, and acquires coordinate information 120A from the lesion area information 120 generated in step ST50 (see FIGS. (See Figure 20). Then, the positional relationship specifying information 104D calculates the degree of overlap 124 using the coordinate information 118A and 120A (see FIGS. 19 and 20). After the process of step ST58 is executed, the display control process moves to step ST60.
  • step ST60 the positional relationship specifying unit 104D determines whether the degree of overlap 124 calculated in step ST58 is greater than or equal to the predetermined degree of overlap. In step ST60, if the degree of duplication 124 is less than the predetermined degree of duplication, the determination is negative and the display control process moves to step ST22 shown in FIG. 25A. In step ST60, if the degree of duplication 124 is equal to or greater than the predetermined degree of duplication, the determination is affirmative and the display control process moves to step ST62.
  • step ST62 the control unit 104E displays the ultrasound image 116 acquired in step ST10 on the first screen 22, and displays the endoscopic image 114 acquired in step ST10 on the second screen 24.
  • the control unit 104E displays the ultrasound image 116 in the third display mode. That is, the control unit 104E highlights the lesion area 116B in the ultrasound image 116 (see FIG. 20).
  • step ST62 the display control process moves to step ST26 shown in FIG. 25A.
  • step ST64 the control unit 104E displays the ultrasound image 116 acquired in step ST10 on the first screen 22, and displays the endoscopic image 114 acquired in step ST10 on the second screen 24.
  • step ST64 the display control process moves to step ST26 shown in FIG. 25A.
  • the detection unit 104B detects the part region 116A and the lesion region 116B from the ultrasound image 116, and the result is the first certainty factor 118C, the second certainty factor 120C, and the part region 116A. It is displayed on the first screen 22 in a display manner according to the positional relationship (for example, degree of overlap 124) between the lesion area 116B and the lesion area 116B. Therefore, it is possible to suppress the occurrence of a situation in which a user or the like recognizes a site and a lesion that have little correlation with each other.
  • a user etc. can recognize a site and a lesion that have a low correlation with each other.
  • the detection unit 104B detects the part region 116A and the lesion region 116B from the ultrasound image 116, and the magnitude relationship between the first certainty factor 118C and the second certainty factor 120C and the region It is displayed on the first screen 22 in a display manner according to the positional relationship (for example, degree of overlap 124) between the region 116A and the lesion region 116B. Therefore, it is possible to suppress the occurrence of a situation in which a user or the like recognizes a site and a lesion that have little correlation with each other.
  • the display mode of the ultrasound image 116 is determined without considering the size relationship between the first certainty factor 118C and the second certainty factor 120C and the positional relationship between the body part region 116A and the lesion region 116B. It is possible to suppress the occurrence of a situation in which a user or the like recognizes a lesion and a site with low relevance. Furthermore, the user or the like can be made aware of the magnitude relationship between the first certainty factor 118C and the second certainty factor 120C through the display mode of the first screen 22. Moreover, since the object to be compared with the second certainty factor 120C is the first certainty factor 118C, there is no need to prepare a threshold value in advance for comparison with the second certainty factor 120C.
  • the display mode is determined according to the magnitude relationship between the first certainty factor 118C and the second certainty factor 120C, but the present invention is not limited to this, and the second certainty factor 120C is the default certainty factor. (for example, 0.7) or more, the display mode may be determined depending on whether the value is greater than or equal to 0.7.
  • the predetermined reliability may be a fixed value or a variable value that is changed according to an instruction received by the receiving device 62 and/or various conditions. If the second certainty factor 120C is equal to or higher than the predetermined certainty factor, the lesion area 116B is displayed more emphasized than when the second certainty factor 120C is less than the predetermined certainty factor.
  • the display intensity of the lesion area 116B may be determined depending on the magnitude of the second certainty factor 120C. For example, the greater the second certainty factor 120C, the higher the display intensity of the lesion area 116B.
  • the display intensity of the lesion area 116B when increasing the display intensity of the lesion area 116B according to the degree of overlap 124, whether the display intensity is determined according to the size of the second certainty factor 120C or whether the display intensity is determined according to the degree of overlap 124 is determined. It may be possible to identify whether the diseased area is present or not by the display mode (for example, the color of the outline of the site region 116A and/or the outline of the lesion region 116B, etc.). Note that the display strength for the part region 116A may also be determined using a similar method using the first certainty factor 128C.
  • the display mode of the ultrasound image 116 is determined according to the positional relationship between one body region 116A and the lesion region 116B
  • the technology of the present disclosure is not limited to this.
  • the display mode of the ultrasound image 116 may be determined according to a plurality of positional relationships.
  • the plurality of positional relationships refers to the positional relationship between a plurality of body regions and the lesion area 116B for a plurality of types of body parts.
  • the detection unit 104B detects body regions 116A and 116C and a lesion region 116B from the ultrasound image 116 in the same manner as in the above embodiment.
  • the region indicated by region 116A and the region indicated by region 116C are different types of regions.
  • the site indicated by the site region 116A is the pancreas
  • the site indicated by the site region 116C is the duodenum.
  • the detection unit 104B generates the lesion area information 120 in the same manner as in the above embodiment. Furthermore, the detection unit 104B generates part area information 118 for each of the plurality of parts. In the example shown in FIG. 26, part area information 118 regarding part area 116A and part area information 118 regarding part area 116C are generated by detection unit 104B.
  • the determination unit 104C refers to the consistency determination table 122 and determines the consistency between the part area 116A and the lesion area 116B and the consistency between the part area 116C and the lesion area 116B.
  • the plural body region information 118 generated by the detection unit 104B, the lesion area information 120 generated by the detection unit 104B, and the determination result by the determination unit 104C are used. Display control processing is performed based on this.
  • FIG. 27A and FIG. This will be explained with reference to FIG. 27B.
  • step ST80 is applied instead of step ST12
  • step ST82 is inserted between step ST80 and step ST50
  • step ST84 and step ST86 are inserted between step ST22 and step ST26. Note that here, the same steps as in the flowcharts shown in FIGS. 25A and 25B are given the same step numbers, and the description thereof will be omitted.
  • step ST80 the detection unit 104B performs an AI-based image recognition process to identify a plurality of body parts (here, as an example, body parts 116A and 116C) from the ultrasound image 116. ) is detected, and the lesion area 116B is detected.
  • the display control process moves to step ST82.
  • step ST82 the detection unit 104B acquires one body part area that is not used in the processing from step ST50 onwards from the plurality of body parts areas detected in step ST80. After the process of step ST82 is executed, the display control process moves to step ST50. After step ST50, processing using one body part area acquired in step ST82 or step ST86 shown in FIG. 27B is performed.
  • step ST84 shown in FIG. 27B the control unit 104E determines whether the processes from step ST50 onward have been performed on all body parts detected in step ST80. In step ST84, if the processes from step ST50 onwards have not been executed for all body parts detected in step ST80, the determination is negative and the display control process moves to step ST86. In step ST84, if the processes from step ST50 onward are executed for all body parts detected in step ST80, the determination is affirmative and the display control process moves to step ST26.
  • step ST86 the detection unit 104B acquires one body part area that has not yet been used in the processes from step ST50 onwards from the plurality of body parts areas detected in step ST80. After the process of step ST86 is executed, the display control process moves to step ST50 shown in FIG. 27A.
  • the ultrasound image 116 is displayed in a display mode determined according to the positional relationship between each of the plurality of body regions and the lesion region 116B. be done.
  • step ST20: N the ultrasound image 116 is displayed in the first display mode.
  • body regions 116A and 116C are displayed together with lesion region 116B, but it is unclear which of body regions 116A and 116C the lesion region 116B is associated with.
  • step ST22 by executing the process of step ST22 shown in FIG.
  • the body part areas 116A and 116C are hidden and the lesion area 116B is displayed. It is possible to prevent a user or the like from erroneously recognizing a region with a low value as a region related to the lesion region 116B.
  • the part area 116A and the lesion area 116B are displayed in the second display mode, and the part area 116C and the lesion area 116B are displayed in the first display mode.
  • body regions 116A and 116C are displayed together with lesion region 116B, but it is unclear which of body regions 116A and 116C the lesion region 116B is associated with.
  • the part area 116C is hidden and the lesion area 116B is displayed by executing the process of step ST22 shown in FIG. 27A, so that the lesion area 116B has low relevance. It is possible to prevent the user or the like from erroneously recognizing the body region 116C as a body region that is related to the lesion region 116B. Further, by executing the process of step ST24 shown in FIG. 27A, the body part area 116A and the lesion area 116B are displayed in a contrastable and distinguishable manner, so that the body part area 116A is highly related to the lesion area 116B. It is possible to make the user etc. aware that this is the case.
  • a display that can be compared and distinguished refers to, for example, a display in a display mode that emphasizes the distinguishability between the body part region 116A and the lesion region 116B. Emphasis on distinctiveness is achieved, for example, by enhancing the color difference and/or brightness difference between the site region 116A and the lesion region 116B.
  • the color difference refers to, for example, complementary colors on the hue wheel.
  • the luminance difference for example, if the lesion area 116B is expressed within the luminance range of "150 to 255 gradations", even if the part area 116A is expressed within the luminance range of "0 to 50 gradations". good.
  • the emphasis on distinctiveness can be achieved by, for example, displaying a frame (e.g., a circumscribing frame or an outer contour) surrounding the site area 116A so that the position of the site region 116A can be specified, and a frame (e.g., a circumscribing frame) surrounding the lesion area 116B so that the position can be specified.
  • a frame e.g., a circumscribing frame or an outer contour
  • This can also be realized by differentiating the display mode (or outer contour).
  • the display mode for each of the plurality of body parts regions may be different depending on each of the plurality of positional relationships.
  • the plurality of positional relationships refers to the positional relationship between a plurality of body regions and the lesion area 116B for a plurality of types of body parts.
  • the part area 116A and the lesion area 116B are displayed in the second display mode. Then, when the degree of overlap 124 between the region 116C and the lesion region 116B is less than the predetermined degree of overlap, the region 116C is displayed on the condition that the degree of overlap 124 is "0". Furthermore, even if the overlap degree 124 between the part region 116C and the lesion region 116B is less than the predetermined overlap degree, if the overlap degree 124 is greater than "0", the part region 116C is hidden.
  • the body part area 116C exists in a position where there is no risk of the body part area 116C being mistakenly recognized by the user as a body part area related to the lesion area 116B, the body part area 116C is displayed.
  • the user or the like can be made aware of the positional relationship between the area 116A and the part area 116C, and the positional relationship between the part area 116C and the lesion area 116B.
  • the degree of overlap 124 between the site area 116C and the lesion area 116B is "0"
  • the longer the distance between the site area 116C and the lesion area 116B the greater the display intensity of the site area 116C (for example, the display intensity of the site area 116C
  • the brightness of the outline and/or the thickness of the outline may be increased.
  • the display mode for each of the plurality of body parts may be changed depending on the positional relationship between the plurality of body parts. For example, if part region 116A and part region 116C overlap with lesion region 116B at a predetermined degree of overlap or more, but region region 116C overlaps with less than the predetermined degree of overlap, part region 116C is hidden, and the lesion region 116B overlaps with a predetermined degree of overlap or more. If part area 116A and part area 116C, which overlap with part area 116B, do not overlap, part area 116C may be displayed.
  • part area 116A and part area 116C that overlap with lesion area 116B at a predetermined degree of overlap or higher do not overlap the display intensity of part area 116C increases as the distance between part area 116A and part area 116C increases. It may be increased.
  • the overlap degree 124 with the lesion area 116B is calculated for each of a plurality of body parts, and all body parts are displayed depending on the calculated degree of overlap 124.
  • the technology of the present disclosure is not limited thereto.
  • the degree of overlap 124 between each of the plurality of body regions and the lesion region 116B may be calculated, and only the body region related to the maximum degree of overlap among the plurality of degree of overlap 124 calculated may be displayed. .
  • the display control processing shown in FIGS. 30A and 30B is executed by the processor 104.
  • the flowcharts shown in FIGS. 30A and 30B differ from the flowcharts shown in FIGS. 27A and 27B in that steps ST100 to ST106 are applied instead of step ST20. Note that, here, the same steps as those in the flowcharts shown in FIGS. 27A and 27B are given the same step numbers, and the description thereof will be omitted.
  • step ST100 the positional relationship specifying unit 104D stores the degree of overlap 124 calculated in step ST18 and the part area information 118 generated in step ST50 in correspondence with each other in the RAM 106.
  • step ST100 the degree of overlap 124 and part area information 118 for each of the plurality of part areas (for example, part areas 116A and 116C) detected in step ST80 are stored in correspondence with each other in RAM 106. That is, a plurality of overlap degrees 124 and a plurality of part area information 118 are stored in the RAM 106 in a one-to-one correspondence.
  • step ST102 the positional relationship specifying unit 104D determines whether the processes from step ST50 onward have been executed for all body parts detected in step ST80. In step ST102, if the processes from step ST50 onwards have not yet been executed for all body parts detected in step ST80, the determination is negative and the display control process moves to step ST86. In step ST102, if the processes from step ST50 onwards are executed for all body parts detected in step ST80, the determination is affirmative and the display control process moves to step ST104 shown in FIG. 27B.
  • step ST104 shown in FIG. 27B the positional relationship specifying unit 104D associates the maximum duplication degree, which is the largest duplication degree 124 among the plurality of duplication degrees 124 stored in the RAM 106, with the maximum duplication degree.
  • Part region information 118 is acquired from RAM 106.
  • step ST106 the positional relationship specifying unit 104D determines whether the maximum degree of duplication acquired in step ST104 is equal to or greater than the predetermined degree of duplication. In step ST106, if the maximum degree of duplication is less than the predetermined degree of duplication, the determination is negative and the display control process moves to step ST22. Then, from step ST22 onwards, processing is performed using the part area information 118 acquired at step ST104 and the lesion area information 120 generated at step ST50. On the other hand, in step ST106, if the maximum degree of duplication is equal to or greater than the predetermined degree of duplication, the determination is affirmative and the display control process moves to step ST24. Then, from step ST24 onwards, processing is performed using the part area information 118 acquired at step ST104 and the lesion area information 120 generated at step ST50.
  • the largest part area which is the part area that has the greatest degree of overlap with the lesion area 116B among the plurality of part areas, and the lesion area 116B are
  • the ultrasound image 116 is displayed in a display mode according to the positional relationship between the two. Therefore, even if a plurality of body parts are detected, the user or the like can be made aware of body parts and lesion areas 116B that are highly related to each other.
  • the ultrasound image 116 is displayed in a display mode according to the positional relationship between the largest region among the plurality of region regions and the lesion region 116B, but the present disclosure
  • the technology is not limited to this.
  • the ultrasound image 116 may be displayed in a display mode depending on the positional relationship with the lesion area 116B.
  • the display control processing shown in FIGS. 31A and 31B is executed by the processor 104.
  • the flowchart shown in FIGS. 31A and 31B differs from the flowchart shown in FIGS. 27A and 27B in that steps ST110 to ST114 are applied instead of step ST82 and step ST50. Note that, here, the same steps as those in the flowcharts shown in FIGS. 27A and 27B are given the same step numbers, and the description thereof will be omitted.
  • step ST110 the detection unit 104B generates a plurality of part region information 118 regarding the plurality of part regions (for example, part regions 116A and 116C) detected in step ST80.
  • the display control process moves to step ST112.
  • step ST112 the detection unit 104B compares the plurality of first certainty factors 118C included in the plurality of body part region information 118 generated in step ST110, and determines the most Part region information 118 that includes a large first certainty factor 118C is identified. Then, from step ST52 onwards, processing using the part area information 118 specified in step ST112 is executed. After the process of step ST112 is executed, the display control process moves to step ST114.
  • step ST114 the detection unit 104B generates lesion area information 120 regarding the lesion area 116B detected in step ST80. After step ST52, processing using the lesion area information 120 generated in step ST114 is executed.
  • step ST22 shown in FIG. 31A the display control process moves to step ST26 shown in FIG. 31B. Further, if the determination is negative in step ST26 shown in FIG. 31B, the process moves to step ST10 shown in FIG. 31A, and if the determination is affirmed in step ST26, the display control process ends.
  • the region that includes the largest first certainty factor 118C among the plurality of first certainty factors 118C acquired from the plurality of part region information 118 is displayed.
  • the ultrasound image 116 is displayed in a display manner according to the positional relationship between the region identified using the region information 118 and the lesion region information 120 and the lesion region 116B. Therefore, even if a plurality of body parts are detected, the user or the like can be made aware of body parts and lesion areas 116B that are highly related to each other.
  • the ultrasound image 116 is displayed in a display mode determined for each frame by performing display control processing on each ultrasound image 116 included in the ultrasound video 26 one frame at a time.
  • FIG. 32 as an example, when the display control process is executed on the plurality of ultrasound images 116 in chronological order, the case where the body part region 116A and the lesion region 116B match is matched.
  • the display mode of the ultrasound image 116 may differ depending on whether the ultrasound image 116 is displayed or not.
  • the ultrasound image 116 is displayed in the first display mode, and when the part area 116A and the lesion area 116B match, the ultrasound image 116 is displayed in the first display mode. It may be displayed in the second display mode.
  • the ultrasound image 116 displayed in the first display mode and several frames of ultrasound images 116 adjacent in the chronological order are displayed in the second display mode, they are displayed in the first display mode.
  • the ultrasound image 116 displayed is originally the ultrasound image 116 that would be displayed in the second display mode.
  • the determination unit 104C determines that the part area 116A and the lesion area 116B match, so that the ultrasound image 116 is There is a high possibility that it was displayed in the display mode.
  • the control unit 104E allows the determination unit 104C to correctly determine the display mode of the ultrasound image 116 that may have been incorrectly determined by the determination unit 104C.
  • the ultrasound image 116 is corrected based on the display mode of the ultrasound image 116.
  • the display mode of the ultrasound image 116 that may have been erroneously determined is the determination when the determination unit 104C determines that the combination of the body region 116A and the lesion region 116B is incorrect (that is, they do not match). Refers to a display mode corresponding to the ultrasound image 116 used as a target.
  • the display mode of the ultrasound image 116 that has been determined correctly refers to the display mode of the ultrasound image 116 that is determined when the determination unit 104C determines that the combination of the body part region 116A and the lesion region 116B is correct (that is, they match). This refers to a display mode corresponding to the ultrasonic image 116 (that is, a display mode determined in the same manner as in the above embodiment).
  • the control unit 104E determines the display mode for each of the plurality of ultrasound images 116 in the manner described in the above embodiment, and displays the plurality of ultrasound images 116 in the order in which the display mode is determined. Maintain in series.
  • the control unit 104E holds a plurality of ultrasound images 116 in a FIFO format. That is, the control unit 104E outputs the oldest frame to the display device 14 every time one new frame is added. In the example shown in FIG. 32, for convenience of illustration, the ultrasound images 116 from the first frame to the seventh frame are held in chronological order by the control unit 104E.
  • the ultrasonic images 116 in the 1st to 3rd frames and the 5th to 7th frames are determined by the determination unit 104C to be correct in the combination of the body part region 116A and the lesion region 116B, and are displayed in the second display mode. It has been decided that The fourth frame of the ultrasound image 116 is determined by the determination unit 104C to be an incorrect combination of the body part region 116A and the lesion region 116B, and is determined to be displayed in the first display mode.
  • each ultrasonic image of three frames before and after the ultrasound image 116 in which the combination of the part region 116A and the lesion region 116B is determined to be incorrect (that is, the fourth frame ultrasound image 116) is shown.
  • the ultrasound image 116 of four or more frames in which the combination of the part area 116A and the lesion area 116B was determined to be correct is compared to the ultrasound image 116 in which the combination of the part area 116A and the lesion area 116B was determined to be incorrect. They may be adjacent before or after in chronological order.
  • the ultrasound image 116 in which the combination of the body part region 116A and the lesion region 116B is determined to be incorrect is one frame, but this is just an example.
  • the number of frames in the ultrasound image 116 in which the combination of part area 116A and lesion area 116B was determined to be incorrect is greater than the number of frames in ultrasound image 116 in which the combination of part area 116A and lesion area 116B was determined to be correct.
  • the number of frames should be sufficiently small.
  • a sufficiently small number of frames refers to a number of frames that is about a fraction to a few hundredths of the number of frames of the ultrasound image 116 in which the combination of the body part region 116A and the lesion region 116B is determined to be correct.
  • the sufficiently small number of frames may be a fixed value or may be a variable value that is changed depending on the instruction received by the receiving device 62 and/or various conditions.
  • the control unit 104E controls the ultrasound images 116 of the 1st frame to 3rd frame and the 5th frame to 7th frame among the plurality of ultrasound images 116 held in time series.
  • the display mode of the fourth frame ultrasonic image 116 is corrected with reference to the display mode determined by the method.
  • the second display mode is determined for the ultrasound images 116 of the 1st frame to 3rd frame and the 5th frame to 7th frame, so the ultrasound image 116 of the 4th frame
  • the first display mode determined for the first display mode is modified to the second display mode.
  • the display mode of all ultrasound images 116 held in chronological order is aligned to the second display mode.
  • the control unit 104E adjusts the display mode of the ultrasound images 116 in the first to seventh frames to the second display mode and outputs the ultrasound images 116 in chronological order to the display device 14, thereby displaying the first screen 22.
  • An ultrasound image 116 is displayed on the screen.
  • step ST146 shown in FIG. 33A the detection unit 104B detects a plurality of body regions (for example, pancreas and kidney) from the ultrasound image 116 by performing an AI-based image recognition process, and detects a plurality of lesion regions. 116B (eg, pancreatic cancer and kidney cancer).
  • the display control process moves to step ST148.
  • step ST148 the detection unit 104B generates a plurality of part region information 118 corresponding to the plurality of part regions detected in step ST80. Furthermore, the detection unit 104B generates a plurality of pieces of lesion area information 120 corresponding to the plurality of lesion areas 116B detected in step ST80. After the process of step ST148 is executed, the display control process moves to step ST149.
  • step ST149 the positional relationship specifying unit 104D selects a lesion area to be processed, which is one unprocessed lesion area 116B after step ST150, from the plurality of lesion areas 116B detected in step ST146. After the process of step ST149 is executed, the display control process moves to step ST150.
  • step ST150 the positional relationship specifying unit 104D obtains coordinate information 118A from each of the plurality of part region information 118 generated in step ST110. Further, the positional relationship specifying unit 104D acquires the coordinate information 120A from the lesion area information 120 corresponding to the processing target lesion area selected in step ST149 from among the plurality of lesion area information 120 generated in step ST148. Then, the positional relationship specifying unit 104D calculates the degree of overlap 124 for each part area and the processing target lesion area detected in step ST80.
  • step ST150 a plurality of overlap degrees 124 are calculated. After the process of step ST150 is executed, the display control process moves to step ST152.
  • step ST152 the positional relationship specifying unit 104D determines whether the maximum degree of overlap 124 exists among the plurality of degrees of overlap 124 calculated in step ST150. In step ST152, if the maximum multiplicity 124 does not exist among the plurality of multiplicities 124, the determination is negative, and the display control process moves to step ST154 shown in FIG. 33B. In step ST152, if the maximum degree of duplication 124 exists among the plurality of degrees of duplication 124, the determination is affirmative and the display control process moves to step 156.
  • the disclosed technology is not limited to this, and even if it is determined whether or not the degree of duplication 124 that is equal to or greater than a certain reference value exists. good.
  • step ST154 shown in FIG. 33B the control unit 104E displays the ultrasound image 116 obtained in step ST10 on the first screen 22, and displays the endoscopic image 114 obtained in step ST10 on the second screen 22. to be displayed.
  • the display control process moves to step ST170.
  • step ST156 shown in FIG. 33A the positional relationship specifying unit 104D determines the maximum overlap degree that is the largest overlap degree 124 among the plurality of overlap degrees 124 calculated in step ST150, and the part associated with the maximum overlap degree.
  • the area information 118 is acquired.
  • step ST158 the positional relationship specifying unit 104D determines whether the maximum degree of duplication acquired in step ST156 is equal to or greater than the predetermined degree of duplication. In step ST158, if the maximum degree of duplication is less than the predetermined degree of duplication, the determination is negative and the display control process moves to step ST154 shown in FIG. 33B. In step ST106, if the maximum degree of duplication is equal to or greater than the predetermined degree of duplication, the determination is affirmative and the display control process moves to step ST160.
  • the determination unit 104C determines whether or not the region to be processed and the lesion region to be processed match in the same manner as the process in step ST16 shown in FIG.
  • the processing target body part area refers to a body part area corresponding to the body part area information 118 acquired in step ST156 (that is, a body part area specified from the body part area information 118 acquired in step ST156).
  • the determination of whether the matching between the processing target region and the processing target lesion region is correct is performed using the region region information 118 acquired in step ST156 and the lesion region information 120 corresponding to the processing target lesion region selected in step ST149. It will be done.
  • the lesion area information 120 corresponding to the lesion area to be processed selected in step ST149 is the lesion area corresponding to the lesion area to be processed selected in step ST149 among the plurality of lesion area information 120 generated in step ST148. Refers to information 120.
  • step ST160 if the processing target region and the processing target lesion region match, the determination is negative and the process moves to step ST154 shown in FIG. 33B. In step ST160, if the processing target region and the processing target lesion region do not match, the determination is affirmative and the process moves to step ST162.
  • step ST162 the positional relationship specifying unit 104D obtains the first certainty factor 118C from the part area information 118 obtained in step ST156. Further, the positional relationship specifying unit 104D acquires lesion area information 120 corresponding to the processing target lesion area selected in step ST149 from the plurality of lesion area information 120 generated in step ST148, and from the acquired variable area information 120. A second certainty factor 120C is obtained. Then, the positional relationship specifying unit 104D determines whether the second certainty factor 120C is greater than the first certainty factor 118C. In step ST162, if the second certainty factor 120C is less than or equal to the first certainty factor 118C, the determination is negative and the process moves to step ST164 shown in FIG. 33B. In step ST162, if the second certainty factor 120C is larger than the first certainty factor 118C, the determination is affirmative and the display control process moves to step ST168 shown in FIG. 33B.
  • first certainty factor 118C>second certainty factor 120C the technology of the present disclosure is not limited to this. For example, it may be determined whether the difference between the first certainty factor 118C and the second certainty factor 120C exceeds a threshold value.
  • the conditions for comparing the first certainty factor 118C and the second certainty factor 120C may be changed depending on the type of the region to be processed. For example, a different threshold value may be provided depending on the type of the processing target region, and the first certainty factor 118C and the second certainty factor 120C, which exceed the threshold value, may be compared.
  • step ST164 shown in FIG. 33B the control unit 104E displays the ultrasound image 116 obtained in step ST10 on the first screen 22, and displays the endoscopic image 114 obtained in step ST10 on the second screen 22. to be displayed.
  • the control unit 104E displays the processing target region in the ultrasound image 116 and hides the processing target lesion region in the ultrasound image 116. For example, if the lesion area to be processed is an area indicating pancreatic cancer and the region to be processed is an area indicating a kidney, the area indicating the kidney is displayed and the area indicating pancreatic cancer is hidden.
  • the concept of "hidden” includes, in addition to a mode in which the display is not completely displayed, a level of perception that does not cause misdiagnosis by the doctor 16 (for example, a level of perception that is determined in advance by a sensory test using an actual device and/or a computer simulation). This also includes a mode in which the display intensity (for example, brightness and/or shading) is reduced to a perceptual level known to the user.
  • the display control process moves to step ST170.
  • step ST168 shown in FIG. 33B the control unit 104E displays the ultrasound image 116 obtained in step ST10 on the first screen 22, and displays the endoscopic image 114 obtained in step ST10 on the second screen 22. to be displayed.
  • the control unit 104E displays the ultrasound image 116 displayed on the first screen 22 in the first display mode. That is, the control unit 104E displays the lesion region to be processed in the ultrasound image 116 and hides the region to be processed in the ultrasound image 116. For example, if the lesion region to be processed is an area indicating pancreatic cancer and the region to be processed is an area indicating kidney, the area indicating pancreatic cancer is displayed and the area indicating kidney is hidden.
  • the display control process moves to step ST170.
  • step ST154 whether the processing target region is finally displayed or hidden depends on the type of the processing target lesion region and/or the processing target region. It may be determined accordingly.
  • a region indicating a specific organ for example, the kidney 66 in a scene where the pancreas 65 is examined.
  • the part area indicating the specific organ is subject to processing related to the redundancy 124 and the first certainty factor 118C and the second certainty factor 120C. (ie, used in the processing of steps ST149 to ST162).
  • step ST170 the positional relationship specifying unit 104D determines whether all of the plurality of lesion areas 116B detected in step ST146 have been used in the processing after step ST150. In step ST170, if all of the plurality of lesion areas 116B detected in step ST146 are not used in the processes after step ST150, the determination is negative and the display control process moves to step ST149 shown in FIG. 33A. do. In step ST170, if all of the plurality of lesion areas 116B detected in step ST146 are used in the processes after step ST150, the determination is affirmative and the display control process moves to step ST26.
  • the lesion area to be processed is The certainty of is determined. That is, the certainty of the lesion area to be processed is determined by performing the processes from step ST149 to step ST162 shown in FIG. 33A. Then, the determination result is displayed on the first screen 22 (see step ST164 and step ST168). Therefore, it is possible to prevent a processing target region from being displayed in an incorrect combination with the processing target lesion region, or from displaying a processing target lesion region having an incorrect combination with the processing target region. As a result, it is possible to prevent misidentification by a user or the like.
  • the positional relationship between the lesion area to be processed and the part area to be processed is a predetermined positional relationship (for example, step ST152: Y), and the part area to be processed and the lesion area to be processed are not aligned. (for example, step ST160: N), and the relationship between the first certainty factor 118C and the second certainty factor 120C is a predetermined certainty relationship (for example, step ST162: Y), the lesion area to be processed is certain. It is determined that there is. That is, as a result of performing the processing in steps ST156 to ST164 shown in FIG. 33A, if the determination in step ST162 is affirmative, it is determined that the lesion area to be processed is certain.
  • the determination result is displayed on the first screen 22 (see step ST168). Therefore, the lesion area to be processed that is determined to be certain is displayed. As a result, the user or the like can grasp the lesion area to be processed that has been determined to be certain.
  • the certainty of the processing target region is determined. That is, the certainty of the region to be processed is determined by performing the processing from step ST149 to step ST162 shown in FIG. 33A. Then, the determination result is displayed on the first screen 22 (see step ST164 and step ST168). Therefore, it is possible to prevent a processing target region from being displayed in an incorrect combination with the processing target lesion region, or from displaying a processing target lesion region having an incorrect combination with the processing target region. As a result, it is possible to prevent misidentification by a user or the like.
  • the bounding box that specifies the body part area 116A may be switched between display and non-display, the display mode such as the outline of the bounding box that specifies the body part area 116A may be changed under the same conditions as in the above embodiment, or the lesion area 116B The display mode such as the outline of the bounding box that specifies may be changed under the same conditions as in the above embodiment.
  • the positional relationship specifying unit 104D may calculate the degree of overlap 124 and/or the distance 126 using the bounding box that specifies the part area 116A and the bounding box that specifies the lesion area 116B.
  • An example of the degree of overlap 124 in this case is an IoU using a bounding box that specifies the body region 116A and a bounding box that specifies the lesion area 116B.
  • the IoU is defined as the bounding box that specifies the site area 116A and the lesion area 116B relative to the combined area of the bounding box that specifies the site area 116A and the bounding box that specifies the lesion area 116B. It refers to the percentage of area that overlaps with the bounding box.
  • the degree of overlap 124 is the ratio of the area in which the bounding box that specifies the body part area 116A and the bounding box that specifies the lesion area 116B overlap with respect to the total area of the bounding box that specifies the lesion area 116B. It's okay. Further, as an example of the distance 126 in this case, a part of the outline of an area that does not overlap with the bounding box that specifies the part area 116A among the bounding boxes that specify the lesion area 116B, and the bounding box that specifies the part area 116A.
  • the part of the outline of the area that does not overlap with the body part area 116A refers to the position furthest from the bounding box that specifies the body part area 116A among the contours of the area that does not overlap with the body part area 116A.
  • the first display mode is an example in which body part areas 116A and 116C are hidden, but this is just an example, and without hiding body part areas 116A and/or 116C,
  • the display intensity of the part areas 116A and /116C may be lower than the display intensity of the lesion area 116B.
  • the strength of the contour of the part region 116A is set according to the degree of overlap 124, but the technology of the present disclosure is not limited to this, and when the degree of overlap 124 is equal to or higher than the predetermined degree of overlap. Furthermore, the display intensity of both the part region 116A and the lesion region 116B may be increased as the degree of overlap 124 increases.
  • the ultrasound image 116 may be displayed on the entire screen of the display device 14.
  • At least the first screen 22 of the 24 screens is displayed.
  • the processor 104 uses cloud computing to display at least the first screen 22 of the first screen 22 and the second screen 24 on the display device 14 or a display device other than the display device 14.
  • An example of the display format is
  • the display control process may be performed on the ultrasound still image.
  • display control processing may be performed on an ultrasound image acquired by an external ultrasound diagnostic apparatus using an external ultrasound probe.
  • a display control process is executed on a medical image obtained by imaging the observation target area of the subject 20 by various modalities such as an X-ray diagnostic device, a CT diagnostic device, and/or an MRI diagnostic device. You may also do so.
  • an extracorporeal ultrasound diagnostic device, an X-ray diagnostic device, a CT diagnostic device, and/or an MRI diagnostic device are examples of the “imaging device” according to the technology of the present disclosure.
  • a device that performs display control processing may be provided outside the display control device 60.
  • Examples of devices provided outside the display control device 60 include the endoscope processing device 54 and/or the ultrasound processing device 58.
  • another example of a device provided outside the display control device 60 is a server.
  • the server is realized by cloud computing.
  • cloud computing is illustrated here, this is just one example.
  • the server may be realized by a mainframe, or may be implemented using fog computing, edge computing, grid computing, etc. It may be realized by network computing.
  • the server is merely an example, and instead of the server, at least one personal computer or the like may be used. Further, the display control processing may be performed in a distributed manner by a plurality of devices including the display control device 60 and at least one device provided outside the display control device 60.
  • the display control processing program 112 may be stored in a portable storage medium such as an SSD or a USB memory.
  • a storage medium is a non-transitory computer-readable storage medium.
  • the display control processing program 112 stored in the storage medium is installed in the computer 100 of the display control device 60.
  • the processor 104 executes display control processing according to the display control processing program 112.
  • the computer 100 is illustrated in the above embodiment, the technology of the present disclosure is not limited thereto, and instead of the computer 100, a device including an ASIC, an FPGA, and/or a PLD may be applied. Further, in place of the computer 100, a combination of hardware configuration and software configuration may be used.
  • processors can be used as hardware resources for executing the display control processing described in the above embodiments.
  • the processor include a processor that is a general-purpose processor that functions as a hardware resource that executes display control processing by executing software, that is, a program.
  • the processor include a dedicated electronic circuit such as an FPGA, a PLD, or an ASIC, which is a processor having a circuit configuration specifically designed to execute a specific process.
  • Each processor has a built-in or connected memory, and each processor uses the memory to execute display control processing.
  • the hardware resources that execute display control processing may be configured with one of these various types of processors, or a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs, or (a combination of a processor and an FPGA). Furthermore, the hardware resource that executes the display control process may be one processor.
  • one processor is configured by a combination of one or more processors and software, and this processor functions as a hardware resource that executes display control processing.
  • a and/or B has the same meaning as “at least one of A and B.” That is, “A and/or B” means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, even when three or more items are expressed by connecting them with “and/or”, the same concept as “A and/or B" is applied.
  • the above processor is detecting a first image area indicating the body part and a second image area indicating the lesion from a medical image obtained by imaging an observation target area including the body part and the lesion;
  • An image processing device that displays results of detecting the first image area and the second image area on a display device in a display mode according to a positional relationship between the first image area and the second image area.
  • the processor obtains a first confidence level that is a confidence level for the result of detecting the first image area and a second confidence level that is a confidence level for the result of detecting the second image area,
  • the display mode is determined according to the first certainty factor, the second certainty factor, and the positional relationship, When the first certainty factor is larger than the second certainty factor and the first image region and the second image region do not overlap, the display mode is different from the first image region in the medical image.
  • the image processing device according to Supplementary Note 1, wherein the second image area is displayed so as to be able to be compared with the second image area.
  • the processor determines whether the combination of the first image area and the second image area is correct or not based on the correspondence between the plurality of types of the parts and the lesions corresponding to each part, The combination of the first image region and the second image region is determined to be correct by the processor, the first certainty factor is greater than the second certainty factor, and the combination of the first image region and the second image region is determined to be correct.
  • the image processing device according to supplementary note 1, wherein the display mode is a mode in which the first image region and the second image region are displayed in a comparable manner in the medical image, when the regions do not overlap.
  • the observation target area includes multiple types of the sites and the lesions
  • the above processor is detecting a plurality of the first image regions and the second image regions indicating a plurality of types of the regions from the medical image;
  • the positional relationship is determined by the degree to which the second image area overlaps with the second image area among the plurality of first image areas. is the relationship between the position of the largest overlapping image area and the position of the second image area.
  • the processor obtains a first confidence level that is a confidence level for the result of detecting the first image area and a second confidence level that is a confidence level for the result of detecting the second image area,
  • the display mode is determined according to the first certainty factor, the second certainty factor, and the positional relationship,
  • the observation target area includes multiple types of the sites and the lesions,
  • the above processor is detecting a plurality of the first image regions and the second image regions indicating a plurality of types of the regions from the medical image;
  • the first certainty factor is the plurality of certainty factors for the plurality of results of detecting the plurality of first image regions.
  • the processor obtains a first confidence level that is a confidence level for the result of detecting the first image area, The image processing device according to supplementary note 1, wherein the display mode is determined according to the first certainty factor and the positional relationship.
  • the processor obtains a second confidence level that is a confidence level for the result of detecting the second image area, The image processing device according to supplementary note 1, wherein the display mode is determined according to the second certainty factor and the positional relationship.

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Medical Informatics (AREA)
  • Veterinary Medicine (AREA)
  • Biophysics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Pathology (AREA)
  • Radiology & Medical Imaging (AREA)
  • Public Health (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Physics & Mathematics (AREA)
  • Molecular Biology (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Vascular Medicine (AREA)
  • Physiology (AREA)
  • Ultra Sonic Daignosis Equipment (AREA)
PCT/JP2023/004985 2022-03-29 2023-02-14 画像処理装置、医療診断装置、超音波内視鏡装置、画像処理方法、及びプログラム Ceased WO2023188903A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
JP2024511381A JPWO2023188903A1 (https=) 2022-03-29 2023-02-14
CN202380030763.8A CN118946315A (zh) 2022-03-29 2023-02-14 图像处理装置、医疗诊断装置、超声波内窥镜装置、图像处理方法、及程序
US18/890,704 US20250009329A1 (en) 2022-03-29 2024-09-19 Image processing apparatus, medical diagnostic apparatus, ultrasonic endoscope apparatus, image processing method, and program

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2022-054506 2022-03-29
JP2022054506 2022-03-29

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/890,704 Continuation US20250009329A1 (en) 2022-03-29 2024-09-19 Image processing apparatus, medical diagnostic apparatus, ultrasonic endoscope apparatus, image processing method, and program

Publications (1)

Publication Number Publication Date
WO2023188903A1 true WO2023188903A1 (ja) 2023-10-05

Family

ID=88201049

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/004985 Ceased WO2023188903A1 (ja) 2022-03-29 2023-02-14 画像処理装置、医療診断装置、超音波内視鏡装置、画像処理方法、及びプログラム

Country Status (4)

Country Link
US (1) US20250009329A1 (https=)
JP (1) JPWO2023188903A1 (https=)
CN (1) CN118946315A (https=)
WO (1) WO2023188903A1 (https=)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010075327A (ja) * 2008-09-25 2010-04-08 Konica Minolta Medical & Graphic Inc 画像診断支援装置、画像診断支援方法及びプログラム
WO2017179350A1 (ja) * 2016-04-11 2017-10-19 富士フイルム株式会社 画像表示制御装置および方法並びにプログラム
JP2018175700A (ja) * 2017-04-20 2018-11-15 キヤノンメディカルシステムズ株式会社 医用画像診断装置、医用画像処理装置、及び医用画像処理プログラム
JP2020192274A (ja) * 2019-05-30 2020-12-03 キヤノンメディカルシステムズ株式会社 医用処理装置及び放射線治療装置
JP2021100555A (ja) * 2019-12-24 2021-07-08 富士フイルム株式会社 医療画像処理装置、内視鏡システム、診断支援方法及びプログラム
JP2021121261A (ja) * 2020-01-31 2021-08-26 学校法人慶應義塾 診断支援プログラム、装置、及び方法

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010075327A (ja) * 2008-09-25 2010-04-08 Konica Minolta Medical & Graphic Inc 画像診断支援装置、画像診断支援方法及びプログラム
WO2017179350A1 (ja) * 2016-04-11 2017-10-19 富士フイルム株式会社 画像表示制御装置および方法並びにプログラム
JP2018175700A (ja) * 2017-04-20 2018-11-15 キヤノンメディカルシステムズ株式会社 医用画像診断装置、医用画像処理装置、及び医用画像処理プログラム
JP2020192274A (ja) * 2019-05-30 2020-12-03 キヤノンメディカルシステムズ株式会社 医用処理装置及び放射線治療装置
JP2021100555A (ja) * 2019-12-24 2021-07-08 富士フイルム株式会社 医療画像処理装置、内視鏡システム、診断支援方法及びプログラム
JP2021121261A (ja) * 2020-01-31 2021-08-26 学校法人慶應義塾 診断支援プログラム、装置、及び方法

Also Published As

Publication number Publication date
JPWO2023188903A1 (https=) 2023-10-05
CN118946315A (zh) 2024-11-12
US20250009329A1 (en) 2025-01-09

Similar Documents

Publication Publication Date Title
US12558068B2 (en) Medical image processing apparatus, endoscope system, medical image processing method, and medical image processing program
WO2024004597A1 (ja) 学習装置、学習済みモデル、医用診断装置、超音波内視鏡装置、学習方法、及びプログラム
JP5527841B2 (ja) 医療画像処理システム
JPWO2022202401A5 (https=)
US20250352169A1 (en) Ultrasonic diagnostic apparatus and method of controlling ultrasonic diagnostic apparatus
US20250090145A1 (en) Diagnostic assistance apparatus, ultrasound endoscope, diagnostic assistance method, and program
EP4609779A1 (en) Medical support device, endoscope system, and medical support method
US20250292400A1 (en) Image processing device, endoscope system, image processing method, and program
WO2023188903A1 (ja) 画像処理装置、医療診断装置、超音波内視鏡装置、画像処理方法、及びプログラム
WO2024004542A1 (ja) 診断支援装置、超音波内視鏡、診断支援方法、及びプログラム
EP4635422A1 (en) Ultrasonic diagnostic device and method for controlling ultrasonic diagnostic device
WO2024202789A1 (ja) 医療支援装置、内視鏡システム、医療支援方法、及びプログラム
JP2025037660A (ja) 医療支援装置、内視鏡装置、医療支援方法、及びプログラム
EP4577120A1 (en) Guided ultrasound imaging for point-of-care staging of medical conditions
JP2025139336A (ja) 画像処理装置、内視鏡システム、画像処理方法、及びプログラム
JP7822849B2 (ja) 画像処理装置、医療診断装置、内視鏡装置、及び医療機器の作動方法
US20230380910A1 (en) Information processing apparatus, ultrasound endoscope, information processing method, and program
EP4681653A1 (en) Ultrasonic diagnostic device and method for controlling ultrasonic diagnostic device
US20250169676A1 (en) Medical support device, endoscope, medical support method, and program
US20230419693A1 (en) Medical image processing apparatus, endoscope system, medical image processing method, and medical image processing program
US20250104242A1 (en) Medical support device, endoscope apparatus, medical support system, medical support method, and program
US20250356494A1 (en) Image processing device, endoscope, image processing method, and program
US20250268577A1 (en) Medical assistance apparatus, ultrasound endoscope, medical assistance method, and program
JPWO2023188903A5 (https=)
US20250366820A1 (en) Ultrasonic diagnostic appratus, ultrasonic diagnostic program, and ultrasonic diagnostic method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23778937

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2024511381

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 202380030763.8

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 23778937

Country of ref document: EP

Kind code of ref document: A1