Classifications

• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
  • A61B8/54—Control of the diagnostic device
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
  • A61B8/46—Ultrasonic, sonic or infrasonic diagnostic devices with special arrangements for interfacing with the operator or the patient
  • A61B8/467—Ultrasonic, sonic or infrasonic diagnostic devices with special arrangements for interfacing with the operator or the patient characterised by special input means
  • A61B8/469—Ultrasonic, sonic or infrasonic diagnostic devices with special arrangements for interfacing with the operator or the patient characterised by special input means for selection of a region of interest
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
  • A61B8/52—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves
  • A61B8/5207—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of raw data to produce diagnostic data, e.g. for generating an image
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
  • A61B8/52—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves
  • A61B8/5215—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of medical diagnostic data
  • A61B8/5238—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of medical diagnostic data for combining image data of patient, e.g. merging several images from different acquisition modes into one image
  • A61B8/5246—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of medical diagnostic data for combining image data of patient, e.g. merging several images from different acquisition modes into one image combining images from the same or different imaging techniques, e.g. color Doppler and B-mode
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
  • A61B8/12—Diagnosis using ultrasonic, sonic or infrasonic waves in body cavities or body tracts, e.g. by using catheters
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
  • A61B8/46—Ultrasonic, sonic or infrasonic diagnostic devices with special arrangements for interfacing with the operator or the patient
  • A61B8/461—Displaying means of special interest
  • A61B8/463—Displaying means of special interest characterised by displaying multiple images or images and diagnostic data on one display
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
  • A61B8/48—Diagnostic techniques
  • A61B8/485—Diagnostic techniques involving measuring strain or elastic properties
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
  • A61B8/48—Diagnostic techniques
  • A61B8/488—Diagnostic techniques involving Doppler signals
• • A—HUMAN NECESSITIES
  • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
  • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
  • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
  • A61B8/52—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves
  • A61B8/5215—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of medical diagnostic data
  • A61B8/5223—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of medical diagnostic data for extracting a diagnostic or physiological parameter from medical diagnostic data

Definitions

• the technology of the present disclosure relates to a diagnosis support device, an ultrasound endoscope, a diagnosis support method, and a program.
• International Publication No. 2020/036121 describes an identification unit that identifies the type of image of a subject, a recognition unit that performs recognition processing to recognize the subject using the image, and a specific type of image identified by the identification unit.
• An endoscope system has been disclosed that includes a notification unit that reports whether or not recognition processing is functioning.
• JP 2021-035442A discloses an ultrasonic diagnostic system that performs diagnostic support when the image mode is B mode or CF mode, and does not perform diagnostic support when the image mode is PW mode.
• Japanese Patent Application Publication No. 2021-083699 discloses a probe that is brought into contact with the breast and outputs a received signal by transmitting and receiving ultrasonic waves to and from the breast, and a probe that outputs a received signal by transmitting and receiving ultrasonic waves to the breast, and based on the received signal, a mammary gland image, a pectoralis major muscle image, and their images.
• an image generation section that generates an ultrasound image including a boundary image between the two; a tilt angle calculation section that computes a tilt angle of the boundary image based on the ultrasound image; and a probe operation based on the tilt angle of the boundary image.
• An ultrasonic diagnostic apparatus is disclosed that includes a support image generation unit that generates a support image to support.
• One embodiment of the technology of the present disclosure is capable of suppressing areas other than the specific area from being erroneously detected as the specific area, and preventing visualization of the detection results of the specific area from interfering with diagnosis.
• a first aspect of the technology of the present disclosure includes a processor, and the processor acquires a first ultrasound image generated by an ultrasound module and includes an observation area, and diagnoses the observation area. Whether the reference information to be referred to is synthesized with the first ultrasound image, whether the first ultrasound image is an image obtained in an auxiliary image mode that is an image mode other than the main image mode, or The first operation mode and the second operation mode are switched according to the setting value that determines the image quality of the first ultrasound image, and the first operation mode is created using the second ultrasound image obtained in the main image mode.
• This is an operation mode in which a specific area is detected from the first ultrasound image based on the detected detection support information, and the second operation mode is not to detect the specific area and output the detection result, or not to detect the specific area.
• a second aspect of the technology of the present disclosure is that the first operation mode is an operation mode used when the reference information is not combined with the first ultrasound image, and the second operation mode is an operation mode that is used when the reference information is not combined with the first ultrasound image.
• This is the diagnosis support device according to the first aspect, which is an operation mode used when the ultrasound images are combined into one ultrasound image.
• a third aspect of the technology of the present disclosure is that the first operation mode is an operation mode used when the first ultrasound image is not an image obtained in the auxiliary image mode, and the second operation mode is an operation mode that is used when the first ultrasound image is not an image obtained in the auxiliary image mode.
• This is the diagnosis support device according to the first aspect or the second aspect, which is an operation mode used when the ultrasound image is an image obtained in the auxiliary image mode.
• a fourth aspect of the technology of the present disclosure is that the first operation mode is an operation mode used when the set value is within a specified range, and the second operation mode is used when the set value is not within the specified range.
• This is a diagnostic support device according to any one of the first to third aspects, which are operating modes used.
• a fifth aspect of the technology of the present disclosure is the diagnostic support according to any one of the first to fourth aspects, in which the reference information includes color information expressing characteristics in the observation target area using colors. It is a device.
• the color information includes a plurality of chromatic pixels
• the first operation mode is a chromatic pixel whose saturation exceeds a first threshold value among the plurality of chromatic pixels.
• the diagnostic support device is an operation mode that is used when the number of objects is less than a second threshold
• the second operation mode is an operation mode that is used when the number is greater than or equal to the second threshold.
• a seventh aspect of the technology of the present disclosure is the diagnosis support device according to any one of the first to sixth aspects, in which the reference information includes text information that assists observation of the observation target area. .
• An eighth aspect of the technology of the present disclosure is the diagnosis support device according to any one of the first to seventh aspects, wherein the reference information includes a measurement line used for measurement within the observation target area. be.
• a ninth aspect of the technology of the present disclosure is the diagnostic support according to any one of the first to eighth aspects, in which the reference information includes treatment auxiliary information that assists treatment using fine-needle aspiration. It is a device.
• the auxiliary image mode is a high-frequency component included in a reflected wave obtained by emitting ultrasonic waves to the observation area and being reflected at the observation area.
• the first image mode is a first image mode that generates an ultrasound image using This is a diagnostic support device.
• An eleventh aspect according to the technology of the present disclosure is the tenth aspect, wherein the first image mode is THI mode, CH mode, or CHI mode, and the second image mode is Doppler mode or elastography mode. This is a diagnostic support device.
• a twelfth aspect of the technology of the present disclosure is that the setting values include a frequency parameter that adjusts the frequency of ultrasound emitted from the ultrasound module, a depth parameter that adjusts the depth expressed in the first ultrasound image, A brightness parameter that adjusts the brightness of the first ultrasound image, a dynamic range parameter that adjusts the dynamic range of the first ultrasound image, and/or a magnification parameter that adjusts the magnification of digital zoom for the first ultrasound image.
• This is a diagnostic support device according to any one of the first to eleventh aspects.
• a thirteenth aspect according to the technology of the present disclosure is any one of the first to twelfth aspects, wherein the ultrasound module has a setting value, and the processor acquires the setting value from the ultrasound module.
• 1 is a diagnostic support device according to two aspects.
• a fourteenth aspect of the technology of the present disclosure is that a text image whose setting value can be specified is synthesized with the frame including the first ultrasound image, and the processor performs image recognition processing on the text image.
• This is a diagnostic support device according to any one of the first to twelfth aspects, in which a setting value is specified by characterizing the setting value, and the first operation mode and the second operation mode are switched according to the characteristic setting value.
• a fifteenth aspect of the technology of the present disclosure is the diagnosis support device according to any one of the first to fourteenth aspects, wherein the processor detects a specific region from the first ultrasound image using an AI method. be.
• a 16th aspect according to the technology of the present disclosure is a 15th aspect from the 1st aspect, wherein the detection support information is a trained model obtained by causing the model to learn teacher data including the second ultrasound image.
• the detection support information is a trained model obtained by causing the model to learn teacher data including the second ultrasound image.
• a diagnostic support device according to any one of the aspects.
• a seventeenth aspect of the technology of the present disclosure is that the processor detects a specific region from the first ultrasound image with a frequency of detecting the specific region from the first ultrasound image according to the reference information, the auxiliary image mode, and/or the setting value.
• This is a diagnosis support device according to any one of the first to sixteenth aspects, in which the accuracy of detecting the specific region and/or the target to be detected as the specific region from the first ultrasound image is varied.
• An eighteenth aspect according to the technology of the present disclosure is the diagnostic support device according to any one of the first to seventeenth aspects, wherein the ultrasound module is an ultrasound endoscope.
• a nineteenth aspect according to the technology of the present disclosure includes a diagnosis support device according to any one of the first to eighteenth aspects, an ultrasound endoscope main body to which an ultrasound diagnostic device is connected, This is an ultrasonic endoscope equipped with.
• a 20th aspect of the technology of the present disclosure is to obtain a first ultrasound image that is generated by an ultrasound module and includes a region to be observed, and a reference that is referred to for diagnosing the region to be observed. whether the information is combined with the first ultrasound image; whether the first ultrasound image is an image obtained in an auxiliary image mode that is an image mode other than the main image mode; switching between a first operating mode and a second operating mode in accordance with a setting value defining image quality of the image, the first operating mode being created using the second ultrasound image obtained in the primary image mode; This is an operation mode in which a specific area is detected from the first ultrasound image based on the detection support information, and the second operation mode is an operation in which the specific area is not detected and the detection result is not output, or the specific area is not detected. This is a diagnostic support method.
• a twenty-first aspect of the technology of the present disclosure is a program for causing a computer to execute processing, the processing acquiring a first ultrasound image generated by an ultrasound module and showing an observation target area. and whether the reference information referred to for diagnosing the observation target area is synthesized with the first ultrasound image, and whether the first ultrasound image is in an auxiliary image mode that is an image mode other than the main image mode.
• the method includes switching between a first operation mode and a second operation mode depending on whether the image is an acquired image or a setting value that determines the image quality of the first ultrasound image, and the first operation mode is the main one.
• FIG. 1 is a conceptual diagram showing an example of a mode in which an endoscope system is used.
• FIG. 1 is a conceptual diagram showing an example of the overall configuration of an endoscope system.
• FIG. 1 is a block diagram showing an example of the configuration of an ultrasound endoscope.
• FIG. 3 is a conceptual diagram illustrating an example of a mode in which a trained model is generated by causing a model to learn a group of B-mode images.
• FIG. 2 is a conceptual diagram showing an example of processing contents of a generation unit.
• FIG. 3 is a conceptual diagram showing an example of processing contents for switching between detection mode and non-detection mode.
• FIG. 1 is a conceptual diagram showing an example of a mode in which an endoscope system is used.
• FIG. 1 is a conceptual diagram showing an example of the overall configuration of an endoscope system.
• FIG. 1 is a block diagram showing an example of the configuration of an ultrasound endoscope.
• FIG. 3 is a conceptual diagram illustrating an
• FIG. 3 is a conceptual diagram showing an example of processing contents for detecting a lesion area from a B-mode image and displaying the B-mode image on a screen of a display device.
• FIG. 2 is a conceptual diagram showing an example of processing contents for generating a Doppler image and displaying it on a screen of a display device.
• It is a flowchart which shows an example of the flow of diagnostic support processing. This is a continuation of the flowchart shown in FIG. 9A.
• It is a flowchart which shows an example of the flow of diagnostic support processing concerning a 1st modification.
• It is a conceptual diagram showing an example of processing contents of a generation part and a control part concerning a 2nd modification.
• CPU is an abbreviation for "Central Processing Unit”.
• GPU is an abbreviation for “Graphics Processing Unit.”
• TPU is an abbreviation for “Tensor Processing Unit”.
• RAM is an abbreviation for "Random Access Memory.”
• NVM is an abbreviation for “Non-volatile memory.”
• EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory.”
• ASIC is an abbreviation for “Application Specific Integrated Circuit.”
• PLD is an abbreviation for “Programmable Logic Device”.
• FPGA is an abbreviation for "Field-Programmable Gate Array.”
• SoC is an abbreviation for “System-on-a-chip.”
• SSD is an abbreviation for “Solid State Drive.”
• USB is an abbreviation for “Universal Serial Bus.”
• HDD is an abbreviation for “Hard Disk Drive.”
• EL is an abbreviation for "Electro-Luminescence”.
• CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor.”
• CCD is an abbreviation for “Charge Coupled Device”.
• PC is an abbreviation for "Personal Computer.”
• LAN is an abbreviation for “Local Area Network.”
• WAN is an abbreviation for “Wide Area Network.”
• AI is an abbreviation for “Artificial Intelligence.”
• BLI is an abbreviation for “Blue Light Imaging.”
• LCI is an abbreviation for "Linked Color Imaging.”
• NN is an abbreviation for “Neural Network”.
• CNN is an abbreviation for “Convolutional neural network.”
• R-CNN is an abbreviation for “Region based Convolutional Neural Network”.
• YOLO is an abbreviation for "You only Look Once.”
• RNN is an abbreviation for "Recurrent Neural Network.”
• FCN is an abbreviation for “Fully Convolutional Network.”
• THI is an abbreviation for "Tissue Harmonic Imaging”.
• CH is an abbreviation for "Compound Harmonic”.
• CHI is an abbreviation for "Contrast Harmonic Imaging”.
• an endoscope system 10 includes an ultrasound endoscope 12 and a display device 14.
• the ultrasound endoscope 12 is a convex type ultrasound endoscope, and includes an ultrasound endoscope main body 16 and a processing device 18 .
• the ultrasound endoscope 12 is an example of an "ultrasonic module” and an “ultrasonic endoscope” according to the technology of the present disclosure.
• the processing device 18 is an example of a "diagnosis support device” according to the technology of the present disclosure.
• the ultrasound endoscope main body 16 is an example of an “ultrasonic endoscope main body” according to the technology of the present disclosure.
• a convex type ultrasound endoscope is used as an example of the ultrasound endoscope 12, but this is just an example, and a radial type ultrasound endoscope is used.
• the technology of the present disclosure is also applicable.
• the ultrasound endoscope main body 16 is used by a doctor 20, for example.
• the processing device 18 is connected to the ultrasound endoscope main body 16 and exchanges various signals with the ultrasound endoscope main body 16. That is, the processing device 18 controls the operation of the ultrasound endoscope body 16 by outputting a signal to the ultrasound endoscope body 16, and controls the operation of the ultrasound endoscope body 16 in response to a signal input from the ultrasound endoscope body 16. Performs various signal processing.
• the ultrasound endoscope 12 is a device for performing medical treatment (for example, diagnosis and/or treatment) on a medical treatment target site (for example, an organ such as the pancreas) in the body of a subject 22, and includes the medical treatment target site.
• An ultrasound image 24 showing the observation target area is generated and output.
• the doctor 20 when observing an observation target region inside the body of the subject 22, the doctor 20 inserts the ultrasound endoscope main body 16 into the subject 22 from the mouth or nose (mouth in the example shown in FIG. 1) of the subject 22. It is inserted into the body and emits ultrasonic waves at locations such as the stomach or duodenum.
• the ultrasonic endoscope main body 16 emits ultrasonic waves to an observation target area inside the body of the subject 22, and detects reflected waves obtained by reflecting the emitted ultrasonic waves at the observation target area.
• FIG. 1 shows an aspect in which an upper gastrointestinal endoscopy is being performed
• the technology of the present disclosure is not limited to this, and is applicable to lower gastrointestinal endoscopy or endobronchial endoscopy.
• the technology of the present disclosure is also applicable to endoscopy and the like.
• the processing device 18 generates an ultrasound image 24 based on the reflected waves detected by the ultrasound endoscope main body 16 and outputs it to the display device 14 or the like.
• the display device 14 displays various information including images under the control of the processing device 18.
• An example of the display device 14 is a liquid crystal display, an EL display, or the like.
• the ultrasound image 24 generated by the processing device 18 is displayed on the screen 26 of the display device 14 as a moving image.
• a mode is shown in which the ultrasound image 24 within the screen 26 includes a lesion area 25 indicating a location corresponding to a lesion.
• a rectangular detection frame 27A that allows the position of the lesion area 25 within the ultrasound image 24 to be specified is displayed on the screen 26.
• the doctor 20 can determine whether or not a lesion is visible in the observation target area, and if a lesion is found, determine whether the lesion is within the observation target area.
• the position of the lesion is specified by referring to the detection frame 27A within the ultrasound image 24.
• the example shown in FIG. 1 shows an example in which the ultrasound image 24 is displayed on the screen 26 of the display device 14, this is just an example; For example, it may be displayed on a display of a tablet terminal.
• the ultrasound images 24 may also be stored on a computer-readable non-transitory storage medium (eg, flash memory, HDD, and/or magnetic tape).
• the display device 14 displays an ultrasound image 24 according to the image mode selected by the doctor 20.
• the image mode refers to a display mode in which reflected waves detected by the ultrasound endoscope body 16 are converted into images and displayed on the display device 14.
• B mode is an example of the "main image mode” according to the technology of the present disclosure
• Doppler mode is an example of the "auxiliary image mode that is an image mode other than the main image mode” according to the technology of the present disclosure.
• B mode is an image mode in which the intensity of reflected waves is converted into brightness and displayed as a two-dimensional tomographic image (hereinafter referred to as "B mode image").
• Doppler mode is an image mode in which hemodynamics identified using the Doppler effect are displayed as color information superimposed on a B-mode image.
• Doppler image an image generated under Doppler mode, that is, an image obtained by superimposing color information indicating hemodynamics on a B-mode image.
• images generated under different image modes such as B-mode images and Doppler images, are simply referred to as “ultrasound images 24" below, when there is no need to distinguish between them and explain them.
• the ultrasound image 24 is a moving image that includes a plurality of frames generated according to a frame rate determined according to the image mode.
• the frame rate of Doppler mode is lower than the frame rate of B mode. Note that although a moving image is illustrated here, this is just an example, and the technology of the present disclosure is valid even if the ultrasound image 24 is a still image.
• the ultrasound endoscope main body 16 includes an operating section 28 and an insertion section 30.
• the insertion portion 30 is formed into a tubular shape.
• the insertion portion 30 has a distal end portion 32, a curved portion 34, and a flexible portion 36.
• the distal end portion 32, the curved portion 34, and the flexible portion 36 are arranged in this order from the distal end side to the proximal end side of the insertion portion 30.
• the flexible section 36 is made of a long, flexible material and connects the operating section 28 and the curved section 34 .
• the bending portion 34 partially curves or rotates around the axis of the insertion portion 30 when the operating portion 28 is operated.
• the insertion section 30 curves depending on the shape of the hollow organ (for example, the shape of the duodenal duct) and rotates around the axis of the insertion section 30 while moving toward the back side of the hollow organ. sent.
• the tip portion 32 is provided with an ultrasonic probe 38 and a treatment tool opening 40.
• the ultrasonic probe 38 is provided on the distal end side of the distal end portion 32.
• the ultrasonic probe 38 is a convex type ultrasonic probe that emits ultrasonic waves and receives reflected waves obtained by reflecting the emitted ultrasonic waves at the observation target area.
• the treatment instrument opening 40 is formed closer to the proximal end of the distal end portion 32 than the ultrasound probe 38 is.
• the treatment tool opening 40 is an opening for allowing the treatment tool 42 to protrude from the distal end portion 32.
• a treatment instrument insertion port 44 is formed in the operation section 28 , and the treatment instrument 42 is inserted into the insertion section 30 from the treatment instrument insertion port 44 .
• the treatment instrument 42 passes through the insertion section 30 and protrudes to the outside of the ultrasound endoscope main body 16 from the treatment instrument opening 40 .
• the treatment instrument opening 40 also functions as a suction port for sucking blood, body waste, and the like.
• a puncture needle is shown as the treatment instrument 42.
• the treatment tool 42 may be a grasping forceps, a sheath, or the like.
• an illumination device 46 and a camera 48 are provided at the tip 32.
• the lighting device 46 emits light.
• Examples of the types of light emitted from the lighting device 46 include visible light (eg, white light, etc.), non-visible light (eg, near-infrared light, etc.), and/or special light.
• Examples of the special light include BLI light and/or LCI light.
• the camera 48 images the inside of the hollow organ using an optical method.
• An example of the camera 48 is a CMOS camera.
• the CMOS camera is just an example, and other types of cameras such as a CCD camera may be used.
• the image obtained by being captured by the camera 48 may be displayed on the display device 14, on a display device other than the display device 14 (for example, a display of a tablet terminal), or on a storage medium (for example, a flash memory). , HDD, and/or magnetic tape).
• the ultrasonic endoscope 12 includes a processing device 18 and a universal cord 50.
• the universal cord 50 has a base end 50A and a distal end 50B.
• the base end portion 50A is connected to the operating portion 28.
• the tip portion 50B is connected to the processing device 18. That is, the ultrasound endoscope main body 16 and the processing device 18 are connected via the universal cord 50.
• the endoscope system 10 includes a reception device 52.
• the reception device 52 is connected to the processing device 18.
• the reception device 52 receives instructions from the user.
• Examples of the reception device 52 include an operation panel having a plurality of hard keys and/or a touch panel, a keyboard, a mouse, a trackball, a foot switch, a smart device, and/or a microphone.
• the processing device 18 performs various signal processing according to instructions received by the reception device 52, and sends and receives various signals to and from the ultrasound endoscope main body 16 and the like. For example, the processing device 18 causes the ultrasound probe 38 to emit ultrasound in accordance with the instruction received by the receiving device 52, and based on the reflected waves received by the ultrasound probe 38, the processing device 18 causes the ultrasound image 24 (see FIG. 1) is generated and output.
• the display device 14 is also connected to the processing device 18.
• the processing device 18 controls the display device 14 according to instructions received by the receiving device 52. Thereby, for example, the ultrasound image 24 generated by the processing device 18 is displayed on the screen 26 of the display device 14 (see FIG. 1).
• the processing device 18 includes a computer 54, an input/output interface 56, a transmitting/receiving circuit 58, and a communication module 60.
• the computer 54 is an example of a "computer" according to the technology of the present disclosure.
• the computer 54 includes a processor 62, a RAM 64, and an NVM 66. Input/output interface 56, processor 62, RAM 64, and NVM 66 are connected to bus 68.
• the processor 62 controls the entire processing device 18.
• the processor 62 includes a CPU and a GPU, and the GPU operates under the control of the CPU and is mainly responsible for executing image processing.
• the processor 62 may be one or more CPUs with integrated GPU functionality, or may be one or more CPUs without integrated GPU functionality.
• the processor 62 may include a multi-core CPU or a TPU.
• the processor 62 is an example of a "processor" according to the technology of the present disclosure.
• the RAM 64 is a memory in which information is temporarily stored, and is used by the processor 62 as a work memory.
• the NVM 66 is a nonvolatile storage device that stores various programs, various parameters, and the like. Examples of the NVM 66 include flash memory (eg, EEPROM) and/or SSD. Note that the flash memory and the SSD are merely examples, and may be other non-volatile storage devices such as an HDD, or a combination of two or more types of non-volatile storage devices.
• the reception device 52 is connected to the input/output interface 56, and the processor 62 acquires instructions accepted by the reception device 52 via the input/output interface 56, and executes processing according to the acquired instructions. .
• a transmitting/receiving circuit 58 is connected to the input/output interface 56.
• the transmitting/receiving circuit 58 generates a pulse waveform ultrasound radiation signal 70 according to instructions from the processor 62 and outputs it to the ultrasound probe 38 .
• the ultrasonic probe 38 converts the ultrasonic radiation signal 70 inputted from the transmitting/receiving circuit 58 into an ultrasonic wave, and radiates the ultrasonic wave to an observation target area 72 of the subject 22 .
• the ultrasonic probe 38 receives a reflected wave obtained when the ultrasonic wave emitted from the ultrasonic probe 38 is reflected by the observation target area 72, and converts the reflected wave into a reflected wave signal 74, which is an electrical signal.
• the transmitting/receiving circuit 58 digitizes the reflected wave signal 74 input from the ultrasound probe 38 and outputs the digitized reflected wave signal 74 to the processor 62 via the input/output interface 56 .
• the processor 62 generates an ultrasound image 24 (see FIG. 1) showing the aspect of the observation target area 72 based on the reflected wave signal 74 input from the transmission/reception circuit 58 via the input/output interface 56.
• a lighting device 46 (see FIG. 2) is also connected to the input/output interface 56.
• the processor 62 controls the lighting device 46 via the input/output interface 56 to change the type of light emitted from the lighting device 46 and adjust the amount of light.
• a camera 48 (see FIG. 2) is also connected to the input/output interface 56.
• the processor 62 controls the camera 48 via the input/output interface 56 and acquires an image obtained by capturing the inside of the subject 22 by the camera 48 via the input/output interface 56 .
• a communication module 60 is connected to the input/output interface 56.
• the communication module 60 is an interface that includes a communication processor, an antenna, and the like.
• the communication module 60 is connected to a network (not shown) such as a LAN or WAN, and manages communication between the processor 62 and external devices.
• the display device 14 is connected to the input/output interface 56, and the processor 62 causes the display device 14 to display various information by controlling the display device 14 via the input/output interface 56.
• the reception device 52 is connected to the input/output interface 56, and the processor 62 acquires instructions accepted by the reception device 52 via the input/output interface 56, and executes processing according to the acquired instructions. .
• a diagnostic support program 76 and a learned model 78 are stored in the NVM 66.
• the processor 62 reads the diagnostic support program 76 from the NVM 66 and executes the read diagnostic support program 76 on the RAM 64 to perform diagnostic support processing.
• the diagnosis support process is a process that detects a lesion from the observation target area 72 using an AI method and supports diagnosis by the doctor 20 (see FIG. 1) based on the detection result.
• the processor 62 detects a lesion from the observation target area 72 by detecting a location corresponding to a lesion from the ultrasound image 24 (see FIG. 1) according to the learned model 78 by performing diagnosis support processing.
• the diagnosis support process is realized by the processor 62 operating as a generation unit 62A, a detection unit 62B, and a control unit 62C according to a diagnosis support program 76 executed on the RAM 64.
• diagnosis support program 76 is an example of a "program” according to the technology of the present disclosure.
• trained model 78 is a trained model that has a data structure used in the process of detecting a lesion from the ultrasound image 24.
• the trained model 78 is an example of "detection support information" and a "trained model” according to the technology of the present disclosure.
• a trained model 78 is generated by training an untrained model 80.
• the B-mode image group 82 is used as teacher data.
• the B-mode image group 82 consists of a plurality of mutually different B-mode images 82A.
• the B-mode image 82A is an example of a "second ultrasound image” according to the technology of the present disclosure.
• the B-mode image group 82 is an example of "teacher data" according to the technology of the present disclosure.
• An example of the model 80 is a mathematical model using a neural network.
• Examples of the type of NN include YOLO, R-CNN, and FCN.
• the NN used in the model 80 may be, for example, a YOLO, an R-CNN, or a combination of an FCN and an RNN.
• RNN is suitable for learning multiple images obtained in time series. Note that the types of NNs mentioned here are just examples, and other types of NNs that can detect objects by learning images may be used.
• a lesion is shown in the plurality of B-mode images 82A. That is, the B-mode image 82A has a lesion area 84 that corresponds to a lesion.
• An annotation 86 is added to the B-mode image 82A.
• the annotation 86 is information that can specify the position of the lesion area 84 in the B-mode image 82A (for example, information that includes a plurality of coordinates that can specify the position of a rectangular frame circumscribing the lesion area 84).
• annotation 86 information that can specify the position of the lesion area 84 in the B-mode image 82A is illustrated as an example of the annotation 86, but this is just an example.
• annotation 86 may include other types of information that specify the lesion shown in the B-mode image 82A, such as information that can identify the type of lesion shown in the B-mode image 82A. good.
• processing using the trained model 78 will be described as processing that is actively performed by the trained model 78 as the main subject. That is, for convenience of explanation, the trained model 78 will be described as having a function of processing input information and outputting a processing result. Further, in the following, for convenience of explanation, a part of the process of learning the model 80 will also be described as a process that is actively performed by the model 80 as the main subject. That is, for convenience of explanation, the model 80 will be described as having a function of processing input information and outputting a processing result.
• Each B-mode image 82A included in the B-mode image group 82 is input to the model 80.
• the model 80 predicts the position of the lesion area 84 from the input B-mode image 82A, and outputs the prediction result.
• the prediction result includes information that allows specifying the position predicted by the model 80 as the position of the lesion area 84 within the B-mode image 82A.
• An example of information that can specify the position predicted by the model 80 is, for example, the position of a bounding box surrounding the predicted position of the lesion area 84 (i.e., the position of the bounding box in the B-mode image 82A).
• An example of this is information that includes multiple coordinates that can specify the location.
• the model 80 is adjusted in accordance with the error between the annotation 86 added to the B-mode image 82A input to the model 80 and the prediction result output from the model 80. That is, the model 80 is optimized by adjusting a plurality of optimization variables (for example, a plurality of connection weights and a plurality of offset values, etc.) in the model 80 so that the error is minimized.
• a model 78 is generated. That is, the data structure of the learned model 78 is obtained by causing the model 80 to learn a plurality of mutually different B-mode images 82A to which the annotations 86 are added.
• the learned model 78 is a mathematical model generated by making the model 80 learn the B-mode image group 82. Therefore, for example, although the trained model 78 can be effectively used when detecting the lesion area 25 (see FIG. 1) from the B-mode image generated as the ultrasound image 24, When detecting the lesion area 25 from the obtained Doppler image, there is a higher possibility that false detection will occur compared to a B-mode image. This is because the model 80 has not learned the color information included in the Doppler image. Furthermore, if the lesion area 25 is detected from the Doppler image and the detection frame 27A (see FIG.
• the color information included in the Doppler image i.e., the color indicating hemodynamic Since the detected information (transformed information) and the detection frame 27A are visualized in a mixed state, the presence of the detection frame 27A may hinder diagnosis for the doctor 20.
• diagnostic support processing is performed as shown in FIGS. 5 to 9B as an example.
• An example of the diagnosis support process will be specifically described below.
• the generation unit 62A sets B mode or Doppler mode according to the image mode instruction.
• B mode is used by the doctor 20 as the main image mode
• Doppler mode is used by the doctor 20 as a sub image mode (ie, an auxiliary image mode).
• the generation unit 62A obtains the B-mode image 24A by obtaining the reflected wave signal 74 from the transmitting/receiving circuit 58 under the B-mode and generating the B-mode image 24A based on the obtained reflected wave signal 74.
• the observation target area 72 is shown in the B-mode image 24A.
• the B-mode image 24A is an image showing a two-dimensional cross section of the observation target region 72.
• a B-mode image 24A having a lesion area 25 is generated by the generation unit 62A, but of course a B-mode image 24A that does not include a lesion is generated by the generation unit 62A.
• the B-mode image 24A is an example of a "first ultrasound image” according to the technology of the present disclosure.
• the lesion area 25 is an example of a "specific area" according to the technology of the present disclosure.
• the generation unit 62A acquires the reflected wave signal 74 from the transmitting/receiving circuit 58 in Doppler mode, and generates the Doppler image 24B based on the acquired reflected wave signal 74.
• the Doppler image 24B is an image obtained by superimposing color information 24B1 expressing the characteristics in the observation target region 72 (see FIG. 3) in colors (that is, chromatic colors) on the B-mode image 24A. .
• the color information 24B1 is information indicating hemodynamics specified using the Doppler effect.
• the color information 24B1 is information referenced by the doctor 20 in order to diagnose the observation target area 72 (see FIG. 3).
• the Doppler image 24B is an example of "an image obtained in the auxiliary image mode" according to the technology of the present disclosure.
• the color information 24B1 is an example of "reference information”, “color information”, and “another image” according to the technology of the present disclosure.
• superimposing the color information 24B1 on the B-mode image 24A is an example of "synthesis” according to the technology of the present disclosure.
• the control unit 62C selects a detection mode and a non-detection mode depending on whether the ultrasound image 24 generated by the generation unit 62A is a B-mode image 24A or a Doppler image 24B. Switch.
• the control unit 62C determines whether the ultrasound image 24 generated by the generation unit 62A is a B-mode image 24A or a Doppler image 24B according to the image mode instruction received by the reception device 52. , switches between detection mode and non-detection mode according to the determination result. That is, the control unit 62C switches between the detection mode and the non-detection mode according to the image mode set according to the image mode instruction received by the reception device. For example, the control unit 62C sets the detection mode in the case of B mode, and sets the non-detection mode in the case of Doppler mode.
• the detection mode is an operation mode in which the lesion area 25 (see FIG. 5) is detected from the B-mode image 24A (see FIG. 5) using the AI method, that is, based on the trained model 78 created using the B-mode image group 82.
• This is an operation mode in which the lesion area 25 (see FIG. 5) is detected from the B-mode image 24A.
• the detection mode is used when the color information 24B1 is not superimposed on the B-mode image 24A. In other words, the detection mode is used when the Doppler image 24B is not generated by the generation unit 62A (that is, when the B-mode image 24A is generated by the generation unit 62A).
• the non-detection mode is an operation mode in which the lesion area 25 is not detected from the ultrasound image 24.
• the non-detection mode an operation mode in which the lesion area 25 is not detected from the ultrasound image 24 is illustrated, but the technology of the present disclosure is not limited to this.
• the non-detection mode is an operation mode in which the lesion area 25 is detected from the ultrasound image 24 and the detection result is not output (in other words, the lesion area 25 is detected by the AI method in the background, but the detection result is visualized). It may also be an operating mode in which it is not allowed.
• the non-detection mode is used when the color information 24B1 is superimposed on the B-mode image 24A. In other words, the non-detection mode is used when the Doppler image 24B is generated by the generation unit 62A.
• the detection unit 62B detects a lesion from the B-mode image 24A generated by the generation unit 62A according to the learned model 78. That is, the detection unit 62B determines the presence or absence of the lesion area 25 in the B-mode image 24A according to the learned model 78, and specifies the position of the lesion area 25 when the lesion area 25 is present in the B-mode image 24A.
• the location identifying information 128 (for example, information including a plurality of coordinates that identifies the location of the lesion area 126) is generated.
• the process by which the detection unit 62B detects a lesion will be explained using the learned model 78 as the main subject.
• the learned model The presence or absence of the lesion area 25 in the B-mode image 24A is determined.
• the trained model 78 determines that the lesion area 25 exists in the B-mode image 24A (that is, when the lesion shown in the B-mode image 24A is detected), it outputs the position specifying information 27.
• the detection frame 27A is a rectangular frame corresponding to a bounding box (for example, a bounding box with the highest reliability score) used when the trained model 78 detects the lesion area 25 from the B-mode image 24A. That is, the detection frame 27A is a frame surrounding the lesion area 25 detected by the learned model 78.
• the detection unit 62B detects a B-mode image 24A corresponding to the location information 27 output from the trained model 78 (i.e., a B-mode image 24A that is input to the learned model 78 for outputting the location information 27) according to the location information 27.
• a detection frame 27A is added to the B-mode image 24A). That is, the detection unit 62B superimposes the detection frame 27A on the B-mode image 24A corresponding to the position specifying information 27 output from the learned model 78 so as to surround the lesion area 25, thereby converting the B-mode image 27 into a B-mode image 27.
• a detection frame 27A is provided.
• the detection unit 62B When the learned model 78 determines that the lesion area 25 is present in the B-mode image 27, the detection unit 62B outputs the B-mode image 24A to which the detection frame 27A is attached to the control unit 62C. Further, when the learned model 78 determines that the lesion area 25 does not exist in the B-mode image 24A, the detection unit 62B outputs the B-mode image 24A to which the detection frame 27A is not attached to the control unit 62C.
• the control unit 62C displays the B-mode image 24A input from the detection unit 62B (that is, the B-mode image 24A reflecting the detection result of the detection unit 62B) on the screen 26 of the display device 14.
• a B-mode image 24A with a detection frame 27A surrounding the lesion area 25 that is, a B-mode image 24A with the detection frame 27A superimposed
• the B-mode image 24A to which the detection frame 27A is not attached that is, the B-mode image 24A output from the learned model 78
• Ru is displayed on the screen 26.
• the control unit 62C acquires the Doppler image 24B generated by the generation unit 62A, and displays the acquired Doppler image 24B on the screen 26 of the display device 14.
• FIGS. 9A and 9B show a processor of the processing device 18 on the condition that diagnosis using the endoscope system 10 has started (for example, that the ultrasound endoscope 12 has started emitting ultrasonic waves).
• An example of the flow of the diagnostic support process performed by 62 is shown.
• the flow of the diagnostic support process shown in FIGS. 9A and 9B is an example of the "diagnosis support method" according to the technology of the present disclosure.
• step ST10 the control unit 62C determines whether the currently set image mode is B mode. In step ST10, if the currently set image mode is not B mode (that is, Doppler mode), the determination is negative and the diagnosis support process moves to step ST26 shown in FIG. 9B. In step ST10, if the currently set image mode is B mode, the determination is affirmative and the diagnosis support process moves to step ST12.
• B mode that is, Doppler mode
• step ST12 the control unit 62C determines whether the currently set operation mode is the non-detection mode. In step ST12, if the currently set operating mode is not the non-detection mode (that is, the detection mode), the determination is negative and the diagnostic support process moves to step ST16. In step ST12, if the currently set operating mode is the non-detection mode, the determination is affirmative and the diagnostic support process moves to step ST14.
• step ST14 the control unit 62C switches the operation mode from the non-detection mode to the detection mode. After the process of step ST14 is executed, the diagnosis support process moves to step ST16.
• step ST16 the generation unit 62A generates the B-mode image 24A based on the reflected wave signal 74 (see FIG. 5) input from the transmission/reception circuit 58.
• the diagnosis support process moves to step ST18.
• step ST18 the detection unit 62B inputs the B-mode image 24A generated in step ST16 to the learned model 78. After the process of step ST18 is executed, the diagnosis support process moves to step ST20.
• step ST20 the detection unit 62B uses the learned model 78 to determine whether a lesion is shown in the B-mode image 24A input to the learned model 78 in step ST18. If a lesion is shown in the B-mode image 24A, the learned model 78 outputs the position specifying information 27.
• step ST20 if no lesion is shown in the B-mode image 24A, the determination is negative and the diagnosis support process moves to step ST24. In step ST20, if a lesion is shown in the B-mode image 24A, the determination is affirmative and the diagnosis support process moves to step ST22.
• step ST20 If the determination in step ST20 is affirmative, the detection unit 62B generates the detection frame 27A based on the position specifying information 128 output from the trained model 78, and , the detection frame 27A is superimposed so as to surround the lesion area 25. Then, in step ST22, the control unit 62C displays the B-mode image 24A in which the lesion area 25 is surrounded by the detection frame 27A on the screen 26 of the display device 14. Since the lesion area 25 in the B-mode image 24A is surrounded by the detection frame 27A, the doctor 20 can visually grasp in which position the lesion appears in the B-mode image 24A. Become. After the process of step ST22 is executed, the diagnosis support process moves to step ST26 shown in FIG. 9B.
• step ST24 the control unit 62C displays the radial ultrasound image 24 generated in step ST16 on the screen 26 of the display device 14. In this case, since the detection frame 27A is not added to the B-mode image 24A, the doctor 20 can visually recognize that no lesion is shown in the B-mode image 24A.
• the diagnosis support process moves to step ST26 shown in FIG. 9B.
• step ST26 shown in FIG. 9B the control unit 62C determines whether the currently set operation mode is the detection mode. In step ST26, if the currently set operating mode is not the detection mode (that is, the non-detection mode), the determination is negative and the diagnostic support process moves to step ST30. In step ST26, if the currently set operating mode is the detection mode, the determination is affirmative and the diagnosis support process moves to step ST28.
• step ST28 the control unit 62C switches the operation mode from the non-detection mode to the detection mode. After the process of step ST28 is executed, the diagnosis support process moves to step ST30.
• step ST30 the generation unit 62A generates the Doppler image 24B based on the reflected wave signal 74 (see FIG. 5) input from the transmission/reception circuit 58.
• the diagnosis support process moves to step ST32.
• step ST32 the control unit 62C displays the Doppler image 24B generated in step ST30 on the screen 26 of the display device 14. Thereby, the doctor 20 can visually recognize hemodynamics from the color information 24B1 (see FIG. 8) included in the Doppler image 24B displayed on the screen 26. Further, since the Doppler image 24B does not include the detection frame 27A, the detection frame 27A does not interfere with observation of the color information 24B1. After the process of step ST32 is executed, the diagnosis support process moves to step ST34.
• step ST34 the control unit 62C determines whether conditions for terminating the diagnostic support process (hereinafter referred to as "diagnostic support terminating conditions") are satisfied.
• An example of the diagnostic support termination condition is that the receiving device 52 has accepted an instruction to terminate the diagnostic support process.
• step ST34 if the diagnostic support end condition is not satisfied, the determination is negative and the diagnostic support process moves to step ST10 shown in FIG. 9A.
• step ST34 if the diagnostic support end condition is satisfied, the determination is affirmative and the diagnostic support process ends.
• the detection mode is an operation mode in which the lesion area 25 is detected from the B-mode image 24A using an AI method using the trained model 78.
• the non-detection mode is an operation mode in which the lesion area 25 is not detected from the ultrasound image 24.
• the trained model 78 is created using the B-mode image group 82 as training data. Therefore, when the process of detecting the lesion area 25 from the Doppler image 24B is performed according to the learned model 78, the lesion area 25 is There is a high possibility that areas other than the above will be erroneously detected as the lesion area 25.
• the detection mode and non-detection mode are switched depending on whether or not the B-mode image 24A is generated. In other words, the detection mode and non-detection mode are switched depending on whether B mode or Doppler mode is set as the image mode.
• the detection mode is an operation mode used when the color information 24B1 is not superimposed on the B-mode image 24A
• the non-detection mode is an operation mode used when the color information 24B1 is superimposed on the B-mode image 24A.
• This is the operating mode used when the In other words, detection mode is the operating mode used when the imaging mode is not Doppler mode (i.e. B mode), and non-detection mode is the operating mode used when the imaging mode is Doppler mode. be. Therefore, by selectively using the detection mode and non-detection mode in the manner described above, it is possible to prevent areas other than the lesion area 25 from being erroneously detected as the lesion area 25 due to the presence of the color information 24B1.
• the color information 24B1 and the detection frame 27A are not displayed in a mixed state on the screen 26, so the visualization of the detection result of the lesion area 25 (that is, the presence of the detection frame 27A) is Obstruction of observation of the color information 24B1 included in the Doppler image 24B can be suppressed.
• the diagnostic support process shown in FIG. 10 is performed by the processor 62.
• the flowchart shown in FIG. 10 differs from the flowchart shown in FIG. 9A in that the process in step ST100 is applied instead of the process in step ST10.
• step ST100 shown in FIG. 10 the control unit 62C performs image analysis processing on the image generated by the generation unit 62A, so that the color information 24B1 is superimposed on the image generated by the generation unit 62A. Determine whether or not there is.
• step ST100 if the color information 24B1 is not superimposed on the image generated by the generation unit 62A, the determination is affirmative and the diagnosis support process moves to step ST12.
• step ST100 if the color information 24B1 is superimposed on the image generated by the generation unit 62A, the determination is negative and the process moves to step ST26 shown in FIG. 9B.
• the detection mode is set as the operation mode, and the color information 24B1 is superimposed on the image generated by the generation unit 62A. If so, non-detection mode is set as the operation mode. Therefore, effects similar to those of the above embodiment can be obtained.
• the detection mode is set as the operation mode for the B-mode image 24A, and the lesion area 25 is detected using the AI method for the B-mode image 24A. Even if it is the B-mode image 24A, the non-detection mode may be set as the operation mode depending on the conditions.
• the control unit 62C sets the non-detection mode as the operation mode.
• the character information 88, the measurement line 90, and the treatment auxiliary information 92 are examples of "reference information” according to the technology of the present disclosure.
• the character information 88 is an example of "character information” according to the technology of the present disclosure.
• the measurement line 90 is an example of a “measurement line” according to the technology of the present disclosure.
• the treatment assistance information 92 is an example of "treatment assistance information" according to the technology of the present disclosure.
• the text information 88 refers to, for example, a text image (that is, an image expressing text) that assists in observing the observation target area 72 (see FIG. 3).
• Assisting the observation of the observation target region 72 means, for example, that it serves as a reference for the doctor 20 when identifying the presence or absence of a lesion and the position of the lesion from the B-mode image 24A.
• the concept of "characters" according to this embodiment also includes numbers, symbols, and the like. Examples of the characters according to this embodiment include characters defined by Unicode.
• information that visualizes the dimensions of the lesion area 25 is shown as an example of the text information 88.
• the measurement line 90 is, for example, a line used for measurement within the observation target area 72 (see FIG. 3).
• a dimension line that allows the length of the lesion area 25 in one direction to be visually specified is shown as an example of the measurement line 90. Note that although a dimension line is illustrated here, this is just an example, and visualization information other than dimensionality (for example, a line used to measure a specified part other than the lesion area 25) It may be.
• the treatment assistance information 92 is information that assists treatment using the fine needle aspiration method.
• a puncture needle see FIG. 2
• an arrow is shown that indicates the direction in which the puncture needle is inserted. Note that although an arrow is illustrated here, this is just an example, and visualization information other than arrows (for example, a broken line or a blinking line) may be used.
• the control unit 62C performs image analysis processing on the image generated by the generation unit 62A, so that the text information 88, the measurement line 90, and/or the treatment auxiliary information 92 are superimposed on the image generated by the generation unit 62A. Determine whether or not. In the example shown in FIG. 11, the control unit 62C determines whether the text information 88, the measurement line 90, and/or the treatment auxiliary information 92 are superimposed on the B-mode image 24A generated by the generation unit 62A.
• the control unit 62C sets the detection mode as the operation mode. Further, when the text information 88, the measurement line 90, and/or the treatment auxiliary information 92 are superimposed on the B-mode image 24A generated by the generation unit 62A, the control unit 62C sets the non-detection mode as the operation mode. . In the example shown in FIG. 11, the text information 88, the measurement line 90, and the treatment assistance information 92 are superimposed on the B-mode image 24A generated by the generation unit 62A, so the non-detection mode is set as the operation mode. In this case, the control unit 62C displays the B-mode image 24A with the text information 88, the measurement line 90, and the treatment assistance information 92 superimposed on the screen 26 of the display device 14.
• FIG. 12 shows an example of the flow of diagnostic support processing according to the second modification.
• the flowchart shown in FIG. 12 differs from the flowchart shown in FIG. 9A in that it includes the process of step ST200 between the process of step ST16 and the process of step ST18, and the process of steps ST202 to ST206 as the process subsequent to step ST24. The difference is that it has
• step ST200 the control unit 62C determines whether character information 88, measurement line 90, and/or treatment auxiliary information 92 are superimposed on the B-mode image 24A generated in step ST16. Determine whether In step ST200, if the character information 88, the measurement line 90, and/or the treatment auxiliary information 92 are not superimposed on the B-mode image 24A generated in step ST16, the determination is negative and the diagnosis support process is continued in step ST18. Move to. In step ST200, if the character information 88, the measurement line 90, and/or the treatment auxiliary information 92 are superimposed on the B-mode image 24A generated in step ST16, the determination is affirmative, and the diagnosis support process is performed in step ST202. Move to.
• step ST202 the control unit 62C determines whether the currently set operation mode is the detection mode. In step ST202, if the currently set operating mode is not the detection mode, the determination is negative and the diagnosis support process moves to step ST206. In step ST202, if the currently set operation mode is the detection mode, the determination is affirmative and the diagnosis support process moves to step ST204.
• step ST204 the control unit 62C switches the operation mode from the detection mode to the non-detection mode. After the process of step ST204 is executed, the diagnosis support process moves to step ST206.
• step ST206 the control unit 62C displays the B-mode image 24A generated in step ST16 (that is, the B-mode image 24A to which the position specifying information 27 is not added) on the screen 26 of the display device 14.
• the diagnosis support process moves to step ST26 shown in FIG. 9B.
• the lesion area 25 is there is a high possibility that the area will be erroneously detected as the lesion area 25. Furthermore, suppose that a process is performed to detect the lesion area 25 from the B-mode image 24A on which the text information 88, the measurement line 90, and/or the treatment auxiliary information 92 are superimposed, according to the learned model 78, and the detection result is a detection frame.
• the presence of the detection frame 27A may interfere with the observation of the text information 88, the measurement line 90, and/or the treatment auxiliary information 92 for the doctor 20.
• Obstructing the observation of the text information 88, the measurement line 90, and/or the treatment auxiliary information 92 means that it obstructs the diagnosis for the doctor 20.
• the detection mode and non-detection mode are switched depending on whether the character information 88, the measurement line 90, and/or the treatment assistance information 92 are superimposed on the B-mode image 24A. That is, when the text information 88, the measurement line 90, and/or the treatment auxiliary information 92 are superimposed on the B-mode image 24A, the non-detection mode is set as the operation mode, and the text information 88, the measurement line 90, and the treatment When none of the auxiliary information 92 is superimposed on the B-mode image 24A, the detection mode is set as the operation mode.
• the detection mode and non-detection mode are switched depending on the presence or absence of the color information 24B1, but the technology of the present disclosure is not limited to this.
• the detection mode is determined according to the number of specific chromatic pixels superimposed on the ultrasound image 24 generated by the generation unit 62A (for example, the number of chromatic pixels included in the color information 24B1). It may be possible to switch between the mode and the non-detection mode.
• the control unit 62C determines the number of chromatic pixels whose saturation is equal to or higher than the first threshold value (hereinafter referred to as “chromatic color The control unit 62C determines whether the number of chromatic pixels is equal to or greater than a second threshold. The control unit 62C determines whether the number of chromatic pixels is less than the second threshold or not.
• the detection mode is set as the operation mode, and when the number of chromatic pixels is equal to or greater than the second threshold, the non-detection mode is set as the operation mode.
• the non-detection mode may be set as the operation mode.
• the first image mode is included in reflected waves (hereinafter simply referred to as "reflected waves") obtained by emitting ultrasonic waves to the observation target area 72 and being reflected by the observation target area 72.
• reflected waves This is an image mode in which an ultrasound image 24 is generated using high frequency components.
• the second image mode is an image mode in which the B-mode image 24A and another image are combined.
• the generation unit 62A selectively sets B mode, Doppler mode, elastography mode, THI mode, CH mode, and CHI mode according to the image mode instruction received by the reception device 52. do.
• the generation unit 62A then generates an ultrasound image 24 according to the set image mode.
• the generation unit 62A generates the Doppler image 24B under the Doppler mode, generates the elastography image 24C under the elastography mode, generates the THI image 24D under the THI mode, and under the CH mode, A CH image 24E is generated, and a CHI image 24F is generated under CHI mode.
• the control unit 62C displays the ultrasound image 24 generated according to the set image mode on the screen 26 of the display device 14. That is, under the Doppler mode, the Doppler image 24B is displayed on the screen 26, under the elastography mode, the elastography image 24C is displayed on the screen 26, and under the THI mode, the THI image 24D is displayed on the screen 26, and the CH Under CH mode, CH image 24E is displayed on screen 26, and under CHI mode, CHI image 24F is displayed on screen 26. Note that the elastography image 24C, the THI image 24D, the CH image 24E, and the CHI image 24F are known ultrasound images, so a description thereof will be omitted here.
• the non-detection mode is set as the operation mode, so the reflected wave It is possible to prevent an area other than the lesion area 25 from being erroneously detected as the lesion area 25 due to the presence of an image area corresponding to a high frequency component included in the lesion area 25 . Furthermore, when observing the ultrasound image 24 (here, as an example, the THI image 24D, CH image 24E, or CHI image 24F), the existence of an image area corresponding to a high frequency component included in the reflected wave is detected in the ultrasound image. It is possible to prevent the diagnosis from being hindered by affecting the visibility of the 24.
• a plurality of parameters 94 are stored in the NVM 66 as setting values that determine the image quality of the ultrasound image 24, and the control unit 62C acquires the plurality of parameters 94 from the NVM 66, The detection mode and non-detection mode are switched according to the plurality of acquired parameters 94.
• the NVM 66 stores a frequency parameter 94A, a depth parameter 94B, a brightness parameter 94C, a dynamic range parameter 94D, and a magnification parameter 94E as a plurality of parameters 94.
• the control unit 62C determines that the frequency parameter 94A is within a first range, the depth parameter 94B is within a second range, the brightness parameter 94C is within a third range, and the dynamic range parameter 94D is within a fourth range. , and when the condition that the magnification parameter 94E is within the fifth range (hereinafter referred to as "parameter condition") is satisfied, the detection mode is set as the operation mode. Further, the control unit 62C sets the non-detection mode as the operation mode when the parameter conditions are not satisfied.
• the first to fifth ranges are ranges specified in advance. The first to fifth ranges are examples of "designated ranges" according to the technology of the present disclosure.
• the first range is the ideal range of the frequency of the ultrasound waves emitted from the ultrasound probe 38.
• An example of an ideal range of ultrasonic frequencies is a frequency range in which areas other than the lesion area 25 are not erroneously detected as the lesion area 25 (for example, a frequency range for all B-mode images 82A included in the B-mode image group 82). frequency range (applicable frequency range).
• the second range is the ideal range of depth expressed in the ultrasound image 24.
• An example of an ideal depth range is a depth range in which areas other than the lesion area 25 are not erroneously detected as the lesion area 25 (for example, a depth range that is applied to all B-mode images 82A included in the B-mode image group 82). depth range).
• the third range is the ideal range of brightness of the ultrasound image 24.
• An example of an ideal brightness range is a brightness range in which areas other than the lesion area 25 are not erroneously detected as the lesion area 25 (for example, a brightness range for all B-mode images 82A included in the B-mode image group 82). range of applied brightness).
• the fifth range is an ideal range of digital zoom magnification for the ultrasound image 24.
• An example of an ideal digital zoom magnification range is a digital zoom magnification range in which areas other than the lesion area 25 are not erroneously detected as the lesion area 25 (for example, when all B-mode images 82A included in the B-mode image group 82 The magnification range of digital zoom applied to
• the detection mode and non-detection mode are switched according to the plurality of parameters 94. That is, the detection mode is set when the parameter conditions are satisfied, and the non-detection mode is set when the parameter conditions are not satisfied. Therefore, it is possible to suppress areas other than the lesion area 25 from being erroneously detected as the lesion area 25 due to the plurality of parameters 94 not being within the ideal range, and to prevent the plurality of parameters 94 from being within the ideal range. It is possible to prevent the visibility of the ultrasonic image 24 from being affected by the fact that the ultrasound image 24 is not displayed to be the same as that of the ultrasound image 24 to hinder diagnosis.
• the detection mode is set when the frequency parameter 94A is within the first range, and the non-detection mode is set when the frequency parameter 94A is not within the first range. Therefore, it is possible to prevent areas other than the lesion area 25 from being erroneously detected as the lesion area 25 due to the frequency parameter 94A not being within the first range, and to prevent the frequency parameter 94A from being within the first range. It is possible to prevent diagnosis from being hindered by affecting the visibility of the ultrasound image 24.
• the detection mode is set when the depth parameter 94B is within the second range, and the non-detection mode is set when the depth parameter 94B is not within the second range. Therefore, it is possible to prevent an area other than the lesion area 25 from being erroneously detected as the lesion area 25 due to the depth parameter 94B not being within the second range, and to prevent the depth parameter 94B from being within the second range. It is possible to prevent diagnosis from being hindered by affecting the visibility of the ultrasound image 24.
• the detection mode is set when the brightness parameter 94C is within the third range, and the non-detection mode is set when the brightness parameter 94C is not within the third range. . Therefore, an area other than the lesion area 25 is prevented from being erroneously detected as the lesion area 25 due to the brightness parameter 94C not being within the third range, and the brightness parameter 94C is not within the third range. It is possible to prevent this from affecting the visibility of the ultrasound image 24 and hindering diagnosis.
• the detection mode is set when the dynamic range parameter 94D is within the fourth range, and the non-detection mode is set when the dynamic range parameter 94D is not within the fourth range. . Therefore, an area other than the lesion area 25 is prevented from being erroneously detected as the lesion area 25 due to the dynamic range parameter 94D not being within the fourth range, and the dynamic range parameter 94D is not within the fourth range. It is possible to prevent this from affecting the visibility of the ultrasound image 24 and hindering diagnosis.
• the detection mode is set when the magnification parameter 94E is within the fifth range, and the non-detection mode is set when the magnification parameter 94E is not within the fifth range. Therefore, it is possible to prevent areas other than the lesion area 25 from being erroneously detected as the lesion area 25 due to the magnification parameter 94E not being within the fifth range, and to prevent the magnification parameter 94E from being within the fifth range. It is possible to prevent diagnosis from being hindered by affecting the visibility of the ultrasound image 24.
• a plurality of parameters 94 are stored in the NVM 66, and the detection mode and non-detection mode are switched by the control unit 62C according to the plurality of parameters 94 in the NVM 66. Therefore, the operation mode can be set according to the plurality of parameters 94 without manually inputting the plurality of parameters 94 required for determining switching between the detection mode and the non-detection mode every time the ultrasound image 24 is generated.
• the technology of the present disclosure is not limited to this, and at least one of the plurality of parameters 94 is within a prespecified range (i.e., the above-mentioned ideal range). )
• the detection mode and the non-detection mode may be switched depending on whether the detection mode is within ).
• a plurality of parameters 94 stored in the NVM 66 are acquired by the control unit 62C, and the control unit 62C switches between the detection mode and the non-detection mode according to the plurality of parameters 94.
• the technology of the present disclosure is applicable even when the plurality of parameters 94 are not stored in the NVM 66.
• the generation unit 62A generates a frame 96 under B mode.
• Frame 96 includes B-mode image 24A, with text image 98 superimposed thereon.
• the text image 98 is an image obtained by converting the plurality of parameters 94 into characters.
• the control unit 62C displays a frame 96 on the screen 26 of the display device 14.
• the B-mode image 24A and the text image 98 are displayed on the screen 26, so that the doctor 20 can observe the B-mode image 24A and grasp the plurality of parameters 94 indicated by the text image 98. Can be done.
• the control unit 62C identifies an area 100 in which the text image 98 exists from the frame 96, and extracts the identified area 100 from the frame 96. Then, the control unit 62C performs image recognition processing (for example, AI-based image recognition processing or non-AI-based image recognition processing using template matching, etc.) on the region 100 extracted from the frame 96. A plurality of parameters 94 are recognized and acquired from a text image 98 included in the text image 100. Then, the control unit 62C switches between the detection mode and the non-detection mode according to the plurality of parameters 94 in the same manner as in the fifth modification. As a result, the same effects as the fifth modification described above can be obtained.
• image recognition processing for example, AI-based image recognition processing or non-AI-based image recognition processing using template matching, etc.
• the processor 62 changes the detection frequency, The detection accuracy and/or the detection target may be different.
• the detection frequency may be different depending on the image mode. Make it different.
• the detection frequency in B mode is increased, and the detection frequency in image modes other than B mode (hereinafter referred to as "other image modes") is increased.
• the detection frequency in B mode is set to every frame, and the detection frequency in other image modes is set to every multiple frames (for example, every two frames).
• a first example of a method for varying the detection accuracy is the amount of training performed on the model 80 to obtain the trained model 78 used in B mode and the amount of training performed on the model 80 to obtain the trained model used in other image modes.
• One method is to vary the amount of learning performed for each student.
• a second example of a method of varying the detection accuracy is a method of varying the number of intermediate layers of the trained model 78 used in the B mode and the number of intermediate layers of the trained model used in other image modes. .
• the detection target is made different depending on the image mode
• the detection target is made different depending on when B mode is used as the main image mode and when another image mode is used as the main image mode.
• the trained model 78 used in the B mode is generated by training the model 80 to detect the first type of lesion, and the obtained trained model is used.
• One method is to generate a trained model used in the image mode by training the model 80 to detect a second type of lesion (that is, a type of lesion different from the first type).
• FIG. 18 shows an example of the flow of diagnostic support processing according to the seventh modification.
• the flowchart shown in FIG. 18 differs from the flowchart shown in FIG. 9A in that the processing in steps ST300 to ST304 is included between the processing in step ST16 and the processing in step ST18.
• step ST300 the control unit 62C determines whether the detection frequency, detection accuracy, and detection target for B mode have been set. In step ST300, if the detection frequency, detection accuracy, and detection target for B mode are set, the determination is affirmative and the diagnosis support process moves to step ST304. In step ST300, if the detection frequency, detection accuracy, and detection target for B mode are not set, the determination is negative and the diagnosis support process moves to step ST302.
• step ST302 the control unit 62C sets the detection frequency, detection accuracy, and detection target for B mode. After the process of step ST302 is executed, the diagnosis support process moves to step ST304.
• step ST304 the control unit 62C determines whether the timing to detect the lesion area 25 has arrived.
• the timing of detecting the lesion area 25 is, for example, a timing separated by a time interval defined by the reciprocal of the frame rate.
• step ST304 if the timing to detect the lesion area 25 has not arrived, the determination is negative and the diagnosis support process moves to step ST24.
• step ST304 when the timing to detect the lesion area 25 has arrived, the determination is affirmative and the diagnosis support process moves to step ST18.
• the detection frequency is varied depending on the image mode, so that it is possible to prevent the number of times the lesion area 25 is detected from being excessive or insufficient depending on the image mode. Can be done. Moreover, according to the seventh modification, since the detection accuracy is varied depending on the image mode, it is possible to suppress a decrease in the accuracy of detecting the lesion area 25 depending on the image mode. Furthermore, according to the seventh modification, since the detection target is changed depending on the image mode, it is possible to suppress detection of an unintended portion as the lesion area 25.
• the seventh modification example has been described using an example in which the detection frequency, detection accuracy, and/or detection target is varied depending on the image mode, the technology of the present disclosure is not limited to this.
• reference information for example, text information 88, measurement line 90, and/or treatment auxiliary information 92, etc.
• setting values that define the image quality of the ultrasound image 24 for example, , a plurality of parameters 94
• the detection frequency, detection accuracy, and/or detection target may be varied.
• the detection mode may be set as the operating mode in the case of the A mode or the M mode, and the detection mode may be set as the operating mode in the case of the image mode determined in advance as a single main image mode.
• the detection mode may be set as the mode.
• the main image mode may be any image mode that corresponds to the type of ultrasound image 24 that the model 80 is to learn.
• the main image mode will be A-mode
• the ultrasound image 24 that the model 80 is to learn is an M-mode
• the main image mode will be M-mode. becomes.
• the technology of the present disclosure is not limited to this, and together with the lesion area 25, or in place of the lesion area 25, other than the lesion area 25 is detected.
• a specific region for example, a specific organ may be detected.
• the technology of the present disclosure is not limited to this.
• the lesion area 25 may be detected using a non-AI method.
• non-AI detection methods include a detection method using template matching.
• the template used for template matching is an example of "detection support information" according to the technology of the present disclosure.
• the ultrasound endoscope 12 is used as an example, but the technology of the present disclosure is also applicable to an external ultrasound diagnostic device.
• the ultrasonic image 24 generated by the processing device 18 and the detection frame 27A are displayed on the screen 26 of the display device 14, but the ultrasonic image 24 to which the detection frame 27A is attached is
• the information may be transmitted to various devices such as a server, a PC, and/or a tablet terminal, and stored in the memory of the various devices. Further, the ultrasound image 24 to which the detection frame 27A is added may be recorded in the report. Further, the position identification information 27 may also be stored in the memory of various devices, or may be recorded in a report. It is preferable that the ultrasound image 24, detection frame 27A, and/or position identification information 27 be stored in a memory or recorded in a report for each subject 22.
• the diagnostic support process may be performed by the processing device 18 and at least one device provided outside the processing device 18, or may be performed by at least one device provided outside the processing device 18 (for example, a The processing may be performed only by an auxiliary processing device connected to the processing device 18 and used to expand the functions of the processing device 18.
• An example of at least one device provided outside the processing device 18 is a server.
• the server may be realized by cloud computing.
• Cloud computing is just one example, and may be network computing such as fog computing, edge computing, or grid computing.
• the server mentioned as at least one device provided outside the processing device 18 is merely an example, and instead of the server, at least one PC and/or at least one mainframe, etc. may be used. Alternatively, it may be at least one server, at least one PC, and/or at least one mainframe.
• the doctor 20 is made to perceive the presence or absence of a lesion and the position of the lesion, but the doctor 20 may be made to perceive the type of lesion and/or the degree of progression of the lesion.
• the model 80 is created using an ultrasound image (B-mode image 82A in the example shown in FIG. 4) as training data with the annotation 86 including information that can identify the type of lesion and/or the degree of progression of the lesion. Just use it for learning.
• processors can be used as hardware resources for executing the diagnostic support processing described in the above embodiments.
• the processor include a processor that is a general-purpose processor that functions as a hardware resource that executes diagnostic support processing by executing software, that is, a program.
• the processor include a dedicated electronic circuit such as an FPGA, a PLD, or an ASIC, which is a processor having a circuit configuration specifically designed to execute a specific process.
• Each processor has a built-in memory or is connected to it, and each processor uses the memory to execute diagnostic support processing.

Landscapes

• Health & Medical Sciences (AREA)
• Life Sciences & Earth Sciences (AREA)
• Engineering & Computer Science (AREA)
• Medical Informatics (AREA)
• Surgery (AREA)
• Pathology (AREA)
• Radiology & Medical Imaging (AREA)
• Biophysics (AREA)
• Biomedical Technology (AREA)
• Heart & Thoracic Surgery (AREA)
• Physics & Mathematics (AREA)
• Molecular Biology (AREA)
• Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
• Animal Behavior & Ethology (AREA)
• General Health & Medical Sciences (AREA)
• Public Health (AREA)
• Veterinary Medicine (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Physiology (AREA)
• Ultra Sonic Daignosis Equipment (AREA)

Priority Applications (2)

Applications Claiming Priority (2)

Related Child Applications (1)

Publications (1)

Family

ID=89382745

Family Applications (1)

Country Status (3)

Cited By (1)

Citations (5)

Family Cites Families (7)

• 2023
  • 2023-06-02 WO PCT/JP2023/020699 patent/WO2024004524A1/ja not_active Ceased
  • 2023-06-02 JP JP2024530600A patent/JPWO2024004524A1/ja active Pending
• 2024
  • 2024-12-02 US US18/964,697 patent/US20250090145A1/en active Pending

Patent Citations (5)

Also Published As

Similar Documents

Legal Events