WO2021044910A1 - 医療画像処理装置、内視鏡システム、医療画像処理方法及びプログラム - Google Patents

医療画像処理装置、内視鏡システム、医療画像処理方法及びプログラム Download PDF

Info

Publication number
WO2021044910A1
WO2021044910A1 PCT/JP2020/032049 JP2020032049W WO2021044910A1 WO 2021044910 A1 WO2021044910 A1 WO 2021044910A1 JP 2020032049 W JP2020032049 W JP 2020032049W WO 2021044910 A1 WO2021044910 A1 WO 2021044910A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
unobserved
image processing
unit
interest
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/032049
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
美沙紀 目黒
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fujifilm Corp
Original Assignee
Fujifilm Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fujifilm Corp filed Critical Fujifilm Corp
Priority to EP20860022.1A priority Critical patent/EP4026479A4/en
Priority to JP2021543711A priority patent/JP7256275B2/ja
Priority to CN202080058661.3A priority patent/CN114269221B/zh
Publication of WO2021044910A1 publication Critical patent/WO2021044910A1/ja
Priority to US17/591,976 priority patent/US12274416B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
    • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
    • A61B1/000094—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope extracting biological structures
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
    • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
    • A61B1/000096—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope using artificial intelligence
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/0002—Operational features of endoscopes provided with data storages
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00043—Operational features of endoscopes provided with output arrangements
    • A61B1/00045—Display arrangement
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00043—Operational features of endoscopes provided with output arrangements
    • A61B1/00055—Operational features of endoscopes provided with output arrangements for alerting the user
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10068—Endoscopic image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/03—Recognition of patterns in medical or anatomical images
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
    • G10L2015/223—Execution procedure of a spoken command

Definitions

  • the present invention relates to a medical image processing apparatus, an endoscopic system, a medical image processing method and a program.
  • a system that introduces AI technology for endoscopic images and supports lesion detection and diagnosis is attracting attention as it reduces the oversight rate of lesions and leads to the reduction of unnecessary biopsies.
  • Such a system operates in real time during the endoscopy, and realizes notification to the doctor such as displaying the recognition result on the screen.
  • AI is an abbreviation for Artificial Intelligence.
  • Patent Documents 1 to 3 describe an endoscopic system that detects a lesion candidate region of an observation image based on a feature amount calculated from an observation image and displays the detected lesion candidate region on a display device. ..
  • a marker image that emphasizes the position of the lesion candidate region is added to the observation image.
  • the system changes the display mode of the marker image according to the length of the detection duration of the lesion candidate region. Specifically, the emphasis process is started when the detection duration of the lesion candidate region reaches the specified period.
  • the endoscope system described in Patent Document 2 stores observation images sequentially output from a video processor during a continuous detection period. If the continuous detection period is less than the specified period, the system stores the observed image in a display area other than the display area for displaying the observed image in the reverse order of the storage order after the specified period elapses. Display the observed image.
  • the endoscopic system described in Patent Document 3 temporarily stores a still image in a memory when a lesion candidate region is detected and notification processing is started.
  • the system adds a marker image to the still image temporarily stored in the memory when the operation switch is turned on, and displays the still image on the display unit.
  • Patent Document 4 describes an electronic endoscopy device that is inserted into a subject and observes the inside of the subject.
  • the apparatus described in the same document automatically temporarily stores a still image in a memory when an abnormal pixel is detected.
  • the device displays thumbnail images of still images temporarily stored in the memory in the thumbnail display area.
  • Patent Documents 1 to 4 do not describe or suggest the above-mentioned information on the overlooked region of interest.
  • Patent Document 1 reduces the burden on the doctor who visually confirms the lesion candidate area by performing emphasis processing on the lesion area or the like when the detection period of the lesion area or the like reaches a specified period. There is. On the other hand, the system described in the same document does not have a component for storing information on a missed lesion candidate region.
  • the system described in Patent Document 2 stores one or more observation images during the period from the start of detection of the region of interest to the interruption, and automatically stores the observation images stored after the detection of the region of interest is interrupted. Is displayed to reduce oversight of lesions. On the other hand, the sequential and automatic redisplay of the observed images may lead to a decrease in the concentration of the doctor and may lead to an oversight of the area of interest.
  • the system described in the same document does not include a component for storing information on a missed lesion candidate region.
  • Patent Document 3 temporarily stores a still image including a lesion candidate region when a lesion candidate region is detected, and displays the still image when the operation switch is turned on.
  • the still image in which the lesion candidate area is detected is selectively stored, and it is considered that the system described in the document stores all the still images including the lesion candidate area. Be done. Then, it is difficult to distinguish whether or not the doctor is observing the temporarily stored still image.
  • Patent Document 4 does not describe that a still image is selectively temporarily stored when an abnormal pixel is detected, and the device described in the same document temporarily stores all the still images when an abnormal pixel is detected. It is thought that he remembers it. Then, it is difficult to distinguish whether or not the doctor is observing the temporarily stored still image.
  • Patent Documents 1 to 4 it is difficult for any of the systems and the like described in Patent Documents 1 to 4 to support the search for an unobserved area of interest that a doctor has overlooked.
  • the present invention has been made in view of such circumstances, and provides a medical image processing apparatus, an endoscopic system, a medical image processing method, and a program that can support a search using information of an unobserved region of interest.
  • the purpose is.
  • the medical image processing device includes an image acquisition unit that acquires an observation image of a subject, a display signal transmission unit that transmits a first display signal representing the observation image to the display device, and a frame image that constitutes the observation image.
  • a medical image including a region of interest detection unit that detects an region of interest from the image, and an unobserved image storage unit that stores a frame image that satisfies an unobserved condition indicating that the region of interest is unobserved from the frame image in which the region of interest is detected. It is a processing device.
  • a frame image including an unobserved region of interest that satisfies the unobserved condition is stored from the observed image in which the region of interest is detected. This can support the search for an unobserved region of interest using an unobserved image.
  • the observation image of the subject may be a moving image or a still image group including a plurality of still images.
  • An example of an observation image is an endoscopic image taken with an endoscope.
  • a frame image is synonymous with one still image included in a still image group including a plurality of still images.
  • An example of an unobserved region of interest is an region of interest in which the user does not operate an imaging device such as a zoom.
  • the second aspect includes an unobserved condition determination unit that determines whether or not the unobserved condition is satisfied for the frame image in which the attention area is detected by using the attention area detection unit in the medical image processing apparatus of the first aspect. It may be configured as a new type.
  • the unobserved image can be determined from the plurality of frame images in which the region of interest is detected based on the unobserved condition.
  • an unobserved condition setting unit for setting an unobserved condition.
  • the unobserved condition determination unit determines that the unobserved condition is satisfied when the number of frame images including the same region of interest is less than or equal to the specified number within the specified period. It may be configured.
  • the number of frame images including the same region of interest can be applied.
  • the unobserved condition determination unit determines that the unobserved condition is satisfied when the amount of change between frame images is equal to or greater than a predetermined threshold value. May be.
  • the amount of change between frame images can be applied as an unobserved condition.
  • the fifth aspect is in the medical image processing apparatus of any one of the second to fourth aspects, when the unobserved condition determination unit stays in an arbitrary area in the screen within a specified period. It may be configured to determine that the unobserved condition is satisfied.
  • the position in the screen of the same area of interest can be applied as an unobserved condition.
  • the sixth aspect represents an unobserved image determined by the unobserved condition determination unit to satisfy the unobserved condition in the medical image processing apparatus according to any one of the second to fifth aspects.
  • the second display signal may be transmitted to the display device.
  • the unobserved image is displayed using the display device. This can support the search for an unobserved region of interest using the unobserved image displayed on the display device.
  • a seventh aspect is the medical image processing apparatus of the sixth aspect, which includes a user input signal acquisition unit that acquires a user input signal transmitted in response to a user operation, and the display signal transmission unit includes a user input signal acquisition unit.
  • the display signal transmission unit includes a user input signal acquisition unit.
  • a display signal representing an unobserved image is transmitted to the display device in response to the acquisition of the user input signal. This can support the search for an unobserved region of interest using the unobserved image displayed on the display device.
  • the first image processing unit that generates the first image by performing the first image processing on the attention area detected by using the attention area detection unit is used.
  • the display signal transmission unit may be configured to transmit a display signal representing the first image to the display device as the first display signal.
  • the first image on which the first image processing has been performed can be displayed on the display device.
  • the first image processing result storage unit for storing the result of the first image processing may be provided.
  • the first image may be applied to the processing result of the first image processing, or a combination of the observation image and the information representing the processing result of the first image processing associated with the observation image may be applied.
  • a ninth aspect is the medical image processing apparatus of the eighth aspect, comprising a second image generation unit that performs a second image processing on an unobserved image to generate a second image, and a display signal transmission unit is a ninth aspect.
  • the display signal representing the two images may be transmitted to the display device as the second display signal.
  • the second image obtained by performing the second image processing on the unobserved image can be displayed on the display device.
  • the first image processing unit performs the enhancement processing of the region of interest on the observed image
  • the second image processing unit performs the enhancement processing on the observed image with respect to the unobserved image. It may be configured to carry out an emphasis process in which the degree of emphasis is stronger than the emphasis process of the area of interest.
  • a second image having a stronger degree of emphasis than the enhancement process of the observed image can be displayed on the display device. This makes it easy to visually recognize the region of interest in the second image.
  • the second image processing unit performs the second image processing on each of the plurality of unobserved images
  • the display signal transmitting unit performs the second image processing.
  • the display signal corresponding to each of the plurality of second images may be transmitted to the display device as the second display signal.
  • a plurality of unobserved images or a plurality of second images can be displayed on the display device. Thereby, the user can search the area of interest in the observed image using the plurality of unobserved images or the plurality of second images.
  • one or more unobserved images are selected from the plurality of unobserved images displayed on the display device, or a plurality of second images displayed on the display device.
  • the configuration may include a selection unit for selecting one or more second images from.
  • the user sets an unobserved image applied to the search for the region of interest in the observed image from a plurality of unobserved images, or applies the search for the region of interest in the observed image from the plurality of second images.
  • a second image to be created can be set.
  • the third image processing is performed on the processing result of the unobserved image or the second image processing displayed on the display device.
  • the configuration may include a third image processing unit.
  • the third image processing can be performed on the unobserved image or the second image.
  • the fourteenth aspect may be a configuration in which the medical image processing apparatus of the thirteenth aspect includes a third image processing result storage unit for storing the processing result of the third image processing.
  • the processing result of the third image processing can be stored.
  • the third image processing result storage unit can store the third image as the processing result of the third image processing.
  • the third image processing result storage unit may store a combination of the image to be processed and the information representing the processing result of the third image processing associated with the image to be processed as the processing result of the third image processing.
  • a fifteenth aspect is the medical image processing apparatus of the thirteenth aspect or the fourteenth aspect, in which the third image processing unit is unobserved when the user input signal acquisition unit acquires a user input signal representing editing of an unobserved image.
  • the image may be edited, or the second image may be edited when the user input signal acquisition unit acquires the user input signal representing the editing of the second image.
  • the unobserved image or the second image can be edited according to the acquisition of the user input signal.
  • a sixteenth aspect is the medical image processing device according to any one of the thirteenth to fifteenth aspects, wherein the third image processing unit acquires a user input signal representing transmission of an unobserved image to an external device.
  • the unit acquires an unobserved image, or when the user input signal acquisition unit acquires a user input signal indicating transmission of the second image to the external device, the second image is externally transmitted. It may be configured to transmit to the device.
  • the unobserved image or the second image can be transmitted to an external device.
  • a seventh aspect is the medical image processing apparatus according to any one of the thirteenth to sixteenth aspects, wherein the third image processing unit obtains a user input signal representing the rearrangement of a plurality of unobserved images.
  • the user input signal acquisition unit acquires a user input signal representing the rearrangement of a plurality of unobserved images or a plurality of second images when the image is acquired by the user input signal acquisition unit, the plurality of second images are rearranged. It may be configured to carry out the rearrangement of.
  • the plurality of unobserved images may be rearranged or the plurality of second images may be rearranged. ..
  • the third image processing unit acquires a user input signal indicating deletion of an unobserved image by the user input signal acquisition unit.
  • the second image may be deleted when the unobserved image is deleted or the user input signal acquisition unit acquires the user input signal indicating the deletion of the second image.
  • the unobserved image or the second image can be deleted according to the acquisition of the user input signal.
  • a nineteenth aspect is the medical image processing apparatus according to any one of the sixth to eighteenth aspects, wherein the display signal transmission unit performs a second display signal after a predetermined period has elapsed from the transmission timing of the second display signal. It may be configured to stop the transmission of.
  • the unobserved image or the second image can be automatically hidden.
  • a twentieth aspect is the medical image processing apparatus according to any one of the sixth to eighteenth aspects, wherein the display signal transmission unit of the second display signal when an unobserved region of interest is displayed on the display device. It may be configured to stop transmission.
  • the unobserved image or the second image can be hidden.
  • the 21st aspect is the medical image processing apparatus according to any one of the 1st to 19th aspects, which includes a user operation unit operated by the user, and the user input signal acquisition unit is used when the user operates the user operation unit. It may be configured to acquire the user input signal transmitted to.
  • the user input signal can be transmitted according to the operation of the user.
  • the 22nd aspect is the medical image processing apparatus according to any one of the 1st to 21st aspects, which includes a voice acquisition unit for acquiring the user's voice, and the user input signal acquisition unit acquires the user's voice using the voice acquisition unit. It may be configured to acquire a user input signal representing the voice of the user.
  • the user input signal can be transmitted according to the user's voice.
  • the 23rd aspect is an observed image showing that the attention area has been observed from the frame image in which the attention area is detected by using the attention area detection unit in the medical image processing apparatus according to any one of the 1st to 22nd aspects. It may be configured to include an observed image storage unit that stores the image.
  • the observed image is stored. This can support the search for the region of interest for observation using the observed image.
  • the 24th aspect may be configured in the medical image processing apparatus of any one of the 1st to 23rd aspects, which includes a notification unit for notifying the detection of the region of interest in the observation image.
  • the user can recognize the detection of the region of interest.
  • the endoscope system includes an endoscope, an endoscope control device that controls the endoscope, and a medical image processing device that processes an endoscopic image acquired by using the endoscope.
  • the medical image processing device includes an image acquisition unit that acquires an observation image of a subject, a display signal transmission unit that transmits a first display signal representing the observation image to the display device, and an endoscope system including.
  • An unobserved image storage unit that detects an attention region from a frame image constituting an observation image and stores an unobserved image satisfying an unobserved condition indicating unobservation of the attention region from the frame image in which the attention region is detected. It is an endoscopic system equipped with a part.
  • the same items as those specified in the 2nd to 24th aspects can be appropriately combined.
  • the component responsible for the processing or function specified in the medical image processing apparatus can be grasped as the component of the endoscope system responsible for the corresponding processing or function.
  • the medical image processing method includes an image acquisition step of acquiring an observation image of a subject, a display signal transmission step of transmitting a first display signal representing the observation image to a display device, and a frame image constituting the observation image.
  • a medical image including a focus area detection step of detecting a region of interest from the image and an unobserved image storage step of storing an unobserved image satisfying an unobserved condition representing the unobserved region of interest from the frame image in which the region of interest is detected. It is a processing method.
  • the same items as those specified in the 2nd to 24th aspects can be appropriately combined.
  • the component responsible for the processing or function specified in the medical image processing apparatus can be grasped as the component of the medical image processing method responsible for the corresponding processing or function.
  • the program according to the 27th aspect pays attention to a computer from an image acquisition function for acquiring an observation image of a subject, a display signal transmission function for transmitting a first display signal representing the observation image to a display device, and a frame image constituting the observation image. It is a program that realizes a region of interest detection function for detecting a region and an unobserved image storage function for storing an unobserved image satisfying an unobserved condition indicating unobserved region of interest from a frame image in which the region of interest is detected.
  • the same items as those specified in the 2nd to 24th aspects can be appropriately combined.
  • the component responsible for the processing or function specified in the medical image processing apparatus can be grasped as the component of the program responsible for the corresponding processing or function.
  • a frame image including an unobserved region of interest that satisfies the unobserved condition is stored from the observed image in which the region of interest is detected. This can support the search for an unobserved region of interest using an unobserved image.
  • FIG. 1 is an overall configuration diagram of an endoscope system including an image processing device according to an embodiment.
  • FIG. 2 is a functional block diagram of the endoscopic system.
  • FIG. 3 is a functional block diagram of the image processing apparatus according to the first embodiment.
  • FIG. 4 is a flowchart showing the procedure of the image processing method according to the first embodiment.
  • FIG. 5 is a schematic view of an observation image.
  • FIG. 6 is a schematic view of the first image.
  • FIG. 7 is a schematic view of a screen when a user input signal representing an unobserved display of a region of interest is acquired.
  • FIG. 8 is a schematic view of the non-display of the unobserved image display area.
  • FIG. 9 is a functional block diagram of the image processing apparatus according to the second embodiment.
  • FIG. 9 is a functional block diagram of the image processing apparatus according to the second embodiment.
  • FIG. 10 is a flowchart showing the procedure of the image processing method according to the second embodiment.
  • FIG. 11 is a schematic view of a second image to which image processing for increasing the degree of emphasis is applied.
  • FIG. 12 is a schematic view of a mode in which a plurality of second images are displayed.
  • FIG. 13 is a schematic diagram of selection of a second image in a mode of displaying a plurality of second images.
  • FIG. 14 is a functional block diagram of the image processing apparatus according to the third embodiment.
  • FIG. 15 is a flowchart showing the procedure of the image processing method according to the third embodiment.
  • FIG. 16 is an explanatory diagram of an example of the third image processing.
  • FIG. 1 is an overall configuration diagram of an endoscope system including an image processing device according to an embodiment.
  • the endoscope system 10 includes an endoscope main body 100, a processor device 200, a light source device 300, and a monitor 400.
  • a part of the tip rigid portion 116 provided in the endoscope main body 100 is enlarged and shown.
  • the endoscope main body 100 includes a hand operation unit 102 and an insertion unit 104.
  • the user grasps and operates the hand operation unit 102, inserts the insertion unit 104 into the body of the subject, and observes the inside of the subject.
  • the user is synonymous with a doctor, a surgeon, and the like.
  • the subject referred to here is synonymous with a patient and a subject.
  • the hand operation unit 102 includes an air supply / water supply button 141, a suction button 142, a function button 143, and an image pickup button 144.
  • the air supply water supply button 141 accepts the operation of the air supply instruction and the water supply instruction.
  • the suction button 142 receives a suction instruction.
  • Various functions are assigned to the function button 143.
  • the function button 143 receives instructions for various functions.
  • the image pickup button 144 receives an image pickup instruction operation. Imaging includes moving image imaging and still image imaging.
  • the hand operation unit 102 functions as a user input unit.
  • a foot switch may be provided as a user input unit.
  • the footswitch comprises a pedal, a pedal indicator and a cable.
  • the cable is connected to the processor device 200.
  • the user can operate the foot switch to transmit a user input signal to the processor device 200.
  • the processor device 200 acquires a user input signal transmitted from the foot switch and performs a process corresponding to the user input signal.
  • the insertion portion 104 includes a soft portion 112, a curved portion 114, and a tip hard portion 116.
  • the soft portion 112, the curved portion 114, and the tip hard portion 116 are arranged in the order of the soft portion 112, the curved portion 114, and the tip hard portion 116 from the side of the hand operation portion 102. That is, the curved portion 114 is connected to the base end side of the tip hard portion 116, the soft portion 112 is connected to the proximal end side of the curved portion 114, and the hand operation portion 102 is connected to the proximal end side of the insertion portion 104.
  • the user can operate the hand operation unit 102 to bend the curved portion 114 and change the direction of the hard tip portion 116 up, down, left and right.
  • the hard tip portion 116 includes an imaging unit, an illumination unit, and a forceps opening 126.
  • FIG. 1 illustrates the photographing lens 132 that constitutes the imaging unit. Further, in the figure, the illumination lens 123A and the illumination lens 123B constituting the illumination unit are shown.
  • the imaging unit is illustrated with reference numeral 130. Further, the illumination unit is illustrated with reference numeral 123 in FIG.
  • wash water is discharged from the water supply nozzle or gas is discharged from the air supply nozzle.
  • the cleaning water and gas are used for cleaning the illumination lens 123A and the like.
  • the water supply nozzle and the air supply nozzle are not shown.
  • the water supply nozzle and the air supply nozzle may be shared.
  • the forceps opening 126 communicates with the pipeline. Treatment tools are inserted into the pipeline. The treatment tool is supported so that it can move forward and backward as appropriate. When removing a tumor or the like, a treatment tool is applied and necessary treatment is performed. It should be noted that the illustration of the pipeline communicating with the forceps opening 126 is omitted.
  • FIG. 2 is a functional block diagram of the endoscope system.
  • the endoscope main body 100 includes an imaging unit 130.
  • the image pickup unit 130 is arranged inside the tip rigid portion 116.
  • the image pickup unit 130 includes a photographing lens 132, an image pickup element 134, a drive circuit 136, and an analog front end 138.
  • AFE shown in FIG. 2 is an abbreviation for Analog Front End.
  • the photographing lens 132 is arranged on the tip end surface 116A of the tip hard portion 116.
  • the image sensor 134 is arranged at a position opposite to the tip end surface 116A of the photographing lens 132.
  • a CMOS type image sensor is applied to the image sensor 134.
  • a CCD type image sensor may be applied to the image sensor 134.
  • CMOS is an abbreviation for Complementary Metal-Oxide Semiconductor.
  • CCD is an abbreviation for Charge Coupled Device.
  • a color image sensor is applied to the image sensor 134.
  • An example of a color image sensor is an image sensor equipped with a color filter corresponding to RGB.
  • RGB is an acronym for Red, Green, and Yellow, which are English notations for red, green, and blue, respectively.
  • a monochrome image sensor may be applied to the image sensor 134.
  • the image sensor 130 can switch the wavelength band of the incident light of the image sensor 134 to perform surface-sequential or color-sequential imaging.
  • the drive circuit 136 supplies various timing signals necessary for the operation of the image pickup device 134 to the image pickup device 134 based on the control signal transmitted from the processor device 200.
  • the analog front end 138 includes an amplifier, a filter and an AD converter.
  • AD is an acronym for analog and digital, which are the English notations for analog and digital, respectively.
  • the analog front end 138 performs processing such as amplification, noise removal, and analog-to-digital conversion on the output signal of the image sensor 134.
  • the output signal of the analog front end 138 is transmitted to the processor device 200.
  • the optical image to be observed is formed on the light receiving surface of the image sensor 134 via the photographing lens 132.
  • the image sensor 134 converts an optical image to be observed into an electric signal.
  • the electric signal output from the image pickup device 134 is transmitted to the processor device 200 via the signal line.
  • the lighting unit 123 is arranged at the tip hard portion 116.
  • the illumination unit 123 includes an illumination lens 123A and an illumination lens 123B.
  • the illumination lens 123A and the illumination lens 123B are arranged at positions adjacent to the photographing lens 132 on the distal end surface 116A.
  • the lighting unit 123 includes a light guide 170.
  • the injection end of the light guide 170 is arranged at a position opposite to the tip end surface 116A of the illumination lens 123A and the illumination lens 123B.
  • the light guide 170 is inserted into the insertion unit 104, the hand operation unit 102, and the universal cable 106 shown in FIG.
  • the incident end of the light guide 170 is arranged inside the light guide connector 108.
  • the endoscope main body 100 described in the embodiment corresponds to an example of an endoscope.
  • the processor device 200 includes an image input controller 202, an image processing unit 204, and a video output unit 206.
  • the image input controller 202 acquires an electric signal corresponding to an optical image to be observed, which is transmitted from the endoscope main body 100.
  • the image processing unit 204 generates an endoscopic image of the observation target based on an imaging signal which is an electrical signal corresponding to the optical image of the observation target.
  • image in the present specification may include the meaning of the image itself and the image data representing the image.
  • the image may include moving and still images.
  • the endoscopic image is illustrated with reference numeral 38 in FIG.
  • the image processing unit 204 can perform image quality correction by applying digital signal processing such as white balance processing and shading correction processing to the image pickup signal.
  • the image processing unit 204 may add incidental information defined by the DICOM standard to the endoscopic image.
  • DICOM is an abbreviation for Digital Imaging and Communications in Medicine.
  • the video output unit 206 transmits a display signal representing an image generated by using the image processing unit 204 to the monitor 400.
  • the monitor 400 displays an image to be observed.
  • the processor device 200 When the image pickup button 144 shown in FIG. 1 is operated, the processor device 200 operates the image input controller 202, the image processing unit 204, and the like in response to the image pickup instruction signal transmitted from the endoscope main body 100.
  • the processor device 200 When the processor device 200 acquires a freeze instruction signal indicating still image imaging from the endoscope main body 100, the processor device 200 applies the image processing unit 204 to generate a still image based on the frame image at the operation timing of the imaging button 144. ..
  • the processor device 200 displays a still image on the monitor 400.
  • the frame image is illustrated with reference numeral 38B in FIG.
  • the still image is illustrated with reference numeral 39 in FIG.
  • the processor device 200 includes a communication control unit 205.
  • the communication control unit 205 controls communication with a device that is communicably connected via an in-hospital system, an in-hospital LAN, and the like.
  • the communication control unit 205 may apply a communication protocol conforming to the DICOM standard.
  • An example of an in-hospital system is HIS (Hospital Information System).
  • LAN is an abbreviation for Local Area Network.
  • the processor device 200 includes a storage unit 207.
  • the storage unit 207 stores an endoscope image generated by using the endoscope main body 100.
  • the storage unit 207 may store various information incidental to the endoscopic image.
  • the processor device 200 includes an operation unit 208.
  • the operation unit 208 outputs an instruction signal according to the user's operation.
  • the operation unit 208 may apply a keyboard, a mouse, a joystick, or the like.
  • the user input signal transmitted from the operation unit 208 is transmitted to the CPU 210.
  • the CPU 210 acquires a user input signal transmitted from the operation unit 208, and performs control corresponding to the acquired user input signal.
  • the operation unit 208 may include the above-mentioned foot switch.
  • the CPU 210 acquires a user input signal transmitted from the hand operation unit 102 shown in FIG. 1, and controls the endoscope main body 100, the processor device 200, and the light source device 300 corresponding to the acquired user input signal. carry out.
  • the processor device 200 includes a voice processing unit 209 and a speaker 209A.
  • the voice processing unit 209 generates a voice signal representing information to be notified as voice.
  • the speaker 209A converts the voice signal generated by using the voice processing unit 209 into voice. Examples of the voice output from the speaker 209A include a message, voice guidance, a warning sound, and the like.
  • the processor device 200 includes a CPU 210, a ROM 211, and a RAM 212.
  • CPU is an abbreviation for Central Processing Unit.
  • ROM is an abbreviation for Read Only Memory.
  • RAM is an abbreviation for Random Access Memory.
  • the CPU 210 functions as an overall control unit of the processor device 200.
  • the CPU 210 functions as a memory controller that controls the ROM 211 and the RAM 212.
  • the ROM 211 stores various programs, control parameters, and the like applied to the processor device 200.
  • the RAM 212 is applied to a temporary storage area for data in various processes and a processing area for arithmetic processing using the CPU 210.
  • the RAM 212 can be applied to the buffer memory when the endoscopic image is acquired.
  • the processor device 200 performs various processes on the endoscope image generated by using the endoscope main body 100, and displays the endoscope image and various information incidental to the endoscope image on the monitor 400. ..
  • the processor device 200 stores the endoscopic image and various information incidental to the endoscopic image.
  • the processor device 200 displays an endoscopic image or the like using the monitor 400, outputs audio information using the speaker 209A, and various types of the endoscopic image. Carry out the process.
  • the processor device 200 can function as an image processing device that performs a predetermined process on a medical image by using a component such as an image processing unit 204.
  • a component such as an image processing unit 204.
  • the prescribed processing include detection processing of the region of interest, enhancement processing of the region of interest, classification processing of the region of interest, discrimination processing of medical images, measurement processing of the region of interest, and the like.
  • the processor device 200 may apply a computer.
  • the computer may apply the following hardware and execute a specified program to realize the functions of the processor device 200.
  • a program is synonymous with software.
  • the processor device 200 can apply various processors as a signal processing unit that performs signal processing.
  • processors include CPUs and GPUs (Graphics Processing Units).
  • the CPU is a general-purpose processor that executes a program and functions as a signal processing unit.
  • the GPU is a processor specialized in image processing.
  • As the hardware of the processor an electric circuit in which an electric circuit element such as a semiconductor element is combined is applied.
  • Each control unit includes a ROM in which a program or the like is stored and a RAM in which a work area for various operations or the like is stored.
  • Two or more processors may be applied to one signal processing unit.
  • the two or more processors may be the same type of processor or different types of processors. Further, one processor may be applied to a plurality of signal processing units.
  • the processor device 200 described in the embodiment corresponds to an example of an endoscope control device.
  • the light source device 300 includes a light source 310, an aperture 330, a condenser lens 340, and a light source control unit 350.
  • the light source device 300 causes the observation light to be incident on the light guide 170.
  • the light source 310 includes a red light source 310R, a green light source 310G, and a blue light source 310B.
  • the red light source 310R, the green light source 310G, and the blue light source 310B emit red, green, and blue narrow-band light, respectively.
  • the light source 310 can generate illumination light in which narrow band lights of red, green and blue are arbitrarily combined.
  • the light source 310 may combine red, green, and blue narrowband light to produce white light.
  • the light source 310 can generate narrow band light by combining any two colors of red, green and blue narrow band light.
  • the light source 310 can generate narrow-band light using any one color of red, green, and blue narrow-band light.
  • the light source 310 may selectively switch and emit white light or narrow band light. Narrow band light is synonymous with special light.
  • the light source 310 may include an infrared light source that emits infrared light, an ultraviolet light source that emits ultraviolet light, and the like.
  • the light source 310 may employ an embodiment including a white light source that emits white light, a filter that allows white light to pass through, and a filter that allows narrow-band light to pass through.
  • the light source 310 of this aspect can selectively emit either white light or narrow band light by switching between a filter that allows white light to pass through and a filter that allows narrow band light to pass through.
  • the filter that passes narrow band light may include a plurality of filters corresponding to different bands.
  • the light source 310 may selectively switch between a plurality of filters corresponding to different bands to selectively emit a plurality of narrow band lights having different bands.
  • the type, wavelength band, etc. can be applied according to the type of observation target, the purpose of observation, and the like.
  • Examples of the types of the light source 310 include a laser light source, a xenon light source, an LED light source, and the like.
  • LED is an abbreviation for Light-Emitting Diode.
  • the observation light emitted from the light source 310 reaches the incident end of the light guide 170 via the diaphragm 330 and the condenser lens 340.
  • the observation light is applied to the observation target via the light guide 170, the illumination lens 123A, and the like.
  • the light source control unit 350 transmits a control signal to the light source 310 and the aperture 330 based on the instruction signal transmitted from the processor device 200.
  • the light source control unit 350 controls the illuminance of the observation light emitted from the light source 310, the switching of the observation light, the on / off of the observation light, and the like.
  • the image processing apparatus shown in the present embodiment reads out an unobserved image, which is a frame image including an unobserved region of interest, when a doctor transmits a user input signal indicating a redisplay of the unobserved region of interest. It is displayed on the monitor 400 shown in 1.
  • the unobserved image is displayed in an area different from the observed image display area where the observed image is displayed.
  • the doctor can compare the observed image with the unobserved image, and can search for the area of interest missed in the observed image.
  • the observation image is a real-time endoscopic image of the observation target displayed on the monitor 400 during endoscopy.
  • the observation image may be a moving image or a still image.
  • the observation target described in the embodiment corresponds to an example of a subject.
  • the monitor 400 described in the embodiment corresponds to an example of a display device.
  • the real-time endoscopic image may include a case where it is displayed on the monitor 400 after a certain delay period due to signal processing or the like has elapsed from the imaging timing.
  • the image processing apparatus shown in this embodiment has the same meaning as a medical image processing apparatus and a medical image processing apparatus. The same applies to the second embodiment and the third embodiment.
  • FIG. 3 is a functional block diagram of the image processing device according to the first embodiment.
  • the image processing device 14 shown in the figure is realized by applying components such as the image processing unit 204 in the processor device 200 shown in FIG.
  • the image processing device 14 includes an image acquisition unit 40, a user input signal acquisition unit 42, a display control unit 44, and a notification unit 46.
  • the image processing device 14 includes a region of interest detection unit 50, an unobserved condition determination unit 51, a first image processing unit 52, an endoscopic image storage unit 70, an unobserved image storage unit 71, an observed image storage unit 71A, and a first image processing device 14.
  • An image processing result storage unit 72 is provided. Each part will be described below.
  • the image acquisition unit 40 acquires an endoscope image 38 imaged using the endoscope main body 100 shown in FIG.
  • the acquisition of the endoscopic image 38 may include the acquisition of the moving image 38A, the acquisition of the frame image 38B, and the acquisition of the still image 39.
  • the image acquisition unit 40 stores the endoscopic image 38 in the endoscopic image storage unit 70.
  • the image acquisition unit 40 may acquire the endoscope image 38 from the processor device 200 via a cable that transmits a signal representing the endoscope image 38.
  • the image acquisition unit 40 may acquire the endoscopic image 38 from the processor device 200 via an information storage medium such as a memory card.
  • the image acquisition unit 40 may acquire the endoscopic image 38 via the communication network.
  • the image acquisition unit 40 can acquire a moving image 38A composed of a time-series frame image 38B.
  • the image acquisition unit 40 can acquire the still image 39 when the still image is captured during the acquisition of the moving image 38A.
  • the image acquisition unit 40 corresponds to the image input controller 202 shown in FIG.
  • the user input signal acquisition unit 42 acquires the user input signal transmitted from the operation unit 30 in response to the operation of the operation unit 30 of the user.
  • the operation unit 30 shown in FIG. 3 may include a hand operation unit 102 shown in FIG. 1, an operation unit 208 shown in FIG. 2, a foot switch (not shown), and the like.
  • the user input signal acquisition unit 42 may include a microphone that acquires the user's voice. That is, the user input signal acquisition unit 42 may acquire the user input signal based on the user's voice.
  • the operation unit 30 described in the embodiment corresponds to an example of the user operation unit.
  • the microphone described in the embodiment corresponds to an example of a voice acquisition unit that acquires a user's voice.
  • the display control unit 44 transmits a display signal representing the endoscopic image 38 to the monitor 400.
  • the monitor 400 displays the endoscopic image 38 as an observation image.
  • the display control unit 44 displays the first image in which the first image processing is performed on the endoscopic image 38 by using the first image processing unit 52.
  • the display signal to be represented is transmitted to the monitor 400.
  • the display control unit 44 transmits the display signal representing the unobserved image to the monitor 400.
  • the monitor 400 displays an unobserved image.
  • the display control unit 44 hides the unobserved image based on the non-display condition of the unobserved image displayed on the monitor 400.
  • the display control unit 44 corresponds to the video output unit 206 shown in FIG.
  • the display control unit 44 described in the embodiment corresponds to an example of a display signal transmission unit that transmits a first display signal representing an observation image to the display device.
  • the display signal representing the endoscopic image 38 described in the embodiment corresponds to an example of the first display signal.
  • the display signal representing the unobserved image described in the embodiment corresponds to an example of the second display signal.
  • the notification unit 46 When the region of interest is detected from the frame image 38B constituting the endoscope image 38, the notification unit 46 performs notification indicating the detection of the region of interest in the endoscope image 38.
  • the notification unit 46 can perform voice notification using the speaker 209A shown in FIG.
  • the notification unit 46 can perform notification of character information or the like using the monitor 400.
  • the attention region detection unit 50 detects the attention region from the frame image 38B acquired by the image acquisition unit 40 by applying a learning device such as a CNN (Convolutional Neural Network).
  • the attention area detection unit 50 derives the feature amount of the frame image 38B and determines the presence or absence of the attention area based on the feature amount.
  • the attention area detection unit 50 is applied with a trained learner that has been trained using a pair of the frame image 38B and the attention area in the frame image 38B as learning data. Examples of detection of the region of interest include detection of lesions and detection of specific organs.
  • the unobserved condition determination unit 51 determines whether or not the frame image 38B in which the region of interest is detected satisfies the condition of the frame image 38B stored in the unobserved image storage unit 71. That is, the unobserved condition determination unit 51 determines whether or not the frame image 38B in which the region of interest is detected is an unobserved image.
  • a specified number of frame images 38B including the same area of interest is set in advance, and the unobserved condition determination unit 51 determines that the number of frame images 38B including the same area of interest is equal to or less than the specified number within the specified period. Judged as an unobserved image.
  • the specified number can be specified according to the frame rate of the imaging unit 130 and the moving speed of the imaging unit 130.
  • the specified number may be any integer greater than or equal to 1.
  • the frame images 38B including the same region of interest are often continuous, but the screen may flicker and the detection of the region of interest may be interrupted.
  • the specified number may be the number of continuous frame images 38B or the number of non-continuous frame images 38B.
  • the movement of the imaging unit 130 represents the movement of the imaging unit 130 along the movement path of the endoscope main body 100 in the endoscopy. Further, the moving speed may include the concept of speed representing the absolute value of the moving speed.
  • the unobserved condition determination unit 51 can determine whether or not the unobserved frame image 38B is based on the amount of change between the frame images 38B such as the change of the movement vector of the imaging unit 130. That is, the unobserved condition determination unit 51 can determine an unobserved image when the amount of change between the frame images 38B is equal to or greater than a predetermined threshold value. As another example of the amount of change between the frame images 38B, the value of the cross-correlation function between the frame images 38B and the like can be mentioned.
  • the unobserved condition determination unit 51 can determine an unobserved image when the same region of interest stays in the non-center portion of the screen within a specified period.
  • the central portion of the image is the center of the screen.
  • it is an area including the center of gravity and does not include the edge of the screen.
  • the outside of the image is an area that does not include the center of the screen or the center of gravity, and is an area that includes the edges of the screen.
  • the unobserved condition determination unit 51 determines that the observation is not performed at least in any one of the number of frame images 38B including the same region of interest, the amount of change between the frame images 38B, and the position of the region of interest on the screen. If so, it can be determined as an unobserved image.
  • the unobserved condition determination unit 51 stores the frame image 38B in which the region of interest is detected in the unobserved image storage unit 71 as an unobserved image when the condition of the unobserved image stored in the unobserved image storage unit 71 is satisfied. To do.
  • the unobserved condition determination unit 51 determines the observed image storage unit as an observed image when the frame image 38B in which the region of interest is detected does not satisfy the condition of the unobserved image stored in the unobserved image storage unit 71. Store in 71A.
  • the first image processing unit 52 When the region of interest is detected from the frame image 38B constituting the endoscope image 38, the first image processing unit 52 performs the first image processing on the frame image 38B including the region of interest.
  • the endoscopic image 38 and the frame image 38B including the region of interest correspond to the above-mentioned observation image.
  • the first image processing can apply the enhancement processing of the region of interest.
  • the first image processing unit 52 generates an enhanced image that emphasizes the region of interest detected from the frame image 38B. Examples of the emphasized image include a bounding box superimposed on the region of interest, a predetermined color and pattern added to the region of interest, and the like.
  • the first image processing unit 52 specifies the position of the region of interest and defines the position of the emphasized image according to the position of the region of interest.
  • the first image processing unit 52 specifies the size of the region of interest and defines the size of the enhanced image according to the size of the region of interest.
  • the first image processing unit 52 stores the processing result of the first image processing in the first image processing result storage unit 72.
  • the first image processing result storage unit 72 can store the first image in which the emphasized image is added to the frame image 38B in which the region of interest is detected.
  • the first image processing result storage unit 72 may store a combination of information of the frame image 38B including the region of interest and the information of the emphasized image associated with the frame image 38B including the region of interest. That is, the first image processing unit 52 may store the frame image 38B file and the emphasized image file as separate files.
  • the first image processing unit 52 uses the display control unit 44 to display the frame image 38B on which the emphasized image is superimposed and displayed as the observation image on the monitor 400.
  • the notification unit 46 may perform notification indicating detection of the region of interest in the endoscopic image 38 by using the frame image 38B on which the emphasized image is superimposed and displayed.
  • the endoscopic image storage unit 70 stores the endoscopic image 38 acquired by using the image acquisition unit 40. When the still image imaging is performed, the endoscopic image storage unit 70 stores the still image 39 generated in the still image imaging.
  • the unobserved image storage unit 71 stores an unobserved image.
  • the display control unit 44 reads out the unobserved image stored in the unobserved image storage unit 71.
  • the unobserved image storage unit 71 may store a new unobserved image in place of the already stored unobserved image when a new unobserved image is generated while the unobserved image is stored. ..
  • the unobserved image storage unit 71 may specify the number of stored unobserved images. Further, the unobserved image storage unit 71 may define the storage period of the unobserved image. That is, the unobserved image storage unit 71 may store the unobserved image that is likely to be redisplayed and delete the unobserved image that is unlikely to be redisplayed. In FIG. 3, the unobserved image is not shown.
  • the observed image storage unit 71A stores the observed image.
  • the display control unit 44 reads out the observed image stored in the observed image storage unit 71A.
  • the first image processing result storage unit 72 stores the processing result of the first image processing performed by using the first image processing unit 52. As the first image processing result, the first image may be applied, or the frame image 38B and the combination of the information of the region of interest associated with the frame image 38B may be applied.
  • Each of the endoscopic image storage unit 70, the unobserved image storage unit 71, the observed image storage unit 71A, and the first image processing result storage unit 72 may apply one or more storage elements. That is, the image processing device 14 may include four storage elements corresponding to each of the endoscopic image storage unit 70, the unobserved image storage unit 71, the observed image storage unit 71A, and the first image processing result storage unit 72. ..
  • the endoscopic image storage unit 70, the unobserved image storage unit 71, the observed image storage unit 71A, and the first image processing result storage unit 72 may be configured by using one storage element of any two to four. ..
  • one storage element may be used to form an endoscopic image storage unit 70, an unobserved image storage unit 71, and an observed image storage unit 71A.
  • FIG. 4 is a flowchart showing the procedure of the image processing method according to the first embodiment.
  • the image processing method shown in this embodiment is synonymous with a medical image processing method, a medical image processing method, and the like. The same applies to the second embodiment and the third embodiment.
  • the image acquisition unit 40 shown in FIG. 3 acquires the endoscopic image 38.
  • the image acquisition unit 40 stores the endoscopic image 38 in the endoscopic image storage unit 70.
  • the process proceeds to the region of interest detection step S12.
  • the attention area detection unit 50 detects the attention area from each frame image 38B constituting the endoscopic image 38.
  • the attention region detection unit 50 may detect the attention region from all the frame images 38B of the moving image 38A, or may detect the attention region from the frame images 38B at predetermined intervals. .. After the attention area detection step S12, the process proceeds to the notification step S14.
  • the notification unit 46 performs notification indicating detection of the region of interest in the frame image 38B.
  • a notification stop step for stopping notification may be performed. After the notification step S14, the process proceeds to the first image processing step S16.
  • the first image processing unit 52 performs the first image processing on the frame image 38B in which the region of interest is detected. After the first image processing step S16, the process proceeds to the first image processing result storage step S17. After performing the first image processing step S16, the notification step S14 may be carried out.
  • the first image processing unit 52 stores the processing result of the first image processing in the first image processing result storage unit 72.
  • the process proceeds to the unobserved condition determination step S18.
  • a display step of displaying the first image may be performed after the first image processing result storage step S17. Further, the first image processing step S16 and the first image processing result storage step S17 may be omitted.
  • the unobserved condition determination unit 51 determines whether or not the frame image 38B in which the region of interest is detected satisfies the unobserved condition. That is, in the unobserved condition determination step S18, the unobserved condition determination unit 51 determines whether or not the frame image 38B in which the region of interest is detected is an unobserved image.
  • the unobserved condition determination step S18 when the unobserved condition determination unit 51 determines that the unobserved condition is not satisfied, a No determination is made. In the case of No determination, the process proceeds to the user input signal acquisition determination step S20.
  • the unobserved condition determination step S18 when the unobserved condition determination unit 51 determines that the unobserved condition is satisfied, a Yes determination is made. In the case of Yes determination, the process proceeds to the unobserved image storage step S19.
  • the unobserved condition determination unit 51 stores the unobserved image in the unobserved image storage unit 71.
  • the process proceeds to the user input signal acquisition determination step S20.
  • An observed image storage step of storing the observed image in the observed image storage unit 71A may be performed before or after the unobserved image storage step S19 or in parallel with the unobserved image storage step S19.
  • the user input signal acquisition unit 42 determines whether or not the user input signal representing the display of the unobserved region of interest has been acquired. In the user input signal acquisition determination step S20, when it is determined that the user input signal acquisition unit 42 has not acquired the user input signal representing the display of the unobserved region of interest, a No determination is made. In the case of No determination, the process proceeds to the final frame image determination step S32.
  • the user input signal acquisition determination step S20 when it is determined that the user input signal acquisition unit 42 has acquired the user input signal representing the display of the unobserved region of interest, the determination is Yes. In the case of Yes determination, the process proceeds to the display signal transmission step S26.
  • the user input signal acquisition determination step S20 corresponds to an example of the user input signal acquisition step.
  • the display control unit 44 transmits a display signal indicating unobserved to the monitor 400.
  • the monitor 400 displays unobserved.
  • the display control unit 44 determines whether or not the non-display condition of the unobserved image displayed on the monitor 400 is satisfied. In the non-display condition determination step S28, when the display control unit 44 determines that the non-display condition of the unobserved image is not satisfied, a No determination is made. In the case of No determination, the non-display condition determination step S28 is continued until a Yes determination is made in the non-display condition determination step S28.
  • non-display condition determination step S28 when it is determined that the display control unit 44 satisfies the non-display condition of the unobserved image, a Yes determination is made. In the case of Yes determination, the process proceeds to the display signal transmission stop step S30. The details of the non-display condition of the unobserved image will be described later.
  • the display control unit 44 stops the transmission of the display signal representing the unobserved image displayed on the monitor 400. That is, in the display signal transmission stop step S30, the display control unit 44 hides the unobserved image. After the display signal transmission stop step S30, the process proceeds to the final frame image determination step S32.
  • the attention region detection unit 50 determines whether or not the attention region detection process for the final frame image 38B in the endoscope image 38 acquired by using the image acquisition unit 40 has been performed.
  • the process proceeds to the endoscopic image acquisition step S10, and each step from the endoscopic image acquisition step S10 to the final frame image determination step S32 is repeatedly performed until a Yes determination is made in the final frame image determination step S32. ..
  • the image processing device 14 ends the image processing method.
  • FIG. 5 is a schematic view of an observation image.
  • the figure shows a schematic view of an observation image of the large intestine as an endoscopic image 38.
  • the endoscope image 38 is displayed in real time as an observation image on the observation image display area 404 on the screen 402 of the monitor 400.
  • the moving image 38A of the endoscopic image 38 during endoscopy is displayed in real time as an observation image.
  • the observation image shown in FIG. 5 shows a case where the region of interest is not detected from the frame image 38B constituting the endoscopic image 38.
  • FIG. 6 is a schematic view of the first image.
  • the first image processing unit 52 shown in FIG. 3 first refers to the frame image 38B according to the detection result of the region of interest 510. Perform image processing.
  • the enhancement process of the attention area 510 is applied as the first image processing, and the bounding box 512 surrounding the attention area 510 is superimposed on the frame image 38B in which the attention area 510 is detected.
  • the displayed first image 500 is displayed.
  • FIG. 6 shows a first image 500 corresponding to the frame image 38B at an arbitrary timing.
  • FIG. 7 is a schematic view of a screen when a user input signal representing an unobserved display of a region of interest is acquired.
  • the doctor grasps the detection of the attention region 510 due to the notification of the detection of the attention region 510, he / she may want to display the missed attention region 510 on the screen 402.
  • the display control unit 44 When the user input signal acquisition unit 42 shown in FIG. 3 acquires a user input signal representing the display of the unobserved region of interest 510, the display control unit 44 reads an unobserved image from the unobserved image storage unit 71 and reads it out.
  • the unobserved image 39A is displayed in the unobserved image display area 406 of the screen 402.
  • the display control unit 44 causes the unobserved image 39A to be displayed in real time on the observed image display area 404 of the screen 402.
  • the doctor can observe the endoscopic image 38 and discover the missed region of interest 510 by utilizing the unobserved image 39A displayed in the unobserved image display region 406.
  • FIG. 7 shows a second image 520 in which the region of interest 510 and the bounding box 512 are superimposed and displayed on the unobserved image 39A. That is, the unobserved image display area 406 may display the unprocessed unobserved image 39A, or the second image 520 subjected to the second image processing may be applied to the unobserved image 39A. .. The details of the second image processing will be described later.
  • FIG. 8 is a schematic view of the non-display of the unobserved image display area.
  • the second image 520 displayed in the unobserved image display area 406 is hidden.
  • FIG. 8 shows a second image 520 hidden by a dotted line.
  • the image processing device 14 includes a timer.
  • the timer measures the period from the display start timing of the second image 520.
  • the display control unit 44 may hide the second image 520 based on the measured value of the timer. The timer is not shown.
  • the display control unit 44 described in the embodiment corresponds to an example of a display signal transmission unit that stops the transmission of the second display signal after a predetermined period has elapsed from the transmission timing of the second display signal.
  • the attention region detection unit 50 shown in FIG. 3 determines the identity between the attention region 510 in the second image 520 and the attention region 510 detected from the endoscopic image 38.
  • the attention region detection unit 50 shown in FIG. 3 compares the feature amount of the attention region 510 in the second image 520 with the feature amount of the attention region 510 detected from the endoscopic image 38, and based on the comparison result, both of them. Identity can be determined.
  • the attention area detection unit 50 can apply information such as a pixel value, the number of pixels, and a shape as a feature amount of the attention area 510.
  • the display control unit 44 When the display control unit 44 obtains a determination result indicating that the attention area 510 in the second image 520 and the attention area 510 in the endoscopic image 38 are the same, the display control unit 44 displays the image in the unobserved image display area 406. The second image 520 is hidden.
  • the notification unit 46 shown in FIG. 3 applies voice notification, character notification, and the like to make the determination.
  • the result can be notified.
  • the non-display condition of the unobserved image display area 406 also applies to the unobserved image 39A when the unobserved image 39A is displayed in the unobserved image display area 406.
  • the image processing apparatus 14 and the image processing method according to the first embodiment can obtain the following effects.
  • the attention region detection unit 50 detects the attention region 510 from the frame image 38B constituting the moving image 38A of the endoscopic image 38.
  • the unobserved image storage unit 71 stores the unobserved image 39A including the unobserved region of interest 510 among the endoscopic images 38 in which the region of interest 510 is detected.
  • the attention region 510 can be searched by using the unobserved image 39A or the like.
  • the display control unit 44 acquires a user input signal representing the display of the unobserved region of interest 510
  • the display control unit 44 reads out the unobserved image 39A stored in the unobserved image storage unit 71, and reads out the unobserved image 39A or the unobserved image 39A.
  • the second image 520 is displayed on the monitor 400.
  • the display control unit 44 causes the unobserved image 39A or the second image 520 to be displayed in the unobserved image display area 406, which is a region different from the observed image display area 404. This makes it easier for the doctor to search for the region of interest 510 in the observation image.
  • the display control unit 44 hides the unobserved image 39A or the second image 520 when the user input signal indicating the non-display of the unobserved image 39A or the second image 520 is acquired. As a result, when the region of interest 510 to be searched is found, the unobserved image 39A that can interfere with the observation can be hidden.
  • the display control unit 44 hides the unobserved image 39A or the second image 520 after a predetermined period has elapsed from the display start timing of the unobserved image 39A or the second image 520. As a result, the unobserved image 39A or the second image 520, which may interfere with the observation, can be automatically hidden.
  • the user input signal acquisition unit 42 acquires the user input signal transmitted from the operation unit and the user input signal according to the user's voice using the microphone. As a result, the user input signal acquisition unit 42 can apply various forms of user input signals.
  • the first image processing unit 52 performs the first processing on the frame image 38B in which the region of interest 510 is detected.
  • the display control unit 44 displays the first image 500 representing the processing result of the first image processing unit 52 in the observation image display area 404. This allows the physician to visually recognize the detection of the region of interest in the observation image.
  • the first image processing unit 52 generates a second image 520 in which the unobserved image 39A is subjected to the second image processing to which the image processing having the same contents as the first image processing is applied.
  • the display control unit 44 displays the second image 520 in the unobserved image display area 406. As a result, the visibility of the region of interest 510 in the second image 520 can be improved.
  • the unobserved image 39A is displayed by using the monitor 400 based on the user input signal, but the unobserved image 39A is displayed by using the monitor 400 based on the generation of the unobserved image 39A. 39A may be displayed.
  • the unobserved image 39A when the unobserved image 39A occurs, the unobserved image 39A can be automatically displayed on the monitor 400.
  • FIG. 9 is a functional block diagram of the image processing apparatus according to the second embodiment.
  • a second image processing unit 54 and a second image processing result storage unit 74 are added to the image processing device 14 shown in FIG.
  • the second image processing unit 54 performs the second image processing on the unobserved image 39A shown in FIG. 7. That is, when the unobserved image 39A is displayed in the unobserved image display area 406, the second image processing unit 54 has a second image 520 with respect to the display mode of the first image 500 to be displayed in the observed image display area 404. The display mode of is changed.
  • the second image processing may be applied with an enhancement process in which the degree of emphasis is stronger than that of the first image 500.
  • the second image processing may apply a batch display of a plurality of different second images 520.
  • As the second image processing a process of selecting one from a plurality of different second images 520 may be applied.
  • the second image processing unit 54 stores the processing result of the second image processing in the second image processing result storage unit 74.
  • the second image 520 can be applied.
  • a combination of the unobserved image 39A and the information representing the processing result of the second processing for the unobserved image 39A may be applied.
  • the second image processing result storage unit 74 stores the processing result of the second image processing performed by using the second image processing unit 54.
  • the second image processing result storage unit 74 may store the processing result of the second image processing selected from the processing results of the plurality of second image processing.
  • FIG. 10 is a flowchart showing the procedure of the image processing method according to the second embodiment.
  • a second image processing step S22 and a second image processing result storage step S24 are added to the flowchart shown in FIG.
  • Each step from the endoscope image acquisition step S10 to the user input signal acquisition determination step S20 shown in FIG. 10 is the same as each step from the endoscope image acquisition step S10 to the user input signal acquisition determination step S20 shown in FIG. Is.
  • the process proceeds to the second image processing step S22.
  • the second image processing unit 54 shown in FIG. 9 performs the second image processing on the unobserved image 39A. After the second image processing step S22, the process proceeds to the second image processing result storage step S24.
  • the second image processing unit 54 stores the result of the second image processing in the second image processing result storage unit 74.
  • the process proceeds to the display signal transmission step S26.
  • the second image processing step S22 and the second image processing result storage step S24 may be performed between the first image processing result storage step S17 and the unobserved condition determination step S18.
  • FIG. 11 is a schematic view of a second image to which image processing for increasing the degree of emphasis is applied.
  • the second image 530 with a stronger degree of emphasis is displayed as compared with the enhancement process of the first image 500 shown in FIG.
  • a bounding box 532 in which the degree of emphasis of the attention area 510 is stronger than that of the bounding box 512 shown in FIG. 6 is applied.
  • the type of the border is changed from a broken line to a solid line, the width of the border is increased, and the color of the border is conspicuous such as red, as compared with the bounding box 512 applied to the first image 500. Color is used.
  • the process of increasing the degree of emphasis is not limited to the mode shown in FIG.
  • at least one of the border type, border line width, and border color of the bounding box 512 shown in FIG. 6 may be changed.
  • the real-time endoscopic image 38 is displayed in the observation image display area 404.
  • the second image 530 can be referred to when the endoscope main body 100 shown in FIG. 1 is operated to search for the region of interest 510.
  • the second image 530 shown in FIG. 11 can improve the visibility of the attention area 510 as compared with the frame image 38B displayed in the unobserved image display area 406 shown in FIG. 7.
  • FIG. 12 is a schematic diagram of a mode in which a plurality of second images are displayed.
  • the second image 540 and the second image 542 shown in FIG. 12 correspond to the unobserved image 39A generated at different timings.
  • the bounding box 512A is superimposed and displayed on the attention area 510A.
  • the bounding box 512B is superimposed and displayed on the attention area 510B.
  • FIG. 13 is a schematic diagram of selection of a second image in a mode of displaying a plurality of second images.
  • the second image 542 selected from the second image 540 and the second image 542 is highlighted. That is, when the user input signal representing the selection of the second image 540 or the like is acquired, the second image processing unit 54 highlights the selected second image 542 based on the user input signal.
  • the unselected second image 540 may be hidden according to the selection of the second image 542.
  • the highlighting of the second image 542 may be hidden in response to the non-selection of the second image 540.
  • the user input signal acquisition unit 42 shown in FIG. 9 corresponds to an example of a selection unit.
  • FIGS. 12 and 13 show an aspect of displaying two second images 540 and the like, three or more second images may be displayed.
  • information indicating each acquisition timing may be added. It is preferable that the plurality of second images 540 and the like are arranged in the order of acquisition timing. Further, in the selection of the second image carried out by selecting the plurality of second images 540 and the like, two or more second images may be selected.
  • a plurality of second images 540 and the like can be used as a plurality of search candidates.
  • one or more search targets can be selected from a plurality of search candidates.
  • At least one of the hue, lightness, and saturation of the region of interest 510 with respect to the first image 500 may be changed.
  • there is a resolution conversion such as lowering the resolution and increasing the resolution.
  • the image processing apparatus 14A and the image processing method according to the second embodiment can obtain the following effects.
  • the second image processing unit 54 generates a second image 530 or the like whose display mode is different from that of the first image 500.
  • the display control unit 44 displays the second image 530 and the like on the monitor.
  • the enhancement processing of the region of interest 510 is applied.
  • the enhancement process according to the second image processing has a higher degree of enhancement than the enhancement process according to the first image processing. As a result, the visibility of the region of interest 510 in the second image 530 and the like can be improved.
  • the monitor 400 displays a plurality of second images 540 and the like having different acquisition timings. As a result, the region of interest 510 missed in the endoscopic image 38 can be searched from among the plurality of second images 540 and the like.
  • One or more second images 542 and the like can be selected from a plurality of second images 540 and the like having different acquisition timings.
  • the second image 542 or the like applied to the search of the region of interest 510 can be set from the plurality of second images 540 or the like having different acquisition timings.
  • FIG. 14 is a functional block diagram of the image processing apparatus according to the third embodiment.
  • a third image processing unit 56 and a third image processing result storage unit 76 are added to the image processing device 14A shown in FIG.
  • the third image processing unit 56 performs the third image processing on the unobserved image 39A or the second image 520 displayed in the unobserved image display area 406.
  • the second image 520 and the like referred to here represent at least one of the second image 520 shown in FIG. 7, the second image 530 shown in FIG. 11, the second image 540 shown in FIG. 12, and the second image 542. The same applies to the following description.
  • the third image processing includes editing the unobserved image 39A or the second image 520 or the like, storing the third image generated by editing the unobserved image 39A or the second image 520 or the like, the unobserved image 39A or the second image. At least one of transmission to an external device 32 such as 520, deletion of the unobserved image 39A or the second image 520, and rearrangement of the plurality of unobserved images 39A or the plurality of second images 520 may be applied.
  • the third image processing result storage unit 76 when the third image processing applied to the third image processing unit 56 is editing of the unobserved image 39A or the second image 520 or the like, the unobserved image 39A or the second image
  • the third image 560 generated by editing 520 and the like is stored.
  • enlargement or reduction of the unobserved image 39A or the second image 520 or the like may be applied.
  • the third image processing unit 56 sends the unobserved image to the external device 32.
  • 39A or the second image 520 or the like is transmitted.
  • the external device 32 may apply a storage device or the like communicatively connected to the image processing device 14B. Note that FIG. 14 illustrates transmission of the second image 520 or the like to the external device 32.
  • the third image processing unit 56 deletes the unobserved image 39A or the second image 520 or the like. carry out.
  • the third image processing unit 56 arranges the unobserved image 39A or the second image 520 or the like. To rearrange.
  • FIG. 15 is a flowchart showing the procedure of the image processing method according to the third embodiment.
  • FIG. 15 shows a procedure in which editing of the second image 530 is applied as the third image processing.
  • a third image processing instruction step S40, a third image processing step S42, and a third image processing result storage step S44 are added to the flowchart shown in FIG.
  • Each step from the endoscopic image acquisition step S10 to the display signal transmission step S26 shown in FIG. 15 is the same as each step from the endoscope image acquisition step S10 to the display signal transmission step S26 shown in FIG. After the display signal transmission step S26, the process proceeds to the third image processing instruction step S40.
  • the user input signal acquisition unit 42 shown in FIG. 14 acquires a third image processing instruction signal representing the processing content of the third image processing.
  • acquisition of the third image processing instruction signal acquisition of a user input signal transmitted from the operation unit 30 shown in FIG. 14 can be mentioned.
  • the process proceeds to the third image processing step S42.
  • the third image processing unit 56 performs the third image processing on the unobserved image 39A, the second image 520, or the like based on the third image processing instruction signal. After the third image processing step S42, the process proceeds to the third image processing result storage step S44.
  • the third image processing result storage unit 76 shown in FIG. 14 stores the processing result of the third image processing.
  • the process proceeds to the non-display condition determination step S28.
  • the third image processing is transmission of the unobserved image 39A or the second image 520 to the external device 32 and deletion of the unobserved image 39A or the second image 520 or the like, the third image processing result storage step S44 is omitted. To.
  • Each step from the non-display condition determination step S28 to the final frame image determination step S32 shown in FIG. 15 is the same as each step from the non-display condition determination step S28 to the final frame image determination step S32 shown in FIG.
  • FIG. 16 is an explanatory diagram of an example of the third image processing.
  • the third image 560 is displayed instead of the second image 520 and the like shown in FIG.
  • the screen 402 may display both the second image 520 and the like and the third image 560.
  • the third image 560 the area of interest 510 of the second image 520 shown in FIG. 7 is enlarged. That is, the third image 560 shown in FIG. 16 includes a region of interest 510C in which the region of interest 510 shown in FIG. 7 is expanded and a bounding box 512C in which the bounding box 512 shown in FIG. 7 is expanded.
  • the image processing apparatus 14B and the image processing method according to the third embodiment can obtain the following effects.
  • a third image processing unit 56 that performs a third image processing on the unobserved image 39A, the second image 520, or the like is provided. As a result, editing of the unobserved image 39A, the second image 520, or the like can be performed.
  • the third image processing includes editing the unobserved image 39A or the second image 520, storing the editing result of the unobserved image 39A or the second image 520, and printing the unobserved image 39A or the second image 520 to an external device. Transmission, deletion of the unobserved image 39A or the second image 520, etc., and rearrangement of the unobserved image 39A or the second image 540, etc. in the embodiment of displaying the plurality of unobserved images 39A or the plurality of second images 540, etc. are applied. To. Thereby, the unobserved image 39A, the second image 520, or the like can be edited.
  • [Modification example of endoscopic system] [Modification of illumination light] As an example of the medical image that can be acquired by using the endoscope system 10 shown in the present embodiment, there is a white band light or a normal light image obtained by irradiating light of a plurality of wavelength bands as white band light. Be done.
  • Another example of a medical image that can be obtained by using the endoscope system 10 shown in the present embodiment is an image obtained by irradiating light in a specific wavelength band.
  • a narrower band than the white band can be applied to a specific wavelength band. The following modifications can be applied.
  • a first example of a particular wavelength band is the blue or green band in the visible range.
  • the wavelength band of the first example includes a wavelength band of 390 nanometers or more and 450 nanometers or less, or 530 nanometers or more and 550 nanometers or less, and the light of the first example is 390 nanometers or more and 450 nanometers or less, or It has a peak wavelength in the wavelength band of 530 nanometers or more and 550 nanometers or less.
  • a second example of a particular wavelength band is the red band in the visible range.
  • the wavelength band of the second example includes a wavelength band of 585 nanometers or more and 615 nanometers or less, or 610 nanometers or more and 730 nanometers or less, and the light of the second example is 585 nanometers or more and 615 nanometers or less, or It has a peak wavelength in the wavelength band of 610 nanometers or more and 730 nanometers or less.
  • the third example of a specific wavelength band includes a wavelength band in which the extinction coefficient differs between oxidized hemoglobin and reduced hemoglobin, and the light in the third example has a peak wavelength in a wavelength band in which the extinction coefficient differs between oxidized hemoglobin and reduced hemoglobin.
  • the wavelength band of the third example includes a wavelength band of 400 ⁇ 10 nanometers, 440 ⁇ 10 nanometers, 470 ⁇ 10 nanometers, or 600 nanometers or more and 750 nanometers or less, and the light of the third example is It has a peak wavelength in the wavelength band of 400 ⁇ 10 nanometers, 440 ⁇ 10 nanometers, 470 ⁇ 10 nanometers, or 600 nanometers or more and 750 nanometers or less.
  • the fourth example of the specific wavelength band is the wavelength band of the excitation light used for observing the fluorescence emitted by the fluorescent substance in the living body and exciting the fluorescent substance.
  • it is a wavelength band of 390 nanometers or more and 470 nanometers or less.
  • the observation of fluorescence may be referred to as fluorescence observation.
  • a fifth example of a specific wavelength band is the wavelength band of infrared light.
  • the wavelength band of the fifth example includes a wavelength band of 790 nanometers or more and 820 nanometers or less, or 905 nanometers or more and 970 nanometers or less, and the light of the fifth example is 790 nanometers or more and 820 nanometers or less. Alternatively, it has a peak wavelength in a wavelength band of 905 nanometers or more and 970 nanometers or less.
  • the processor device 200 may generate a special optical image having information in a specific wavelength band based on a normal optical image obtained by imaging with white light. Note that the generation here includes acquisition. In this case, the processor device 200 functions as a special optical image acquisition unit. Then, the processor device 200 obtains a signal in a specific wavelength band by performing an operation based on color information of red, green and blue, or cyan, magenta and yellow contained in a normal optical image.
  • Cyan, magenta, and yellow may be expressed as CMY using the acronyms Cyan, Magenta, and Yellow, which are the respective English notations.
  • Example of generating a feature image As a medical image, at least one of a white band light, a normal light image obtained by irradiating light in a plurality of wavelength bands as white band light, and a special light image obtained by irradiating light in a specific wavelength band.
  • a feature image can be generated using the based operation.
  • the above-mentioned image processing apparatus and image processing method can be configured as a program that realizes a function corresponding to each part in the image processing apparatus or each process in the image processing method by using a computer.
  • a computer may be configured with a program that realizes an image acquisition function, a display signal transmission function, a region of interest detection function, an unobserved image storage function, and a user input signal acquisition function.
  • the image acquisition function corresponds to the image acquisition unit 40 and the endoscopic image acquisition step S10.
  • the display signal transmission function corresponds to the display control unit 44 and the display signal transmission step S26.
  • the attention area detection function corresponds to the attention area detection unit 50 and the attention area detection step S12.
  • the unobserved image storage function corresponds to the unobserved image storage unit 71 and the unobserved condition determination step S18.
  • the user input signal acquisition function corresponds to the user input signal acquisition unit 42 and the user input signal acquisition determination step S20.
  • the program may include a first image processing unit 52 and a first image processing corresponding to the first image processing step S16.
  • the program may include a first image processing result storage unit 72 and a first image processing result storage function corresponding to the first image processing result storage step S17.
  • the program may include a second image processing function corresponding to the second image processing unit 54 and the second image processing step S22.
  • the program may include a second image processing result storage unit 74 and a second image processing result storage function corresponding to the second image processing result storage step S24.
  • the program may include a third image processing function corresponding to the third image processing unit 56 and the third image processing step S42.
  • the mode in which the program is stored and provided in the non-temporary information storage medium the mode in which the program signal is provided via the communication network is also possible.
  • the constituent requirements can be appropriately changed, added, or deleted without departing from the gist of the present invention.
  • the present invention is not limited to the embodiments described above, and many modifications can be made by a person having ordinary knowledge in the art within the technical idea of the present invention.
  • Endoscope system 14 Image processing device 14A Image processing device 38 Endoscope image 38A Moving image 39 Still image 39A Unobserved image 40 Image acquisition unit 42 User input signal acquisition unit 44 Display control unit 46 Notification unit 50 Attention area detection unit 51 Unobserved condition determination unit 52 First image processing unit 54 Second image processing unit 56 Third image processing unit 70 Endoscopic image storage unit 71 Unobserved image storage unit 71A Observed image storage unit 72 First image processing result storage Part 74 Second image processing result storage part 76 Third image processing result storage part 100 Endoscope body 102 Hand operation part 104 Insertion part 106 Universal cable 108 Light guide connector 112 Flexible part 114 Curved part 116 Tip hard part 116A Tip side end face 123 Illumination unit 123A Illumination lens 123B Illumination lens 126 Force opening 130 Imaging unit 132 Imaging lens 134 Imaging element 136 Drive circuit 138 Analog front end 141 Air supply / water supply button 142 Suction button 143 Function button 144 Imaging button 170 Light guide 200 Processor device 202 Image input

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Surgery (AREA)
  • Physics & Mathematics (AREA)
  • Medical Informatics (AREA)
  • General Health & Medical Sciences (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Public Health (AREA)
  • Pathology (AREA)
  • Veterinary Medicine (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Animal Behavior & Ethology (AREA)
  • Biophysics (AREA)
  • Optics & Photonics (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Theoretical Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Computational Linguistics (AREA)
  • Human Computer Interaction (AREA)
  • Acoustics & Sound (AREA)
  • Quality & Reliability (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Software Systems (AREA)
  • Primary Health Care (AREA)
  • Epidemiology (AREA)
  • Endoscopes (AREA)
  • Closed-Circuit Television Systems (AREA)
PCT/JP2020/032049 2019-09-03 2020-08-25 医療画像処理装置、内視鏡システム、医療画像処理方法及びプログラム Ceased WO2021044910A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
EP20860022.1A EP4026479A4 (en) 2019-09-03 2020-08-25 MEDICAL IMAGE PROCESSING DEVICE, ENDOSCOPE SYSTEM, MEDICAL IMAGE PROCESSING METHOD AND PROGRAM
JP2021543711A JP7256275B2 (ja) 2019-09-03 2020-08-25 医療画像処理装置、内視鏡システム、医療画像処理装置の作動方法及びプログラム
CN202080058661.3A CN114269221B (zh) 2019-09-03 2020-08-25 医疗图像处理装置、内窥镜系统、医疗图像处理方法以及记录介质
US17/591,976 US12274416B2 (en) 2019-09-03 2022-02-03 Medical image processing apparatus, endoscope system, medical image processing method, and program

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2019-160463 2019-09-03
JP2019160463 2019-09-03

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/591,976 Continuation US12274416B2 (en) 2019-09-03 2022-02-03 Medical image processing apparatus, endoscope system, medical image processing method, and program

Publications (1)

Publication Number Publication Date
WO2021044910A1 true WO2021044910A1 (ja) 2021-03-11

Family

ID=74852339

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/032049 Ceased WO2021044910A1 (ja) 2019-09-03 2020-08-25 医療画像処理装置、内視鏡システム、医療画像処理方法及びプログラム

Country Status (5)

Country Link
US (1) US12274416B2 (https=)
EP (1) EP4026479A4 (https=)
JP (1) JP7256275B2 (https=)
CN (1) CN114269221B (https=)
WO (1) WO2021044910A1 (https=)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020059526A1 (ja) * 2018-09-18 2020-03-26 富士フイルム株式会社 医用画像処理装置、プロセッサ装置、医用画像処理方法、及びプログラム
WO2025106737A1 (en) * 2023-11-15 2025-05-22 The Board Of Trustees Of The Leland Stanford Junior University Apparatuses and methods involving sensor-based sensory extrapolation

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006198106A (ja) 2005-01-19 2006-08-03 Olympus Corp 電子内視鏡装置
JP2006293237A (ja) * 2005-04-14 2006-10-26 Olympus Medical Systems Corp 画像表示装置
JP2007318431A (ja) * 2006-05-25 2007-12-06 Fujifilm Corp 表示制御システムおよび表示制御方法
WO2017073337A1 (ja) 2015-10-27 2017-05-04 オリンパス株式会社 内視鏡装置
WO2017216922A1 (ja) 2016-06-16 2017-12-21 オリンパス株式会社 画像処理装置及び画像処理方法
WO2018198161A1 (ja) 2017-04-24 2018-11-01 オリンパス株式会社 内視鏡画像処理装置及び内視鏡画像処理方法
WO2018216617A1 (ja) * 2017-05-25 2018-11-29 日本電気株式会社 情報処理装置、制御方法、及びプログラム
WO2019078237A1 (ja) * 2017-10-18 2019-04-25 富士フイルム株式会社 医療画像処理装置、内視鏡システム、診断支援装置、並びに医療業務支援装置

Family Cites Families (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
ATE225964T1 (de) * 1993-03-31 2002-10-15 Luma Corp Informationsverwaltung in einem endoskopiesystem
US7986337B2 (en) * 2004-09-27 2011-07-26 Given Imaging Ltd. System and method for editing an image stream captured in vivo
JP5203648B2 (ja) * 2007-07-20 2013-06-05 オリンパス株式会社 画像抽出装置および画像抽出プログラム
JP5802364B2 (ja) * 2009-11-13 2015-10-28 オリンパス株式会社 画像処理装置、電子機器、内視鏡システム及びプログラム
JP5492729B2 (ja) * 2010-09-28 2014-05-14 富士フイルム株式会社 内視鏡画像記録装置、及び内視鏡画像記録装置の作動方法、並びにプログラム
US8870751B2 (en) * 2010-09-28 2014-10-28 Fujifilm Corporation Endoscope system, endoscope image recording apparatus, endoscope image acquisition assisting method and computer readable medium
US9345389B2 (en) * 2010-11-12 2016-05-24 Emory University Additional systems and methods for providing real-time anatomical guidance in a diagnostic or therapeutic procedure
US8900131B2 (en) * 2011-05-13 2014-12-02 Intuitive Surgical Operations, Inc. Medical system providing dynamic registration of a model of an anatomical structure for image-guided surgery
US10064541B2 (en) * 2013-08-12 2018-09-04 Endochoice, Inc. Endoscope connector cover detection and warning system
WO2015029584A1 (ja) * 2013-08-30 2015-03-05 オリンパスメディカルシステムズ株式会社 画像管理装置
CN105683889B (zh) * 2013-11-21 2019-08-13 奥林巴斯株式会社 图像显示装置
EP3080998B1 (en) * 2013-12-11 2023-08-02 Given Imaging Ltd. System and method for controlling the display of an image stream
CN105992546B (zh) * 2015-01-21 2018-08-28 Hoya株式会社 内窥镜系统
JP6389136B2 (ja) * 2015-03-30 2018-09-12 富士フイルム株式会社 内視鏡撮影部位特定装置、プログラム
CN108135457B (zh) 2015-10-26 2020-02-21 奥林巴斯株式会社 内窥镜图像处理装置
WO2017122400A1 (ja) * 2016-01-15 2017-07-20 オリンパス株式会社 内視鏡画像観察支援システム
JP2018033657A (ja) * 2016-08-31 2018-03-08 コニカミノルタ株式会社 医用画像システム及びプログラム
JP2019180966A (ja) * 2018-04-13 2019-10-24 学校法人昭和大学 内視鏡観察支援装置、内視鏡観察支援方法、及びプログラム

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006198106A (ja) 2005-01-19 2006-08-03 Olympus Corp 電子内視鏡装置
JP2006293237A (ja) * 2005-04-14 2006-10-26 Olympus Medical Systems Corp 画像表示装置
JP2007318431A (ja) * 2006-05-25 2007-12-06 Fujifilm Corp 表示制御システムおよび表示制御方法
WO2017073337A1 (ja) 2015-10-27 2017-05-04 オリンパス株式会社 内視鏡装置
WO2017216922A1 (ja) 2016-06-16 2017-12-21 オリンパス株式会社 画像処理装置及び画像処理方法
WO2018198161A1 (ja) 2017-04-24 2018-11-01 オリンパス株式会社 内視鏡画像処理装置及び内視鏡画像処理方法
WO2018216617A1 (ja) * 2017-05-25 2018-11-29 日本電気株式会社 情報処理装置、制御方法、及びプログラム
WO2019078237A1 (ja) * 2017-10-18 2019-04-25 富士フイルム株式会社 医療画像処理装置、内視鏡システム、診断支援装置、並びに医療業務支援装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4026479A4

Also Published As

Publication number Publication date
CN114269221A (zh) 2022-04-01
US20220151467A1 (en) 2022-05-19
EP4026479A4 (en) 2022-11-02
JP7256275B2 (ja) 2023-04-11
US12274416B2 (en) 2025-04-15
EP4026479A1 (en) 2022-07-13
CN114269221B (zh) 2025-08-26
JPWO2021044910A1 (https=) 2021-03-11

Similar Documents

Publication Publication Date Title
JP6577088B2 (ja) 内視鏡システム
JP6834184B2 (ja) 情報処理装置、情報処理装置の作動方法、プログラム及び医療用観察システム
JP7374280B2 (ja) 内視鏡装置、内視鏡プロセッサ、及び内視鏡装置の作動方法
JP7387859B2 (ja) 医用画像処理装置、プロセッサ装置、内視鏡システム、医用画像処理装置の作動方法及びプログラム
JP7146318B1 (ja) コンピュータプログラム、学習モデルの生成方法、及び手術支援装置
JP7600250B2 (ja) 画像処理システム、プロセッサ装置、内視鏡システム、画像処理方法及びプログラム
JP6640866B2 (ja) 画像処理装置、内視鏡システム、及び画像処理方法
JPWO2018159363A1 (ja) 内視鏡システム及びその作動方法
JPWO2017057574A1 (ja) 画像処理装置、内視鏡システム、及び画像処理方法
JPWO2017057572A1 (ja) 画像処理装置、内視鏡システム、及び画像処理方法
JPWO2020184257A1 (ja) 医用画像処理装置及び方法
JP6203088B2 (ja) 生体観察システム
JP7146925B2 (ja) 医用画像処理装置及び内視鏡システム並びに医用画像処理装置の作動方法
JP7256275B2 (ja) 医療画像処理装置、内視鏡システム、医療画像処理装置の作動方法及びプログラム
JP6210923B2 (ja) 生体観察システム
WO2016121811A1 (ja) 画像処理装置及び画像処理方法、並びに内視鏡システム
CN115315210B (zh) 图像处理装置、图像处理方法、导航方法以及内窥镜系统
CN115381379A (zh) 医疗图像处理装置、内窥镜系统及医疗图像处理装置的工作方法
US12327396B2 (en) Image processing device, image processing method, and program
JP2026034758A (ja) 医療画像処理装置、医療画像処理方法、プログラム、及び記録媒体
JP2023178526A (ja) 画像処理装置、内視鏡システム、画像処理装置の作動方法、及び画像処理装置用プログラム
JP2024160185A (ja) 内視鏡システム、プロセッサ装置及びその作動方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20860022

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021543711

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2020860022

Country of ref document: EP

Effective date: 20220404

WWG Wipo information: grant in national office

Ref document number: 202080058661.3

Country of ref document: CN