WO2021199294A1 - 情報処理装置、表示方法、及びプログラムが格納された非一時的なコンピュータ可読媒体 - Google Patents

情報処理装置、表示方法、及びプログラムが格納された非一時的なコンピュータ可読媒体 Download PDF

Info

Publication number
WO2021199294A1
WO2021199294A1 PCT/JP2020/014850 JP2020014850W WO2021199294A1 WO 2021199294 A1 WO2021199294 A1 WO 2021199294A1 JP 2020014850 W JP2020014850 W JP 2020014850W WO 2021199294 A1 WO2021199294 A1 WO 2021199294A1
Authority
WO
WIPO (PCT)
Prior art keywords
lesion
image
detected
information processing
control unit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/014850
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
郁磨 ▲高▼橋
達 木村
公康 田光
上條　憲一
翔太 大塚
元靖 奥津
真善 山田
隆二 浜本
齋藤　豊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NEC Corp
National Cancer Center Japan
National Cancer Center Korea
Original Assignee
NEC Corp
National Cancer Center Japan
National Cancer Center Korea
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by NEC Corp, National Cancer Center Japan, National Cancer Center Korea filed Critical NEC Corp
Priority to JP2022513004A priority Critical patent/JP7448923B2/ja
Priority to PCT/JP2020/014850 priority patent/WO2021199294A1/ja
Priority to US17/915,946 priority patent/US12419492B2/en
Priority to EP20929609.4A priority patent/EP4129151A4/en
Publication of WO2021199294A1 publication Critical patent/WO2021199294A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
    • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
    • A61B1/000094—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope extracting biological structures
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
    • A61B1/045—Control thereof
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00043—Operational features of endoscopes provided with output arrangements
    • A61B1/00045—Display arrangement
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00043—Operational features of endoscopes provided with output arrangements
    • A61B1/00055—Operational features of endoscopes provided with output arrangements for alerting the user
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
    • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
    • A61B1/000096—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope using artificial intelligence
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00043—Operational features of endoscopes provided with output arrangements
    • A61B1/00045—Display arrangement
    • A61B1/0005—Display arrangement combining images e.g. side-by-side, superimposed or tiled

Definitions

  • the present disclosure relates to an information processing device, a display method, and a non-temporary computer-readable medium in which a program is stored.
  • Patent Document 1 discloses an image display device that displays an image taken by a capsule endoscope. This image display device displays a slider indicating the shooting time of the image currently displayed in the main image display area on the time bar indicating the imaging period of the capsule endoscope.
  • Patent Document 1 does not assume that the user confirms the lesion detected by the detection process using an endoscope. Therefore, in the technique, the lesion detected by the detection process is not assumed. I can't figure out how far before the shooting time of.
  • the present disclosure provides an information processing device, a display method, and a program that can easily grasp during an examination how far before the imaging time of a lesion detected by a detection process on an image captured by an endoscope is from the present.
  • the purpose is to do.
  • the information processing device is An image acquisition unit that sequentially acquires the images currently taken by the endoscope, and A lesion detection unit that sequentially performs a lesion site detection process on images sequentially acquired by the image acquisition unit, and a lesion detection unit. It has a display control unit that displays on a display device the degree of passage of time up to the current time point at the time of taking the image in which the lesion site is detected by the lesion detection unit.
  • the program according to the third aspect of the present disclosure is An image acquisition step to sequentially acquire the images currently taken by the endoscope, and A lesion detection step in which lesion site detection processing is sequentially performed on sequentially acquired images, and At the time of taking the image in which the lesion site is detected, the computer is made to perform a display control step of displaying the degree of passage of time up to the present time on the display device.
  • an information processing device a display method, and a program that can easily grasp during an examination how far before the imaging time of a lesion detected by a detection process on an image captured by an endoscope is from the present. can.
  • FIG. It is a block diagram which shows an example of the structure of the information processing apparatus which concerns on Embodiment 1.
  • FIG. It is a block diagram which shows the structure of the inspection support system which concerns on Embodiment 2.
  • FIG. It is a schematic diagram which shows an example of the display screen generated by a processor apparatus. It is a block diagram which shows an example of the functional structure of the information processing apparatus which concerns on Embodiment 2.
  • FIG. It is a schematic diagram which shows the display example based on the control of a display control unit.
  • It is a schematic diagram which shows an example of the hardware configuration of the information processing apparatus which concerns on Embodiment 2.
  • FIG. It is a flowchart which shows an example of the operation of the information processing apparatus during inspection by an endoscope.
  • It is a schematic diagram which shows the display example based on the control of a display control unit.
  • FIG. 1 is a block diagram showing an example of the configuration of the information processing apparatus 1 according to the first embodiment.
  • the information processing device 1 is a device that supports an examination by a user (for example, a doctor) using an endoscope, and performs a lesion detection process using an image taken by the endoscope and a display process based on the detection result.
  • the information processing device 1 includes an image acquisition unit 2, a lesion detection unit 3, and a display control unit 4.
  • the image acquisition unit 2 sequentially acquires the images currently taken by the endoscope. Specifically, the image acquisition unit 2 sequentially acquires each frame image constituting the moving image taken by the endoscope.
  • the lesion detection unit 3 sequentially performs a lesion site detection process on the images sequentially acquired by the image acquisition unit 2.
  • the lesion detection unit 3 performs an arbitrary image recognition process to detect the lesion site captured in the image.
  • the lesion refers to an abnormality in a living tissue caused by a disease or the like, and includes, but is not limited to, for example, a polyp or a tumor.
  • the display control unit 4 controls the display of information on the display device. Specifically, the display control unit 4 displays on the display device the degree of passage of time up to the current time point at the time of taking an image in which the lesion site is detected by the lesion detection unit 3. This degree of progress can be said to be an index indicating how long ago the lesion was detected in the image taken from the present. Any display mode can be adopted as the display mode of the degree of progress. As will be described in the embodiment described later, the display control unit 4 may graphically display the degree of elapsed time, or may display a numerical value indicating the elapsed time up to the current time point.
  • the display control unit 4 controls to display the degree of passage of time up to the current time point at the time of taking an image in which the lesion site is detected by the lesion detection unit 3. Therefore, the user can easily grasp when the lesion detected by the detection process was photographed during the examination. That is, according to the information processing device 1 or the display method realized by the above processing, it is being inspected how far before the imaging time of the lesion detected by the detection processing on the image captured by the endoscope is from the present. Can be easily grasped.
  • the information processing device 1 includes a processor and a memory as a configuration (not shown).
  • the processor reads a computer program in which the above-mentioned processing of the information processing apparatus 1 is implemented from the memory, and executes the computer program. As a result, the processor realizes the functions of the image acquisition unit 2, the lesion detection unit 3, and the display control unit 4.
  • the image acquisition unit 2, the lesion detection unit 3, and the display control unit 4 may each be realized by dedicated hardware.
  • a part or all of each component of each device may be realized by a general-purpose or dedicated circuitry, a processor, or a combination thereof. These may be composed of a single chip or may be composed of a plurality of chips connected via a bus.
  • a part or all of each component of each device may be realized by a combination of the above-mentioned circuit or the like and a program.
  • a processor a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like can be used as a processor.
  • each component of the information processing device 1 when a part or all of each component of the information processing device 1 is realized by a plurality of information processing devices and circuits, the plurality of information processing devices and circuits may be centrally arranged. It may be distributed.
  • the information processing device, the circuit, and the like may be realized as a form in which each of the client-server system, the cloud computing system, and the like is connected via a communication network.
  • the function of the information processing device 1 may be provided in the SaaS (Software as a Service) format.
  • FIG. 2 is a block diagram showing a configuration of the inspection support system 10 according to the second embodiment.
  • the inspection support system 10 includes an endoscope system 100, an information processing device 200, a display device 300, and a speaker 400.
  • the endoscopic system 100 is used to inspect the lumen in the living body of the subject to be inspected. For example, it is used to examine the large intestine, but it may also be used to examine other gastrointestinal tracts and the like.
  • the endoscope system 100 includes an endoscope 110, a light source device 120, a processor device 130, and a display device 140.
  • the endoscope 110 is optically connected to the light source device 120 and further electrically connected to the processor device 130.
  • the endoscope 110 has an insertion unit 111 to be inserted into the body of a person to be inspected, and an operation unit 112 for operating the direction of the tip of the insertion unit 111.
  • the endoscope 110 is provided with an imaging unit 113 for photographing the inside of the body.
  • the image pickup unit 113 includes, for example, various lenses, an image pickup sensor, a signal processing circuit, and the like.
  • As the image pickup sensor for example, a sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor) is used.
  • the various lenses and the image sensor are arranged at the tip of the insertion unit 111, for example, and the other signal processing circuits are arranged at the operation unit 112, for example.
  • the imaging unit 113 outputs an image signal of the captured image to the processor device 130 under the control of the processor device 130.
  • the insertion portion 111 Inside the insertion portion 111, a light guide that propagates the illumination light from the light source device 120 to the tip of the insertion portion 111 is provided, and the inside of the body can be illuminated by the illumination light. Further, the insertion portion 111 is provided with a treatment tool insertion passage for guiding a treatment tool such as an electronic knife from the operation portion 112 to the tip of the insertion portion 111. Therefore, the user (doctor) can excise the lesion site or the like with the treatment tool while looking at the image taken by the endoscope 110. Further, the insertion portion 111 is provided with a nozzle for injecting air, water, or the like from the tip of the insertion portion 111.
  • a treatment tool insertion passage for guiding a treatment tool such as an electronic knife from the operation portion 112 to the tip of the insertion portion 111. Therefore, the user (doctor) can excise the lesion site or the like with the treatment tool while looking at the image taken by the endoscope 110. Further
  • the light source device 120 is a device that supplies illumination light to the above-mentioned light guide provided in the endoscope 110 under the control of the processor device 130.
  • the illumination light output from the light source device 120 passes through the light guide and is emitted from the tip of the endoscope 110. As a result, the observation site in the body is irradiated.
  • the processor device 130 is electrically connected to the endoscope 110, the light source device 120, the display device 140, and the information processing device 200.
  • the processor device 130 comprehensively controls the operation of the endoscope system 100.
  • the processor device 130 performs predetermined image processing on the image signal received from the endoscope 110 to generate a captured image to be displayed on the display device 140.
  • the processor device 130 arranges the photographed image in the photographed image area 50, and arranges characters or images of various reference information such as information of the inspection target person in the non-photographed image area 51.
  • Image 52 is generated.
  • the display image 52 is an image displayed on the entire screen of the display device 140.
  • the processor device 130 controls the display image 52 to be displayed on the display device 140.
  • the processor device 130 outputs the display image 52 to the information processing device 200.
  • the processor device 130 may output the captured image to the information processing device 200.
  • the processor device 130 outputs the display image 52 or the captured image to the information processing device 200, so that the current captured image by the endoscope 110 is sequentially output to the information processing device 200.
  • the processor device 130 outputs to the information processing device 200 in real time a moving image taken by the endoscope 110, that is, a series of images taken continuously in time of the lumen in the living body.
  • the processor device 130 includes, for example, a memory and a processor such as a CPU and a GPU, and the processor reads software (computer program) including one or more instructions from the memory and executes it to realize the processing of the processor device 130. do.
  • software computer program
  • the display device 140 displays the display image 52 generated by the processor device 130.
  • the display device 140 is a platform panel display such as a liquid crystal display, a plasma display, or an organic EL (Electro-Luminescence) display.
  • the display device 300 is a device that is electrically connected to the information processing device 200 and displays an image under the control of the information processing device 200.
  • the display device 300 is a platform panel display such as a liquid crystal display, a plasma display, or an organic EL display.
  • the speaker 400 is electrically connected to the information processing device 200, and outputs sound under the control of the information processing device 200.
  • the information processing device 200 corresponds to the information processing device 1 of FIG. 1 and is a device that supports the inspection of a user (for example, a doctor). The details of the information processing apparatus 200 will be described below.
  • FIG. 4 is a block diagram showing an example of the functional configuration of the information processing device 200.
  • the information processing apparatus 200 includes an image acquisition unit 210, a lesion detection unit 220, a sound control unit 230, and a display control unit 240.
  • the image acquisition unit 210 corresponds to the image acquisition unit 2 in FIG. 1, and sequentially acquires the images currently captured by the endoscope 110. More specifically, the image acquisition unit 210 sequentially acquires a series of images in which the lumen in the living body is continuously photographed in real time. That is, the image acquisition unit 210 sequentially acquires each frame image (still image) constituting the captured moving image in real time. When the output from the processor device 130 is the display image 52, the image acquisition unit 210 performs a process of cutting out the captured image arranged in the captured image area 50 from the display image 52.
  • the lesion detection unit 220 corresponds to the lesion detection unit 3 in FIG. 1, and sequentially performs a lesion site detection process on the images sequentially acquired by the image acquisition unit 210. That is, the lesion site is sequentially detected for each frame image constituting the captured moving image.
  • the lesion detection unit 220 performs real-time detection processing of the lesion site. For example, it is preferable that the lesion detection unit 220 performs the detection process at a processing speed faster than the frame rate of the captured moving image.
  • the lesion detection unit 220 performs a known image recognition process to detect the lesion site from the image. In this detection process, the position of the lesion site in the image is also detected.
  • the lesion detection unit 220 performs a lesion site detection process by inputting an image acquired by the image acquisition unit 210 into a model learned in advance by a machine learning algorithm.
  • This model is, for example, a model learned by deep learning such as CNN (Convolution Neural Network), but may be a model learned by using another machine learning algorithm.
  • the lesion detection unit 220 displays the image based on whether or not the index value (accuracy) indicating the probability that the lesion site is captured in the image, which is output from the above-mentioned model, exceeds a predetermined threshold value. Determine if the lesion site is visible.
  • an image including a lesion site that is, an image in which a lesion site is detected is also referred to as a lesion image.
  • the lesion detection unit 220 detects the lesion site, the lesion image, the position information in the image of the detected lesion site in the image, the above-mentioned index value, and the time when the lesion image is taken are stored in a storage device such as a memory 291 described later.
  • Information (hereinafter referred to as shooting time information) representing the image is stored.
  • the shooting time information may be any information that can identify how far the time when the shooting was performed is in the past from the current time.
  • the shooting time information is the system time at the time when shooting is performed.
  • the captured image is acquired by the image acquisition unit 210 in real time, and the lesion detection process is performed in real time.
  • the lesion detection unit 220 may use the time when the information processing apparatus 200 acquires the image as the image acquisition time point information, or the lesion detection unit 220 may use the time when the lesion detection process is performed as the image acquisition time point information. May be.
  • the shooting time information may be attached to the image data by the processor device 130 as incidental information. As described above, the shooting time information may be information that can identify how far the shooting time is from the current time, so the frame number of the moving image is used instead of the system time. May be good. This is because, which is the captured image in which the lesion was detected based on the frame number of the latest captured image (current captured image), the frame number of the past captured image in which the lesion was detected, and the frame rate of the moving image. This is because it is possible to identify whether the image was taken in the past.
  • the sound control unit 230 controls the output of the speaker 400.
  • the sound control unit 230 outputs a sound notifying the detection from the speaker 400. This makes it possible to notify the user of the detection of the lesion site. For example, a user who hears this sound can search for a lesion site detected by the information processing apparatus 200 by adjusting the imaging position of the endoscope 110.
  • the display device 300 displays an index indicating how long before the present time the lesion site was detected in the image taken. Therefore, by confirming this, the user can easily grasp how much the position of the endoscope 110 should be returned from the current position. Therefore, by performing the display control described later, the user can easily adjust the position of the endoscope 110 in the living body so that the detected lesion is photographed again.
  • the display control unit 240 controls the display of the display device 300.
  • the display control unit 240 corresponds to the display control unit 4 of FIG. Therefore, in particular, the display control unit 240 displays on the display device 300 the degree of passage of time up to the current time point with respect to the time point of photographing the lesion image found by the processing of the lesion detection unit 220.
  • the display control unit 240 specifies the degree of progress of the lesion image by referring to the information at the time of photographing the lesion image and calculating the elapsed time since the lesion image was photographed.
  • FIG. 5 is a schematic view showing a display example of the display device 300 based on the control of the display control unit 240.
  • the display control of the display control unit 240 of the present embodiment will be described with reference to FIG.
  • the display control unit 240 displays the image 61 in which the lesion site is detected by the lesion detection unit 220 in the lesion image area 60 on the screen of the display device 300.
  • the latest detected lesion image is displayed in the lesion image area 60.
  • the display control unit 240 further displays the position of the lesion site detected by the lesion detection unit 220 in the image 61.
  • the display control unit 240 displays the frame 62 surrounding the lesion site on the image 61.
  • the display control unit 240 determines the display position of the frame 62 based on the position information of the lesion site output by the lesion detection unit 220. Since the display indicating the position of the lesion site is performed in this way, the user can easily confirm where the information processing apparatus 200 has detected as the lesion.
  • the display control unit 240 displays the current captured image by the endoscope 110 sequentially acquired from the processor device 130 by the image acquisition unit 210 in the captured video area 63 on the screen of the display device 300.
  • the lesion image in the lesion image region 60 and the image currently displayed by the endoscope 110 are displayed side by side on the display device 300.
  • the display control unit 240 displays the degree of passage of time for the lesion image identified during the examination in the time passage area 64 on the screen of the display device 300.
  • the display control unit 240 displays a mark 66 indicating the time of taking an image in which the lesion site is detected by the lesion detection unit 220 on the time axis 65 starting from the current time point. Display the degree of progress.
  • the mark 66 is a circle mark, but a mark of another design may be used.
  • One end 65a of the time axis 65 represents the current time point
  • the other end 65b of the time axis 65 represents a time point that goes back by a predetermined period from the current time point.
  • the time from the present to 5 seconds before is represented by the time axis 65, but the predetermined period is not limited to 5 seconds and any time can be set.
  • each mark 66 represents the time when the lesion image was taken. As the time advances, the time point at which the lesion image is taken shifts to a point in the distant past. Therefore, the display control unit 240 moves the display position of the mark 66 on the time axis 65 with the transition of the time. In the example shown in FIG.
  • the display control unit 240 displays the mark 66 so as to flow downward along the time axis 65 as the time advances.
  • the mark 66 indicating the time of imaging of the lesion image is displayed on the time axis 65, the degree of passage of time at the time of imaging is graphically displayed. Therefore, the user can easily visually grasp the degree of passage of time when the lesion image is taken.
  • the display control unit 240 is between the images sequentially acquired by the image acquisition unit 210 from the current time point to a time point preceding a predetermined period (5 seconds as an example in FIG. 5).
  • the progress of the captured image is displayed. That is, the display control unit 240 does not display the degree of progress for all the lesions detected during the examination, but displays the degree of progress for the lesions detected in the latest period. If the degree of progress is displayed for the entire period from the start of the inspection to the present time, a large amount of information will be displayed on a limited screen size, and the visibility of the information will be reduced.
  • the degree of progress only for the lesions detected in the latest period it is possible to easily display the degree of progress of the lesions detected in the latest period.
  • the need for the user to confirm the lesion detected by the information processing apparatus 200 using the endoscope 110 that is, the need for adjusting the position of the endoscope 110 so that the lesion is photographed again is this lesion. Often occurs immediately after is detected. Therefore, it is possible to improve the convenience of the user by displaying the elapsed time only in the latest period.
  • the display control unit 240 may display the degree of progress for the entire period from the start of the inspection to the present time.
  • the display control unit 240 may display the mark 66 according to the accuracy of detection in the detection process. Specifically, the display control unit 240 displays the mark 66 according to the index value of the image determined to be the lesion image by the lesion detection unit 220. That is, the display control unit 240 may display the mark 66 having a different design depending on the index value. As described above, the index value represents the probability that the lesion site is captured in the image, and is output by the lesion detection unit 220. For example, the display control unit 240 may display the marks 66 of different colors according to the index value. The shape or pattern may be changed instead of the color. By doing so, the user can easily grasp the reliability of the detection result by the information processing apparatus 200.
  • the display control unit 240 may change the display mode of the mark 66 depending on other factors, not limited to the accuracy of detection. For example, the display control unit 240 may display the mark 66 according to the size of the lesion site detected by the lesion detection unit 220 or the diagnosis result (whether benign or not, or the degree of malignancy, etc.).
  • the display control unit 240 may display a mark 66 according to the number of detected lesion sites. That is, the display control unit 240 may display the mark 66 having a different design depending on the number of detected lesion sites. Further, the display control unit 240 may display the value of the number of detected lesion sites in the vicinity of the mark 66. By doing so, the user can easily grasp the number of detected lesion sites.
  • FIG. 6 is a schematic diagram showing an example of the hardware configuration of the information processing apparatus 200.
  • the information processing apparatus 200 includes an input / output interface 290, a memory 291 and a processor 292.
  • the input / output interface 290 is an input / output circuit for communicating with any other device such as the processor device 130, the display device 300, and the speaker 400.
  • the memory 291 is composed of, for example, a combination of a volatile memory and a non-volatile memory.
  • the memory 291 is used to store software (computer program) including one or more instructions executed by the processor 292, data used for various processes of the information processing apparatus 200, and the like.
  • the processor 292 reads software (computer program) from the memory 291 and executes it to process each component shown in FIG. Specifically, the processor 292 processes the image acquisition unit 210, the lesion detection unit 220, the sound control unit 230, and the display control unit 240.
  • the processor 292 may be, for example, a CPU, a GPU, or the like. Processor 292 may include a plurality of processors. As described above, the information processing apparatus 200 has a function as a computer.
  • Non-transitory computer-readable media include various types of tangible storage media (tangible storage media).
  • Examples of non-temporary computer-readable media include magnetic recording media (eg, flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (eg, magneto-optical disks), CD-ROMs (Read Only Memory) CD-Rs, CDs. -R / W, including semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)).
  • the program may also be supplied to the computer by various types of temporary computer-readable media.
  • Examples of temporary computer-readable media include electrical, optical, and electromagnetic waves.
  • the temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.
  • FIG. 7 is a flowchart showing an example of the operation of the information processing apparatus 200 during the inspection by the endoscope 110.
  • an operation example will be described with reference to the flowchart of FIG. 7.
  • step S100 the image acquisition unit 210 acquires the current captured image by the endoscope 110 output from the processor device 130.
  • step S101 the lesion detection unit 220 performs a lesion detection process on the image acquired in step S100.
  • the process proceeds to step S103. If no lesion is detected in the image (No in step S102), the process proceeds to step S104.
  • step S103 the sound control unit 230 outputs a sound notifying that the lesion has been detected from the speaker 400. After step S103, the process proceeds to step S104.
  • step S104 the display control unit 240 calculates the elapsed time from the acquisition of the lesion image that has already been detected between the start of the examination and the present. As a result, the display control unit 240 specifies the degree of passage of time for the detected lesion image.
  • step S105 the display control unit 240 determines the lesion image of the target to be displayed with the degree of progress from the lesion images already detected from the start of the examination to the present. Specifically, the display control unit 240 sets the lesion image whose elapsed time is within the range of the time axis 65 as the display target of the elapsed time. That is, the display control unit 240 sets the lesion image taken between the current time point and the previous time point by a predetermined period (5 seconds as an example in FIG. 5) as the display target of the degree of progress.
  • step S106 the display control unit 240 determines the display content of the display device 300. Specifically, the display control unit 240 displays the latest detected lesion image in the lesion image area 60, and displays a frame 62 surrounding the lesion site on the lesion image. In addition, the display control unit 240 displays the currently captured image by the endoscope 110 in the captured image area 63. Then, the display control unit 240 displays the elapsed degree of the lesion image determined as the display target of the elapsed degree in step S105 in the time elapsed area 64. After step S106, the process returns to step S100, and the above-described process is performed on the next image.
  • the display control unit 240 uses the time axis 65 and the mark 66 to determine the degree of passage of time from the time of taking the image in which the lesion site is detected by the lesion detection unit 220 to the current time point. indicate. Therefore, since the degree of progress is displayed graphically, the user can easily visually grasp when the lesion detected by the detection process of the information processing apparatus 200 was photographed during the examination. Can be done. That is, the user can easily visually grasp during the examination how far before the imaging time of the lesion detected by the detection process of the information processing apparatus 200 is from the present. Therefore, the user can easily guess how much the endoscope 110 should be moved in order to re-photograph the lesion with the endoscope 110.
  • the display control unit 240 displays one lesion image in the lesion image area 60 on the screen of the display device 300, but a plurality of lesion images may be displayed.
  • the points different from those of the second embodiment will be described, and the description of the overlapping configuration and processing will be omitted.
  • the display control unit 240 displays the plurality of lesion images in the lesion image area 60 as shown in FIG. ..
  • the display control unit 240 may set an upper limit on the number of lesion images to be displayed. For example, the display control unit 240 may display a maximum of N (where N is a positive integer) lesion images detected most recently. When the value of N is 1, one lesion image is displayed as in the second embodiment.
  • the display control unit 240 may display a line 67 that associates the position in the image of the lesion site detected by the lesion detection unit 220 with the mark 66. By doing so, the degree of progress of each lesion image can be easily determined.
  • the line 67 that associates the position of the lesion site in the image with the mark 66 may be displayed regardless of the number of displayed lesion images. That is, the line 67 may be displayed even when only one lesion image is displayed. Further, when a plurality of lesion sites are detected in one lesion image, a line 67 may be displayed for each lesion site.
  • the fourth embodiment will be described.
  • the lesion detection unit 220 detects the same lesion site in a plurality of images
  • the degree of progress can be displayed for each image. Therefore, the visibility of the degree of progress may decrease. Therefore, in the present embodiment, the degree of progress is not displayed for each image for the same lesion site.
  • the information processing device 200 is replaced by the information processing device 500.
  • FIG. 9 is a block diagram showing an example of the functional configuration of the information processing apparatus 500 according to the fourth embodiment.
  • the information processing device 500 is different from the information processing device 200 in that it further includes an image selection unit 250.
  • the processing of the image selection unit 250 is realized, for example, by the processor 292 reading software (computer program) from the memory 291 and executing the processing.
  • the image selection unit 250 identifies a plurality of images in which the same lesion site is detected by the detection process of the lesion detection unit 220, and one image (hereinafter, referred to as a representative image) from the specified plurality of images. Select. That is, the image selection unit 250 identifies a plurality of lesion images in which the same lesion site is detected, and selects a representative image from them. For example, the image selection unit 250 identifies a plurality of lesion images in which the same lesion site is detected by comparing the lesion images with each other. More specifically, the image selection unit 250 performs object tracking processing using the feature points of the lesion site on the continuous captured images (frame images) constituting the captured moving image, thereby performing the same lesion site.
  • the image selection unit 250 selects a representative image from a plurality of lesion images in which the same lesion site is detected.
  • the image selection unit 250 selects, for example, a lesion image having the highest detection accuracy in the detection process from among these plurality of lesion images.
  • an image in which the lesion site can be easily seen may be selected from among these images.
  • the image selection unit 250 may select an image in which the position of the lesion site is closest to the center in the image as an image in which the lesion site is easy to see, or an image in which the contrast between the lesion site and the non-lesion site is maximum. May be selected.
  • the image selection unit 250 may specify a plurality of lesion images in which the same lesion site is detected by performing a process of calculating the similarity of images instead of the object tracking process. In this case, it is possible to identify a plurality of images showing the same lesion site even if the same lesion site is not continuously photographed in time.
  • the display control unit 240 of the present embodiment sets only the lesion image selected by the image selection unit 250 as a display target of the degree of passage of time for a plurality of lesion images in which the same lesion site is detected. do. That is, the display control unit 240 does not display the degree of passage of time for the lesion images that are not selected by the image selection unit 250 among the plurality of lesion images in which the same lesion site is detected. More specifically, in the present embodiment, the display control unit 240 is a lesion image whose elapsed time is within the range of the time axis 65, and satisfies either (1) or (2) below. The lesion image is used as a display target for the degree of progress.
  • the display control unit 240 of the present embodiment displays only the lesion image selected by the image selection unit 250 in the lesion image area 60 for a plurality of lesion images in which the same lesion site is detected. And. That is, the display control unit 240 does not display the lesion image not selected by the image selection unit 250 among the plurality of lesion images in which the same lesion site is detected in the lesion image area 60. More specifically, in the present embodiment, the display control unit 240 has a maximum of N most recently detected lesion images (where N is positive) among the lesion images satisfying either (1) or (2) below. The lesion image (integer) is displayed.
  • the embodiment 4 has been described above. According to the present embodiment, even when the lesion detection unit 220 detects the same lesion site in a plurality of images, the image selection unit 250 selects an image to be displayed. Therefore, the deterioration of the visibility of the display is suppressed.
  • the inspection support system 10 includes the display device 140 and the display device 300, but the display device 140 may be omitted.
  • the display control unit 240 displays the image in which the lesion site is detected in the lesion image area 60, and displays the current image captured by the endoscope 110 in the captured image area 63. The display may be omitted for either or both of these.
  • (Appendix 1) An image acquisition unit that sequentially acquires the images currently taken by the endoscope, and A lesion detection unit that sequentially performs a lesion site detection process on images sequentially acquired by the image acquisition unit, and a lesion detection unit.
  • An information processing device having a display control unit that displays on a display device the degree of passage of time up to the current time point at the time of taking the image in which the lesion site is detected by the lesion detection unit.
  • the display control unit displays the degree of progress by displaying a mark indicating the time of taking the image in which the lesion site was detected by the lesion detection unit on the time axis starting from the current time point.
  • Appendix 6 It further has an image selection unit that identifies a plurality of images in which the same lesion site is detected in the detection process and selects a representative image from the specified plurality of images. For the plurality of images in which the same lesion site is detected, the display control unit sets only the image selected by the image selection unit as the display target of the elapsed degree.
  • the information processing device described in the section. (Appendix 7) The information processing device according to any one of Supplementary note 1 to 6, wherein the display control unit further displays the image in which the lesion site is detected by the lesion detection unit. (Appendix 8) The information processing device according to Appendix 7, wherein the display control unit further displays the position of the lesion site detected by the lesion detection unit in the image.
  • Appendix 9 The information processing device according to Appendix 7 or 8, wherein the display control unit further displays an image currently captured by the endoscope, which is sequentially acquired by the image acquisition unit.
  • Appendix 10 The display control unit The degree of progress is displayed by displaying a mark indicating the time of taking the image in which the lesion site is detected by the lesion detection unit on the time axis starting from the current time point. The image in which the lesion site is detected by the lesion detection unit is displayed, and the image is displayed.
  • the information processing apparatus according to Appendix 1 which displays a line corresponding to the position in the image of the lesion site detected by the lesion detection unit and the mark.
  • Information processing device 2 Image acquisition unit 3 Disease detection unit 4 Display control unit 10 Inspection support system 65 Time axis 66 Mark 67 Line 100 Endoscope system 110 Endoscope 111 Insertion unit 112 Operation unit 113 Imaging unit 120 Light source device 130 Processor Device 140 Display device 200 Information processing device 210 Image acquisition unit 220 Disease detection unit 230 Sound control unit 240 Display control unit 250 Image selection unit 300 Display device 400 Speaker 500 Information processing device

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Surgery (AREA)
  • Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Pathology (AREA)
  • Radiology & Medical Imaging (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Biophysics (AREA)
  • Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Animal Behavior & Ethology (AREA)
  • General Health & Medical Sciences (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Signal Processing (AREA)
  • Endoscopes (AREA)
PCT/JP2020/014850 2020-03-31 2020-03-31 情報処理装置、表示方法、及びプログラムが格納された非一時的なコンピュータ可読媒体 Ceased WO2021199294A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
JP2022513004A JP7448923B2 (ja) 2020-03-31 2020-03-31 情報処理装置、情報処理装置の作動方法、及びプログラム
PCT/JP2020/014850 WO2021199294A1 (ja) 2020-03-31 2020-03-31 情報処理装置、表示方法、及びプログラムが格納された非一時的なコンピュータ可読媒体
US17/915,946 US12419492B2 (en) 2020-03-31 2020-03-31 Information processing device, display method, and non-transitory computer-readable medium for storing a program for lesion detection processing supporting decision making using machine learning
EP20929609.4A EP4129151A4 (en) 2020-03-31 2020-03-31 Information processing device, display method, and non-transitory computer-readable medium having program stored therein

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/014850 WO2021199294A1 (ja) 2020-03-31 2020-03-31 情報処理装置、表示方法、及びプログラムが格納された非一時的なコンピュータ可読媒体

Publications (1)

Publication Number Publication Date
WO2021199294A1 true WO2021199294A1 (ja) 2021-10-07

Family

ID=77927055

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/014850 Ceased WO2021199294A1 (ja) 2020-03-31 2020-03-31 情報処理装置、表示方法、及びプログラムが格納された非一時的なコンピュータ可読媒体

Country Status (4)

Country Link
US (1) US12419492B2 (https=)
EP (1) EP4129151A4 (https=)
JP (1) JP7448923B2 (https=)
WO (1) WO2021199294A1 (https=)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2025012557A (ja) * 2023-07-13 2025-01-24 キヤノン株式会社 医療システムとその制御方法、及びプログラム

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006061469A (ja) 2004-08-27 2006-03-09 Olympus Corp 画像表示装置および画像表示プログラム
JP2015173921A (ja) * 2014-03-17 2015-10-05 オリンパス株式会社 画像処理装置、画像処理方法、及び画像処理プログラム
WO2017216922A1 (ja) * 2016-06-16 2017-12-21 オリンパス株式会社 画像処理装置及び画像処理方法
WO2019088121A1 (ja) * 2017-10-30 2019-05-09 公益財団法人がん研究会 画像診断支援装置、資料収集方法、画像診断支援方法および画像診断支援プログラム
WO2020021864A1 (ja) * 2018-07-27 2020-01-30 富士フイルム株式会社 医療画像処理装置
WO2020039685A1 (ja) * 2018-08-20 2020-02-27 富士フイルム株式会社 医療画像処理装置
WO2020039929A1 (ja) * 2018-08-23 2020-02-27 富士フイルム株式会社 医用画像処理装置及び内視鏡システム並びに医用画像処理装置の作動方法
WO2020054604A1 (ja) * 2018-09-11 2020-03-19 日本電気株式会社 情報処理装置、制御方法、及びプログラム

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008301968A (ja) * 2007-06-06 2008-12-18 Olympus Medical Systems Corp 内視鏡画像処理装置
JP2009050321A (ja) * 2007-08-23 2009-03-12 Olympus Corp 画像処理装置
JP6485694B2 (ja) * 2015-03-26 2019-03-20 ソニー株式会社 情報処理装置および方法
WO2018159363A1 (ja) * 2017-03-01 2018-09-07 富士フイルム株式会社 内視鏡システム及びその作動方法
WO2019198637A1 (ja) * 2018-04-13 2019-10-17 富士フイルム株式会社 画像処理装置、内視鏡システム、及び画像処理方法
WO2020036109A1 (ja) * 2018-08-17 2020-02-20 富士フイルム株式会社 医用画像処理装置及び内視鏡システム並びに医用画像処理装置の作動方法
CN112584739B (zh) * 2018-08-20 2024-04-05 富士胶片株式会社 医疗图像处理系统
CN112584741B (zh) * 2018-08-20 2024-10-29 富士胶片株式会社 医疗图像处理系统
EP3858222B1 (en) * 2018-09-28 2023-11-22 FUJIFILM Corporation Medical image processing device, medical image processing method, program, diagnosis assistance device, and endoscope system
CN113164023B (zh) * 2018-11-28 2024-02-23 奥林巴斯株式会社 内窥镜系统和内窥镜用图像处理方法、以及计算机可读存储介质

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006061469A (ja) 2004-08-27 2006-03-09 Olympus Corp 画像表示装置および画像表示プログラム
JP2015173921A (ja) * 2014-03-17 2015-10-05 オリンパス株式会社 画像処理装置、画像処理方法、及び画像処理プログラム
WO2017216922A1 (ja) * 2016-06-16 2017-12-21 オリンパス株式会社 画像処理装置及び画像処理方法
WO2019088121A1 (ja) * 2017-10-30 2019-05-09 公益財団法人がん研究会 画像診断支援装置、資料収集方法、画像診断支援方法および画像診断支援プログラム
WO2020021864A1 (ja) * 2018-07-27 2020-01-30 富士フイルム株式会社 医療画像処理装置
WO2020039685A1 (ja) * 2018-08-20 2020-02-27 富士フイルム株式会社 医療画像処理装置
WO2020039929A1 (ja) * 2018-08-23 2020-02-27 富士フイルム株式会社 医用画像処理装置及び内視鏡システム並びに医用画像処理装置の作動方法
WO2020054604A1 (ja) * 2018-09-11 2020-03-19 日本電気株式会社 情報処理装置、制御方法、及びプログラム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4129151A4

Also Published As

Publication number Publication date
EP4129151A1 (en) 2023-02-08
JP7448923B2 (ja) 2024-03-13
US12419492B2 (en) 2025-09-23
JPWO2021199294A1 (https=) 2021-10-07
EP4129151A4 (en) 2023-12-13
US20230337893A1 (en) 2023-10-26

Similar Documents

Publication Publication Date Title
JP4153963B2 (ja) 内視鏡挿入形状検出装置
US20120033062A1 (en) Method and device for examining or imaging an interior surface of a cavity
EP3136943A1 (en) System and method of scanning a body cavity using a multiple viewing elements endoscope
CN106659362A (zh) 图像处理装置、图像处理方法、图像处理程序以及内窥镜系统
US12488584B2 (en) Information processing device, display method, and non-transitory computer-readable medium for storing program
JP2024169523A (ja) 情報処理装置、情報処理方法、及びコンピュータプログラム
JP6840263B2 (ja) 内視鏡システム及びプログラム
US20230215059A1 (en) Three-dimensional model reconstruction
EP4111938A1 (en) Endoscope system, medical image processing device, and operation method therefor
WO2021176664A1 (ja) 検査支援システム、検査支援方法、及び、プログラム
WO2023126999A1 (ja) 画像処理装置、画像処理方法、及び、記憶媒体
JP7448923B2 (ja) 情報処理装置、情報処理装置の作動方法、及びプログラム
US20250086838A1 (en) Medical support device, endoscope apparatus, medical support method, and program
JP7823750B2 (ja) 画像処理装置、画像処理方法及びプログラム
JP2025026062A (ja) 医療支援装置、内視鏡装置、医療支援方法、及びプログラム
WO2021176665A1 (ja) 手術支援システム、手術支援方法、及び、プログラム
JP2021194268A (ja) 血管観察システムおよび血管観察方法
JP7609278B2 (ja) 画像処理装置、画像処理方法及びプログラム
US20250387008A1 (en) Medical support device, endoscope system, medical support method, and program
US20240112407A1 (en) System, methods, and storage mediums for reliable ureteroscopes and/or for imaging
US20240335093A1 (en) Medical support device, endoscope system, medical support method, and program
US20250241514A1 (en) Image display device, image display method, and recording medium
WO2024185357A1 (ja) 医療支援装置、内視鏡システム、医療支援方法、及びプログラム
WO2024190272A1 (ja) 医療支援装置、内視鏡システム、医療支援方法、及びプログラム
JP2025184545A (ja) 内視鏡検査支援装置、内視鏡検査システム、処理方法、およびプログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20929609

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022513004

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2020929609

Country of ref document: EP

Effective date: 20221031

WWG Wipo information: grant in national office

Ref document number: 17915946

Country of ref document: US