WO2023089717A1

WO2023089717A1 - Information processing device, information processing method, and recording medium

Info

Publication number: WO2023089717A1
Application number: PCT/JP2021/042366
Authority: WO
Inventors: 憲一上條; 翔太大塚; 達木村; 敦丸亀; 雅弘西光; 亮作志野
Original assignee: 日本電気株式会社
Priority date: 2021-11-18
Filing date: 2021-11-18
Publication date: 2023-05-25

Abstract

An endoscopic image acquisition means acquires an endoscopic image. A still image acquisition means acquires a plurality of still images in each of which an image of a lesion included in the endoscopic image is captured. An identity determination means determines the identity of lesions respectively included in the plurality of still images. A representative image selection means selects a representative image in which the lesion is best shown among a plurality of still images that are determined as corresponding to the same lesion and then displays the selected image on a display device.

Description

Information processing device, information processing method, and recording medium

This disclosure relates to processing of information regarding endoscopy.

During an endoscopy, the doctor may take dozens of images, but if there are many images taken, it takes time to select the images to attach to the inspection report. . In addition, there is AI (Artificial Intelligence) that makes qualitative judgments on areas suspected of being lesions, but if the AI performs judgment processing every time a doctor takes an image of a suspected lesion, checking the judgment results becomes complicated. , the load of AI processing also increases. For example, Patent Literature 1 proposes a method of selecting and outputting an image of interest from a group of time-series images obtained by capturing the inside of a lumen of a subject in time series.

JP 2011-024727 A

If multiple images taken by a doctor are of the same lesion in the same area, it is desirable to limit the number of images and use them as images attached to reports or images for qualitative judgment.

One object of the present disclosure is to provide an information processing device capable of selecting images attached to reports and images used for AI qualitative judgment from a huge number of images taken in endoscopy. to do.

In one aspect of the present disclosure, an information processing device includes:
endoscopic image acquisition means for acquiring an endoscopic image;
a still image acquiring means for acquiring a plurality of still images of lesions included in the endoscopic image;
Identity determination means for determining identity of lesions included in the plurality of still images;
selecting means for selecting a representative image that best represents the lesion from a plurality of still images determined to correspond to the same lesion.

In another aspect of the present disclosure, an information processing method includes:
Acquiring endoscopic images,
Acquiring a plurality of still images of lesions included in the endoscopic image,
determining the identity of the lesions included in the plurality of still images;
A representative image that best represents the lesion is selected from a plurality of still images determined to correspond to the same lesion.

In yet another aspect of the present disclosure, the recording medium comprises
Acquiring endoscopic images,
Acquiring a plurality of still images of lesions included in the endoscopic image,
determining the identity of the lesions included in the plurality of still images;
A program is recorded that causes a computer to select a representative image that best represents the lesion from a plurality of still images determined to correspond to the same lesion.

According to the present disclosure, it is possible to select an image attached to a report or an image used for qualitative judgment of AI from among the images taken in the endoscopy.

1 is a block diagram showing a schematic configuration of an endoscopy system; FIG. 2 is a block diagram showing the hardware configuration of the image processing device; FIG. 2 is a block diagram showing the functional configuration of the image processing device; FIG. Schematic representation of the structure of the large intestine. A display example of an inspection report is shown. 4 shows another display example of an inspection report. 4 is a flowchart of display processing by the image processing device; It is a block diagram which shows the functional structure of the information processing apparatus of 2nd Embodiment. 9 is a flowchart of processing by the information processing apparatus of the second embodiment;

Preferred embodiments of the present disclosure will be described below with reference to the drawings.
<First embodiment>
[System configuration]
FIG. 1 shows a schematic configuration of an endoscopy system 100. As shown in FIG. The endoscopy system 100 acquires an image captured by an endoscopy examiner and an image captured by an AI during an examination (including treatment) using an endoscope. Then, if there are a plurality of images of the same lesion among the images taken by the examiner or AI, the endoscopy system 100 selects and displays a representative image from among them. In particular, the endoscopy system 100 of this embodiment is characterized in that an image that best represents a lesion is selected as a representative image and displayed. This saves the inspector the trouble of selecting an image to be attached to the inspection report, and enables efficient creation of the inspection report.

As shown in FIG. 1, the endoscopy system 100 mainly includes an image processing device 1, a display device 2, and an endoscope 3 connected to the image processing device 1.

The image processing apparatus 1 acquires from the endoscope 3 an image captured by the endoscope 3 during the endoscopy (hereinafter also referred to as an "endoscopic image Ic"), and performs the endoscopy. The display device 2 is caused to display display data for confirmation by the inspector. Specifically, the image processing apparatus 1 acquires, as an endoscopic image Ic, a moving image of the interior of an organ captured by the endoscope 3 during an endoscopy. Further, when the examiner finds a lesion during the endoscopy, the examiner operates the endoscope 3 to input an instruction to photograph the lesion position. The image processing apparatus 1 generates a lesion image showing a lesion position based on an imaging instruction from an examiner. Specifically, the image processing apparatus 1 generates a lesion image, which is a still image, from the endoscopic image Ic, which is a moving image, based on the imaging instruction of the examiner. At this time, if the generated lesion image is not clear, the image processing apparatus 1 may notify the examiner to input the imaging instruction again.

The display device 2 is a display or the like that performs a predetermined display based on a display signal supplied from the image processing device 1 .

The endoscope 3 mainly includes an operation unit 36 for inputting air supply, water supply, angle adjustment, imaging instruction, etc. by the examiner, and a flexible endoscope which is inserted into an organ to be examined of the examinee. a flexible shaft 37 , a distal end portion 38 containing an imaging unit such as an ultra-compact imaging device, and a connecting portion 39 for connecting to the image processing apparatus 1 .

In the following, explanation will be given mainly on the premise of processing in endoscopic examination of the large intestine, but the inspection object is not limited to the large intestine, but the gastrointestinal tract (digestive organ) such as the stomach, esophagus, small intestine, duodenum, etc. good too.

In addition, the part to be detected in the endoscopy is not limited to the lesion part, and may be any part that the examiner needs to pay attention to (also called "part of attention"). Such points of interest include a lesion site, an inflamed site, a surgical scar or other cut site, a fold or projection site, and the tip portion 38 of the endoscope 3 in the lumen. It may be a place that is easy to contact (easy to get stuck) on the wall surface of the.

[Hardware configuration]
FIG. 2 shows the hardware configuration of the image processing apparatus 1. As shown in FIG. The image processing apparatus 1 mainly includes a processor 11, a memory 12, an interface 13, an input unit 14, a light source unit 15, a sound output unit 16, and a database (hereinafter referred to as "DB") 17. ,including. Each of these elements is connected via a data bus 19 .

The processor 11 executes a predetermined process by executing a program or the like stored in the memory 12. The processor 11 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit). Note that the processor 11 may be composed of a plurality of processors. Processor 11 is an example of a computer.

The memory 12 is composed of various volatile memories used as working memory, such as RAM (Random Access Memory) and ROM (Read Only Memory), and non-volatile memory for storing information necessary for processing of the image processing apparatus 1. be done. Note that the memory 12 may include an external storage device such as a hard disk connected to or built into the image processing apparatus 1, or may include a storage medium such as a detachable flash memory or disk medium. The memory 12 stores a program for the image processing apparatus 1 to execute each process in this embodiment.

In addition, under the control of the processor 11, the memory 12 temporarily stores a series of endoscopic images Ic captured by the endoscope 3 during endoscopic examination. In addition, the memory 12 temporarily stores lesion images captured based on imaging instructions from the examiner during the endoscopy. These images are stored in the memory 12 in association with, for example, subject identification information (for example, patient ID), time stamp information, and the like.

The interface 13 performs an interface operation between the image processing device 1 and an external device. For example, the interface 13 supplies the display data Id generated by the processor 11 to the display device 2 . Also, the interface 13 supplies illumination light generated by the light source unit 15 to the endoscope 3 . The interface 13 also supplies the processor 11 with electrical signals indicating the endoscopic image Ic supplied from the endoscope 3 . The interface 13 may be a communication interface such as a network adapter for performing wired or wireless communication with an external device, and may be a hardware interface conforming to USB (Universal Serial Bus), SATA (Serial AT Attachment), or the like. may

The input unit 14 generates an input signal based on the operation of the inspector. The input unit 14 is, for example, a button, touch panel, remote controller, voice input device, or the like. The light source unit 15 generates light to be supplied to the distal end portion 38 of the endoscope 3 . The light source unit 15 may also incorporate a pump or the like for sending out water or air to be supplied to the endoscope 3 . The sound output unit 16 outputs sound under the control of the processor 11 .

The DB 17 stores endoscopic images and lesion information acquired by the subject's past endoscopic examinations. The lesion information includes lesion images and related information. The DB 17 may include an external storage device such as a hard disk connected to or built into the image processing apparatus 1, or may include a storage medium such as a detachable flash memory. Note that instead of providing the DB 17 in the endoscopy system 100, the DB 17 may be provided in an external server or the like, and related information may be acquired from the server through communication.

[Function configuration]
FIG. 3 is a block diagram showing the functional configuration of the image processing apparatus 1. As shown in FIG. The image processing device 1 functionally includes a position detection unit 21 , an inspection data generation unit 22 , an AI determination unit 23 , and a display data generation unit 24 .

An endoscope image Ic is input from the endoscope 3 to the image processing device 1 . The endoscopic image Ic is input to the position detection unit 21 , inspection data generation unit 22 and AI determination unit 23 . The position detection unit 21 detects the position of the endoscope 3, that is, the imaging position of the endoscope image, based on the endoscope image Ic. Specifically, the position detection unit 21 detects the imaging position by image analysis of the input endoscope image Ic. Here, the imaging position may be three-dimensional coordinates of the organ to be inspected, but may be information indicating at least one of a plurality of parts of the organ to be inspected. For example, as shown in FIG. 4, the large intestine is composed of multiple parts such as the cecum, ascending colon, transverse colon, descending colon, sigmoid colon, and rectum. Therefore, when the examination target is the large intestine, the position detection unit 21 should be able to detect at least which of the above parts the imaging position belongs to.

Specifically, the position detection unit 21 detects the image at that time based on the pattern of the mucous membrane, the presence or absence of folds, the shape of the folds, the number of folds passed by the movement of the endoscope 3, etc. included in the endoscopic image. It is possible to estimate which part of the large intestine the position belongs to. Further, the position detection unit 21 estimates the moving speed of the endoscope 3 based on the endoscopic image, calculates the moving distance in the large intestine based on the moving speed and time, and determines the imaging position. can be estimated. Furthermore, the position detection unit 21 may detect the imaging position using the insertion length of the endoscope 3 inserted into the organ in addition to the image analysis of the endoscopic image. In this embodiment, the method of detecting the imaging position in the organ to be inspected is not limited to a specific method. The position detection unit 21 outputs the detected imaging position to the inspection data generation unit 22 .

In addition, patient information of a patient who is a subject is input to the image processing apparatus 1 through the input unit 14 . The patient information is information that uniquely identifies a patient, and may use a patient name, an ID uniquely assigned to each patient, or a personal identification number such as my number. Patient information is input to the examination data generator 22 .

The inspection data generation unit 22 generates inspection data of the imaging position based on the endoscopic image Ic and the imaging position detected by the position detection unit 21 and temporarily stores it in the memory 12 . Specifically, when the inspector operates the endoscope 3, the inspection data generator 22 generates an endoscopic image Ic captured by the endoscope 3, the patient name, the patient ID, the date and time of the inspection, the image Inspection data including imaging information including positions and the like are generated and stored in the memory 12 . The test data generation unit 22 outputs the generated test data to the AI determination unit 23 and the display data generation unit 24 .

The AI determination unit 23 performs image analysis based on the endoscopic image Ic and determines the presence or absence of a lesion. The AI determination unit 23 uses an image recognition model or the like prepared in advance to detect a lesion-like portion included in the endoscopic image. When detecting a lesion-like portion, the AI determination unit 23 outputs the position (lesion position) and the lesion-likeness score as a determination result. On the other hand, when the AI determination unit 23 does not detect a lesion-like portion, it outputs a determination result indicating that there is no lesion.

Also, the AI determination unit 23 acquires a lesion image, which is a still image generated based on the imaging instruction of the examiner, from the test data generated by the test data generation unit 22 . Then, the AI determination unit 23 groups images related to the same lesion from among the plurality of lesion images. Specifically, the AI determination unit 23 performs pattern matching of lesion sites on a plurality of lesion images captured within a predetermined period of time, and determines whether or not the images relate to the same lesion. When the AI determination unit 23 determines that the images are related to the same lesion, the images are grouped. Note that the AI determination unit 23 may determine whether or not the images relate to the same lesion by using the above-described pattern matching and lesion position information.

Furthermore, the AI determination unit 23 selects a representative image from among the grouped images. Here, the representative image is an image that best represents the lesion in the grouped image group. Specifically, the AI determination unit 23 selects a representative image using criteria exemplified below. In the first example, the AI determination unit 23 selects the image in which the lesion is shown the largest among the grouped images as the representative image. In a second example, the AI determination unit 23 selects the image in which the lesion is most centered among the grouped images as the representative image. In the third example, the AI determination unit 23 selects the most focused image among the grouped images as the representative image. In a fourth example, the AI determination unit 23 selects the image with the highest lesion-likeness score among the grouped images as the representative image. Note that the AI determination unit 23 may select a representative image based on a combination of these criteria.

It should be noted that the representative image is not limited to one. An examiner may image a lesion using different types of illumination light during an endoscopy. Therefore, for example, when the examiner switches the illumination light of the endoscope from white light to special light and photographs the same lesion, the representative image may be selected for each type of light.

The AI determination unit 23 outputs the representative image thus selected to the display data generation unit 24 . At this time, the AI determination unit 23 may perform qualitative determination on the representative image and output the result of the qualitative determination to the display data generation unit 24 .

The lesion image is not limited to a still image generated based on the imaging instruction of the examiner, and may be a still image generated by the AI determination unit 23 by estimating the lesion site by itself based on the endoscopic image Ic. There may be.

The display data generation unit 24 generates display data Id using the inspection data input from the inspection data generation unit 22 and the representative image data input from the AI determination unit 23 , and outputs the display data Id to the display device 2 .

[Display example]
Next, display examples by the display device 2 will be described.
(inspection report)
FIG. 5 shows a display example of an inspection report. In this example, a representative image of a patient's lesion and information related to the lesion (hereinafter referred to as "related information") are displayed.

Specifically, in the display example of FIG. 5, related information 41 and a lesion image 42 are displayed in the display area 40 in addition to basic information such as the patient name, patient ID, and examination date and time. The related information 41 is information about lesions included in the lesion image 42 . The related information 41 includes lesion position, lesion size, macroscopic type (endoscopic findings), tissue type (pathological diagnosis result), treatment, and the like. A lesion image 42 is a representative image selected by the AI determination unit 23 from a plurality of images grouped as corresponding to the same lesion. That is, the representative image is the image that best represents the lesion from among the grouped images. Related information 41 and lesion image 42 are displayed for each detected lesion. The lesion displayed in this manner may be an image captured by the examiner, or may be an image captured automatically by the AI determination unit 23 .

As described above, in this embodiment, the image processing apparatus 1 selects a representative image from among a plurality of images corresponding to the same lesion captured during endoscopic examination, and attaches it to the examination report. This saves the inspector the trouble of selecting an image, and makes it possible to efficiently create an inspection report.

FIG. 6 shows another display example of an inspection report. In this example, a plurality of still images determined to be the same lesion and a representative image selected from the plurality of still images are displayed in association with the imaging time of the endoscopic image Ic. . Note that the display example of FIG. 6 is used in addition to the inspection report shown in FIG. 5 as necessary. In this example, during an endoscopy of a certain patient, there are multiple images captured by the examiner between 14:10 and 14:20, and images captured by the AI between 14:20 and 14:30. Suppose there are multiple At this time, in the display area 50, a plurality of still images and a representative image selected from the plurality of still images are displayed on the timeline indicating the elapsed time of the examination.

Specifically, the image processing apparatus 1 groups images corresponding to the same lesion among the images captured between 14:10 and 14:20. Then, the image processing apparatus 1 displays the grouped image group 51 and the representative image 53 selected from the image group 51 on the timeline. In addition, the image processing apparatus 1 groups images corresponding to the same lesion among the images captured between 14:20 and 14:30. Then, the image processing apparatus 1 displays the grouped image group 52 and the representative image 54 selected from the image group 52 on the timeline. In this way, by displaying the representative images in a manner associated with the imaging time and the group of images corresponding to the same lesion, the examiner can easily view the images captured by himself and the images captured by the AI in chronological order. It is possible to grasp the

[Image display processing]
Next, display processing for performing the display as described above will be described. FIG. 7 is a flow chart of display processing by the image processing device 1 . This processing is realized by executing a program prepared in advance by the processor 11 shown in FIG. 2 and operating as each element shown in FIG.

First, the examination data generation unit 22 acquires a lesion image taken by a doctor (step S11). Also, the AI determination unit 23 independently captures a lesion image based on the endoscopic image Ic, and the examination data generation unit 22 acquires the image captured by the AI determination unit 23 (step S12). Note that step S12 may be executed before step S11 or may be executed simultaneously with step S11.

Next, the AI determination unit 23 groups images related to the same lesion from a plurality of lesion images captured within a predetermined period of time (step S13). Next, the AI determination unit 23 determines a representative image that best represents the lesion from each grouped image group (step S14). Further, the AI determination unit 23 performs qualitative determination on the lesion of the representative image (step S15).

Next, the display data generation unit 24 uses the inspection data generated by the inspection data generation unit 22, the representative image determined by the AI determination unit 23, and the qualitative determination result of the AI determination unit 23 to generate the image shown in FIG. Display data as exemplified is generated and output to the display device 2 (step S16). The display device 2 displays the received display data (step S17). Thus, a display such as the display example of FIG. 5 is performed.

<Second embodiment>
FIG. 8 is a block diagram showing the functional configuration of the information processing apparatus according to the second embodiment. The information processing device 70 includes endoscope image acquisition means 71 , still image acquisition means 72 , identity determination means 73 , representative image selection means 74 , and display device 75 .

FIG. 9 is a flowchart of processing by the information processing apparatus of the second embodiment. The endoscopic image acquisition means 71 acquires an endoscopic image (step S71). Next, the still image acquiring means 72 acquires a plurality of still images of lesions included in the endoscopic image (step S72). The identity determining means 73 determines the identity of the lesions included in the plurality of still images (step S73). The representative image selection means 74 selects a representative image that best represents the lesion from a plurality of still images determined to correspond to the same lesion, and displays it on the display device 75 (step S74).

According to the information processing apparatus 70 of the second embodiment, it is possible to save the examiner the trouble of selecting images to be attached to the examination report after the endoscopy, and to efficiently create the examination report.

Some or all of the above embodiments can also be described as the following additional remarks, but are not limited to the following.

(Appendix 1)
endoscopic image acquisition means for acquiring an endoscopic image;
a still image acquiring means for acquiring a plurality of still images of lesions included in the endoscopic image;
Identity determination means for determining identity of lesions included in the plurality of still images;
selection means for selecting a representative image that best represents the lesion from a plurality of still images determined to correspond to the same lesion;
Information processing device.

(Appendix 2)
The information processing apparatus according to appendix 1, wherein the representative image is a still image showing the largest lesion among the plurality of still images determined to correspond to the same lesion.

(Appendix 3)
The information processing apparatus according to appendix 1, wherein the representative image is a still image in which the lesion is most centered among the plurality of still images determined to correspond to the same lesion.

(Appendix 4)
The information processing apparatus according to appendix 1, wherein the representative image is the most focused still image among the plurality of still images determined to correspond to the same lesion.

(Appendix 5)
comprising determination means for determining the likelihood of a lesion for the plurality of still images;
The information processing apparatus according to appendix 1, wherein the representative image is a still image that is determined to be most likely to be a lesion by the determining means.

(Appendix 6)
display means for displaying an image showing a plurality of still images determined to correspond to the same lesion and a representative image selected from the plurality of still images in association with the imaging time of the endoscopic image; The information processing apparatus according to any one of appendices 1 to 5, comprising:

(Appendix 7)
7. The information processing apparatus according to any one of appendices 1 to 6, wherein the plurality of still images include still images captured based on an imaging instruction by an examiner.

(Appendix 8)
7. The information processing apparatus according to any one of appendices 1 to 6, wherein the plurality of still images include still images automatically captured by AI imaging means that detects and captures a lesion in the endoscopic image. .

(Appendix 9)
Acquiring endoscopic images,
Acquiring a plurality of still images of lesions included in the endoscopic image,
determining the identity of the lesions included in the plurality of still images;
An information processing method for selecting a representative image that best represents a lesion from a plurality of still images determined to correspond to the same lesion.

(Appendix 10)
Acquiring endoscopic images,
Acquiring a plurality of still images of lesions included in the endoscopic image,
determining the identity of the lesions included in the plurality of still images;
A recording medium recording a program for causing a computer to select a representative image that best represents a lesion from a plurality of still images determined to correspond to the same lesion.

Although the present disclosure has been described above with reference to the embodiments and examples, the present disclosure is not limited to the above embodiments and examples. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

Reference Signs List 1 image processing device 2 display device 3 endoscope 11 processor 12 memory 17 database (DB)
21 position detection unit 22 inspection data generation unit 23 AI determination unit 24 display data generation unit 100 endoscopy system

Claims

endoscopic image acquisition means for acquiring an endoscopic image;
a still image acquiring means for acquiring a plurality of still images of lesions included in the endoscopic image;
Identity determination means for determining identity of lesions included in the plurality of still images;
selection means for selecting a representative image that best represents the lesion from a plurality of still images determined to correspond to the same lesion;
Information processing device.
The information processing apparatus according to claim 1, wherein the representative image is a still image showing the largest lesion among the plurality of still images determined to correspond to the same lesion.
The information processing apparatus according to claim 1, wherein the representative image is a still image in which the lesion is most centered among the plurality of still images determined to correspond to the same lesion.
The information processing apparatus according to claim 1, wherein the representative image is the most focused still image among the plurality of still images determined to correspond to the same lesion.
comprising determination means for determining the likelihood of a lesion for the plurality of still images;
2. The information processing apparatus according to claim 1, wherein the representative image is a still image that is determined to be most lesion-like by the determining means.
display means for displaying an image showing a plurality of still images determined to correspond to the same lesion and a representative image selected from the plurality of still images in association with the imaging time of the endoscopic image; The information processing apparatus according to any one of claims 1 to 5, comprising:
The information processing apparatus according to any one of claims 1 to 6, wherein the plurality of still images include still images taken based on an imaging instruction from an examiner.
7. The information processing according to any one of claims 1 to 6, wherein said plurality of still images include still images automatically captured by AI imaging means for detecting and capturing a lesion in said endoscopic image. Device.
Acquiring endoscopic images,
Acquiring a plurality of still images of lesions included in the endoscopic image,
determining the identity of the lesions included in the plurality of still images;
An information processing method for selecting a representative image that best represents a lesion from a plurality of still images determined to correspond to the same lesion.
Acquiring endoscopic images,
Acquiring a plurality of still images of lesions included in the endoscopic image,
determining the identity of the lesions included in the plurality of still images;
A recording medium recording a program for causing a computer to select a representative image that best represents a lesion from a plurality of still images determined to correspond to the same lesion.