WO2023195103A1

WO2023195103A1 - Inspection assistance system and inspection assistance method

Info

Publication number: WO2023195103A1
Application number: PCT/JP2022/017187
Authority: WO
Inventors: 尚希深津; 浩正藤田; 憲輔三宅
Original assignee: オリンパスメディカルシステムズ株式会社
Priority date: 2022-04-06
Filing date: 2022-04-06
Publication date: 2023-10-12

Abstract

A display control unit 120 displays an insertion guidance pertaining to how to insert an endoscope when an insertion step is taking place in an inspection process for endoscope inspection. The display control unit 120 displays observation assistance information for providing assistance for observation when an observation step is taking place in the inspection process.

Description

Inspection support system and inspection support method

The present disclosure relates to an examination support system and an examination support method that support endoscopy.

Patent Document 1 is an endoscope system equipped with a function of detecting lesions, which determines whether the endoscopy process is an insertion process or an extraction process, and in the extraction process, the endoscope system is equipped with a function to detect lesions. An endoscope system that notifies the detection of a lesion with relatively high notification power is disclosed.

International Publication No. 2020/017212

The inspection process of colonoscopy consists of an insertion process in which the endoscope is inserted from the anus to the cecum, and an observation process in which the inside of the large intestine from the cecum to the anus is observed while withdrawing the endoscope. During the insertion process, the doctor needs to quickly and carefully insert the endoscope into the cecum in order to reduce the burden on the patient. Furthermore, in the observation process, the doctor needs to carefully observe the endoscopic image while pulling out the endoscope so as not to miss any lesions. As described above, the insertion process and observation process of endoscopy require different examination skills, and it is desired to develop a technology that effectively supports the examination work of doctors in the insertion process and observation process.

The present disclosure has been made in view of these circumstances, and its purpose is to provide a technology that supports a doctor's inspection work in the insertion process and observation process of endoscopy.

In order to solve the above problems, an inspection support system according to an aspect of the present invention includes one or more processors having hardware. The one or more processors display an insertion guide on how to insert the endoscope when the inspection process in endoscopy is an insertion process, and display an insertion guide for supporting observation when the inspection process is an observation process. Display observation support information.

Another aspect of the present invention is an inspection support method, which displays an insertion guide regarding how to insert an endoscope when the inspection process in endoscopy is an insertion process, and the inspection process is an observation process. display observation support information to support observation.

Note that any combination of the above components and the expressions of the present disclosure converted between methods, devices, systems, recording media, computer programs, etc. are also effective as aspects of the present disclosure.

1 is a diagram showing the configuration of an inspection support system according to an embodiment. FIG. 3 is a diagram showing functional blocks of a server device. FIG. 2 is a diagram showing functional blocks of an endoscopic observation device. FIG. 3 is a diagram showing a first display area where an endoscopic image is displayed in the insertion process. It is a figure which shows the 2nd display area in which an endoscopic image is displayed in an observation process. FIG. 3 is a diagram comparing a first display area and a second display area. It is a figure showing a flow chart of inspection support processing in an embodiment. FIG. 3 is a diagram showing an example of an endoscopic image displayed in a first display area. FIG. 7 is a diagram showing an example of an insertion guide displayed around the first display area. FIG. 7 is a diagram showing another example of an insertion guide displayed around the first display area. FIG. 7 is a diagram showing an example of an endoscopic image displayed in a second display area. FIG. 7 is a diagram showing an example of observation support information displayed around the second display area.

FIG. 1 shows the configuration of an inspection support system 1 according to an embodiment. The examination support system 1 is installed in a medical facility such as a hospital that performs endoscopy. In the examination support system 1, a server device 2, an image analysis device 3, an image storage device 8, an endoscope system 9, and a terminal device 10b are communicably connected via a network 4 such as a LAN (local area network). be done. The endoscope system 9 is installed in an examination room and includes an endoscope observation device 5 and a terminal device 10a. In the examination support system 1, the server device 2, image analysis device 3, and image storage device 8 may be provided outside the medical facility, for example, as a cloud server.

The endoscope observation device 5 is connected to an endoscope 7 that is inserted into the patient's digestive tract. The endoscope 7 has an insertion section that is inserted into a subject, an operation section provided on the proximal end side of the insertion section, and a universal cord extending from the operation section. The endoscope 7 is detachably connected to the endoscope observation device 5 by a scope connector provided at the end of the universal cord.

The elongated insertion section has a hard distal end, a curved section formed to be freely curved, and a flexible elongated tube section in order from the distal end to the proximal end. A plurality of magnetic coils are arranged at predetermined intervals along the longitudinal direction of the insertion section inside the distal end, curved section, and flexible tube section, and the magnetic coils are supplied from the endoscope observation device 5. generates a magnetic field according to the coil drive signal.

The endoscope 7 has a light guide for illuminating the inside of the digestive tract by transmitting the illumination light supplied from the light source section of the endoscope observation device 5, and has a distal end section that transmits illumination light that is transmitted by the light guide. An illumination window for emitting illumination light to the living tissue and a photographing unit that photographs the living tissue at a predetermined period and outputs an imaging signal to the endoscopic observation device 5 are provided. The imaging unit includes a solid-state image sensor (for example, a CCD image sensor or a CMOS image sensor) that converts incident light into an electrical signal.

The endoscopic observation device 5 generates an endoscopic image by performing image processing on the imaging signal photoelectrically converted by the solid-state image sensor of the endoscope 7, and displays the image in real time on a display device 6 provided in the examination room. indicate. In addition to normal image processing such as A/D conversion and noise removal, the endoscopic observation device 5 may have a function of performing special image processing for the purpose of highlighting and the like. The imaging frame rate of the endoscope 7 is preferably 30 fps or more, and may be 60 fps. The endoscopic observation device 5 generates endoscopic images at a cycle of the imaging frame rate. The endoscopic observation device 5 may be configured by one or more processors having dedicated hardware, but may also be configured by one or more processors having general-purpose hardware. The endoscope 7 of the embodiment is a flexible endoscope and has a forceps channel for inserting an endoscopic treatment tool. By inserting biopsy forceps into the forceps channel and manipulating the inserted biopsy forceps, the doctor can perform a biopsy and collect a portion of the diseased tissue during an endoscopy.

The doctor operates the endoscope 7 according to the examination procedure and observes the endoscopic image displayed on the display device 6. The inspection process in endoscopy consists of an "insertion process" in the outward path from when the endoscope 7 enters the inside of the subject until it turns back at a predetermined turn-around point, and an "insertion process" on the outward path after the endoscope 7 turns back at the turn-back point. , and an "observation process" on the return trip until it is pulled out of the subject. The observation process is a process of observing the inside of the large intestine while withdrawing the endoscope 7 from within the subject, and may also be referred to as a "removal process." In colonoscopy, the insertion process is a process of inserting the endoscope 7 for colon examination from the anus to the cecum, which is the turning point, and the observation process is the process of inserting the endoscope 7 for colon examination from the anus to the cecum, which is the turning point.The observation process is the process of pulling out the endoscope 7 from the cecum to the anus. This is the process of observing the inside of the large intestine.

In the observation process, when the living tissue to be captured is displayed on the display device 6, the doctor operates the release switch of the endoscope 7. The endoscopic observation device 5 captures an endoscopic image at the timing when the release switch is operated, and stores the captured endoscopic image together with information (image ID) for identifying the endoscopic image in the image storage device 8. Send to. The endoscopic observation device 5 may assign image IDs including serial numbers to endoscopic images in the order in which they are captured. Note that the endoscopic observation device 5 may send a plurality of captured endoscopic images together to the image storage device 8 after the end of the examination. The image storage device 8 records the endoscopic image transmitted from the endoscopic observation device 5 in association with the examination ID that identifies the endoscopic examination.

In the embodiment, "imaging" means an operation in which the solid-state imaging device of the endoscope 7 converts incident light into an electrical signal. Note that "imaging" may include the operation from the converted electrical signal to the endoscope observation device 5 generating an endoscopic image, and may further include the operation until displaying it on the display device 6. . In the embodiment, "capture" means an operation of acquiring an endoscopic image generated by the endoscopic observation device 5. Note that "capture" may include an operation of saving (recording) an acquired endoscopic image. In the embodiment, a photographed endoscopic image is captured when the doctor operates the release switch, but the photographed endoscopic image may be automatically captured regardless of the operation of the release switch. good.

The terminal device 10a is provided in the examination room and includes an information processing device 11a and a display device 12a. The terminal device 10a may be used by a doctor, a nurse, or the like to check information regarding the biological tissue being imaged in real time during an endoscopy. Note that in the embodiment shown below, the endoscopic image taken by the endoscope 7 is displayed on the display device 6 connected to the endoscopic observation device 5, but in another embodiment, the endoscopic image is , may be displayed on the display device 12a of the terminal device 10a.

The terminal device 10b includes an information processing device 11b and a display device 12b, and is installed in a room other than the examination room. The terminal device 10b is used by a doctor when creating a report of an endoscopy. In a medical facility, the

terminal devices

10a, 10b may be configured with one or more processors having general-purpose hardware.

In the examination support system 1 of the embodiment, the endoscopic observation device 5 displays the endoscopic image on the display device 6 in real time, and also sends the endoscopic image to the image analysis device 3 together with the meta information of the image. Supply in real time. Here, the meta information includes at least the frame number of the image and the shooting time information. The frame number may be information indicating the number of frames after the endoscope 7 starts imaging.

The image analysis device 3 is an electronic computer that analyzes endoscopic images, detects lesions included in the endoscopic images, and qualitatively diagnoses the detected lesions. The image analysis device 3 may be a CAD (computer-aided diagnosis) system having an AI (artificial intelligence) diagnosis function. The image analysis device 3 may be composed of one or more processors with dedicated hardware, but may also be composed of one or more processors with general-purpose hardware.

The image analysis device 3 performs machine learning using learning endoscopic images, information indicating organs and parts included in the endoscopic images, and information regarding lesion areas included in the endoscopic images as training data. Use the trained model generated by . Annotation work on endoscopic images is performed by an annotator with specialized knowledge such as a doctor, and a type of deep learning such as CNN, RNN, LSTM, etc. may be used for machine learning. When this trained model receives an endoscopic image, it outputs information indicating the photographed organ, information indicating the photographed region, and information regarding the photographed lesion (lesion information). The lesion information output by the image analysis device 3 includes at least lesion presence/absence information indicating whether a lesion is included in the endoscopic image (a lesion is captured). If a lesion is included, the lesion information includes information indicating the size of the lesion, information indicating the position of the outline of the lesion, information indicating the shape of the lesion, information indicating the depth of invasion of the lesion, and qualitative diagnosis results of the lesion. The qualitative diagnosis result of the lesion may include information indicating the type of the lesion.

During an endoscopic examination, the image analysis device 3 is provided with endoscopic images from the endoscopic observation device 5 in real time together with meta information of the endoscopic images, and is provided with information indicating organs, information indicating parts, and lesions. Generate information. Hereinafter, the information indicating the organ, the information indicating the site, and the lesion information generated by the image analysis device 3 will be collectively referred to as "image analysis information." The image analysis device 3 provides the generated image analysis information to the endoscope observation device 5. Therefore, the endoscopic observation device 5 can display information regarding the image analysis results on the display device 6 while displaying the endoscopic image.

When the user operates the release switch (capture operation), the endoscope observation device 5 provides the frame number of the captured endoscopic image to the image analysis device 3 along with information indicating that the capture operation has been performed (capture operation information). do. When the image analysis device 3 acquires the capture operation information, it provides the server device 2 with the examination ID, a frame number, photographing time information, and image analysis information corresponding to the frame number. Here, the frame number, photographing time information, and image analysis information constitute "additional information" that expresses the characteristics and properties of the endoscopic image. When the image analysis device 3 acquires the capture operation information, it transmits the additional information together with the examination ID to the server device 2, and the server device 2 records the additional information in association with the examination ID. Note that the additional information may include an image ID.

When the user finishes the endoscopic examination, he or she operates the end examination button on the endoscopic observation device 5. The operation information of the examination end button is supplied to the server device 2 and the image analysis device 3, and the server device 2 and the image analysis device 3 recognize the end of the endoscopy.

Note that the period from when the test start button is operated until the test end button is operated is the period before the endoscope is inserted into the subject, and the period when the endoscope is completely removed from the subject after observation is completed. During the period after being removed, the endoscope photographs the outside of the subject (the space inside the examination room). The trained model of the embodiment is configured to output information indicating that the image is an image outside the subject when an endoscopic image taken during that period is input.

FIG. 2 shows functional blocks of the server device 2. The server device 2 includes a communication section 20, a processing section 30, and a storage device 60. The communication unit 20 transmits and receives information such as data and instructions to and from the image analysis device 3, endoscope observation device 5, image storage device 8, terminal device 10a, and terminal device 10b via the network 4. The processing section 30 includes an order information acquisition section 40 and an additional information acquisition section 42. The storage device 60 includes an order information storage section 62 and an additional information storage section 64.

The server device 2 includes a computer, and the various functions shown in FIG. 2 are realized by the computer executing programs. A computer includes a memory for loading a program, one or more processors for executing the loaded program, an auxiliary storage device, other LSI, and the like as hardware. A processor is constituted by a plurality of electronic circuits including a semiconductor integrated circuit and an LSI, and the plurality of electronic circuits may be mounted on one chip or on a plurality of chips. The functional blocks shown in FIG. 2 are realized by the cooperation of hardware and software, and therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways by only hardware, only software, or a combination thereof. It is understood.

The order information acquisition unit 40 acquires order information for endoscopy from the hospital information system. For example, before the start of a day's testing work at a medical facility, the order information acquisition section 40 acquires the order information for that day from the hospital information system and stores it in the order information storage section 62. Before the start of the examination, the endoscopic observation device 5 or the information processing device 11a may read order information for the examination to be performed from the order information storage unit 62 and display it on the display device.

The additional information acquisition unit 42 acquires the examination ID and the additional information of the endoscopic image from the image analysis device 3, and stores the additional information in the additional information storage unit 64 in association with the examination ID. Additional information on the endoscopic image includes a frame number, imaging time information, and image analysis information.

FIG. 3 shows functional blocks of the endoscopic observation device 5. The endoscopic observation device 5 has a function of controlling the endoscope 7 and displaying images taken by the endoscope 7 on the display device 6 in real time. The endoscopic observation device 5 includes a receiving antenna 76, a communication section 78, and a control device 80. The communication unit 78 transmits and receives information such as data and instructions to and from the server device 2, image analysis device 3, image storage device 8, and

terminal devices

10a and 10b via the network 4. The control device 80 includes an endoscope control section 82, a signal processing section 84, an operation information acquisition section 86, a shape information acquisition section 90, an image providing section 92, an image analysis information acquisition section 94, an endoscope image acquisition section 100, and a process. It includes a specifying section 102, an operation guide generation section 110, an observation support information generation section 112, and a display control section 120.

The endoscopic observation device 5 includes a computer, and various functions shown in FIG. 3 are realized by the computer executing programs. A computer includes a memory for loading a program, one or more processors for executing the loaded program, an auxiliary storage device, other LSI, and the like as hardware. A processor is constituted by a plurality of electronic circuits including a semiconductor integrated circuit and an LSI, and the plurality of electronic circuits may be mounted on one chip or on a plurality of chips. The functional blocks shown in FIG. 3 are realized by the cooperation of hardware and software, and therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways by only hardware, only software, or a combination thereof. It is understood.

<Basic operation by control device 80>
The basic operation of the control device 80 will be explained below.
The endoscope control unit 82 has a light source including an LED or a lamp, and supplies the endoscope 7 with illumination light for illuminating the inside of the subject. Furthermore, the endoscope control section 82 has a drive circuit and generates a coil drive signal for driving a plurality of magnetic coils provided in the insertion section of the endoscope 7. The endoscope control unit 82 supplies coil drive signals to the plurality of magnetic coils, so that the plurality of magnetic coils of the endoscope 7 generate magnetic fields.

The receiving antenna 76 has a plurality of coils that three-dimensionally detects the magnetic field generated by each of the plurality of magnetic coils. The receiving antenna 76 detects the magnetic field generated by each of the plurality of magnetic coils, and outputs a magnetic field detection signal corresponding to the strength of the detected magnetic field to the shape information acquisition section 90.

The shape information acquisition unit 90 acquires the positions of the plurality of magnetic coils within the subject based on the magnetic field detection signal output from the receiving antenna 76. Specifically, the shape information acquisition unit 90 acquires a plurality of three-dimensional coordinate values in a virtual spatial coordinate system with a predetermined position (anus, etc.) of the subject as the origin or reference point as the positions of the plurality of magnetic coils. You may do so. The shape information acquisition unit 90 generates insertion shape information indicating the shape of the endoscope inserted into the subject from the three-dimensional coordinate values of the plurality of magnetic coils. In the embodiment, the shape information acquisition section 90 provides the generated insertion shape information to the process identification section 102 and the operation guide generation section 110.

The signal processing unit 84 performs image processing such as A/D conversion, noise removal, and optical correction (shading correction) on the imaging signal supplied from the endoscope 7 to generate an endoscopic image. The signal processing unit 84 outputs the generated endoscopic image to the display control unit 120 and the endoscopic image acquisition unit 100. The display control unit 120 displays the endoscopic image in a predetermined display area on the display screen of the display device 6.

After acquiring the endoscopic image, the endoscopic image acquisition unit 100 supplies the endoscopic image to the image providing unit 92 and the operation guide generation unit 110. The image providing unit 92 transmits the endoscopic image to the endoscopic observation device 5 via the communication unit 78. The image analysis device 3 generates image analysis information indicating the result of analyzing the endoscopic image, and sends it back to the endoscopic observation device 5. The image analysis information acquisition unit 94 acquires the image analysis information generated by the image analysis device 3 and supplies it to the process identification unit 102 and the observation support information generation unit 112.

The operation information acquisition unit 86 acquires operation information (capture operation information) regarding the user's operation of the release switch. When the operation information acquisition unit 86 acquires the capture operation information, the image providing unit 92 adds an image ID to the endoscopic image, and sends the endoscopic image together with the image ID and meta information via the communication unit 78. The data is transmitted to the storage device 8. At this time, the image providing unit 92 notifies the image analysis device 3 of the frame number of the endoscopic image together with the capture operation information, and the image analysis device 3 transmits image analysis information corresponding to the notified frame number to the server device 2. Register.

<Inspection support operation by control device 80>
The endoscopy support operation performed by the control device 80 will be described below.
In recent years, the resolution of endoscopic images has been increasing, and the screen size of the display device 6 that displays endoscopic images has become larger. If the screen size is large, the entire displayed endoscopic image may not fit within the user's field of view. need to be moved to some extent.

In the insertion process of endoscopy, the doctor performs the insertion operation of the endoscope 7 while viewing the endoscopic image displayed on the display device 6. It is preferable to see the whole thing. On the other hand, in the observation process of endoscopy, the doctor locally observes the endoscopic image while moving his/her line of sight so as not to miss any lesions, so the display device 6 displays high-definition endoscopic images. It is preferable that it is displayed. Therefore, the control device 80 of the embodiment has a function of displaying endoscopic images in different manners in the insertion process and the observation process.

Specifically, the display control unit 120 displays the endoscopic image in the first display area on the display screen of the display device 6 in the insertion step. Further, in the observation step, the display control unit 120 displays the endoscopic image in a second display area larger than the first display area on the display screen of the display device 6.
FIG. 4 shows a first display area 130 where an endoscopic image is displayed during the insertion process, and FIG. 5 shows a second display area 132 where an endoscopic image is displayed during the observation process. FIG. 6 shows a comparison diagram comparing the first display area 130 and the second display area 132.

As shown in FIG. 6, the second display area 132 in the observation process is larger than the first display area 130 and includes the first display area 130. The shape of the second display area 132 and the shape of the first display area 130 are similar, and are set at screen positions having a common center point. Therefore, the user can view endoscopic images during the insertion process and observation process without significantly changing the viewing direction.

In the insertion step, the display control unit 120 displays the endoscopic image in the relatively small first display area 130 so that the entire endoscopic image is within the user's visual field, so that the user can see the endoscopic image within the user's visual field. It becomes possible to concentrate on the insertion operation of the endoscope 7 without moving the line of sight toward the endoscopic image. On the other hand, in the observation process, the display control unit 120 displays the endoscopic image in the relatively large second display area 132, and the user can miss a lesion by observing the high-resolution endoscopic image. It is possible to reduce the

FIG. 7 shows a flowchart of inspection support processing in the embodiment. When the user operates an examination start button provided on the endoscopic observation device 5, the endoscopy is started (S10). After the start of the examination, the process identification unit 102 acquires examination status information indicating the status of the endoscopy, and determines the examination process based on the examination status information. The examination status information may be at least one of information obtained by analyzing an endoscopic image, information indicating the shape of the endoscope, and information indicating the position of the endoscope. In the embodiment, the examination status information may be image analysis information provided from the image analysis information acquisition section 94 and/or insertion shape information of the endoscope 7 provided from the shape information acquisition section 90.

The process specifying unit 102 determines whether the inspection process is an insertion process, an observation process, or a process that is neither an insertion process nor an observation process, based on the inspection status information. Processes that are neither the insertion process nor the observation process include a preparation process before inserting the endoscope 7 into the subject, and a process after the endoscope 7 is completely removed from the subject after the observation is completed. and a termination step. In the embodiment, the process specifying unit 102 has a function of determining an inspection process based on image analysis information and/or insertion shape information.

As described above, the image analysis device 3 generates image analysis information including information indicating the part inside the subject (part information) for the endoscopic image taken inside the subject, and images the outside of the subject. Image analysis information including information indicating that the endoscopic image is an image outside the subject is generated. Therefore, the process specifying unit 102 can determine the inspection process by referring to the site information included in the image analysis information or the information indicating that the image is an external image of the subject.

Immediately after the start of the examination, the endoscope 7 has not yet been inserted into the subject and exists outside the subject. At this time, the endoscope 7 is photographing the space inside the examination room, and since the image analysis information includes information indicating that the image is outside the subject, the endoscope 7 It is determined that the camera is photographing the outside of the subject and has not yet been inserted into the subject (N in S12).

If the image analysis information includes site information, the process specifying unit 102 determines that the endoscope 7 is inserted into the subject and the examination process is an insertion process (Y in S12). Note that in colonoscopy, the first site into which the endoscope 7 is inserted is the "rectum," so the process identification unit 102 uses the image analysis information acquired in chronological order to identify images of the outside of the subject. When the information indicating "rectal" is no longer present and the information indicating "rectum" is included, it may be determined that the inspection process is an insertion process. In the insertion step, the display control unit 120 displays the endoscopic image in the first display area 130 (S14).

FIG. 8 shows an example of an endoscopic image displayed in the first display area 130. The display control unit 120 displays the endoscopic image in the first display area 130 at a first resolution. As will be described later, the first resolution of the endoscopic image in the insertion step is lower than the second resolution of the endoscopic image in the observation step. In the insertion process, the endoscopic image is displayed in the small first display area 130, so that the entire endoscopic image is within the user's visual field, and the user therefore moves his/her line of sight toward the endoscopic image. This allows the user to concentrate on the insertion operation of the endoscope 7.

In the insertion process, the operation guide generation unit 110 generates an insertion guide regarding how to insert the endoscope 7. The operation guide generating unit 110 acquires at least one of an endoscopic image and information indicating the insertion shape of the endoscope 7, generates an insertion guide based on at least one of the endoscopic image and the insertion shape information, and displays the generated insertion guide. The control unit 120 displays the insertion guide in an area different from the first display area 130 (S16).

FIG. 9 shows an example of an insertion guide displayed around the first display area 130. The operation guide generation unit 110 estimates the direction in which the lumen exists based on the endoscopic image. Various methods have been proposed in the past for estimating the direction in which a lumen exists from an endoscopic image, and the operation guide generation unit 110 may use any of the methods to estimate the direction in which a lumen exists. . For example, the operation guide generation unit 110 may recognize the lumen position in the endoscopic image using an image recognition function, or may recognize the lumen position in the endoscopic image by inputting endoscopic image data into a machine-learned lumen direction estimation model. The direction in which the cavity exists may also be obtained. The insertion guide 140 shown in FIG. 9 is a mark indicating the direction in which the lumen exists, and in this example indicates that the lumen exists below the endoscopic image.

It is preferable that the display control unit 120 arranges the insertion guide 140 indicating the lumen direction in an area outside the first display area 130 and inside the second display area 132. In FIG. 9, the boundary of the second display area 132 is shown with a broken line for reference. By displaying the insertion guide 140 in the area inside the second display area 132, the blank space created by reducing the first display area 130 compared to the second display area 132 can be effectively used. Furthermore, the display control unit 120 may determine the position where the insertion guide 140 is placed depending on the direction in which the lumen exists. In the example shown in FIG. 9, the insertion guide 140 indicates that the lumen is present at the bottom of the screen, so the display control unit 120 moves the insertion guide 140 downward with respect to the first display area 130. It is placed. For example, if the insertion guide 140 indicates that the lumen exists on the right side of the screen, the display control unit 120 positions the insertion guide 140 on the right side with respect to the first display area 130, and the insertion guide 140 However, if the lumen is shown to exist in the upper left direction of the screen, the display control unit 120 arranges the insertion guide 140 in the upper left side with respect to the first display area 130. Since the direction in which the lumen exists indicates the direction in which the distal end of the endoscope 7 should be directed (curved), the user can determine the operation of the endoscope 7 to be performed based on the orientation and placement position of the insertion guide 140. can be understood intuitively.

FIG. 10 shows another example of the insertion guide displayed around the first display area 130. The operation guide generation unit 110 derives recommended operations based on the insertion shape information of the endoscope 7. The insertion guide 142 shown in FIG. 10 is a guide showing a recommended operation, and in this example shows a loop release operation to be performed when passing through the sigmoid colon. The operation guide generation unit 110 displays the insertion guide 142 at a position close to the endoscopic image displayed in the first display area 130. By displaying the insertion guide 142 at a position close to the endoscopic image, the user can confirm the insertion guide 142 without significantly moving his/her line of sight. Note that when the insertion guide 140 indicating the lumen direction is also displayed at the same time, the display control unit 120 preferably sets the display position of the insertion guide 142 so as not to overlap the insertion guide 140. Note that the operation guide generation unit 110 may generate a schema diagram showing the insertion status of the endoscope 7, and the display control unit 120 may display the schema diagram as an insertion guide.

The process specifying unit 102 refers to the site information in the image analysis information and determines whether the endoscope 7 has reached the "cecum" which is the turning point (S18). The process specifying unit 102 determines that the inspection process is not an observation process (N in S18) but an insertion process (Y in S12) until it is determined that the endoscope 7 has reached the cecum. In the insertion step, the display control unit 120 displays an endoscopic image in the first display area 130 (S14), and also displays an operation guide as necessary (S16).

On the other hand, after the process specifying unit 102 refers to the site information in the image analysis information and determines that the endoscope 7 has reached the "cecum" which is the turning point, it determines that the inspection process is an observation process. Determine (Y in S18). In the observation step, the display control unit 120 displays the endoscopic image in the second display area 132 (S20).

FIG. 11 shows an example of an endoscopic image displayed in the second display area 132. The display control unit 120 displays the endoscopic image in the second display area 132 at a second resolution higher than the first resolution. In the observation process, a high resolution endoscopic image is displayed in the second display area 132, allowing the user to find even small lesions.

In the observation process, the observation support information generation unit 112 generates observation support information for supporting observation. The observation support information generation unit 112 may generate observation support information from the image analysis information provided by the image analysis information acquisition unit 94. The display control unit 120 displays the observation support information in an area different from the second display area 132 (S22).

FIG. 12 shows an example of observation support information displayed around the second display area 132. The observation support information generation unit 112 may generate the observation support information 150 using the lesion information included in the image analysis information. The observation support information 150 is information regarding lesions, and specifically, a mark (in this example, A frame surrounding the lesion) is added. Note that the observation support information 150 may add a mark indicating the position of the detected lesion to the endoscopic image displayed in the second display area 132. The display control unit 120 displays the observation support information 150 near the endoscopic image or superimposed on the endoscopic image. By looking at the observation support information 150, the user can recognize that the lesion is automatically detected by the image analysis device 3.

Furthermore, the observation support information generation unit 112 may generate observation support information 152 indicating that a lesion has been detected, and the display control unit 120 may display the observation support information 152. When the image analysis information includes lesion information, the observation support information generation unit 112 generates observation support information 152, which is an alert mark to alert the user. The display control unit 120 may display the observation support information 152 in an area outside the second display area 132 while a lesion is being detected. For example, the alert mark may be displayed in a color that stands out against the endoscopic image. By looking at the observation support information 152, the user can recognize that the lesion has been automatically detected by the image analysis device 3.

When the observation support information generating unit 112 detects an area (unobserved area) that the user has not confirmed well, it may generate information regarding the unobserved area of the subject as observation support information. For example, if a lesion is detected by the image analysis device 3 on the outward path in the insertion process, but is not detected by the image analysis device 3 on the return path in the observation process, that is, it has not been imaged by the endoscope 7. In this case, the observation support information generation unit 112 may generate information indicating that the lesion (unobserved area) has been passed, and the display control unit 120 may display the information on the display screen as observation support information.

The process specifying unit 102 refers to the site information in the image analysis information and determines whether the endoscope 7 has been completely removed to the outside of the subject (S24). The process specifying unit 102 determines that the observation process is in progress (Y in S18) until the endoscope 7 is completely removed to the outside of the subject (N in S24). In the observation step, the display control unit 120 displays the endoscopic image in the second display area 132 (S20), and also displays observation support information as necessary (S22).

If the image analysis information includes information indicating that the image is outside the subject, the process identification unit 102 determines that the endoscope 7 has been completely removed from the subject (S24). Y), the observation process ends.

The present disclosure has been described above based on the embodiments. Those skilled in the art will understand that these embodiments are illustrative, and that various modifications can be made to the combinations of their constituent elements and processing processes, and that such modifications are also within the scope of the present disclosure. be. In the embodiment, the endoscope observation device 5 is equipped with the control device 80, but the terminal device 10a may be equipped with the control device 80, and the function of the control device 80 is different from that of the endoscope observation device 5 and the server device. 2. It may be distributed to the image analysis device 3 and/or the terminal device 10, and realized in cooperation with a plurality of devices.

In the embodiment, the process identification unit 102 determines the inspection process based on image analysis information provided from the image analysis information acquisition unit 94. In a modified example, the process specifying unit 102 may generate image analysis information using a learned model that the image analysis device 3 has. Further, the process specifying unit 102 performs the inspection process using a trained model that can determine whether the endoscope is inside or outside the subject and whether it has reached the cecum inside the subject. May be determined.

Further, the process specifying unit 102 may determine the inspection process based on the insertion shape information of the endoscope 7 provided from the shape information acquiring unit 90. Specifically, the process specifying unit 102 determines whether the tip of the endoscope has reached the cecum based on the insertion shape of the endoscope 7 and/or the position of the tip of the endoscope 7, and therefore determines whether the tip of the endoscope has reached the cecum. It may be determined whether the inspection process is an insertion process or an observation process.

In the embodiment, the display control unit 120 sets the display mode of the endoscopic image according to the inspection process determined by the process specifying unit 102, but the user may be able to forcibly change the display mode. . For example, in the insertion process, if the user finds a lesion while the display control unit 120 is displaying a small endoscopic image, the display control unit 120 can display a high-resolution image by manually forcibly switching the display mode. The endoscopic image may be displayed in a large size.

The present disclosure can be used in the technical field of displaying endoscopic images.

DESCRIPTION OF SYMBOLS 1... Examination support system, 2... Server device, 3... Image analysis device, 4... Network, 5... Endoscope observation device, 6... Display device, 7... Endoscope, 8... Image storage device, 9... Endoscope system, 10a, 10b... Terminal device, 11a, 11b... Information processing device, 12a, 12b... Display device, 20 ... Communication department, 30 ... Processing section, 40 ... Order information acquisition section, 42 ... Additional information acquisition section, 60 ... Storage device, 62 ... Order information storage section, 64 ... - Additional information storage section, 76... Reception antenna, 78... Communication section, 80... Control device, 82... Endoscope control section, 84... Signal processing section, 86... Operation Information acquisition unit, 90... Shape information acquisition unit, 92... Image providing unit, 94... Image analysis information acquisition unit, 100... Endoscopic image acquisition unit, 102... Process identification unit, 110... Operation guide generation unit, 112... Observation support information generation unit, 120... Display control unit, 130... First display area, 132... Second display area, 140, 142... - Insertion guide, 150, 152...Observation support information.

Claims

An inspection support system comprising one or more processors having hardware;
The one or more processors include:
When the inspection process in endoscopy is an insertion process, display an insertion guide on how to insert the endoscope,
when the inspection process is an observation process, displaying observation support information for supporting observation;
An inspection support system characterized by:
The one or more processors include:
In the insertion step, displaying an endoscopic image in a first display area,
In the observation step, displaying the endoscopic image in a second display area larger than the first display area;
The inspection support system according to claim 1, characterized in that:
The second display area includes the first display area,
The inspection support system according to claim 2, characterized in that:
The one or more processors include:
In the insertion step, displaying the endoscopic image in the first display area at a first resolution;
In the observation step, displaying the endoscopic image in the second display area at a second resolution higher than the first resolution;
The inspection support system according to claim 2, characterized in that:
The one or more processors include:
Obtaining examination status information indicating the status of the endoscopy,
determining whether the inspection process is the insertion process or the observation process based on the inspection status information;
The inspection support system according to claim 1, characterized in that:
The examination status information is at least one of information obtained by analyzing an endoscopic image, information indicating the shape of the endoscope, and information indicating the position of the endoscope.
The inspection support system according to claim 5, characterized in that:
The insertion process is a process on the outward path from when the endoscope enters the inside of the subject until it turns back at the turning point, and the observation process is a process from when the endoscope turns back at the turning point to the subject. This is a process on the return journey until the specimen is pulled out of the specimen.
The inspection support system according to claim 1, characterized in that:
The examination status information is generated based on an endoscopic image,
The one or more processors include:
From the inspection status information, it is determined that the inspection step is the insertion step until it is determined that the endoscope has reached the cecum;
After determining from the examination status information that the endoscope has reached the cecum, determining that the examination step is the observation step;
The inspection support system according to claim 5, characterized in that:
The one or more processors include:
in the insertion step, generating the insertion guide based on at least one of an endoscopic image and information indicating the shape of the endoscope;
The inspection support system according to claim 1, characterized in that:
the insertion guide is a mark indicating the direction in which the lumen exists;
The inspection support system according to claim 9, characterized in that:
The one or more processors include:
determining a position to place the mark according to the direction in which the lumen exists;
The inspection support system according to claim 10.
The insertion guide is a guide indicating recommended operations;
The inspection support system according to claim 9, characterized in that:
The insertion guide is a schema diagram showing a state of insertion of the endoscope.
The inspection support system according to claim 9, characterized in that:
The observation support information is generated from an endoscopic image.
The inspection support system according to claim 1, characterized in that:
The observation support information is information regarding a lesion;
The inspection support system according to claim 14.
The observation support information is information regarding an unobserved region of the subject;
The inspection support system according to claim 14.
An inspection support method, comprising:
When the inspection process in endoscopy is an insertion process, display an insertion guide on how to insert the endoscope,
when the inspection process is an observation process, displaying observation support information for supporting observation;
An inspection support method characterized by:
to the computer,
When the inspection process in endoscopy is an insertion process, a function of displaying an insertion guide on how to insert the endoscope;
When the inspection process is an observation process, a function of displaying observation support information for supporting observation;
A recording medium that stores a program to realize the