JP2023107399A - Endoscope system and operation method thereof - Google Patents

Abstract

To provide an endoscope system which performs a detection of an abnormality of an apparatus due to a failure without using a special sensor for detecting the failure and a determination of the type of the failure and efficiently analyzes a cause of the failure by recording a detection result for each failure, and an operation method thereof.SOLUTION: A detection of a failure of an endoscope and a determination of the type of the failure are performed from a medical moving image photographed by the endoscope, and a failure moving image including a failure scene at the timing when the failure is detected from the medical moving image is extracted. The extracted contents of the failure moving image are determined according to a storage mode. The endoscope system is equipped with a storage mode for storing the entire extracted failure moving image and a storage mode for selecting the failure moving image to be stored by temporarily storing. The endoscope system stores failure information including at least the type of the failure and the failure moving image in a main storage region.SELECTED DRAWING: Figure 3

Description

The present invention relates to an endoscopic system and method of operation thereof.

In a medical device such as an endoscope, a system that automatically detects an abnormality such as a device failure and saves it as an error log is common. In addition to the error type and date and time of occurrence, the error log records the device settings, program processing status, and communication status between modules when the error occurred. By using the error log, it is possible to grasp the fact that an abnormality such as a failure has occurred and the circumstances of the occurrence. It is difficult to identify the specific details of the failure and the cause of the failure by detecting failures and the like using error logs. Therefore, it is required to grasp the specific details of the failure using data other than the error log.

Specifically, in Japanese Unexamined Patent Application Publication No. 2002-100002, a plurality of status signals representing operating statuses of equipment are inspected. Detects that the status signal is out of the allowable range, and records the time of detection, the elapsed time from start-up, and the subsequent status. Also, the type of abnormality is displayed by lighting, sound, or the like. In Patent Document 2, an abnormality is detected from data indicating the operating state of equipment, and the operating state is recorded together with the type of the detected abnormality. It is determined whether or not the abnormality is determined to be a failure, and the recorded data is erased when an abnormality that is not determined to be a failure is detected.

Japanese Patent No. 3207874 Japanese Patent No. 3549505

In Patent Document 1, an abnormality is detected using a device status signal, and in Patent Document 2, an abnormality occurs temporarily when an extremely high load is applied, and the possibility of self-recovery over time is high. , to determine the kind of what is not. In Patent Literatures 1 and 2, a signal indicating the operating state of a device is acquired, abnormality is detected, recorded, and determined to determine the type of abnormality. Although it is possible to determine that an abnormality such as a temporary high load due to external noise or the like is not due to a failure, it is difficult to determine that it is a failure and to identify its cause.

In addition, in devices such as endoscopes, which have size restrictions, it is not possible to install a dedicated sensor for detecting failure, or the cost is high. Furthermore, when analyzing all the data obtained from the test results, it takes time and effort for the user and is inefficient. Therefore, it is required to efficiently confirm the contents of the failure and identify the cause of the failure in the detection and recording of the failure from the inspection results obtained from the ordinary inspection equipment.

The present invention detects equipment abnormalities due to failures and determines the type of failure without using a dedicated sensor for detecting failures, and records the detection results for each failure to efficiently analyze the cause of the failure. It is an object of the present invention to provide a scope system and method of operation thereof.

The endoscope system of the present invention comprises a processor, and the processor detects a failure of the endoscope from a medical video taken by the endoscope, determines the type of failure, and detects the timing of failure detection from the medical video. is extracted, and stored in the main memory area together with failure information including at least the type of failure.

It is preferable to extract a failure animation having a plurality of frame images that are not a failure scene before or before or after the failure scene.

It is preferable to determine the extraction range of the failure animation to be extracted based on the failure scene and failure information corresponding to the failure scene.

It is preferable to determine the extraction range of the failure animation to be extracted according to the failure scene and the operation history at the timing near the failure scene.

Temporarily save failure videos in a temporary storage area, sort out failure videos based on failure information,
It is preferable to store the screened failure animation in the main memory area.

It is preferable to calculate the degree of importance of failure animations and perform selection based on the degree of importance.

Preferably, the selection is based on the chronological order of failure scenes.

Preferably, at least one failure animation is stored in the main memory area for each failure type.

It is preferable to perform sorting by user operation.

It is preferable to acquire the shape information of the endoscope and store the shape information in the main storage area together with the failure animation and the failure information when saving the failure animation.

It is preferable to extract failure animations using shape information.

It is preferable to detect failures during endoscopy and extract failure animations.

It is possible to temporarily save failure videos in non-volatile memory, select the temporarily stored failure videos regardless of the end status of the endoscopy, and save the selected failure videos from the non-volatile memory to the main memory area. preferable.

When extracting a failure movie, a different identifier is set for each failure movie, the identifier of the failure movie saved in the main memory area is confirmed, and failure movies having the same identifier as the confirmed identifier are stored in a non-volatile memory. It is preferably deleted from memory.

A method for operating an endoscope system according to the present invention comprises the steps of detecting a failure of an endoscope from a medical video captured by the endoscope, determining the type of failure, and determining the timing at which the failure is detected from the medical video. and storing the failure animation together with failure information including at least the type of failure in the main memory area.

According to the present invention, without using a dedicated sensor for detecting a failure, an abnormality in a device caused by a failure is detected, the type of failure is determined, and the detection result is recorded for each failure, thereby efficiently analyzing the cause of the failure. can do

FIG. 3 is an explanatory diagram showing connected devices of the endoscope system; 3 is a block diagram showing functions of a processor device; FIG. FIG. 10 is an explanatory diagram of a medical moving image for which failure detection processing is performed; FIGS. 4A, 4B, and 4C are explanatory diagrams of respective frame images detected in the failure detection process. FIG. 4 is an explanatory diagram of an extraction range for a failure scene when extracting a failure video; FIG. 4 is an explanatory diagram of each save mode; FIG. 11 is an explanatory diagram of a case where extracted failure animations are saved without being sorted; FIG. 10 is an explanatory diagram of a case where extracted failure animations are sorted and stored; FIG. 11 is an image diagram displaying a playback screen of a failure video; FIG. 10 is an image diagram showing thumbnails of failure animations; 4 is a flow chart showing a series of processes of the present invention; FIGS. 12A, 12B, and 12C are explanatory diagrams for acquiring shape information of the insertion portion of the endoscope in the second embodiment. FIG. 11 is an explanatory diagram of temporarily storing a failure video in a nonvolatile memory in the third embodiment;

[First embodiment]
FIG. 1 is a diagram showing connected devices of an endoscope 11 in an endoscope system 10 according to an embodiment of the present invention. The endoscope system 10 has an endoscope 11 , a light source device 12 , a processor device 13 , a database 14 , a display 15 and a user interface (UI) 16 . The processor device 13 is electrically connected with the light source device 12 , the database 14 , the display 15 and the user interface 16 . The endoscope 11 has an elongated shape, and has an insertion section 11a that is inserted into a patient's body to acquire an image, and an operation section 11b that receives user operations such as bending of the insertion section 11a and zooming when acquiring an image. . A distal end portion 11c having a photographing function and performing irradiation of illumination light and projection of the treatment instrument is provided at the distal end of the insertion portion 11a.

The light source device 12 is optically connected to the endoscope 11 and supplies illumination light to the endoscope 11 during endoscopy. The database 14 is a device capable of storing images and transmitting/receiving data to/from the processor device 13, and may be a recording medium such as a USB (Universal Serial Bus) or HDD (Hard Disc Drive). The display 15 displays images acquired by the processor device 13 . The user interface 16 is an input device for input to the processor unit 13, and may use a foot pedal, a gesture recognizer, or a voice recognizer in addition to or instead of a keyboard or mouse. The user interface 16 is not limited to the input means provided in the medical device, such as the switch of the endoscope 11 . Note that images include moving images, still images, and frame images that constitute moving images.

The database 14 stores moving images and still images of medical examinations created by the processor device 13 and other medical equipment. Unless otherwise specified, white light is used as the illumination light for photographing medical examinations, a video signal of 60 frames per second (60 fps (frame per second)) is obtained, and the photographing time is recorded. Also, when the video signal is 60 fps, it is preferable to count the time in units of 1/100 seconds.

As shown in FIG. 2, in the endoscope system 10, a program in a program memory is operated by a central control unit (not shown) configured by an image control processor, so that an image acquisition unit 21, an input The functions of the receiving unit 22, the main storage area 23, the display control unit 24, and the failure animation creating unit 30 are realized. Further, with the functional realization of the failure video creation unit 30, the functions of the failure detection unit 31, the extraction range determination unit 32, the extraction unit 35, the temporary storage unit 36, and the selection unit 37 are implemented. The extraction range determination unit 32 further includes the functions of a mode control unit 33 and an image information management unit 34 , and the selection unit 37 further includes the function of an importance calculation unit 38 . The image acquisition unit 21 receives data such as medical videos captured by the endoscope 11 and medical videos stored in the database 14 . The acquired medical animation is transmitted to the failure animation creating unit 30 as the medical animation 40 .

The input receiver 22 connects with the user interface 16 . The main storage area 23 stores the acquired medical videos or the failure videos extracted by the failure detection unit 31 . Instead of the main storage area 23, the database 14 connected to the processor device 13 may have the function. The display control unit 24 performs control to display an image on the display 15 . In the processor unit 13, programs related to processing such as image processing are stored in a program memory (not shown).

As shown in FIG. 3, when creating a failure movie, the failure movie creator 30 determines the type of each frame constituting the medical movie by failure detection processing. The medical animations 40 to be input to the failure animation creating unit 30 are continuous in time series in which the inside of the body is photographed, that is, they are temporally continuous in the same examination. Each frame image constituting the medical video 40 is subjected to failure detection processing by the failure detection unit 31, and normal frame images 41 without failure, failure frame images 42 having an abnormality due to failure, and abnormal but not failure frame images 41 are detected. It is sorted into the no-failure/abnormal frame image 43 . Also, the type of failure is determined.

In the medical moving image 40 subjected to the failure detection process, failure frame images 42 having the same failure type and detected continuously or with a certain frequency or more are determined to be the same failure. A frame image group that constitutes the same failure is detected as a failure scene. The medical video 40 undergoes failure video extraction processing for each failure scene. Each rectangle shown in FIG. 3 is one frame that constitutes a medical video, and three consecutive frames are failure scenes. .

As shown in FIG. 4, in the failure detection processing of a medical moving image 40, any one of a normal frame image 41, a failure frame image 42, or a failure-free abnormal frame image 43 is detected. Each frame image indicated by a rectangle in FIG. 3 was actually photographed in the state shown in FIGS. 4(A) to 4(C). As shown in FIG. 4A, normal frame images 41 are normal frame images in which no abnormalities have been detected in the medical moving image 40 . As shown in FIG. 4B, the failure frame image 42 is a frame image in which a failure of the endoscope 11 is detected in the medical video 40, and vertical and horizontal lines run through the image due to "disconnection". A frame image showing a disconnection C and a frame image including a non-lesion abnormal region T in which a part of the image is painted over regardless of the object to be observed due to "scope dirt". Further, as shown in FIG. 4(C), the failure-free abnormal frame image 43 is a frame image that is abnormal but not a failure. For example, it is a frame image in which part or all of the image is not displayed due to "external noise", or a lesion area R is shown in the image as a severe "lesion". Since the fault-free abnormal frame image 43 is not a fault, it is handled in the same manner as the normal frame image 41 in extraction after fault detection processing.

The functions of the failure animation creation unit 30 are as follows. The failure animation creation unit 30 receives a medical animation 40 captured by the endoscope 11 , which is acquired by the image acquisition unit 21 from the database 14 or the endoscope 11 . The failure detection unit 31 performs failure detection processing on the received medical moving image 40 . In the failure detection process, the presence or absence of a failure scene, which is one or more frame images in which a failure has been detected, is determined, and if there is a failure scene, the type of failure is determined. The extraction unit 35 extracts a short-time failure movie 44 for each failure scene, which is one or a plurality of frame images showing the same failure detected by the failure detection process. The failure animation 44 is stored in the main memory area 23 together with failure information including at least failure information.

The failure scene is not limited to the consecutive failure frame images 42, and may include the normal frame images 41 or the failure-free abnormal frame images 43 at a certain rate or less. However, failure frame images that are continuous but have different failure types are different failure scenes. In other words, the failure scene is detected continuously or at a certain frequency or more and is composed of failure frame images 42 of the same type. Further, even when only one failure frame image 42 is detected in the failure detection process, it may be regarded as a failure scene.

If the failure animation 44 consists only of a failure scene, it is difficult to compare the frame image with a normal state, so there is a possibility that the cause of the failure cannot be analyzed accurately. Therefore, when extracting the failure animation 44 from the medical animation 40, it is preferable to acquire the normal frame images 41 together before or after the failure scene.

As shown in FIG. 5, in extracting the failure animation 44, the failure animation is extracted from the medical animation 40 based on the set extraction range. A first extraction range in which a failure scene and a plurality of normal frame images 41 at timings before and after the failure scene are extracted, and a second extraction in which a failure scene and a plurality of normal frame images 41 at timings before the failure scene are extracted. An extraction range, either the range or a third extraction range for extracting only failure scenes, is set in advance before extraction. As for extraction, the fault-free abnormal frame image 43 is handled in the same way as the normal frame image 41 because it is not a fault.

In the confirmation of the failure by screen display and the investigation of the cause of the failure by image analysis, the failure video 44 has multiple frame images that are not the failure scene, for example, the normal frame image 41 at the timing before the failure scene, and compares before and after the failure occurrence. It is preferable to set the first extraction range that is possible. If the failure that has occurred is not temporary and continues from the occurrence until the end of the endoscopy, the upper limit of failure scenes to be extracted is set, such as stopping detection as failure scenes after a certain period of time has elapsed. In that case, since there is no normal frame image 41 after the occurrence of the failure, it is preferable to use the second extraction range. When saving a large amount of failure animations 44, for example, when priority is given to reducing the data capacity, it is preferable to extract the failure animations 44 in the third extraction range. Regarding the extraction of the failure video 44 in each storage mode described below, extraction processing is performed in the first extraction range unless otherwise specified.

The failure detection unit 31 executes failure detection processing for detecting the presence/absence of failures such as image distortion and dirt on the received medical moving image 40 for each frame. The failure detection unit 31 has a function of a learned model required for failure detection processing, for example. That is, it is a computer algorithm consisting of a neural network that performs machine learning, and detects the presence or absence of a failure for each medical video 40 input according to the learning content, and if there is a failure, performs specific inference about the type of failure. , to determine the type of fault. As for the inference result, it is preferable to acquire information such as the rate of matching with previously learned content in addition to the type of failure.

In the failure detection process, failure detection is performed by detecting failure frame images 42 from frame images in the acquired medical video 40, and failure detection such as "disconnection", "scope contamination", and "lens damage" is performed on the failure frame images 42. Perform fault judgment for judging the type. The determination result is associated with the failure frame image 42 as failure information. In the failure detection process, abnormality detection and failure extraction may be performed to determine whether or not there is a failure in the medical video 40 . In the abnormality detection, a frame image other than the normal frame image 41 having an abnormality such as image disturbance or noise is detected. It is determined whether the frame image 43 is a "failure-free" failure-free abnormal frame image 43 that is not a failure of the mirror.

The extraction range determination unit 32 combines the failure scene detected in the failure detection process, the setting of the save mode controlled by the mode control unit 33, and information such as the operation history and failure information referred to by the image information management unit 34. to determine the range to be extracted from the medical video 40 , that is, the structure of the failure video 44 . Moreover, you may set by user operation. For example, it determines whether to leave the normal image frames before and after the selected failure scene, and determines the time range of the failure animation 44 to be extracted from the type of failure scene and image information.

The mode control unit 33 receives and controls mode selection related to setting of extraction contents of the failure video 44 . The mode control unit 33 sets the normal mode or the save mode, and further sets the save mode to one of the first to sixth save modes. Mode selection is performed by user operation, for example. In the normal mode, at least one of saving the medical video 40 in which the fault is detected in the main storage area 23 and transmitting it to the display control unit 24 is executed.

As shown in FIG. 6, there are six types of storage modes, which can be classified according to conditions and methods for extracting and storing failure animations 44 from medical animations 40 . In the first saving mode and the second saving mode, all the failure animations 44 extracted for each failure scene are saved in the main memory area 23 without being sorted. In the third to sixth storage modes, failure movies 44 extracted for each failure scene are temporarily stored in a temporary storage area, and failure movies 44 to be stored are selected. The sorted failure animation 44 is stored in the main memory area 23 . Details of extraction and sorting in each storage mode will be described later.

The image information management unit 34 acquires and manages the failure information including the type of failure acquired by the failure detection unit 31, the operation history of the endoscope 11, and the like. The failure information includes, for example, the severity of the failure, the area of the abnormal area caused by the failure, the duration of the failure scene, and the like. The operation history of the endoscope 11 includes angle operations, use of treatment tools, zoom operations, and the like.

The extraction unit 35 extracts a failure video 44 having at least a failure scene from the medical video 40 based on the extraction range determined by the extraction range determination unit 32 . The extracted failure video 44 is transmitted to the main storage area 23 or the temporary storage unit 36 depending on the storage mode. If the save mode selects the failure video 44 to be saved, it is transmitted to the temporary storage unit 36 and is saved in the main storage area 23 without selection, or if the failure video 44 to be saved is selected, The failure animation 44 is transmitted to the temporary storage unit 36 .

When the failure movie storage mode is set to the first storage mode or the second storage mode, since sorting is not performed, the extracted failure movie 44 is transmitted and stored in the main storage area 23 . In the case of the third saving mode to the sixth saving mode, in order to sort out the failure animation 44 to be saved, it is transmitted to the temporary saving unit 36 and temporarily saved.

The temporary storage unit 36 temporarily stores each failure movie 44 that is a storage candidate when sorting out the extracted failure movies 44 in creating a failure movie. The sorting is preferably performed by the sorting unit 37 after all the extracted failure animations 44 are temporarily stored. Temporary storage is realized by the temporary storage unit 36 storing in a temporary storage area (not shown) that is a volatile memory or non-volatile memory within the processor device 13 . It should be noted that the temporary storage area may be physically the same device as the main storage area 23 if it is a non-volatile memory.

The sorting unit 37 sorts the failure animations 44 to be stored as failure animations according to the selection criteria defined by the third to sixth storage modes. By performing the sorting, it is possible to narrow down failure scenes that are highly likely to be useful for failure cause analysis. The sorting is performed by at least one of comparing failure scenes and failure information in the failure animation 44, and determining whether the failure scene satisfies the criteria set in each storage mode. For example, in the third storage mode, sorting is performed based on the degree of importance of failure scenes.

The importance calculator 38 calculates the importance of the failure scene detected by the failure detector 31 . The degree of importance can be expressed, for example, as a percentage (%) or graded evaluation (high, medium, low). The degree of importance is calculated from at least one of the order in which failures occur, the types of failures, the frequency of occurrence of failures, the severity of failures, the certainty of failure detection, the visibility of failures, and the degree of impact. The more important a failure is, the more likely it is that the video will be useful for analyzing the failure. Image information for calculating the degree of importance is acquired by the image information management unit 34 .

The selected failure animation 44 is transmitted to and stored in the main memory area 23 . Moreover, in order to check the stored failure animation 44, it may be transmitted to the display control unit 24 and displayed on the screen. Note that the failure animation 44 may be stored in the database 14 instead of the main storage area 23 or similarly.

The display control unit 24 controls screen display of the medical video 40 or the failure video 44 received from the failure video creation unit 30 . Further, when displaying the failure animation 44 , editing of image information including the configuration of the failure animation and failure information by the user via the user interface 16 is accepted. When the failure animation 44 is displayed, the failure scene can be confirmed and the failure information can be referred to.

Determining the save mode will be explained. The save mode is preferably set by the user based on the contents of the endoscopy. The storage mode includes the first and second storage modes in which the failure animation 44 extracted for each failure scene is directly stored in the main memory area 23, and the extracted failure animation 44 is transmitted to the temporary storage unit 36 and temporarily stored in the temporary storage area. There are third to sixth modes for selecting failure animations 44 to be stored and stored in the main memory area 23 .

As shown in FIG. 7, a case in which the extracted failure animation 44 is not selected, which is common to the first and second storage modes, will be described. A failure scene of a failure frame image 42, a failure scene of a failure frame image 42a, and a failure scene of a failure frame image 42b, which are different failure scenes, are detected by failure detection processing in the medical video 40, and the detected failure information is is transmitted to the extraction unit 35 together with the The extraction unit 35 extracts a failure animation 44 for each failure scene. A failure movie 44 having a failure scene of the failure frame image 42, a failure movie 44a having a failure scene of the failure frame image 42a, and a failure movie 44b having a failure scene of the failure frame image 42b are created. The created failure animations 44, 44a, 44b are stored in the main memory area 23 in the order of extraction.

In the first save mode, the extraction range of the failure video 44 to be extracted is determined based on the failure scene included in the medical video 40 and the failure information corresponding to the failure scene. Specifically, the extraction range of the failure video 44 for each failure scene is determined based on the type of failure, the duration of the failure, and the average area of abnormal parts due to the failure in the frame images detected in the failure detection process. For example, in a failure scene with a long duration or in a failure scene whose type of failure is "disconnection", the duration of the normal frame images 41 to be extracted, that is, the range of the failure moving image 44 to be extracted is lengthened. The extraction range may be determined by the number of frame images, or may be determined by a time range based on the number of frame images per second.

In the second storage mode, the extraction range of the failure video 44 to be extracted is determined according to the failure scene included in the medical video 40 and the operation history of the endoscope 11 at timings near the failure scene. Specifically, they are an angle operation, a treatment instrument operation, a zoom operation, an imaging mode switching operation, and the like. This leads to identification of failures related to the operation of the endoscope 11 . In some cases, failures such as wire breakage are associated with the angle operation, and in some cases, failures related to the treatment tool can be confirmed by copying the treatment tool into an image. Since a failure can occur after several seconds have passed since the operation of the endoscope, the operation history of the timing before the failure scene is also targeted.

Determining the range of the failure animation 44 to be extracted according to the operation history is, for example, when the treatment tool is operated, the treatment tool can be protruded or retracted with a single operation, so a failure related to the treatment tool A failure scene due to a change in protrusion or retraction and a normal frame image 41 that can be compared with the failure scene are extracted in a relatively short range. When the angle operation is performed, the angle operation may be performed once over several seconds, or may be performed multiple times to gradually bend the insertion portion 11a of the endoscope 11 . Therefore, by extracting a large number of normal frame images 41, it becomes easier to confirm a failure so small that it is not detected as a failure, or to confirm a change in which the area of the abnormal region increases due to the failure.

A case will be described in which the extracted failure animation 44 is sorted based on the set extraction range, which is common to the third to sixth storage modes. When sorting is performed, the failure movie 44 temporarily stored in the temporary storage unit 36 is sorted based on the failure information, and the sorted failure movie 44 is stored in the main storage area 23 .

As shown in FIG. 8, for example, in a medical video 40, failure scenes of a failure frame image 42, a failure frame image 42a, and a failure frame image 42b, which are different failure scenes, are detected by failure detection processing, and failure scenes are detected together with failure information. It is transmitted to the extraction unit 35 . The extraction unit 35 extracts a failure movie 44 having a failure scene of the failure frame image 42, a failure movie 44a having a failure scene of the failure frame image 42a, and a failure movie 44b having a failure scene of the failure frame image 42b. The extracted failure videos 44, 44a, and 44b are temporarily stored in the temporary storage unit 36 in order. After all the extracted failure animations 44 are temporarily stored, the failure scenes are compared and selected by the selection unit 37 . When the failure animation 44 is selected, the failure animations 44 a and 44 b are deleted and the failure animation 44 is saved in the main memory area 23 .

In the third storage mode, the degree of importance of the failure animation 44 is calculated, and the failure animation 44 is selected based on the degree of importance. When the importance is used for sorting, the importance calculating unit 38 calculates the importance for each failure scene and associates it with the failure information of the failure scene. The degree of importance indicates the possibility of a serious failure and its usefulness in analyzing the cause of the failure. Calculation results that are less than or "low" are deleted. A method of calculating the degree of importance will be described later. Also, a certain number or a certain ratio of failure movies 44 may be selected in descending order of importance instead of the case where the importance is evaluated as a certain value or more.

In the fourth save mode, failure movies to be saved in the main memory area 23 are selected based on the chronological order of failure scenes detected from the medical movie 40 . For example, among a plurality of failure scenes detected in the medical video 40, the first or last N failure scenes or a failure video 44 having one selected failure scene at regular intervals is saved. In the first failure scene, a short period of time has passed since the endoscopy was started, and it is easy to detect failures other than those caused by the endoscopy, such as dirt on the scope. In the failure scene on the last side, when multiple failures occur, it is easy to detect multiple types of failures in one failure scene. In addition, it is possible to check whether there is a failure that occurs in the entire endoscopy by sorting at regular intervals.

In the fifth save mode, failure movies are sorted based on the types of failure scenes detected. When multiple failures occur, at least one failure animation 44 is saved in the main storage area 23 for each type. Among a plurality of failures, even if a type of failure that occurs with high frequency and has a large abnormal area and a type of failure that occurs with low frequency and has a small abnormal area are detected in the medical video 40, they are saved. When checking failure videos, you can check without overlooking each type of failure scene. Also, if different types of failure scenes are detected simultaneously or at short intervals, the failure animation 44 including both of them may be created as different types of failure scenes.

In the sixth save mode, the failure animation 44 to be saved in the main memory area 23 is sorted by user selection. Specifically, both the temporarily stored failure video 44 and image information including at least failure information are displayed on the display 15 , and the failure video 44 reproduced by the user and the image information are confirmed and stored in the main storage area 23 . A failure animation 44 is determined. The image information to be displayed may be information of the failure movie 44 or information for each displayed frame image.

As shown in FIG. 9, the display 15 has an image display area 50 for reproducing a failure animation 44 and confirming the contents of the failure, and an image information display column 51 for displaying image information including failure information of the failure animation 44 to be reproduced. indicate. In the image display area 50 , for example, the extracted failure animations 44 are reproduced in order, and whether or not to individually store them in the main storage area 23 is selected. The failure information includes, for example, failure type, failure scene time, abnormal area average area, and operation history.

User operations in the sorting process are performed via the user interface 16, and there are methods such as executing a playback command or a save command provided in the command area 52 using a keyboard, mouse, foot pedal, or the like. Also, image information may be edited by user operation.

As shown in FIG. 10 , instead of the method of sequentially reproducing and confirming failure animations 44 , a list display using thumbnails may be performed in the image display area 50 for sorting. A plurality of displayed thumbnail images of the failure animation 44 is selected, and the image information of the selected failure animation 44 is displayed in the image information display column 51 . Whether or not to use the command area 52 may be determined based on the image information. When the selected malfunction video 44 is to be reproduced, it may be reproduced in the size displayed as the thumbnail, or may be temporarily enlarged and reproduced as shown in FIG.

The failure movie storage mode is not limited to the contents of the first to sixth storage modes, and the failure movie 44 may be stored by combining the extraction criteria and selection conditions of each storage mode. Further, after storing the failure animation 44 in each storage mode in the main storage area 23, the stored failure animation 44 may be displayed on the display 15 for confirmation.

A series of operations for controlling storage of the failure animation 44 according to the present embodiment will be described along the flowchart shown in FIG. 11 . The endoscope system 10 acquires from the database 14 and the endoscope 11 the medical video 40 of the medical examination, which is the extraction source of the failure video 44 (step ST110). One of the storage modes for extracting and storing the failure animation 44 is selected according to the examination contents of the acquired medical animation 40 and the expected failure contents. (Step ST120). A failure detection process is executed to detect a failure scene from the medical moving image 40 (step ST130). If no failure scene is detected from the medical video 40 by the failure detection process (N in step ST140), it is determined that there is no failure. Another medical video 40 is acquired in order to extract the failure video 44 (step ST110). If a failure scene is detected in the medical video 40 by the failure detection process (Y in step ST140), the type of failure for each detected failure scene is determined (step ST150). A failure animation 44 having at least a failure scene is extracted from the medical animation 40 according to the selected storage mode and the extraction range according to the result of the failure detection processing (step ST160).

All of the extracted failure animations 44 may be stored as they are, or the failure animations 44 to be stored may be selected. Whether or not sorting is performed is determined by the save mode. If the save mode is the first save mode or the second save mode, that is, if the failure movie 44 is not selected (N in step ST170), all the extracted failure movies 44 are saved in the main storage area 23 (step ST200). .

If the save mode is the third to sixth save modes, that is, if failure scenes are to be screened (Y in step ST170), the failure animation 44 is temporarily saved together with image information including failure information (step ST180). The temporarily stored failure video 44 is
A judgment as to whether or not a certain criterion is satisfied is made, and the failure animations 44 are compared with each other to select the failure animation 44 to be stored (step ST190). The selected failure animation 44 is stored in the main memory area 23 (step ST200).

When the number of failure videos 44 is insufficient for analyzing the cause of the failure, or when changing the storage mode when extracting from the same medical video 40 (N in step ST210), the medical video 40 is acquired again. Then, it is preferable to detect the failure, extract the failure animation 44, and store it (step ST110). If enough failure animations 44 for failure cause analysis can be saved (Y in step ST210), a series of flow is terminated.

[Second embodiment]
In the above-described first embodiment, a failure scene is detected from each frame constituting the medical video 40 in the failure detection process, and the failure video 44 is extracted and stored for each detected failure scene. In the second embodiment, in addition to that, when the shape information of the endoscope 11 is acquired and the extracted failure animation 44 is stored, both the shape information and the failure information are stored. note that. The description of the contents common to the above embodiment will be omitted.

When saving the failure animation 44 , the failure animation 44 in which the shape information and the failure information are linked is saved in the main storage area 23 . The shape information may be used to analyze the failure cause of the failure animation 44 . Also, the shape information of the endoscope 11 is used to extract the failure animation 44 . Shape information may be used not only for extraction but also for selection.

As shown in FIG. 12, in the failure detection process of the medical moving image 40, the bending of the insertion portion 11a at a certain distance from the distal end portion 11c of the endoscope 11 as shown in FIG. 12(B), the bending of the insertion portion 11a at a certain distance from the distal end portion 11c of the endoscope 11 is about 90 degrees with respect to the operation portion 11b. A failure may or may not be detected depending on the physical shape of the endoscope 11 , such as when the endoscope 11 is bent to a degree and disconnection occurs. In that case, disconnection does not occur with a small number of angle operations such as one or two times, and the angle operation may be bent gradually over time rather than in a short period of time, so the operation history does not necessarily track sufficiently. Not always possible. Also, when the curvature of the insertion portion 11a at a certain distance from the distal end portion 11c of the endoscope 11 is approximately 180 degrees with respect to the operation portion 11b, as shown in FIG. A bend of about 90 degrees may be detected as a more severe fault. Therefore, the failure detection processing including the shape information of the insertion portion 11a of the endoscope 11 can perform failure detection processing with higher accuracy than the failure detection processing of the medical animation 40 by machine learning or the like.

Specifically, by using the endoscope system 10 having a scope shape information sensor, failure detection processing and change in the shape information of the endoscope 11 are combined on a frame-by-frame basis. It is preferable to store the shape information with failure scene detection as a trigger. This prevents a failure such as disconnection that occurs during an endoscopy from being determined to be due to external noise, or a failure scene whose importance is calculated to be low, enabling highly accurate detection. In addition, in the analysis of the cause of the failure, it becomes easier to repair the failure by obtaining the result that the occurrence of the failure requires a specific shape.

As the shape information sensor of the insertion portion 11a of the endoscope 11, it is preferable to use a function of acquiring shape information by generating magnetism from the outside, such as an endoscope insertion shape observation device (coronavi). In addition to the shape information sensor, the endoscope 11 may acquire shape information using a function not intended to detect failure of the distal end, such as a dynamic sensor or an insertion depth gauge.

[Third embodiment]
The first embodiment and the second embodiment described above are forms in which the medical video 40 obtained by photographing the entire endoscopy is acquired, and the failure video 44 is extracted and stored. In the present embodiment, a mode will be described in which failure detection processing is performed on frame images received by the processor device while endoscopic inspection is being performed in real time, and a failure moving image 44 is extracted. Note that the description of the contents common to the above embodiment will be omitted.

Perform fault detection processing in real time during endoscopy. When the endoscopy starts, the image acquiring unit 21 receives one or a few frame images and inputs them to the failure animation creating unit 30 . The failure detection unit 31 sequentially performs failure detection processing on the frame images input to the failure animation creation unit 30 to detect failure scenes.

A failure animation 44 is extracted for each failure scene. In the case of a storage mode in which selection is not performed, the extracted failure animation 44 is stored in the main memory area 23 . In the case of the storage mode for sorting, the extracted failure animation 44 is temporarily stored in the temporary storage unit 36 . The temporarily stored failure animation 44 is sorted according to the storage mode, and is stored in the main storage area 23 after sorting. Note that it is not necessary to create the medical moving image 40 that records the entire medical examination.

The function of the temporary storage unit 36 may be implemented by a non-volatile memory that maintains stored data even when the processor device 13 is not powered. If the power of the endoscope 11 or the processor device 13 is turned off during the endoscopic examination, or if the processor device 13 is not completely saved in the main storage area 23 after the endoscopic examination is completed, A situation may occur in which the device 13 is powered off. Even in such a case, the data temporarily stored in the temporary storage unit 36, which is a non-volatile memory, can be prevented from being lost when the power is turned off.

In the storage mode for selection, the failure animation 44 extracted for each failure is temporarily stored in the temporary storage unit 36, which is a non-volatile memory. When the power of the endoscope 11 or the processor device 13 is turned off during the endoscopy, the endoscopy is finished at that time, and when the power is turned on again, sorting of the failure animation 44 is started. do. After the endoscopy is completed, sorting is performed, and the sorted failure animation 44 is stored in the main memory area 23 .

As shown in FIG. 13, a case where the power is turned off during inspection in real-time detection in which data is temporarily stored in a temporary storage area, which is a non-volatile memory, will be described. Examination A is an examination when the endoscope examination is completed without turning off the power, and examination B is an examination when the power of the endoscope 11 or the processor device 13 is turned off during real-time detection. In the examination A for which the user has operated the end of the endoscopy, sorting and saving are performed on the temporarily saved failure video 44 after the end of the endoscopy. In inspection B, which was forcibly terminated when the power was turned off instead of being terminated by a user's operation, when the power was turned on again, the failure movie 44 that was saved in the non-volatile memory at the time the power was turned off are selected and stored. By temporarily storing it in the non-volatile memory, it is possible to sort out the failure animation 44 regardless of the completion status of the endoscopy, such as when the power is turned off during the examination.

After the sorting is finished, the sorted failure video 44 is saved from the non-volatile memory to the main memory area 23. However, if the processor device 13 is powered off during saving or if the main memory area 23 is removed from the processor device 13, If it is possible, the connection may be cut off. If the saving process is interrupted, it may not be possible to distinguish between the failure video 44 saved in the main memory area 23 and the failure video 44 that is not saved. Therefore, an identifier such as a flag is set when extracting the failure animation 44 for automatic determination.

When the failure animations 44 are temporarily saved, different identifiers are set for the respective failure animations 44 when extracting the failure animations. The identifier is a flag or the like, and is preferably set as image information. When resuming the interrupted saving process, the identifier of the failure movie 44 saved in the main memory area 23 is confirmed, and the presence or absence of the failure movie 44 having the same identifier is determined in the temporary storage area, which is a non-volatile memory. The failure video 44 having the same identifier as the confirmed identifier is deleted from the non-volatile memory without being saved.

If the failure video 44 for which the same identifier has not been confirmed exists in the temporary storage area, it is determined that the failure video 44 is not saved in the main storage area 23, and storage processing is performed. It should be noted that the identifier may be used not only when the saving process is interrupted, but also for confirming whether the saving is surely performed during the normal saving process.

In the third embodiment (real-time failure detection), the sorting of failure videos 44 may be performed immediately after extraction of the failure videos 44 without waiting for the timing when all the failure videos 44 are available, that is, until the end of the inspection. For example, the level of importance of the m-th extracted failure video 44 that has already been temporarily stored is compared with the newly extracted (m+1)-th failure video 44, and the one with the lower level of importance is deleted. As a result, the memory area can be saved when the area for temporary storage is limited.

In the above embodiment, the central control unit, the image acquisition unit 21, the input reception unit 22, the display control unit 24, and the failure detection unit 31 included in the failure video creation unit 30, the extraction range determination unit 32, the mode control unit 33, the image information The hardware structure of the processing unit (processing unit) that executes various processes such as the management unit 34, the extraction unit 35, the temporary storage unit 36, the selection unit 37, and the importance calculation unit 38 includes various processor. Various processors include CPUs (Central Processing Units) and FPGAs (Field Programmable Gate Arrays), which are general-purpose processors that run software (programs) and function as various processing units. Programmable Logic Devices (PLDs), which are processors, and dedicated electric circuits, which are processors having circuitry specifically designed to perform various processes.

One processing unit may be composed of one of these various processors, or composed of a combination of two or more processors of the same type or different types (for example, a plurality of FPGAs or a combination of a CPU and an FPGA). may be Also, a plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units in one processor, first, as represented by computers such as clients and servers, one processor is configured by combining one or more CPUs and software, There is a form in which this processor functions as a plurality of processing units. Secondly, as typified by System On Chip (SoC), etc., there is a form of using a processor that realizes the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip. be. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

Further, the hardware structure of these various processors is, more specifically, an electrical circuit in the form of a combination of circuit elements such as semiconductor elements. The hardware structure of the storage unit is a storage device such as an HDD (hard disc drive) or an SSD (solid state drive).

10 endoscope system 11 endoscope 11a insertion portion 11b operation portion 11c tip portion 12 light source device 13 processor device 14 database 15 display 16 user interface 21 image acquisition portion 22 input reception portion 23 main storage area 24 display control portion 30 failure video Creation unit 31 Failure detection unit 32 Extraction range determination unit 33 Mode control unit 34 Image information management unit 35 Extraction unit 36 Temporary storage unit 37 Selection unit 38 Importance calculation unit 40 Medical video 41 Normal frame image 42 Failure frame image 42a Failure frame image 42b Failure frame image 43 Failure-free abnormal frame image 44 Failure animation 44a Failure animation 44b Failure animation 50 Image display area 51 Image information display area 52 Command area C Disconnection R Lesion area T Non-lesion abnormal area

Claims (15)

with a processor
The processor
Detect failure of the endoscope from medical videos taken with the endoscope,
determining the type of failure;
extracting a failure video having a failure scene at the timing at which the failure was detected from the medical video;
An endoscope system that stores the failure video in a main storage area together with failure information including at least the type of failure. The processor
2. The endoscope system according to claim 1, wherein the failure animation having a plurality of frame images that are not the failure scene is extracted before, before, or after the failure scene. The processor
3. The endoscope system according to claim 1, wherein an extraction range of said failure animation to be extracted is determined based on said failure scene and said failure information corresponding to said failure scene. The processor
4. The endoscope system according to any one of claims 1 to 3, wherein an extraction range of said failure animation to be extracted is determined according to said failure scene and an operation history at a timing near said failure scene. The processor
temporarily storing the failure video in a temporary storage area;
sorting the failure video based on the failure information;
3. The endoscope system according to claim 1, wherein the screened failure animation is stored in a main storage area. The processor
Calculate the importance of the failure video,
6. The endoscope system according to claim 5, wherein said sorting is performed based on said degree of importance. The processor
6. The endoscope system according to claim 5, wherein the sorting is performed based on the chronological order of the failure scene detected from the medical moving image. The processor
6. The endoscope system according to claim 5, wherein at least one failure animation is stored in a main storage area for each failure type. The processor
6. The endoscope system according to claim 5, wherein the sorting is performed by a user's operation. The processor
Acquiring shape information of the endoscope,
10. The endoscope system according to any one of claims 1 to 9, wherein the shape information is stored in the main storage area together with the failure animation and the failure information when storing the failure animation. The processor
The endoscope system according to claim 10, wherein the failure animation is extracted using the shape information. The processor
The endoscope system according to any one of claims 1 to 9, wherein the failure is detected during endoscopy and the failure animation is extracted. The processor
temporarily storing the failure video in a non-volatile memory;
Regardless of the completion status of the endoscopy, sort out the temporarily stored failure video,
13. The endoscope system according to claim 12, wherein the selected failure animation is saved from the nonvolatile memory to the main storage area. The processor
setting different identifiers for each of the failure videos when extracting the failure videos;
confirming the identifier of the failure video saved in the main storage area;
14. The endoscope system according to claim 13, wherein the failure animation having the same identifier as the confirmed identifier is deleted from the nonvolatile memory. detecting a failure of the endoscope from a medical video taken by the endoscope;
determining the type of failure;
a step of extracting a failure video having a failure scene at the timing at which the failure was detected from the medical video;
A method of operating an endoscope system, comprising a step of storing the failure video together with failure information including at least the type of failure in a main storage area.

Images