CN117893953A

CN117893953A - Soft digestive tract endoscope operation standard action evaluation method and system

Info

Publication number: CN117893953A
Application number: CN202410296759.2A
Authority: CN
Inventors: 刘与之; 赵锐; 张海瑞
Original assignee: Sichuan Deep Blue Bird Technology Co ltd
Current assignee: Sichuan Deep Blue Bird Technology Co ltd
Priority date: 2024-03-15
Filing date: 2024-03-15
Publication date: 2024-04-16

Abstract

The application discloses a soft digestive tract endoscope operation specification action evaluation method and system, and relates to the technical field of medical systems. The method comprises the steps of collecting images of targets involved in digestive tract endoscope operation, splitting actions of the targets in the digestive tract endoscope operation, and establishing a digestive tract endoscope standard operation specification database; comparing the identified actions of the target in the digestive tract endoscope operation with corresponding actions in the standard operation specification database; outputting the comparison result of the current action of the target and the standard data in real time, and reminding when the current action evaluation is not standard; after the operation of the digestive tract endoscope is integrally completed, all comparison results are summarized and evaluated, the operation state of an endoscope doctor is detected in real time, the standard is corrected immediately, the consistency and quality of the operation skills of the doctor are effectively improved, and the integral level and diagnosis and treatment quality of the endoscope diagnosis and treatment technology are improved.

Description

Soft digestive tract endoscope operation standard action evaluation method and system

Technical Field

The application relates to the field of medical systems, in particular to a soft digestive tract endoscope operation specification action evaluation method and system.

Background

Soft digestive tract endoscopy is a common medical diagnostic tool, and comprises an upper digestive tract endoscope, an esophagoscope, a gastroscope, a colorectal endoscope, a sigmoidoscope and other electronic endoscopes or fiber endoscopes, which are used for checking the internal conditions of upper and lower digestive tract organs of human bodies such as throat, esophagus, stomach, duodenum, colon, rectum and the like. The examination may help doctors to find and evaluate a number of diseases and lesions, such as ulcers, inflammation, bleeding, tumors, stenosis, and foreign bodies. It can also be used for sampling biopsy to diagnose diseases, and can be used for treating tumors by minimally invasive operations, such as mucoectomy or submucosal dissection of early cancers of the digestive tract, and can also be used for hemostasis treatment of acute digestive tract hemorrhage. Therefore, the soft gastrointestinal endoscope plays an extremely important role in diagnosis and treatment of total gastrointestinal diseases.

However, improper upper gastrointestinal endoscopy operation may lead to the following adverse consequences, including:

complications increase: the endoscope should be inserted carefully and with proper technique to ensure smooth passage through the esophagus, stomach and duodenum. If the insertion process is too rapid or too rough, severe complications such as mucosal injury, bleeding or pain, and even perforation may result. Irregular operation may lead to increased incidence of complications such as bleeding, perforation, infection, etc., which pose a threat to the health of the patient.

Pain and discomfort: improper insertion, advancement, or biopsy procedures may cause pain, discomfort, etc. to the patient and affect patient comfort and experience.

The inspection result is inaccurate: the irregular operation may result in poor quality biopsy samples, failing to provide accurate pathological diagnosis results, thereby affecting the judgment of the disease and the selection of treatment schemes.

The early digestive tract cancer detection rate is low: the irregular operation can lead to incomplete examination and dead angles, and lead to missed diagnosis of early esophageal cancer, gastric cancer and colorectal cancer.

In conclusion, when a digestive endoscopist performs soft gastrointestinal endoscopy, related operation standards and technical requirements should be strictly adhered to, safety and comfort of patients are emphasized, and accuracy and success rate of examination are improved. However, the standardized diagnosis and treatment level of the soft digestive tract endoscope in China and the international advanced level have a certain gap. The operation of the soft type digestive endoscopy has no unified standard, the teaching generally adopts a mode of 'master band freehand', the homogenization is easy, the mode is backward, the operation method and the posture are various, the standardization of the operation of the endoscope is difficult to achieve, and the operation level of digestive endoscopists in China, especially basic-level endoscopists, is low, and the missed diagnosis rate is high. Therefore, in endoscopy, compliance with the technical specifications of soft digestive tract endoscope operation should be emphasized, and indications and contraindications of endoscope diagnosis and treatment should be strictly grasped. And (3) culturing standardized operation of each endoscope doctor, establishing standard operation technical actions of various soft digestive endoscopes including upper digestive tract endoscopes, esophagoscope, gastroscopes, colorectal lenses, sigmoidoscopes and the like), further strengthening standardized diagnosis and treatment of the endoscopes, continuously improving diagnosis and treatment technology and quality of the domestic endoscopes, improving early diagnosis and early treatment level of digestive tract cancers, and better serving patients. At present, the operation assessment of the digestive endoscopist is mainly scored by side observation of the superior physicians with abundant experience, the efficiency is low, and the operation assessment is difficult to popularize into the quality control of daily endoscope diagnosis and treatment work. In summary, it is imperative to establish a soft-type enteroscope operation specification action evaluation method and system for specifying an endoscopist's soft-type enteroscope operation.

Disclosure of Invention

Aiming at the problems that the conventional soft digestive tract endoscope operation lacks uniform standard and the operation specification difference among different doctors causes the non-standardization of the endoscope operation, the application provides an innovative soft digestive tract endoscope operation specification action evaluation method and system. Through establishing the standard operation specification database of the digestive tract endoscope, and adopting the artificial intelligence technology to carry out standardized evaluation on the operation actions of doctors, the patent aims at unifying the operation standards of the endoscope and improving the consistency and quality of the operation skills of the doctors, thereby continuously improving the overall level and diagnosis and treatment quality of the domestic endoscope diagnosis and treatment technology. The specific technical scheme is as follows:

in a first aspect of the present application, there is provided a soft-type enteron endoscope operation specification action evaluation method, comprising:

image acquisition is carried out on a target involved in the digestive tract endoscope operation, the motion of the target in the digestive tract endoscope operation is split, the target and the motion of the target are identified, and a digestive tract endoscope standard operation specification database is established;

the method comprises the steps of collecting video streams in real time in the operation process of the digestive tract endoscope, extracting features of the video streams collected in real time, and identifying actions of the targets in the video streams in the operation of the digestive tract endoscope;

Comparing the identified actions of the target in the digestive tract endoscope operation with corresponding actions in the standard operation specification database;

outputting the comparison result of the current action of the target and the standard data in real time, and reminding when the current action evaluation is not standard; and (5) carrying out summarization evaluation on all comparison results after the whole digestive tract endoscope operation is completed.

In an embodiment of the present application, in the step of establishing the database of standard operation specifications of the digestive tract endoscope, a specific operation process of the digestive tract endoscope acquires a video stream through a camera and presents the video stream, and performs target detection and human body key point detection on the video stream:

the target detection includes identifying a doctor, a patient, an assistant, an endoscope, and a biopsy forceps;

the human body key point detection comprises the steps of identifying the gesture of a patient, identifying the action of a doctor and identifying the action of an assistant, specifically identifying preset skeleton points of the doctor, the patient and the assistant, and identifying the action of the target through the position, the angle and the gesture change of the preset skeleton points;

and judging the video stream collected by the camera, storing the video stream of which the action of the target belongs to the standard operation, and arranging according to the sequence of the standard operation of the digestive tract endoscope to form the standard operation specification database for establishing the digestive tract endoscope.

In an embodiment of the present application, the step of performing real-time video stream acquisition on the operation process of the digestive tract endoscope specifically includes in-vitro video acquisition and in-vivo video acquisition:

the external video acquisition is that a doctor controls the rotation, the forward and backward movement, the water injection and the gas injection and the object taking of the endoscope through different postures and different operation control host machine knobs in different steps, the external video acquisition device acquires external operation videos and decomposes actions of the doctor, an assistant and a patient, compares the actions with corresponding actions in the standard operation specification database, and outputs comparison results;

the in-vivo video acquisition is to acquire in-vivo video of a patient through an endoscope, transmit the in-vivo video of the patient to a doctor in real time, compare the in-vivo video with corresponding actions in the standard operation specification database and output a comparison result.

In an embodiment of the present application, in the step of performing real-time collection of video stream during the operation process of the digestive tract endoscope, the method specifically includes the steps of further including in-vivo and in-vitro video matching, when the in-vivo video of the patient is collected by the endoscope, and a preset condition occurs, and when a doctor and an assistant need to make corresponding actions in vitro, the in-vitro video collector collects the in-vitro video, and the actions of the doctor and the assistant are collected by the in-vitro image collector and compared with the standard operation specification database, and the comparison result is output.

In an embodiment of the present application, the step of comparing the identified action with the canonical data of the action in the database specifically includes:

dynamic time normalization, which is to perform time axis normalization for time sequence data of operation actions;

pattern feature extraction, namely extracting key pattern features from the normalized time sequence through a deep learning network comprising a convolutional neural network and a cyclic neural network;

similarity scoring, namely calculating the similarity between the real-time action characteristics and the standard action mode characteristics by a measurement method comprising cosine similarity;

and judging and feeding back the threshold value, judging the normalization of the operation according to the similarity score and a preset threshold value, and providing real-time feedback.

In an embodiment of the present application, the step of arranging according to the sequence of continuous actions specifically includes:

arranging the key point coordinates and time information of doctors, patients and assistants in the video stream into sequence data, wherein each sequence represents a key point track of a person, the time is one dimension in the sequence, and a time sequence data set is obtained, and each sequence is associated with the action of a target;

Inputting a sequence data set containing key point coordinates and time information into a trained RNN model, and predicting the action of a target;

comparing the predicted result with the action of the standard operation specification database to judge whether the predicted result meets the requirements; if the action does not meet the specification, the system gives a prompt.

In an embodiment of the present application, before the step of identifying the motion of the target in the operation of the digestive tract endoscope, the method further includes video stream preprocessing, where the video stream preprocessing includes image adjustment and background segmentation, and the image adjustment is to unify parameter adjustment including brightness and contrast of a real-time video stream with a corresponding video stream in the standard operation specification database; the background segmentation separates the target object from the background by a deep learning algorithm.

In an embodiment of the present application, in the step of summarizing and evaluating all the comparison results after the overall operation of the digestive tract endoscope is completed, setting scores and corresponding scoring criteria for corresponding actions in the standard operation specification database, scoring the actions identified in the overall operation flow of the digestive tract endoscope one by one, integrating all the scores after the overall operation flow is completed, and generating an operation report.

In an embodiment of the present application, after the step of outputting, in real time, a comparison result of the current action of the target and the specification data, and when the current action evaluation is not specified, the step of prompting further includes:

when a doctor considers that the action currently performed is not irregular according to the self condition of the patient, the reminding is canceled;

after the integral operation of the digestive tract endoscope is finished, judging a corresponding action for canceling reminding by a doctor, and if the corresponding action belongs to standard operation, storing a corresponding video stream into the standard operation specification database; and if the corresponding action belongs to standard operation which needs the pre-condition to exist, supplementing the pre-condition, and if the pre-condition appears again, using the corresponding video stream of the corresponding action as standard operation, wherein the pre-condition comprises physiological data of the patient.

In a second aspect of the present application, there is provided a soft enteron endoscope operation specification action evaluation system comprising:

the standard database module is used for carrying out image acquisition on a target involved in digestive tract endoscope operation, splitting actions carried out on the target in the digestive tract endoscope operation, identifying the target and the actions carried out on the target, and establishing a digestive tract endoscope standard operation specification database;

The data acquisition module is used for carrying out real-time acquisition on video streams in the operation process of the digestive tract endoscope, extracting characteristics of the video streams acquired in real time and identifying actions of the targets in the video streams in the operation of the digestive tract endoscope;

an action comparison module for comparing the identified actions of the target in the digestive tract endoscope operation with the corresponding actions in the standard operation specification database;

the result output module outputs the comparison result of the action currently done by the target and the standard data in real time, and reminds when the action evaluation currently done is not standard; and (5) carrying out summarization evaluation on all comparison results after the whole digestive tract endoscope operation is completed.

In an embodiment of the present application, in the standard database module, a specific operation process of the digestive tract endoscope acquires a video stream through a camera, and performs target detection and human body key point detection on the video stream:

In an embodiment of the present application, the data acquisition module specifically includes an in vitro video acquisition sub-module and an in vivo video acquisition sub-module:

the external video acquisition submodule is used for enabling a doctor to control the rotation, the forward and backward movement, the water injection and the gas injection and the object taking of the endoscope through different postures and different operation control host machine knobs of different steps, enabling an external video acquisition device to acquire external operation videos, decomposing actions of the doctor, an assistant and a patient, comparing the actions with corresponding actions in the standard operation specification database, and outputting comparison results;

the in-vivo video acquisition submodule acquires in-vivo videos of a patient through an endoscope, transmits the in-vivo videos of the patient to a doctor in real time, compares the in-vivo videos with corresponding actions in the standard operation specification database, and outputs a comparison result.

In an embodiment of the present application, the data acquisition module further includes an in-vivo and in-vitro video matching sub-module, when the endoscope acquires the in-vivo video of the patient and the preset condition occurs, and when a doctor and an assistant need to make corresponding actions in vitro, the in-vitro video acquisition device acquires the in-vitro video, and the in-vitro image acquisition device acquires actions of the doctor and the assistant and compares with the standard operation specification database, and outputs a comparison result.

In an embodiment of the present application, the data comparison module specifically includes:

the time normalization sub-module is used for carrying out dynamic time normalization and carrying out time axis normalization on time sequence data of the operation action;

the feature extraction sub-module is used for extracting pattern features, and extracting key pattern features from the normalized time sequence through a deep learning network comprising a convolutional neural network and a cyclic neural network;

the evaluation sub-module is used for scoring the similarity and calculating the similarity between the real-time action characteristics and the standard action mode characteristics through a measurement method comprising cosine similarity;

and the feedback sub-module is used for judging and feeding back the threshold value, judging the normalization of the operation according to the similarity score and the preset threshold value and providing real-time feedback.

In an embodiment of the present application, the arranging according to the sequence of continuous actions specifically includes:

In an embodiment of the present application, the data acquisition module further includes a video stream preprocessing sub-module, where the video stream preprocessing includes image adjustment and background segmentation, and the image adjustment is to unify parameter adjustment including brightness and contrast of a real-time video stream with a corresponding video stream in the standard operation specification database; the background segmentation separates the target object from the background by a deep learning algorithm.

In an embodiment of the present application, the ending output module specifically includes setting scores and corresponding scoring criteria for corresponding actions in the standard operation specification database, scoring actions identified in the overall procedure of the digestive tract endoscope operation one by one, integrating all scores after the overall procedure is completed, and generating an operation report.

In an embodiment of the present application, the ending output module further includes:

The application has the following beneficial effects:

splitting actions generated by each target in the operation of the digestive tract endoscope one by one, storing a standard operation video into a database, collecting the operation process of a doctor in real time by establishing a standard operation specification database of the digestive tract endoscope and unifying the standard, identifying the step of the current action as the whole operation process, comparing the step with the corresponding action in the standard operation specification database, prompting when the evaluation of the current action is not normal, namely when the difference between the current action and the corresponding action in the standard operation specification database exceeds a threshold value, carrying out corresponding action adjustment by the doctor, avoiding adopting a mode of 'master zone free' which is easy to homogenize, has various operation methods and gestures and is difficult to achieve standardization of the operation of the endoscope, detecting the operation state of the endoscope doctor in real time by an action detection system, correcting the standardization operation of the endoscope in real time, establishing a unified standard, improving the technical and quality of diagnosis and treatment of the doctor in China, and better serving patients.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure.

Fig. 1 is a schematic structural diagram of an electronic device in a hardware running environment according to an embodiment of the present application.

Fig. 2 is a flowchart of steps provided in an embodiment of the present application.

Fig. 3 is a schematic diagram of a functional module provided in an embodiment of the present application.

Fig. 4 is a schematic diagram of a technical route provided in an embodiment of the present application.

Fig. 5 is a schematic diagram of a YOLOv8 model network structure according to an embodiment of the present application.

Fig. 6 is a schematic diagram of key points at the human body 17 according to an embodiment of the present application.

Fig. 7 is a schematic diagram of a patient's left lateral recumbent position provided in an embodiment of the present application.

Fig. 8 is a schematic view of a doctor standing position provided in an embodiment of the present application.

Fig. 9 is a schematic diagram of a left upper arm patch body according to an embodiment of the present application.

Fig. 10 is a schematic diagram providing a left hand mirror height in accordance with an embodiment of the present application.

Fig. 11 is a schematic diagram of a right-hand mirror gesture provided in an embodiment of the present application.

Fig. 12 is a schematic diagram of a right hand position provided in an embodiment of the present application.

Detailed Description

In order that the above-recited objects, features and advantages of the present application will become more readily apparent, a more particular description of the invention briefly described above will be rendered by reference to specific embodiments that are illustrated in the appended drawings. It will be apparent that the embodiments described are some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.

The following further describes the aspects of the present application with reference to the accompanying drawings.

As shown in fig. 1, the electronic device may include: a processor 1001, such as a central processing unit (Central Processing Unit, CPU), a communication bus 1002, a user interface 1003, a network interface 1004, a memory 1005. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display, an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a WIreless interface (e.g., a WIreless-FIdelity (WI-FI) interface). The Memory 1005 may be a high-speed random access Memory (Random Access Memory, RAM) Memory or a stable nonvolatile Memory (NVM), such as a disk Memory. The memory 1005 may also optionally be a storage device separate from the processor 1001 described above.

Those skilled in the art will appreciate that the structure shown in fig. 1 is not limiting of the electronic device and may include more or fewer components than shown, or may combine certain components, or may be arranged in different components.

As shown in fig. 1, an operating system, a data storage module, a network communication module, a user interface module, and an electronic program may be included in the memory 1005 as one type of storage medium.

In the electronic device shown in fig. 1, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the electronic device of the present invention may be provided in the electronic device, and the electronic device invokes a soft digestive tract endoscope operation specification action evaluation method stored in the memory 1005 through the processor 1001, and executes a soft digestive tract endoscope operation specification action evaluation system provided in the embodiment of the present application.

Based on the foregoing hardware operating environment and system architecture, in a first aspect of the present application, referring to fig. 2, there is provided a soft-type enteron endoscope operation specification action evaluation method, including:

In this embodiment, the actions generated by each target in the operation of the digestive tract endoscope are split one by one, the standard operation video is stored in the database, the operation process of the doctor is collected in real time by establishing the standard operation specification database of the digestive tract endoscope and unifying the standard, the step of the current action as the whole operation process is identified, and compared with the corresponding actions in the standard operation specification database, when the evaluation of the current action is not normal, that is, when the difference between the current action and the corresponding action in the standard operation specification database exceeds the threshold value, the doctor is reminded, and the corresponding action adjustment is performed, so that the mode of easy homogenization, various modes of behind the mode and various operation modes and difficult to reach the standardization of the operation of the endoscope is avoided.

In an embodiment of the present application, in the step of establishing the database of standard operation specifications of the digestive tract endoscope, a specific operation process of the digestive tract endoscope is presented by capturing a video stream through a camera, as shown in fig. 4, and performing target detection and human body key point detection on the video stream:

In the embodiment, the targets involved in the digestive tract endoscope operation process and the corresponding actions of the targets are respectively identified and detected, wherein the targets comprise doctors, patients, assistants, endoscopes and biopsy forceps at first, and the identities of the doctors, the patients and the assistants can be accurately identified by introducing the appearance information as a measurement standard, so that the accuracy of an operation object, the operation tools such as the endoscopes and the biopsy forceps is ensured; meanwhile, the human body key point detection is used for identifying the gesture of a patient, the action of a doctor and the action of an assistant, wherein the human body gesture estimation has wide application in computer vision, including behavior identification, human body tracking and the like. This technique allows the system to accurately identify and track critical skeletal points of the human body, providing a basis for in-depth analysis. The use of human pose estimation for upper gastrointestinal endoscopy operation specification motion detection has the following advantages:

Accurate action positioning: the human body pose estimation provides detailed information on the physician's operation actions, including hand position, angle and pose changes. This provides a very detailed analysis of the operational actions and allows for a deep understanding of the accuracy and consistency of each critical action.

Real-time feedback: in contrast to conventional video analysis techniques, human body pose estimation can provide real-time motion recognition and analysis, providing immediate feedback to doctors, helping them immediately correct possible errors.

Deep action analysis: human posture estimation provides the ability to analyze the physician's operating skills in depth, as subtle motion differences can be captured, thereby ensuring that they follow the operating specifications.

Evaluation based on criteria: human body posture estimation ensures that each operation meets the established medical and technical standards by comparing with the predefined standard operation posture

Adaptability to multiple applications: in addition to motion detection, human body pose estimation may also assist other applications, such as human body tracking and behavior recognition, providing a more comprehensive operational assessment.

Therefore, the human body posture estimation is selected to identify various actions of the doctor in the operation of the upper gastrointestinal endoscope, and then the actions are compared with the actions in the standard behavior library, so that the normalization of the doctor operation is ensured in real time; specifically, 17 human body key points of a doctor, a patient and an assistant are detected respectively, and then according to the change condition of the key points of adjacent frames of video in space, the actions of the doctor, the patient and the assistant can be simplified, so that whether the operation of the doctor is consistent with the actions in a standard library, whether the lateral posture of the patient is compliant and whether the operation of the assistant is standard can be identified, and 17 key points of a human body are shown in fig. 6.

Specifically, the human body posture is identified by taking the YOLOv8 model as a target detection and human body key point identification algorithm. The pedestrian re-recognition technology can accurately recognize the same person under different environments and illumination conditions and can distinguish the identities of different persons, so that the pedestrian re-recognition technology is used for locking the identities of doctors, patients and assistants in the inspection process.

The YOLOv8 is mainly composed of a feature extraction network (backbone network), a feature fusion network (neck) and a detection head (head), and the network structure is shown in fig. 5. The feature extraction network is responsible for feature extraction, and the extracted feature graphs with different scales are then sent to the feature fusion network and the detection head. The feature graphs are sent into a feature pyramid through processing of a C2f convolution module and other convolution modules, and finally the bounding box positioning and classification of the targets are completed.

The feature extraction network is a series of depth optimizations and improvements based on YOLOv 5. The feature extraction network adopts an innovative method, namely, the original input is subjected to channel expansion through two-dimensional convolution with the step length of 2 and the core size of 3, and a Focus module is avoided, so that the parameter quantity is reduced. Based on CSPNet (Cross Stage Partial Network), the C2F module in the network replaces the C3 module in YOLOv5 and optimizes the process of feature extraction and fusion. In the C2F module, the features are first subjected to 1x1 convolution to perform channel merging, then the feature tensor is divided into two parts, one part enters the Bottleneck block to further extract the features, and the other part is subjected to channel stitching with the features processed by the Bottleneck block. The Bottleneck block utilizes a residual structure, extracts features through two convolutions, and then performs element-by-element addition operation with the original input. The structure not only strengthens the feature extraction capability of the convolutional neural network and reduces the time consumption of memory access, but also can reserve the basic features of the original input and avoid the problem of gradient disappearance. In addition, the network also maintains an SPPF pooling module, and the adaptability of the model to input images with different sizes is enhanced. Thus, the feature extraction network of YOLOv8 can efficiently extract features of doctors, patients, and assistants.

The feature fusion network employs a modified PANet (Path Aggregation Network) architecture that is optimized based on the original FPN (Feature Pyramid Networks). This improved network architecture is primarily intended to more effectively fuse features at different levels. Specifically, by upsampling the smaller feature layer and splicing and fusing with the larger feature layer, the network can directly output rich large-dimension feature map information. Further, the PANet can downsample the large-dimension features and re-fuse the large-dimension features with the small-dimension feature map so as to enrich the context information of the model and enhance the expression capability of the small-dimension features. Meanwhile, the step of channel integration is omitted before the up-sampling of the small-dimension feature map, and a C2F module is used for replacing a C3 module, so that the image features are extracted and fused more accurately, and the recognition and positioning accuracy of the target is optimized.

The decoupling detection head is used for the detection head, so that the network is allowed to respectively classify and return at the head part through two different convolution modules, and the flexibility and accuracy of the model are enhanced. The design refers to the structures of YOLOX and YOLOv 6. After the feature map enters the header portion, feature extraction is performed by two identical convolutional pooling activation modules. Since DFL (Distribution Focal Loss) is used to calculate the loss, the number of channels in the regression head is specifically adjusted by the network, specifically, the channel adjustment is performed by (1×1) convolution with the output channel (4×reg_max) in order to calculate the regression loss. Meanwhile, the model is correspondingly adjusted in the classification head part, and the channel adjustment is carried out by adopting (1 multiplied by 1) convolution with the same output channel number as the class number, so as to calculate the classification loss. The design of the decoupling detection head enables the model to be more accurate and efficient in processing classification and regression tasks.

The classification Loss employs a VFL Loss, the main improvement being the introduction of an asymmetric weighting operation. The expression of VFL Loss is as follows, where q is the label and in the classification Loss (VFL Loss), the distinction between positive and negative samples is typically defined based on their relationship to the target (group trunk). In the object detection task of this patent:

positive samples: refers to a prediction box (pro-sams or bounding boxes) that has significant overlap with the real target (ground truth bounding boxes), such as the values calculated by IoU-Intersection over Union. In other words, if the degree of overlap of a prediction block with a certain real object is higher than a certain preset threshold (e.g., 0.5), this prediction block is regarded as a positive sample. In VFL Loss, the label q of the positive sample is the IoU value of the predicted box and the real target box, which indicates the degree of matching between the predicted box and the real target.

Negative sample: in contrast, negative examples are those prediction frames that overlap with the real object to a very low degree or no overlap at all. This means that these prediction boxes do not effectively capture any object of interest. In the definition of VFL Loss, the label q for the negative samples is set to 0, indicating that there is no or little overlap between the samples and any real target. For positive samples, q is IoU for bbox and ground trunk, which is the "binding box," i.e., the prediction box. In the object detection task, the model outputs a prediction box for each identified object, which identifies the location and size of the object that the model deems to be. Thus, bbox herein refers to a model-generated rectangular box surrounding a detected object (e.g., doctor, patient, etc.), the ground truth being a real box or a real object bounding box. These boxes are calibrated in advance by the human annotators in the training data, accurately representing the position and size of each target. In training the target detection model, these real frames are used to guide model learning so that it can accurately predict the position of the target, and in VFL Loss, the label q of the positive sample is the ratio (IoU) of the intersection of the predicted frame (bbox) and the real frame (ground trunk), which is used to measure the degree of overlap between the predicted frame and the real target frame. If the degree of overlap is high (i.e., ioU value is high), the prediction block is more likely to be considered as a positive sample, and vice versa. In the target detection task, ioU is often used as a criterion for judging whether a predicted frame is sufficiently close to a real frame, thereby deciding whether the predicted frame is a positive sample or a negative sample; and for negative samples q=0.

Where p represents the model's predicted probability for a certain class. In the context of object detection, this generally refers to the probability that a model predicts that a particular bounding box contains an object of a certain class.

For positive samples (i.e., where the predicted and real frames overlap significantly), q is IoU (cross-over) between the predicted and real frames (bbox, ground trunk). For negative samples (i.e., where the prediction block overlaps with the real block not significantly or not at all), q is set to 0.

Alpha is a weighting factor used to balance the importance of positive and negative samples in the loss function. In many cases, the number of negative samples in the training dataset is far greater than the positive samples, and this parameter helps to prevent the model from being excessively biased towards negative samples.

Gamma is an adjustment parameter used to control the degree of weighting of samples that are difficult to classify or misclassified in the loss function. This parameter is derived from Focal Loss, which reduces the weight of the easily classified samples, making the model more focused on samples that are difficult to classify and misclassified.

Regression Loss uses a combination of CIOU Loss and DFL (Distribution Focal Loss). The formula is expressed as follows:

the first part is the cross-ratio loss of prediction and real frames (CIoU), and the expression of CIoU loss is expressed as follows:

The CIoU (Complete Intersection over Union) penalty is a penalty function used to optimize the model regression prediction box position in the objective detection task, which is an extension of the IoU (Intersection over Union) penalty. CIoU loss considers three important factors: shape similarity, center point distance and aspect ratio to improve the positioning accuracy of the target detection model.

IoU in the CIoU formula is the intersection ratio of the predicted and real frames, which is the intersection area of the predicted and real frames divided by their union area. IoU is a criterion for measuring the degree of overlap of a predicted frame with a real frame, and has a value ranging from 0 to 1, and a larger value indicates a higher degree of overlap.

In the CIoU loss formula, ρ represents the euclidean distance between the prediction box center and the true box center. Specifically, if the center coordinates of the prediction frame areThe center coordinates of the real frame are +.>Then ρ isIs a measure of the distance between these two center points and can be calculated by the following formula:

here, theIt is the square of this distance, namely:

alpha is a weight parameter that is used to balance the various parts of the CIoU penalty. In practice, α is typically dynamically adjusted according to the value of the aspect ratio consistency term v to ensure that when the aspect ratio of the predicted and real frames differs significantly, the contribution of this difference to the loss function increases accordingly.

v is an indicator of the aspect ratio consistency of the predicted and real frames. Aspect ratio refers to the ratio of the width to the height of a rectangular box. The calculation of v generally involves the width and height of the prediction frame and the real frame, the purpose of which is to ensure that the prediction frame is not only close in position to the real frame, but also similar in shape (especially aspect ratio) to the real frame.

c represents the diagonal length between the two bounding box minimum closed rectangles.

b represents the prediction frame center;

representing the center of a real frame;

a and v are aspect ratios, b represents a prediction frame center coordinate parameter,representing the real frame center coordinate parameter, c represents the diagonal length between the minimum closed rectangle of two bounding frames, the minimum closed rectangle representing that c in the CIoU loss formula represents that two boundaries are containedThe diagonal length of the minimum closed rectangle of the boxes (i.e., the predicted and real boxes). In particular, the smallest closed rectangle of two bounding boxes refers to the smallest rectangular area that can contain both bounding boxes. Imagine that if there is a predicted frame and a real frame, they may overlap partially, completely, or not overlap at all. Regardless of the relative position between them, a minimum rectangle can always be drawn, which can just contain the two bounding boxes. The diagonal length c of this smallest closed rectangle is the straight line distance from one corner of this rectangle to the other diagonal corner. By calculating the value of c, CIoU loss takes into account the geometric distance between the prediction box and the real box, which helps to more fully evaluate the positioning accuracy of the prediction box. That is, in addition to considering the degree of overlap (by IoU) and the center point distance (by ρ) between the predicted and real frames, CIoU loss also ensures that the predicted frame is spatially as close as possible to the real frame by c, thereby further improving the positioning accuracy of target detection.

The second part is a distributed focus loss DFL, which mainly models the position of the bounding box as a generalized distribution, which allows the network to quickly focus on a position distribution closer to the target position, thereby accelerating the convergence speed of the model and improving the positioning accuracy of the model. The expression of DFL penalty is as follows, where y is the value of the real label,and->Representing the nearest value to y, respectively, which represents the true value of a particular attribute of the bounding box in the context of the DFL (Distribution Focal Loss) penalty. This attribute may be the center coordinate, width, height, or other relevant attribute of the bounding box. In short, y is the true target attribute value that we try to predict through the target detection model. For example, in an object detection task, if we focus on the abscissa of the center point of the bounding box, then y is the true value of this abscissa, +.>And->Then correspond to->And->Is a probability of (2).

By introducing a distribution idea when positioning the target, the regression loss method realizes more accurate target positioning, so that the model can obtain better performance on the fine granularity level of the object position.

In addition, the specific implementation manner of comparing the identified action of the target in the operation of the digestive tract endoscope with the corresponding action in the standard operation specification database comprises the following steps:

By using a target detection algorithm, including the YOLOv8 model, the location of devices including doctors, patients, assistants, endoscopes, and bioptomes can be accurately identified. Meanwhile, a DeepSort algorithm is also applied to detect and track targets of doctors, patients and assistants. The deep sort algorithm includes three main components: detection, apparent feature extraction and data correlation. First, the target is detected using YOLOv 8. Then, the apparent characteristics of the target are extracted by adopting a pedestrian re-recognition network model, and a group of vectors are generated. Next, the cosine distance is used to calculate the similarity between the trajectory and the target to achieve data correlation. The cosine distance is calculated as follows:

in the course of this formula (ii) the formula,the cosine similarity between the target track and the detection result is calculated. The cosine distance is used to calculate the similarity between the target trajectory and the detection result, which can both be represented as vectors by their apparent features. In the context of object tracking and pedestrian Re-identification (Re-ID), apparent features generally refer to features extracted from an image region of object detection that are capable of capturing visual appearance information of the object, such as color, texture, shape, and the like.

Vector representation of target trajectory: for target trajectories, each trajectory may be represented by an aggregation of historically detected apparent features of the target. For example, if a person is continuously tracked in a video sequence, the detection of the person in different frames (i.e., image regions at different points in time) may be used to extract apparent features, which may then be integrated (e.g., averaged or maximally pooled) to form a feature vector representing the entire trajectory.

Vector representation of detection results: for each new detection result, i.e. the object detected in the current frame, it can also be represented as a vector by extracting the apparent features from the detected image area.

Cosine similarity is obtained by calculating the dot product of the two eigenvectors and dividing by their respective modulo lengths, as follows:

wherein,is the apparent eigenvector of the new detection result, +.>Is track->T represents the transpose of the vector. The cosine similarity has a value in the range of-1 to 1,wherein 1 represents the exact same direction, -1 represents the exact opposite direction, and 0 represents the orthogonality.

In practical applications, to calculate the cosine distance, a value of 1 minus cosine similarity is generally used, namely:

The cosine distance ranges from 0 to 2, where 0 represents the exact same direction and 2 represents the exact opposite direction. In this way, the similarity between the target track and the new detection result can be quantified, and an effective measure is provided for data association.

The cosine distance between the two can be obtained by subtracting the cosine similarity from 1. The smaller the value of the cosine distance, the higher the similarity between the target track and the detection result.

Representing an indicator for determining whether the cosine distance is smaller than +.>To determine if a match has occurred. By introducing appearance information as a metric, the identity of the doctor, patient and assistant can be accurately identified.

By identifying the bbox of the assistant and the endoscope and the biopsy forceps, ioU of the assistant and the endoscope and the biopsy forceps can be calculated, so that whether the assistant is in contact with the endoscope and the biopsy forceps is judged. Assume that the bounding box of the assistant is defined by the following four values: (x 1, y1, x2, y 2), wherein (x 1, y 1) is the coordinates of the upper left corner and (x 2, y 2) is the coordinates of the lower right corner. Likewise, the bounding box of the endoscope is also defined by similar coordinates: (x 1) _a , y1 _a , x2 _a , y2 _a ). IoU between the assistant and the endoscope can be calculated by the following formula:

If IoU >0.2, the system is regarded as an assistant operating the endoscope, judges the violation and reminds.

A human critical point algorithm including YOLOv8 is combined with the recurrent neural network to capture both spatial and temporal features. Spatial features including YOLOv8 models are first used to extract human keypoints and then these features are combined with time series data in a recurrent neural network. This approach can better capture the spatial relationship between keypoints. Specifically, the system uses YOLOv8 to detect the key points of the human body of doctors, patients and assistants respectively, and then can identify whether the operation of the doctors is consistent with the action of the standard library, whether the lateral postures of the patients are compliant and whether the operation of the assistants is standard according to the change condition of the key points of the adjacent frames of the video.

The in-vivo and in-vitro video matching is to be noted, for example, a camera at the working end of the instrument displays that the current working end is attached to the intestinal wall, at the moment, a standard database displays that the endoscope is required to be rotated or is retracted, an in-vitro image collector is operated to collect actions of a doctor to be compared with the standard database, and a comparison result is output;

in this embodiment, the action comparison includes the operation comparison outside the patient body and the operation comparison inside the patient body, and the two positions are different and adopt different cameras to acquire images, and further includes the matching comparison inside and outside the patient body, that is, the comparison of corresponding actions is made correspondingly outside the patient body according to the internal condition of the patient body, so that the actions inside and outside the patient body are smoother, and the operation of the digestive tract endoscope is complete and uniform.

Note that, dynamic time normalization: for time series data of the operation actions, a dynamic time warping (Dynamic Time Warping, DTW) technique is used for time axis normalization to eliminate the effect of differences in execution speed on the results.

And (3) extracting mode characteristics: key pattern features are extracted from the normalized time series using deep learning techniques, such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs).

Similarity score: and calculating the similarity between the real-time action characteristics and the standard action mode characteristics by using cosine similarity or other advanced similarity measurement methods.

Threshold decision and feedback: and judging the normalization of the operation according to the similarity score and a preset threshold value, and providing real-time feedback.

In addition, the method can also adopt a plurality of comparison algorithms to carry out complete similarity comparison, comprising the following steps:

the acquisition algorithm is used for synchronously acquiring multi-mode data: and combining a high-resolution video, a depth sensor and a motion capture technology, and synchronously acquiring visual images, three-dimensional gestures and key point data of an operator.

Intelligent noise reduction treatment: and a machine learning algorithm, such as a random forest or a Support Vector Machine (SVM), is utilized to automatically identify and filter background noise and irrelevant motion, so that the accuracy and efficiency of data acquisition are improved.

Feature extraction algorithm, depth feature coding: the improved CNN model, such as ResNet or acceptance network, is used to extract high-level visual and dynamic features.

And (3) space-time key point analysis: and (3) carrying out deep analysis on the acquired key point time sequence data by combining a long-short-term memory network (LSTM) or a gating cycle unit (GRU), and capturing the time characteristics of the action.

The method for establishing the identification model comprises the following steps of: a deep learning model is trained by using a large number of standard operation videos and corresponding canonical action annotations through migration learning and incremental learning strategies.

Model optimization: and the model structure and parameters are optimized by adopting a multitask learning and model distillation technology, so that the recognition accuracy and instantaneity are improved.

Model verification: through cross-validation and actual operation scene testing, the generalization capability and practicability of the model are verified.

It should be noted that, in the present embodiment, the "upper gastrointestinal tract detection action specification standard specification" is to divide different actions, and whether the actions are compliant or not is described, but there is no related concept related to time, so that the whole endoscope operation flow is somewhat unclear, and what is done for each action is also unclear, so that in the present embodiment, a time concept is introduced for the actions, and the sequence of the digestive tract endoscope operation is sorted, for example, the biopsy step includes:

(1) Sufficiently approaches the biopsy target and is clamped with the mucous membrane as perpendicular as possible.

(2) Bleeding can make lesions inaccurate again, and the first biopsy should be selected to be the most positive site, while allowing blood to flow to the low off-lesion site.

(3) For those with bleeding tendency, if aspirin, warfarin, etc. are taken for a long time, 5-7 d should be stopped for further examination.

(4) It is not necessary to biopsy at vascular venous tumor or coarse veins.

(5) The specimen is immediately put into a fixed container, the name, sex, age, date, position and the like are marked on the container, and a pathological examination application form is filled in, wherein the application form content comprises the contents of hospitalization number, gastroscopy number, endoscope diagnosis, application doctor and the like besides the recorded content on the container. When the pathological diagnosis is inconsistent with the endoscopic diagnosis, the pathologist should be contacted for consultation.

In an embodiment of the present application, before the step of identifying the motion of the target in the operation of the digestive tract endoscope, the method further includes video stream preprocessing, where the video stream preprocessing includes image adjustment and background segmentation, and the image adjustment is to unify parameter adjustment including brightness and contrast of a real-time video stream with a corresponding video stream in the standard operation specification database; the background segmentation separates the target object from the background through a deep learning algorithm, so that the calculated amount is reduced, and the comparison speed is effectively improved.

In this embodiment, different scoring items, scores and corresponding scoring criteria may be set according to actual situations, which are exemplified as follows:

patient left side recumbent position: the compliance standard is that the right leg of the patient is bent to form an included angle of 30-60 degrees, and the left leg is straightened or both legs are bent to form an included angle of 30-60 degrees; the right leg is over-bent or not in place and the left leg is not straightened (included angle is less than 170 °); excessive or insufficient bending of the legs; the interactive objects involved in the picture are: doctor, patient, hospital bed, see fig. 7;

White balance: a compliance standard, aligning the endoscope with white gauze or a calibration box and operating a machine for calibration; the front end of the endoscope is not aligned to the gauze or the calibration box without the compliance standard and the operation machine acts; the interactive objects involved in the picture are: doctor, endoscope host;

water and gas injection attraction test: compliance standard, placing the endoscope into a liquid container (water cup, water bucket, etc.); the endoscope is not put into a liquid container (a water cup, a water bucket and the like) without compliance standard; the interactive objects involved in the picture are: doctor, endoscope, liquid container;

and (3) endoscope angle test: the compliance standard is that the size knob of the endoscope operation part is operated by a left hand, so that the front end of the endoscope is bent up and down and left and right; the operation of the left-hand operation knob is not detected without compliance standard, and the front end of the endoscope is not bent and changed; the interactive objects involved in the picture are: doctor, endoscope;

the right hand position is parallel to the patient's mouth: compliance standard, doctor right hand is located on patient's head; the right hand of the doctor is positioned at other positions of the patient without compliance standard; the interactive objects involved in the picture are: doctor, patient and sickbed;

doctor standing position: compliance standard, doctor is within 15cm from sickbed; the doctor is far from the sickbed than 15cm without compliance standard; the interactive objects involved in the picture are: doctor, patient, endoscope, see fig. 8;

The upper left arm is attached to the body: the compliance standard is that the upper left arm should be kept close to the trunk in the whole inspection process, and the vertical angle between the left shoulder joint and the left elbow joint is not more than 10 degrees; if the standard is not compliant, the upper left arm leaves the body in the examination, the rule is violated, and if the rule appears three times, the project does not score. The vertical angle between the left shoulder joint and the left elbow joint exceeds 10 degrees; the interactive objects involved in the picture are: the doctor, see fig. 9;

left hand mirror height: compliance standard, the left hand mirror should be 15cm below the shoulder and above the waist; the non-compliance standard is that the shoulder is buckled at the lower side of the shoulder at the upper side of the double-breast connecting line, the waist is buckled at the lower side of the waist, and the shoulder is buckled at the upper side of the shoulder by 0 minutes; the interactive objects involved in the picture are: doctor, endoscope host, see fig. 10;

left hand mirror pose: compliance standard, deep lens holding method: the left ring finger and the little finger are in a natural state to hold the operation part; the size knob is adjusted by the thumb and the middle finger, and the air injection button, the water injection button and the suction button are controlled by the index finger; "big nursery flow" mirror holding method: the endoscope operation part is clamped by the thumb and the ring finger of the left hand, the big knob and the small knob are controlled, and the small finger is in a natural state to hold the operation part; the index finger controls the suction button, and the middle finger controls the air injection and water injection buttons; standard lens holding method: the left ring finger and the little finger are in a natural state to hold the operation part; the size knob is adjusted by the thumb; the index finger controls the suction button, and the middle finger controls the air injection and water injection buttons; the standard lens holding method is not matched, the first two lens holding methods are detected to be scored, and the standard lens holding method is not scored; the interactive objects involved in the picture are: doctor, endoscope host;

Right hand mirror pose: compliance standard, the right hand and the mouth of the patient should be the same plane, and the left-right deviation is less than or equal to 30 degrees; the right hand position is higher or lower than the mouth plane without compliance standard, and the deviation is more than 30 degrees; the interactive objects involved in the picture are: doctor, endoscope, see fig. 11;

right hand position: compliance standard, right hand mirror; the method is characterized in that the method does not conform to the standard, and the knob touching the endoscope operation part by right hand operation is illegal; the interactive objects involved in the picture are: doctor, endoscope host;

right hand position: compliance standard, the right hand distance is 20-25cm from the mouth; the right hand distance from the mouth is too long or too short without compliance standards; the interactive objects involved in the picture are: doctor, patient, see fig. 12;

description: a doctor is locked by the object of the hand-held mirror at the beginning of the system, a patient is locked by lying on a sickbed, a nurse is left to lock, and three objects are tracked by the Reid (re-identification) technology in the checking process to judge the identity;

the assistant helps access the biopsy forceps: compliance standards, the assistant does not help access the biopsy forceps; the assistant helps access the biopsy forceps without compliance criteria; the interactive objects involved in the picture are: an assistant, biopsy forceps;

The assistant holds the mirror: compliance standards, the assistant should not touch the mirror during the inspection; a contact mirror is contacted by an assistant in the checking process without compliance standard; the interactive objects involved in the picture are: an assistant, a mirror;

the assistant lifts the mandible: compliance standard, assistant manually lifts the patient's lower jaw; the assistant does not raise the patient's mandible by hand, which is not a compliance standard; the interactive objects involved in the picture are: an assistant, a patient;

left hand biopsy forceps of assistant: compliance standard, the assistant rotates the biopsy forceps with the left hand; non-compliance standard, the assistant uses the right hand or does not help rotate the bioptome; the interactive objects involved in the picture are: an assistant, biopsy forceps;

mirror entering condition: compliance standard, the gastroscope enters once; the gastroscope is not in compliance with the standard, and 1 score is obtained after twice entrance and 0 score is obtained after more than twice entrance; the interactive objects involved in the picture are: doctor, endoscope;

body rotation judgment: compliance standard, body does not rotate greatly; non-compliance standard, body rotation is great: the trunk is taken as an axis, and the leftward rotation angle of the trunk is more than or equal to 45 degrees; the interactive objects involved in the picture are: a doctor;

whether to straighten the mirror body: compliance standard, there is a straightening mirror body action; the lens body is not straightened due to the non-compliance standard; the interactive objects involved in the picture are: doctor, endoscope;

Normalized image evaluation of upper gastrointestinal endoscopy:

the number of pictures taken by the hypopharynx, esophagus, cardia, stomach and duodenum is more than or equal to 60, and the number of pictures does not reach the standard and is withheld;

the examination site should include: palate, uvula and tonsillar fossa, oropharynx, hypopharynx, glossoepiglottis, larynx, bilateral piriform fossa, esophageal entrance, esophageal neck segment, upper, middle and lower chest segment, cardiac frontal, cardiac reversal of various angles, fundus, upper, middle and lower stomach segment four walls, gastric angle (frontal and anterior and posterior), stomach Dou Si quadrant, stomach Dou Yuan near, pylorus, duodenal bulbar, descending deep. The throat and esophagus need to have electronic dye pictures and white light pictures.

Taking fixed picture, and taking more than 3 times of non-fixed picture, wherein the number is 0;

starting from the entrance of the endoscope to the exit of the endoscope from the entrance, performing routine examination for a total duration of 8 minutes or more and less than 8 minutes, if the dyeing examination duration is more than or equal to 10 minutes, if the examination duration is more than 15 minutes, the routine examination is performed (without biopsy time);

the pictures with more ghost images and mucus foam are unqualified pictures, and the unqualified pictures are more than 5 buckles.

Standardized biopsy evaluation under endoscope:

when in oesophageal biopsy, the biopsy forceps cannot extend too long, and the biopsy forceps are buckled when the length is more than 1 cm;

The biopsy forceps are parallel to the focal mucosa, and the biopsy forceps cannot be buckled;

the biopsy should be performed under direct vision, if blind vision appears, the biopsy is deducted;

the biopsy forceps are pulled out quickly, and if the biopsy forceps are too slow, the biopsy forceps are buckled;

taking a picture after biopsy;

after biopsy, observing whether bleeding exists on the wound surface, wherein the observation time is longer than 10 seconds, and no obvious bleeding score is observed when no observer takes 0 minutes;

summarizing the results: and integrating all the identification and evaluation results to generate a detailed report.

Visual display: and displaying key points, action recognition results and normalization evaluation in the operation process through a graphical interface.

Through the detailed data acquisition and processing flow, the system can monitor and evaluate the normalization of the operation of the soft digestive tract endoscope in real time, thereby improving the operation safety and diagnosis and treatment efficiency. In addition, the system can provide precious real-time feedback and training support for doctors and nurses, and further improves the medical service quality.

In the embodiment, considering that the situation of each patient is different, a doctor can adopt an irregular operation action according to the situation of each patient, so in the embodiment, after the doctor confirms, the doctor cancels the reminding by voice control or assistant operation, avoids meaningless reminding of the doctor for a long time, can easily lead the doctor to distraction, gives the doctor free authority in the actual operation, achieves the aim of prompting the doctor operation by reminding, assisting the doctor to better execute the action, and gives the doctor greater authority under the special condition; further, after the overall operation of the digestive tract endoscope is finished, judging the corresponding action of canceling reminding by a doctor, and if the corresponding action belongs to standard operation, storing the corresponding video stream into the standard operation specification database; when the corresponding action belongs to standard operation which needs to exist the pre-condition, the pre-condition is supplemented, when the pre-condition appears again, the corresponding action is used for corresponding video stream as the standard operation, the pre-condition comprises physiological data of a patient, so that when the same condition is met again in the follow-up process, the follow-up operation can be automatically switched, the fact that the correct operation is misjudged is avoided, redundant reminding possibly brings influence to doctors is avoided, through long-time accumulation and enrichment of cases, the standard operation specification database aiming at patients with various special physiological data can be executed by using the digestive tract endoscope operation specification, and the applicability is wider.

In a second aspect of the present application, there is provided a soft-type enteron endoscope operation specification action evaluation system, as shown in fig. 3, comprising:

It should be noted that, the specific implementation of the operation specification action evaluation system for a soft digestive tract endoscope in the embodiment of the present application refers to the specific implementation of the operation specification action evaluation method for a soft digestive tract endoscope set forth in the first aspect of the embodiment of the present application, and is not described herein again.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that an article or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such article or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other like elements in an article or apparatus that comprises the element.

The above provides a method for evaluating operation specification actions of a soft digestive tract endoscope, and specific examples are applied to illustrate the principles and embodiments of the present application, and the above examples are only used to help understand the operation specification actions of a soft digestive tract endoscope of the present application and the core ideas thereof; meanwhile, as those skilled in the art will vary in the specific embodiments and application scope according to the ideas of the present application, the contents of the present specification should not be construed as limiting the present application in summary.

Claims

1. A method for evaluating operation specification actions of a soft digestive tract endoscope, comprising the steps of:

image acquisition is carried out on a target involved in the digestive tract endoscope operation, the motion of the target in the digestive tract endoscope operation is split, the target and the motion of the target are identified, and a standard operation specification database of the digestive tract endoscope is established;

2. The method for evaluating operation specification actions of a soft digestive tract endoscope according to claim 1, wherein in the step of establishing a standard operation specification database of the digestive tract endoscope, a specific digestive tract endoscope operation process is represented by capturing a video stream through a camera, and performing target detection and human body key point detection on the video stream:

3. The method for evaluating operation specification actions of a soft digestive tract endoscope according to claim 2, wherein the step of performing real-time video stream acquisition on the operation process of the digestive tract endoscope specifically comprises in-vitro video acquisition and in-vivo video acquisition:

4. The method for evaluating the operation standard action of the soft digestive tract endoscope according to claim 3, wherein the step of collecting the video stream in real time during the operation process of the digestive tract endoscope specifically comprises the steps of matching in-vitro and in-vivo videos, when the preset condition occurs in the in-vivo videos of the patient collected by the endoscope, and a doctor and an assistant are required to make corresponding actions in vitro, the in-vitro video collector collects the in-vitro videos, and the in-vitro image collector collects the actions of the doctor and the assistant to be compared with the standard operation standard database, and outputs a comparison result.

5. The method according to claim 1, wherein the step of comparing the recognized motion of the target in the operation of the digestive tract endoscope with the corresponding motion in the standard operation specification database comprises:

6. The method for evaluating the operation specification action of a soft digestive tract endoscope according to claim 2, wherein the step of arranging according to the sequence of standard operation of the digestive tract endoscope specifically comprises the steps of:

7. The method according to claim 1, wherein before the step of identifying the motion of the target in the operation of the flexible endoscope, the method further comprises preprocessing the video stream, wherein the preprocessing of the video stream comprises image adjustment and background segmentation, and the image adjustment is used for unifying the parameter adjustment of the real-time video stream comprising brightness and contrast with the corresponding video stream in the standard operation specification database; the background segmentation separates the target object from the background by a deep learning algorithm.

8. The method for evaluating operation specification actions of a soft endoscope according to any of claims 1-7, wherein the step of summarizing and evaluating all comparison results after the operation of the endoscope is completed, includes setting scores and corresponding scoring criteria for corresponding actions in the standard operation specification database, scoring the actions identified in the overall process of the operation of the endoscope one by one, integrating all scores after the completion of the overall process, and generating an operation report.

9. The method for evaluating the operation specification action of the soft digestive tract endoscope according to claim 8, wherein the step of outputting the comparison result of the action currently performed by the target and the specification data in real time, and prompting when the evaluation of the action currently performed is not specification, further comprises:

10. A soft-tip operation specification action evaluation system, comprising: