CN114782855A - Cataract surgery evaluation method, system and medium based on deep learning - Google Patents

Abstract

The present invention belongs to the technical field of deep learning, and provides a cataract surgery evaluation method, system and medium based on deep learning, comprising the steps of: S1. Using the characteristics of surgical instruments in a video frame combined with the background characteristics of the eye, determine the location of the video frame where the video frame is located. Classify the operation stage; S2, train a general-purpose network according to the initial feature labels of each link of the operation to extract the evaluation features used for surgical evaluation in the video frames corresponding to each link; S3, obtain the extracted evaluation features according to the preset labels for each link of the operation The quantitative information is input into the pre-trained classification evaluation network to classify and evaluate each link of the operation. The advantage of the present invention lies in establishing the quantitative relationship between the descriptive evaluation index in the ICO‑OSCAR standard and the characteristics of the operation that can be learned by the deep learning network, so that the artificial intelligence technology can replace the expert doctor to participate in the whole operation training, and the objectivity and reliability of the training effect can be improved. , and the response rate.

Description

A method, system and medium for cataract surgery evaluation based on deep learning

technical field

The present invention relates to the technical field of deep learning, and in particular, to a method, system and medium for cataract surgery evaluation based on deep learning.

Background technique

Cataract is the first blinding disease. In my country, cataract patients account for about half of the total number of blind people. Surgery is the main way to help patients regain their light. At present, the surgery rate in my country is still at a low level. Improving the cataract surgery rate and ensuring the effect of surgery is one of the main problems to be solved urgently in the current blindness prevention and blindness treatment. Cataract surgery has entered the era of refraction. The individual needs of patients and the pursuit of postoperative visual effects have put forward higher requirements for the location of the cataract surgery incision, the size of the capsulorhexis, and the centering of the intraocular lens after implantation. Intraoperative evaluation and feedback information on incision, capsulorhexis and intraocular lens implantation is particularly important for improving the surgical skills of novice doctors in cataract surgery training.

Standardized training for cataract surgery is aimed at shortening the learning curve, standardizing surgical procedures, and reducing surgical complications. Traditional cataract surgery training feedback usually uses the International Ophthalmology Association's Ophthalmic Surgery Competency Assessment Standard (ICO-OSCAR) to track and evaluate doctors, and expert doctors score each part of cataract surgery according to the standard, which is time-consuming and subjective. It is difficult to meet clinical requirements through training in a short period of time.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a cataract surgery evaluation method, system and medium based on deep learning to solve the above problems.

In order to achieve the above object, the technical scheme adopted in the present invention is:

A cataract surgery evaluation method based on deep learning, including steps:

S1, using the surgical instrument feature in the video frame combined with the eye background feature, through a preset classification network to classify the surgical stage where the video frame is located; the surgical stage includes an incision link, a capsulorhexis link and an intraocular lens implantation link;

S2, training a general-purpose network according to the initial feature labels of each link of the operation to extract the evaluation features used for surgical evaluation in the video frames corresponding to each link;

S3. Obtain the quantitative information of each link of the operation according to the extracted evaluation feature according to the preset label, and input the quantitative information into the preset classification evaluation network after training to classify and evaluate each link of the operation.

Further, the step of classifying the surgical stage in which the video frame is located includes:

S11. Perform hierarchical sampling processing on the video frame;

S12, obtaining the surgical instrument and the eye background area in the video frame through the trained preset target detection model, and batch cropping the surgical instrument and the eye background area;

S13, send the cropped surgical instrument and the eye background area, and the corresponding video frame to the trained classifier respectively, and output the classification result of the surgical stage where the video frame is located through the classifier.

Further, the step of training the general-purpose network includes:

T1, randomly crop an image block from the video frame, and input it and the front and rear frame images to the spatial feature encoder to calculate the corresponding spatial feature respectively;

T2. The differentiable tracker is used to calculate the positioning parameter corresponding to the image block that best matches the image block in the corresponding spatial features of the front and rear frame images, and the bilinear sampling is performed by the bilinear sampler to obtain the space that best matches the image block. feature;

T3. Perform end-to-end training on the spatial feature encoder and the differentiable tracker through steps T1 and T2, so as to obtain a trained general-purpose network.

Further, the evaluation features for surgical evaluation include surgical instrument position information, optical flow field information, limbal morphology and position features, and intraocular lens position features.

Further, the steps of classifying and evaluating the incision link in the operation include:

A1. Fitting the limbus to establish the limbal center according to the positional feature of the limbus;

A2. Taking the center of the corneal limbus as the reference point, the relative motion trajectory of the surgical instrument is obtained according to the position information of the surgical instrument;

A3. Input the relative motion trajectory of the surgical instrument and the morphological characteristics of the corneal limbus into the preset classification evaluation network to evaluate the surgical incision according to the ICO-OSCAR standard, and obtain the operation score of the surgical incision.

Further, the steps of classifying and evaluating the capsulorhexis link in the operation are as follows:

Input the optical flow field information into the preset classification evaluation network to evaluate the capsulorhexis in the operation according to the ICO-OSCAR standard, and obtain the operation score of the capsulorhexis in the operation.

Further, the steps of classifying and evaluating the intraocular lens implantation link in the operation are as follows:

B1. Fit the limbus position feature and intraocular lens position feature extracted by the universal network to obtain the respective center point position information of the two;

B2. Input the position information of the center points of the two into the preset classification evaluation network to evaluate the intraoperative intraocular lens implantation according to the ICO-OSCAR standard, and obtain the operation score of the intraoperative intraocular lens implantation.

A second aspect of the present invention provides a deep learning-based cataract surgery evaluation system, comprising at least one processor and at least one memory, wherein the memory stores a computer program, and when the program is executed by the processor When executed, the processor is enabled to execute the deep learning-based cataract surgery evaluation method.

In a third aspect of the present invention, a computer-readable storage medium is provided, when instructions in the storage medium are executed by a processor in the device, the device can perform the deep learning-based cataract surgery evaluation method.

Compared with the prior art, the present invention at least includes the following beneficial effects:

(1) The present invention takes the surgical instruments involved in different surgical links as the main learning feature, combines the eye background information, and realizes the accurate classification of video frames through the accurate classifier after training, and provides algorithm support for data processing before surgical evaluation;

(2) The present invention achieves the same network extraction of quantifiable surgical features in different surgical links by developing a general-purpose multi-feature extraction network. Labeling to achieve skill evaluation of surgical incision, capsulorhexis and intraocular lens implantation;

(3) Establish the quantitative relationship between the descriptive evaluation indicators in the ICO-OSCAR standard and the surgical path, limbal morphology, and optical flow information of surgical instruments that can be learned by the deep learning network, so as to realize the replacement of expert doctors with artificial intelligence technology to participate in the whole surgical training. , to improve the objectivity, reliability, and response rate of the training effect.

Description of drawings

Fig. 1 is the flow chart of the cataract surgery evaluation method based on deep learning in the embodiment of the present invention;

Fig. 2 is the flow chart of classifying the operation stage in which video frame is located in the embodiment of the present invention;

3 is a flowchart of training a general-purpose network in an embodiment of the present invention;

4 is a schematic diagram of general-purpose network training in an embodiment of the present invention;

Fig. 5 is the flow chart that the incision link in the operation is classified and evaluated in the embodiment of the present invention;

Fig. 6 is a flow chart of classifying and evaluating intraoperative intraocular lens implantation in an embodiment of the present invention.

Detailed ways

It should be noted that the technical solutions between the various embodiments of the present invention can be combined with each other, but must be based on the realization by those of ordinary skill in the art. When the combination of technical solutions is contradictory or cannot be realized, it should be considered that such technical solutions The combination does not exist and is not within the scope of protection claimed by the present invention.

The following are specific embodiments of the present invention, and the technical solutions of the present invention will be further described in conjunction with the accompanying drawings, but the present invention is not limited to these embodiments.

As shown in Figure 1, a method for evaluating cataract surgery based on deep learning of the present invention includes the steps:

S1. Using the characteristics of surgical instruments in the video frame combined with the background characteristics of the eye, classify the surgical stage in which the video frame is located through a preset classification network; the surgical stage includes the incision link, the capsulorhexis link and the intraocular lens implantation link.

Wherein, as shown in Figure 2, the step of classifying the surgical stage at which the video frame is located includes:

S11. Perform hierarchical sampling processing on the video frame.

Due to the uneven level of surgical skills of surgeons and different requirements for different stages of surgery, the video recording time is long, the duration of each stage of surgery is different, and the surgical scene is complex and changeable. The present invention uses stratified sampling to process video frames, which can effectively Improve the training efficiency of the network model.

S12. Obtain the surgical instrument and the eye background area in the video frame through the trained preset target detection model, and cut the surgical instrument and the eye background area in batches.

The present invention trains the target detection model based on Yolov3 according to the surgical instruments and eye background labels, obtains the surgical instruments and eye background areas that are focused on in the video frame, and cuts them in batches as the input of the classifier, so as to improve the video frame Layout fine-grained information.

S13, send the cropped surgical instrument and the eye background area, and the corresponding video frame to the trained classifier respectively, and output the classification result of the surgical stage where the video frame is located through the classifier.

The global and local features are extracted by feeding surgical instruments, eye background regions and corresponding video frames into the ResNet network respectively. Then, the features output by the fully connected layer of each classifier are concatenated as a fine-grained feature representation of the entire frame, which improves the accuracy of video frame classification.

According to the surgical instruments involved in different surgical links as the main learning feature, the present invention realizes accurate classification of video frames by combining eye background information, and provides algorithm support for subsequent data processing.

S2. Train a general-purpose network according to the initial feature labels of each link of the operation to extract the evaluation features used for surgical evaluation in the video frames corresponding to each link.

In order to improve the versatility of the feature extraction network, reduce the task amount of labeling, and improve the efficiency of feature extraction, the present invention learns the visual correspondence according to the unlabeled video, and trains the patch (image block) extracted from the image by tracking the forward direction. After tracking in several consecutive frames, it learns the feature space and computes the visual similarity of target patches along consecutive frames in the time series. When extracting the features of incision, capsulorhexis, and intraocular lens implantation, given the labels of incision, capsulorhexis, and intraocular lens implantation, the network can directly extract the features required for evaluation of each segment of the operation.

Wherein, as shown in Figure 3 and Figure 4, in the present invention, the step of training the general-purpose network includes:

T1. An image block is randomly cropped from the video frame, and input to the spatial feature encoder together with the frame images before and after to obtain the corresponding spatial feature respectively.

T2. The differentiable tracker is used to calculate the positioning parameter corresponding to the image block that best matches the image block in the corresponding spatial features of the front and rear frame images, and the bilinear sampling is performed by the bilinear sampler to obtain the space that best matches the image block. feature.

T3. Perform end-to-end training on the spatial feature encoder and the differentiable tracker through steps T1 and T2, so as to obtain a trained general-purpose network.

The present invention calculates the spatial features x ^P and x ^I respectively by inputting the randomly cropped patch (P _t ) and the front and rear frame images (I _t-1 ) from the video frame into the spatial feature encoder Φ based on the ResNet network, respectively, and The channel dimension of spatial features is normalized to facilitate subsequent similarity calculation.

The differentiable tracker T first measures the similarity between the coordinates of x ^I and x ^P , and then calculates the localization parameter θ corresponding to the image patch in the feature x ^I that best matches x ^P. Bilinear sampling is performed on x ^I and θ to generate a new patch feature x ^P' , and then the similarity between the new patch feature and its previous and previous frame images is calculated to form a loop and realize the training of the network.

After the training of the universal network is completed, by giving different feature labels, it can realize the feature extraction of incision, capsulorhexis, and intraocular lens implantation in surgery, and obtain the position of surgical instruments, optical flow field, corneal limbus shape, intraocular lens Location and other information for use in the next step of the classification and evaluation of each link.

S3. Obtain the quantitative information of each link of the operation according to the extracted evaluation feature according to the preset label, and input the quantitative information into the preset classification evaluation network after training to classify and evaluate each link of the operation.

According to the information of surgical instrument position, optical flow field, corneal limbus shape and intraocular lens position obtained from the general network, it is converted into quantitative information that can be evaluated, such as the movement trajectory of the surgical instrument, the movement speed and direction of the surgical instrument (ie Optical flow field information), and combined with the limbal position, morphological changes and other information, combined with the classification labels of experts and doctors according to the ICO-OSCAR standard, the classification models were trained separately to realize the classification and evaluation of incision, capsulorhexis and intraocular lens implantation. Work.

Among them, as shown in Figure 5, the steps of classifying and evaluating the incision link in the operation include:

A1. Fitting the limbus to establish the limbal center according to the positional feature of the limbus;

A2. Taking the center of the corneal limbus as the reference point, the relative motion trajectory of the surgical instrument is obtained according to the position information of the surgical instrument;

A3. Input the relative motion trajectory of the surgical instrument and the morphological characteristics of the corneal limbus into the preset classification evaluation network to evaluate the surgical incision according to the ICO-OSCAR standard, and obtain the operation score of the surgical incision.

The steps to classify and evaluate the capsulorhexis in the operation are as follows:

Input the optical flow field information into the preset classification evaluation network to evaluate the capsulorhexis in the operation according to the ICO-OSCAR standard, and obtain the operation score of the capsulorhexis in the operation.

As shown in Figure 6, the steps for classifying and evaluating intraocular lens implantation during surgery are as follows:

B1. Fit the limbus position feature and intraocular lens position feature extracted by the universal network to obtain the respective center point position information of the two;

B2. Input the position information of the center points of the two into the preset classification evaluation network to evaluate the intraoperative intraocular lens implantation according to the ICO-OSCAR standard, and obtain the operation score of the intraoperative intraocular lens implantation.

The invention establishes the quantitative relationship between the descriptive evaluation index in the ICO-OSCAR standard and the surgical instrument path, corneal limbus shape, and optical flow information of the surgical instrument that can be learned by the deep learning network, so as to realize the replacement of expert doctors by artificial intelligence technology to participate in the whole operation. Training improves the objectivity, reliability, and response rate of the training effect.

In another embodiment of the present invention, a deep learning-based cataract surgery evaluation system is also provided, comprising at least one processor and at least one memory, wherein the memory stores a computer program, and when the program is executed by the When the processor executes, the processor can execute the above-mentioned deep learning-based cataract surgery evaluation method.

In another embodiment of the present invention, a computer-readable storage medium is also provided. When the instructions in the storage medium are executed by a processor in the device, the device can perform the above-mentioned deep learning-based cataract surgery. Evaluation method.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which the present invention pertains can make various modifications or additions to the described specific embodiments or substitute in similar manners, but will not deviate from the spirit of the present invention or go beyond the definitions of the appended claims range.

Claims (9)

1. A cataract surgery evaluation method based on deep learning is characterized by comprising the following steps:

s1, classifying the operation stage of the video frame through a preset classification network by using the characteristics of the surgical instruments in the video frame in combination with the eye background characteristics; the operation stage comprises an incision link, a capsulorhexis link and an artificial lens implantation link;

s2, training a universal network according to the initial feature tags of each link of the operation to extract evaluation features for operation evaluation in the video frames corresponding to each link;

and S3, acquiring quantitative information of each link of the operation according to the extracted evaluation features and the preset labels, and inputting the quantitative information into a trained preset classification evaluation network to perform classification evaluation on each link of the operation.

2. The cataract surgery evaluation method based on deep learning of claim 1, wherein the step of classifying the surgery stage of the video frame comprises:

s11, carrying out hierarchical sampling processing on the video frame;

s12, acquiring surgical instruments and eye background areas in the video frame through the trained preset target detection model, and cutting the surgical instruments and the eye background areas in batches;

and S13, respectively sending the cut surgical instruments, the eye background area and the corresponding video frame into a trained classifier, and outputting the classification result of the surgical stage of the video frame through the classifier.

3. The method for evaluating cataract surgery based on deep learning of claim 1, wherein the step of training the universal network comprises:

t1, randomly cutting an image block from the video frame, inputting the image block and the previous and next frame images into a spatial feature encoder to respectively calculate corresponding spatial features;

t2, calculating the positioning parameters corresponding to the image block which is most matched with the image block in the space characteristics corresponding to the front and rear frame images through a micro tracker, and performing bilinear sampling through a bilinear sampler to obtain the space characteristics of the most matched image block;

and T3, performing end-to-end training on the space feature encoder and the micro tracker through steps T1 and T2, and thus obtaining the trained universal network.

4. The cataract surgery evaluation method based on deep learning of claim 1, wherein the evaluation features for surgery evaluation comprise surgical instrument position information, optical flow field information, corneal limbus morphology and position features and artificial lens position features.

5. The cataract surgery evaluation method based on deep learning of claim 4, wherein the step of classifying and evaluating incision links in surgery comprises:

a1, fitting the corneal limbus according to the corneal limbus position characteristics to establish a corneal limbus center;

a2, obtaining the relative motion track of the surgical instrument by taking the corneal limbus center as a reference point according to the position information of the surgical instrument;

and A3, inputting the relative motion track of the surgical instrument and the corneal limbus morphological characteristics into a preset classification evaluation network, and evaluating the surgical incision link according to ICO-OSCAR standard to obtain the operation score of the surgical incision link.

6. The cataract surgery evaluation method based on deep learning of claim 4 is characterized in that the step of classifying and evaluating the capsulorhexis link in the surgery is as follows:

and inputting the optical flow field information into a preset classification evaluation network to evaluate the capsulorhexis link in the operation according to ICO-OSCAR standard so as to obtain the operation score of the capsulorhexis link in the operation.

7. The cataract surgery evaluation method based on deep learning of claim 4 is characterized in that the step of classifying and evaluating the intraocular lens implantation link in surgery comprises:

b1, fitting the corneal limbus position feature artificial lens position features extracted through the universal network to obtain respective central point position information of the corneal limbus position feature artificial lens and the corneal limbus position feature artificial lens;

and B2, inputting the central point position information of the two into a preset classification evaluation network, and evaluating the intraocular lens implantation link in the operation according to ICO-OSCAR standard to obtain the operation score of the intraocular lens implantation link in the operation.

8. A deep learning-based cataract surgery evaluation system comprising at least one processor and at least one memory, wherein the memory stores a computer program which, when executed by the processor, enables the processor to carry out the deep learning-based cataract surgery evaluation method according to any one of claims 1 to 7.

9. A computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor in a device, enable the device to perform the method for deep learning-based cataract surgical assessment of any one of claims 1-7.

Images