CN116597943A - Forward track prediction method and equipment for instrument operation in minimally invasive surgery - Google Patents
Forward track prediction method and equipment for instrument operation in minimally invasive surgery Download PDFInfo
- Publication number
- CN116597943A CN116597943A CN202310501391.4A CN202310501391A CN116597943A CN 116597943 A CN116597943 A CN 116597943A CN 202310501391 A CN202310501391 A CN 202310501391A CN 116597943 A CN116597943 A CN 116597943A
- Authority
- CN
- China
- Prior art keywords
- instrument
- motion
- primitive
- distance
- adopting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000002324 minimally invasive surgery Methods 0.000 title claims abstract description 33
- 239000000203 mixture Substances 0.000 claims abstract description 22
- 238000003909 pattern recognition Methods 0.000 claims abstract description 14
- 230000006870 function Effects 0.000 claims description 15
- 230000003044 adaptive effect Effects 0.000 claims description 12
- 238000004422 calculation algorithm Methods 0.000 claims description 12
- 230000015654 memory Effects 0.000 claims description 12
- 238000001514 detection method Methods 0.000 claims description 10
- 238000003860 storage Methods 0.000 claims description 7
- 238000013213 extrapolation Methods 0.000 claims description 6
- 238000012937 correction Methods 0.000 claims description 5
- 238000012549 training Methods 0.000 claims description 5
- 230000009466 transformation Effects 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000013519 translation Methods 0.000 claims description 4
- 238000007476 Maximum Likelihood Methods 0.000 claims description 3
- 238000002372 labelling Methods 0.000 abstract description 2
- 238000001356 surgical procedure Methods 0.000 description 18
- 238000010586 diagram Methods 0.000 description 8
- 230000009471 action Effects 0.000 description 7
- 238000004458 analytical method Methods 0.000 description 7
- 238000004891 communication Methods 0.000 description 5
- 230000006399 behavior Effects 0.000 description 4
- 241000229175 Calotes Species 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 238000012512 characterization method Methods 0.000 description 2
- 238000002192 cholecystectomy Methods 0.000 description 2
- 238000010219 correlation analysis Methods 0.000 description 2
- 238000000354 decomposition reaction Methods 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000001727 in vivo Methods 0.000 description 2
- 238000005065 mining Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- 101100391172 Dictyostelium discoideum forA gene Proteins 0.000 description 1
- 206010048038 Wound infection Diseases 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 210000001367 artery Anatomy 0.000 description 1
- 238000009412 basement excavation Methods 0.000 description 1
- 230000003542 behavioural effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000012567 pattern recognition method Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
- G16H20/40—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
- G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
- A61B34/10—Computer-aided planning, simulation or modelling of surgical operations
- A61B2034/107—Visualisation of planned trajectories or target regions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20076—Probabilistic image processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Biomedical Technology (AREA)
- Medical Informatics (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Surgery (AREA)
- Evolutionary Computation (AREA)
- Molecular Biology (AREA)
- Public Health (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Computational Linguistics (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Biophysics (AREA)
- Primary Health Care (AREA)
- Epidemiology (AREA)
- Urology & Nephrology (AREA)
- Business, Economics & Management (AREA)
- Veterinary Medicine (AREA)
- General Business, Economics & Management (AREA)
- Animal Behavior & Ethology (AREA)
- Heart & Thoracic Surgery (AREA)
- Radiology & Medical Imaging (AREA)
- Quality & Reliability (AREA)
- Robotics (AREA)
- Databases & Information Systems (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The application belongs to the technical field of medical instrument control, and particularly discloses a forward track prediction method and equipment for instrument operation in minimally invasive surgery. Comprising the following steps: acquiring a motion trail of a surgical instrument in a two-dimensional image coordinate system; analyzing a motion mode of the motion of the instrument, converting the motion trail into a two-dimensional polar coordinate system, and splitting the continuous motion trail of the instrument in the motion into discrete motion primitives; establishing a distance primitive by adopting a Gaussian mixture model, and determining the tip orientation by adopting instrument gesture recognition; on-line updating distance primitive parameters by adopting self-adaptive pattern recognition, and on-line predicting the direction deviation of the tip orientation by adopting motion direction autoregressive; the forward distance of the instrument is predicted on line by adopting Gaussian mixture regression, and the forward direction of the instrument is corrected on line by adopting internal weighting of multiple motion primitives; and fusing the predicted advancing distance and the advancing direction to obtain a predicted result of the advancing track of the instrument motion in the image. The method does not need to rely on a large number of labeling data sets, and has high robustness and strong generalization capability.
Description
Technical Field
The application belongs to the technical field of medical instrument control, and particularly relates to a forward track prediction method and equipment for instrument operation in minimally invasive surgery.
Background
Minimally invasive surgery has gradually replaced open surgery and has become the mainstream of many important surgical procedures due to the advantages of low morbidity and mortality, less pain, less wound infection, short hospital stay, and the like. During the last decade, robot-assisted minimally invasive surgery (RAMIS) has been popular because it is more dexterous and efficient than general minimally invasive surgery. Most RAMIS systems use robotically driven surgical devices that are still operated remotely by the surgeon. The search for how to implement autonomous RAMIS has attracted considerable attention, which requires more insight into the situation, namely surgical context awareness, during surgery. Surgical context awareness integrates: 1. surgical workflow identification-perception and understanding of current status; 2. surgical workflow prediction-prediction future state.
In previous studies, the granularity of the surgical workflow was arranged from coarse to fine into stages, steps and actions. Following this concept, the newly published work on surgical situational awareness focuses on the existing granularity level. A common idea in surgical context awareness is that finer granularity of surgical workflow analysis can provide more efficient clinical information. Therefore, analysis of the action level of fine granularity requires further mining. In addition, existing research has focused on analyzing and understanding current surgical procedures, or reviewing analysis history procedures. Considering that the surgical procedure is complex and has high uncertainty, the prediction of future states in the surgical procedure is a still-to-be-explored field.
On the other hand, end-to-end methods have been widely used in surgical context awareness to date with significant success. However, the following obstacles remain in clinical use. On the one hand, the end-to-end approach shows limited accuracy and robustness in vivo scenarios. The complexity of the process, the variability of the situation and the hard-frame characteristics of the in-vivo images interfere with the extraction of effective features from the deep network; end-to-end training is prone to overfitting due to the lack of available samples caused by difficulty in collecting and annotating. On the other hand, end-to-end recognition works well for coarse-grained components-stages and steps. However, for fine-grained components, any slight disturbance can greatly alter the performance of the deep network. Thus, learning implicit recognition and prediction functions through end-to-end training is very challenging. Furthermore, the end-to-end approach mainly focuses on performance while ignoring interpretability, which results in semantic gap between algorithm and surgeon.
In summary, in the scene perception of real surgery, the following problems still exist: (1) The analysis of the action layer of fine granularity still needs further excavation; (2) Lack of prediction of future states in a surgical workflow; (3) poor interpretability of surgical context awareness algorithms.
Disclosure of Invention
Aiming at the defects or improvement demands of the prior art, the application provides a forward track prediction method and equipment for instrument operation in minimally invasive surgery, and the future state prediction of the surgical flow is performed according to the fine granularity component-action level of the current laparoscopic cholecystectomy surgery, namely, the future state of the surgical operation is predicted at the initial stage of single surgical operation. Specifically, the motion track of the instrument in the two-dimensional image coordinate system is extracted and converted into the two-dimensional polar coordinate system, and the starting point of the instrument motion is used as a far point to realize the alignment of the operation starting point. Then in a polar coordinate system, respectively establishing primitive models of the offline movement distance and direction of the instrument so as to represent a complex movement track mode, and simultaneously, aiming at the online movement, considering the difference of movement tracks of multiple operations, the application provides a self-Adaptive Pattern Recognition (APR) method. For forward distance, APR-GMM is adopted to correct a distance primitive model; and dynamically correcting the forward direction model by adopting APR-PE aiming at the forward direction. And finally, online updating of the motion primitive model parameters is realized.
To achieve the above object, according to one aspect of the present application, there is provided a forward trajectory prediction method of an instrument operation in a minimally invasive surgery, comprising the steps of:
s1: acquiring a motion trail of a surgical instrument in a two-dimensional image coordinate system;
s2: converting the motion trail into a two-dimensional polar coordinate system by analyzing the motion mode of the motion of the instrument, and further splitting the continuous motion trail of the instrument in the motion into discrete motion primitives;
s3: based on the motion primitive, a Gaussian mixture model is adopted to establish a distance primitive, and instrument gesture recognition is adopted to determine instrument tip orientation;
s4: on-line updating distance primitive parameters by adopting self-adaptive pattern recognition, and on-line predicting the direction deviation of the instrument tip orientation by adopting motion direction autoregressive;
s5: based on the updated distance primitive parameters, online predicting the instrument advancing distance by adopting Gaussian mixture regression, and online correcting the instrument advancing direction by adopting multi-motion primitive internal weighting;
s6: and fusing the predicted advancing distance and the advancing direction to obtain a predicted result of the advancing track of the instrument motion in the image.
As a further preferred aspect, in step S1, a YOLO target detection algorithm is used to extract a motion trajectory of the surgical instrument in the two-dimensional image coordinate system.
As a further preferred aspect, in step S2, the converting the motion trajectory into a two-dimensional polar coordinate system includes: the initial point of the operation of the instrument is taken as the origin of a polar coordinate system, the variation of the movement distance of the instrument is used for reflecting the difference of the operation speed of each primitive, and the variation of the movement direction of the instrument is used for reflecting the difference of the operation angle of each primitive.
As a further preferred embodiment, in step S3, the motion primitive is represented by using a gaussian mixture model, that is, a gaussian kernel is matched with the motion primitive to encode the execution distance of the instrument, so as to build a distance primitive, where a probability density function of the gaussian mixture model is represented as:
f k (r t )＝N(r t |μ k ,Σ k )
in the formula ,πk Coefficients of the kth Gaussian component, f k (r t ) Kth kernel, r representing gaussian mixture model t Mu, the movement distance at time t k Is the mean of the kth gaussian kernel.
Preferably, the direction of the instrument is deduced according to the posture of the instrument, the instrument main body and the tip are taken as targets, a target detection model yolo_v5 is adopted to obtain a binding box of the instrument main body and the tip, the center of the binding box is taken as the positions of the left end, the right end and the left end and right end connecting centers of the instrument on the image frame, and at the moment, the vector of the instrument direction is expressed as:
at t, the estimated angle of instrument directionThe method comprises the following steps:
in the formula ,is the vector of the instrument direction, +.>Vector connecting center to right end of instrument for right and left ends,/->The vector connecting the center to the left end of the instrument for the left and right ends.
As a further preferred aspect, in step S4, the online updating the distance primitive parameters using adaptive pattern recognition includes:
k of Gaussian mixture model th Performing off-line training parameterization on the core to obtainTo be used forAs initial conditions, per core +.>Translation vector +.> and />Is expressed as an adaptive primitive parameter, k th The starting point of a primitive is defined as the switching point from the last primitive to the current primitive, denoted as sp k For the initial->Parameters are carried out, and a distance primitive model after correction at the moment t is calculated>
As a further preferable mode, the distance primitive model corrected at the time tComprising the following steps:
wherein ,is k th Updating starting point, mu of primitive k,0 Is the mean value of the kth gaussian kernel at time t=0, pi k Is the weight of the kth gaussian kernel, Σ k,t Is the variance of the kth gaussian kernel at time t;
adaptive primitive parametersThe calculation method of (1) comprises the following steps:
the maximum likelihood estimate is expressed as:
calculating the rotation angle of the kth Gaussian kernel estimated by a formula through logarithmic transformation and space-time regularization
Where p and σ are penalty parameters, and />Representing spatially and temporally canonical terms, respectively.
As a further preferred aspect, in step S4, the predicting the deviation of the direction of the instrument tip on line using autoregressive motion direction includes:
adjusting the estimated direction based on the backward directionIntroducing a velocity extrapolation to correct +.>Specifically, the backward average value is used as the result of the velocity extrapolation, i.e. +.>Then, estimated heading +.>Is determined by the following means:
in the formula ,representing confidence coefficient->Is the estimated heading.
According to another aspect of the present application, there is also provided a forward trajectory prediction system for instrument operation in minimally invasive surgery, comprising:
the first main control module is used for acquiring the motion trail of the surgical instrument in the two-dimensional image coordinate system;
the second main control module is used for converting the motion trail into a two-dimensional polar coordinate system by analyzing the motion mode of the motion of the instrument, and further splitting the continuous motion trail of the instrument in the motion into discrete motion primitives;
the third main control module is used for establishing a distance primitive by adopting a Gaussian mixture model based on the motion primitive and determining the tip orientation of the instrument by adopting instrument gesture recognition;
the fourth main control module is used for online updating distance primitive parameters by adopting self-adaptive pattern recognition and online predicting the direction deviation of the instrument tip orientation by adopting motion direction autoregressive;
the fifth main control module is used for predicting the forward distance of the instrument on line by adopting Gaussian mixture regression based on the updated distance primitive parameters, and correcting the forward direction of the instrument on line by adopting internal weighting of multiple motion primitives;
and the sixth main control module is used for fusing the predicted advancing distance and the advancing direction to acquire a predicted result of the advancing track of the instrument motion in the image.
According to another aspect of the present application, there is also provided a computer readable storage medium having stored thereon a forward trajectory prediction program of an instrument operation in a minimally invasive surgery, which when executed by a processor, implements the steps of the forward trajectory prediction method of an instrument operation in a minimally invasive surgery as described above.
According to another aspect of the present application, there is also provided a terminal device comprising a memory, a processor and a forward trajectory prediction program of an instrument operation in a minimally invasive surgery stored on the memory and operable on the processor, the forward trajectory prediction program of an instrument operation in a minimally invasive surgery being configured to implement the steps of the forward trajectory prediction method of an instrument operation in a minimally invasive surgery as described above.
In general, compared with the prior art, the above technical solution conceived by the present application mainly has the following technical advantages:
1. the application extracts the motion trail of the instrument in the two-dimensional image coordinate system, converts the motion trail into the two-dimensional polar coordinate system, and uses the starting point of the instrument motion as a far point to realize the alignment of the operation starting point. Then in a polar coordinate system, respectively establishing primitive models of the offline movement distance and direction of the instrument so as to represent a complex movement track mode, and simultaneously, aiming at the online movement, considering the difference of movement tracks of multiple operations, the application provides a self-Adaptive Pattern Recognition (APR) method. For forward distance, APR-GMM is adopted to correct a distance primitive model; and dynamically correcting the forward direction model by adopting APR-PE aiming at the forward direction. And finally, online updating of the motion primitive model parameters is realized. I.e., for fine-grained components-actions in the surgical procedure. Break down fine-grained surgical procedures into finer-grained motion primitives and enable future state predictions of intra-operative surgical instrument operations. Furthermore, context awareness is provided for the creation of the RAMIS system, and on the other hand, interpretability analysis is provided for understanding the behavioral patterns of the surgeon's surgical procedure.
2. According to the application, the motion track of the instrument in the two-dimensional image coordinate system extracted by YOLO_v5 is converted into the two-dimensional polar coordinate system, so that the mining operation behavior mode is realized. Characterization of complex trajectory patterns is achieved by decomposing a continuous instrument trajectory into discrete motion primitives. And establishing a distance primitive model through the Gaussian mixture model, and determining the head direction of the instrument through instrument gesture recognition, so as to establish a direction primitive model.
3. In the scene of online execution of operation, the application provides an adaptive pattern recognition method for correcting the deviation between an offline model and an online track. The predicted forward track is adjusted only by using the backward track of the on-line motion of the instrument, so that the future state prediction with high precision and low calculation consumption is realized.
4. According to the application, by analyzing the operation behavior mode of a surgeon, a future state prediction algorithm of instrument operation is designed, on one hand, the algorithm does not need to rely on a large scale of labeling data sets, and has high robustness and strong generalization capability, and on the other hand, a RAMIS navigation function lays a foundation for assisting the research and development of an operation module.
Drawings
FIG. 1 is a flow chart of a method of forward trajectory prediction for instrument operation in accordance with a preferred embodiment of the present application;
FIG. 2 is a flow chart of the instrument operation trajectory extraction based on YOLO_v5 in the present application;
FIG. 3 is a schematic view of a trajectory of a single motion performed by an operating instrument in accordance with the present application;
FIG. 4 is a schematic diagram showing correlation analysis between a backward track and a forward track in the present application;
FIG. 5 is a schematic representation of the pattern of motion elements of the instrument in distance and direction according to the present application;
fig. 6 (a) is a distance primitive model of the separating-jaw operation according to the present application, and fig. 6 (b) is a direction primitive model of the separating-jaw operation according to the present application;
fig. 7 (a) is a schematic diagram of the true trajectory and predicted forward trajectory of the split-split operation; fig. 7 (b) is a schematic diagram of the actual trajectory and predicted forward trajectory of the clip applier-ligating operation; fig. 7 (c) is a schematic diagram of the actual trajectory and predicted forward trajectory of the surgical scissors-cutting operation.
Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application. In addition, the technical features of the embodiments of the present application described below may be combined with each other as long as they do not collide with each other.
As shown in fig. 1, the method for predicting the forward trajectory of the operation of an instrument according to the embodiment of the present application includes the following steps:
(1) YOLO target detection algorithm for extracting motion trail of surgical instrument in two-dimensional image coordinate system
As shown in fig. 2, in a single surgical operation, the body of operation is an instrument operated by the surgeon. Thus, in laparoscopic images, the state of the surgical procedure may be characterized by the trajectory of the instrument. Therefore, the patent samples the target detection algorithm yolo_v5 to capture the motion track of the instrument in the image coordinate system, and provides a data base for further decomposition of the surgical operation. The basic principle of the object detection algorithm yolo_v5 is as follows:
the loss function of the target detection task is generally composed of two parts, classificition Loss (classification loss function) and Bounding Box Regeression Loss (regression loss function).
Wherein I is an indicator function, determining whether an obj center falls in the grid, i=1 when the obj center falls in the grid, if not, i=0, p (c) is a probability distribution of each detected class,each class of theoretical probability distributions, i being the ith grid, s 2 Is the total number of grids.
In yolo_v5, bounding Box Regeression Loss is a compromise between three important aggregate factors: CIOU_Loss of overlapping area, center point distance, aspect ratio:
wherein IOU is the intersection ratio of the Prediction box (Prediction boundary) and the group trunk box (actual boundary), distance_2 2 distance_C is Euclidean Distance between two center points of the Prediction box and the group trunk box 2 And firstly obtaining a union set for the Prediction Box and the group trunk Box, and obtaining the diagonal distance of the minimum external Box of the union set. V is a parameter that measures aspect ratio uniformity and can be defined as:
wherein ,wgt ，h gt Width and height of the group trunk, w P ，h P The width and height of the Prediction box.
Classificition Loss (classification loss function) and Bounding Box Regeression Loss (regression loss function) are used as optimization functions, category information and position information of the binding box are updated, and finally detection of the target instrument is achieved. And sequentially connecting the centers of the instrument binding boxes detected in the continuous frames in one operation, so as to obtain the instrument motion trail in the action.
(2) Excavating instrument operation track mode based on behavior analysis of surgeon
In one particular surgical action, the trajectory of the tool exhibits a partially observable pattern on the laparoscopic image, depending on the operating experience of the surgeon. As shown in fig. 3, three key actions during Calot triangle (cholecysto triangle) region exposure are analyzed as a specified trajectory pattern. The surgeon's operation can be divided into three phases: rapidly approaching a Calot triangle; separating the balloon catheter from the balloon artery with the tip oscillated; through an anatomic window to ensure separation. Likewise, the movements of other instruments may be broken down into similar phases. Thus, the present application breaks down the motion of the surgical instrument in one operation into motion primitives to describe its trajectory pattern in order to express the trajectory pattern. This further decomposition of the surgical procedure can support accurate trajectory prediction by modeling and combining the motion primitives of the instrument. For future prediction of the motion state of the instrument, the independent variable is a backward track, namely a historical track of the motion of the instrument; the dependent variable is the forward trajectory, i.e. the future motion trajectory of the instrument. Setting the forward steps as 3, 6, 9 and 12 steps, and carrying out correlation analysis on independent variables and dependent variables, wherein the results are shown in figure 4.
From the analysis of fig. 4, it can be seen that the forward and backward movements of the instrument have an image correlation only in a uniform dimension, and that the correlation decays as the number of forward steps increases. This indicates that the source error exists in predicting the forward trajectory directly from the backward trajectory of the instrument in the two-dimensional image coordinate system, and the mode of the motion trajectory of the instrument must be further mined. Since the initial pose of the instrument operated by the surgeon once is different, the difference of each primitive is reflected by the difference of the operation speed. Therefore, it is considered to convert the trajectory in the original image coordinate system into a polar coordinate system, to reflect the difference in operation speed of each primitive with the start point of the operation of the instrument as the origin of the polar coordinate system, to reflect the difference in operation angle of each primitive with the change in the movement distance of the instrument. The results after transformation are shown in FIG. 5.
(3) Characterization and modeling of instrument motion trajectory patterns in a single surgical procedure
In the field of robot teaching learning, motion primitives are often used to encode teaching behaviors. The application uses the concept and the coding mode of the motion element for the first time in the operation of the surgical instrument for analyzing the laparoscopic images. In particular, the motion primitives are presented in the form of a Gaussian Mixture Model (GMM) whose gaussian kernels are matched to the primitives to encode the execution distance of the instrument. The Probability Density Function (PDF) of the GMM may be expressed as:
wherein ,πk Representing the coefficients of the kth gaussian component. f (f) k (r t ) K representing GMM th And (3) a core. It is generally distributed in a conditional gaussianIs expressed in terms of (a). Introducing maximum expectation Expectation Maximization (EM) algorithm to estimate parameter of GMM +.>The modeling result is shown in fig. 6 (a).
For the instrument movement direction, the instrument is moved substantially in the direction of the tip orientation during execution. Thus, in one embodiment of the application, the tip orientation of the instrument is determined using the method of instrument pose recognition. As shown in fig. 6 (b), the orientation of the instrument can be deduced from its pose. For Pose Estimation (PE) of an instrument, an instrument is modeled as an articulated object. As shown in fig. 5, the right angle split-pliers in this embodiment define the articulation of the instrument as left (L), right (R) and center (C) connecting R and L. The surgical instrument body and the tip are respectively targeted, and the instrument body and the tip are obtained by using a target detection model yolo_v5. The positions of C, L and R on the image frame are taken as the center of the bounding box, respectively. The vector of the instrument direction can be defined asAt t, the estimated angle of instrument directionThe method comprises the following steps:
(4) On-line Adaptive Pattern Recognition (APR) updating primitive parameters
APR-GMM correction distance primitive: offline GMM represents the distance of tool movement within a particular action, its parameters being trained with historical data setsAnd (5) presenting. As for changes in instrument trajectory, offline GMMs are often unable to maintain sufficient accuracy for long-term prospective predictions at an early stage of motion. To this end, an online adaptation mechanism for GMM parameters is adjusted, and the offline model is modified to adapt to the particular situation. Conventional adaptive GMM methods tend to mix current observed trajectory data with historical data to generate a new GMM, which is inefficient in terms of the amount of data and computational effort to achieve model convergence. From observations, GMM cores operating in the same type have similar shapes, while the location and orientation of the cores differ from one motion trial to another. This observation inspires the present application to generate GMMs that adapt to unknown trajectories by updating the location and orientation of the kernel while maintaining the shape of the kernel. Thus in one embodiment of the application, an APR-GMM method is proposed, the GMM being updated by affine transformation of its kernel. K (K) th The Gaussian kernel for offline training is parameterized +.>To->As initial conditions, per core +.>Translation vector +.> and />Is expressed as an adaptive primitive parameter. k (k) th The starting point of a primitive is defined as the switching point from the last primitive to the current primitive, denoted as sp k . After determining these parameters, byStarting fromParameters are carried out, and a distance primitive model after correction at the moment t is calculated>
wherein ,is k th The update start point of the primitive. Equation (6) shows ∈>Representing the target position of the translation. Adaptive primitive parameters->The calculation method of (1) is that
Maximum Likelihood Estimation (MLE) can be expressed as:
by logarithmic transformation and spatio-temporal regularization, the rotation of the problem estimation can be optimized by
Wherein ρ isAnd σ is the penalty parameter. and />Representing spatially and temporally regularized terms, respectively, to improve the robustness of the algorithm.
APR-PE corrects distance primitive: in clinical surgery, the instrument is always moved in the direction of its tip, which can be used to predict its movement tendency. In practice, the actual direction of the toolIs not exactly equal to the estimated +.>But rather wave around it. In order to achieve an accurate direction estimation, in a preferred embodiment of the application, an APR-PE is proposed which adjusts the estimated direction according to the backward direction>Furthermore, the present application introduces a velocity extrapolation to correct +.>The application uses the backward average value as the result of the velocity extrapolation, i.e./taking account of the track noise>Then, estimated heading +.>The determination may be made by:
wherein ,representing the confidence coefficient, is typically smaller in the execute primitive (execution primitive) than the other primitives, with a direction approximately equal to the instrument tip orientation.
(5) Forward prediction of instrument operation trajectory in an online surgical procedure
Forward distanceIs a primitive of update->The results were then generated using Gaussian Mixture Regression (GMR).And (4) forward direction->In combination to characterize the forward trajectory, the predicted results are shown in fig. 7.
The embodiment of the application predicts the future state of the operation flow according to the fine granularity component-action level of the current laparoscopic cholecystectomy operation, namely, predicts the future state of the operation in the initial stage of single operation. In a single surgical procedure, the subject of the procedure is the instrument operated by the surgeon. Thus, in laparoscopic images, the state of the surgical procedure may be characterized by the trajectory of the instrument. Future state predictions within a single operation are equivalent to predictions of the forward trajectory of the surgical instrument. The precondition for prediction is to determine the pattern of the instrument movement, and the correct pattern recognition can only generate a reliable track prediction result. In addition, the mode determination can only provide a substantially accurate prediction result, and the prediction algorithm needs to have adaptability to the difference in consideration of the difference between data.
According to another aspect of the present application, there is also provided a forward trajectory prediction system for instrument operation in minimally invasive surgery, comprising:
the first main control module is used for acquiring the motion trail of the surgical instrument in the two-dimensional image coordinate system;
the second main control module is used for converting the motion trail into a two-dimensional polar coordinate system by analyzing the motion mode of the motion of the instrument, and further splitting the continuous motion trail of the instrument in the motion into discrete motion primitives;
the third main control module is used for establishing a distance primitive by adopting a Gaussian mixture model based on the motion primitive and determining the tip orientation of the instrument by adopting instrument gesture recognition;
the fourth main control module is used for online updating distance primitive parameters by adopting self-adaptive pattern recognition and online predicting the direction deviation of the instrument tip orientation by adopting motion direction autoregressive;
the fifth main control module is used for predicting the forward distance of the instrument on line by adopting Gaussian mixture regression based on the updated distance primitive parameters, and correcting the forward direction of the instrument on line by adopting internal weighting of multiple motion primitives;
and the sixth main control module is used for fusing the predicted advancing distance and the advancing direction to acquire a predicted result of the advancing track of the instrument motion in the image.
According to another aspect of the present application, there is also provided a computer readable storage medium having stored thereon a forward trajectory prediction program of an instrument operation in a minimally invasive surgery, which when executed by a processor, implements the steps of the forward trajectory prediction method of an instrument operation in a minimally invasive surgery as described above.
The method of the embodiment of the application is realized by the electronic equipment, so that the related electronic equipment is necessary to be introduced. To this end, an embodiment of the present application provides an electronic device, as shown in fig. 3, including: at least one processor (processor), a communication interface (Communications Interface), at least one memory (memory) and a communication bus, wherein the at least one processor, the communication interface, and the at least one memory communicate with each other via the communication bus. The at least one processor may invoke logic instructions in the at least one memory to perform all or part of the steps of the methods provided by the various method embodiments described above.
According to another aspect of the present application, there is also provided a terminal device comprising a memory, a processor and a forward trajectory prediction program of an instrument operation in a minimally invasive surgery stored on the memory and operable on the processor, the forward trajectory prediction program of an instrument operation in a minimally invasive surgery being configured to implement the steps of the forward trajectory prediction method of an instrument operation in a minimally invasive surgery as described above.
Further, the logic instructions in at least one of the memories described above may be implemented in the form of a software functional unit and may be stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present application without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. Based on this knowledge, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In summary, the present application first performs pattern recognition of instrument motion: and (3) splitting the continuous instrument motion trail in the motion into discrete motion primitives by analyzing the motion mode of the motion. The motion trail of the instrument is generated by the operation of the surgeon, and the operating habits of different surgeons, the human environments of multiple operations and the camera visual angles of multiple operations are different. Therefore, the patent extracts the motion trail of the instrument in the two-dimensional image coordinate system, converts the motion trail into the two-dimensional polar coordinate system, and uses the starting point of the instrument motion as a far point so as to realize the alignment of the operation starting points. And then in a polar coordinate system, respectively establishing primitive models of the offline movement distance and direction of the instrument so as to represent a complex movement track mode.
In addition, the application aims at the difference of motion trajectories of online motion considering multiple operations, and the patent proposes a scheme of Adaptive Pattern Recognition (APR). And designing an error correction scheme of the offline primitive model and the online backward track according to the backward distance and the backward direction provided by the backward track of the instrument. For forward distance, APR-GMM is adopted to correct a distance primitive model; and dynamically correcting the forward direction model by adopting APR-PE aiming at the forward direction. And finally, online updating of the motion primitive model parameters is realized.
It will be readily appreciated by those skilled in the art that the foregoing description is merely a preferred embodiment of the application and is not intended to limit the application, but any modifications, equivalents, improvements or alternatives falling within the spirit and principles of the application are intended to be included within the scope of the application.
Claims (10)
1. A method for predicting forward trajectory of instrument operation in minimally invasive surgery, comprising the steps of:
s1: acquiring a motion trail of a surgical instrument in a two-dimensional image coordinate system in an image on the hand;
s2: converting the motion trail into a two-dimensional polar coordinate system by analyzing the motion mode of the motion of the instrument, and further splitting the continuous motion trail of the instrument in the motion into discrete motion primitives;
s3: based on the motion primitive, a Gaussian mixture model is adopted to establish a distance primitive, and instrument gesture recognition is adopted to determine instrument tip orientation;
s4: on-line updating distance primitive parameters by adopting self-adaptive pattern recognition, and on-line predicting the direction deviation of the instrument tip orientation by adopting motion direction autoregressive;
s5: based on the updated distance primitive parameters, online predicting the instrument advancing distance by adopting Gaussian mixture regression, and online correcting the instrument advancing direction by adopting multi-motion primitive internal weighting;
s6: and fusing the predicted advancing distance and the advancing direction to obtain a predicted result of the advancing track of the instrument motion in the image.
2. The method for predicting the forward trajectory of an instrument operation in minimally invasive surgery according to claim 1, wherein in step S1, a YOLO target detection algorithm is used to extract the motion trajectory of the surgical instrument in a two-dimensional image coordinate system.
3. The method for predicting the forward trajectory of an instrument operation in minimally invasive surgery according to claim 1, wherein in step S2, the converting the motion trajectory into a two-dimensional polar coordinate system includes: the initial point of the operation of the instrument is taken as the origin of a polar coordinate system, the variation of the movement distance of the instrument is used for reflecting the difference of the operation speed of each primitive, and the variation of the movement direction of the instrument is used for reflecting the difference of the operation angle of each primitive.
4. The method according to claim 1, wherein in step S3, a gaussian mixture model is used to represent motion primitives, i.e. a gaussian kernel is matched with the motion primitives to encode the execution distance of the instrument, thereby creating distance primitives, and wherein the probability density function of the gaussian mixture model is expressed as:
f k (r t )＝N(r t |μ k ,Σ k )
in the formula ,πk Coefficients of the kth Gaussian component, f k (r t ) Indicating high levelK of the mixture model th Core, r t Mu, the movement distance at time t k Is the mean of the kth gaussian kernel.
Deducing the direction of the instrument according to the posture of the instrument, taking the instrument main body and the tip as targets, adopting a target detection model YOLO_v5 to obtain a binding box of the instrument main body and the tip, respectively taking the center of the binding box as the positions of the left end, the right end and the left end and the right end connecting center of the instrument on an image frame, and at the moment, the vector of the instrument direction is expressed as follows:
at t, the estimated angle of instrument directionThe method comprises the following steps:
in the formula ,is the vector of the instrument direction, +.>Vector connecting center to right end of instrument for right and left ends,/->The vector connecting the center to the left end of the instrument for the left and right ends.
5. The method for predicting the forward trajectory of an instrument operation in minimally invasive surgery according to claim 1, wherein in step S4, the online updating of the distance primitive parameters using adaptive pattern recognition includes:
gaussian will be takenK of hybrid model th Performing off-line training parameterization on the core to obtainTo->As initial conditions, per core +.>Translation vector +.> and />Is expressed as an adaptive primitive parameter, k th The starting point of a primitive is defined as the switching point from the last primitive to the current primitive, denoted as sp k For the initialParameters are carried out, and a distance primitive model after correction at the moment t is calculated>
6. The method for predicting the forward trajectory of instrument operation in minimally invasive surgery according to claim 5, wherein the distance primitive model corrected at time tComprising the following steps:
wherein ,is k th Updating starting point, mu of primitive k,0 Is the mean value of the kth gaussian kernel at time t=0, pi k Is the weight of the kth gaussian kernel, Σ k,t Is the variance of the kth gaussian kernel at time t;
adaptive primitive parametersThe calculation method of (1) comprises the following steps:
the maximum likelihood estimate is expressed as:
calculating the rotation angle of the kth Gaussian kernel estimated by a formula through logarithmic transformation and space-time regularization
Where p and σ are penalty parameters, and />Representing spatially and temporally canonical terms, respectively.
7. The method for predicting the forward trajectory of an instrument operation in minimally invasive surgery according to claim 1, wherein in step S4, the predicting the deviation of the direction of the instrument tip orientation on line using autoregressive motion direction includes:
adjusting the estimated direction based on the backward directionIntroducing a velocity extrapolation to correct +.>Specifically, the backward average value is used as the result of the velocity extrapolation, i.e. +.>Then, estimated heading +.>Is determined by the following means:
in the formula ,representing confidence coefficient->Is the estimated heading.
8. A forward trajectory prediction system for instrument manipulation in minimally invasive surgery, comprising:
the first main control module is used for acquiring the motion trail of the surgical instrument in the two-dimensional image coordinate system;
the second main control module is used for converting the motion trail into a two-dimensional polar coordinate system by analyzing the motion mode of the motion of the instrument, and further splitting the continuous motion trail of the instrument in the motion into discrete motion primitives;
the third main control module is used for establishing a distance primitive by adopting a Gaussian mixture model based on the motion primitive and determining the tip orientation of the instrument by adopting instrument gesture recognition;
the fourth main control module is used for online updating distance primitive parameters by adopting self-adaptive pattern recognition and online predicting the direction deviation of the instrument tip orientation by adopting motion direction autoregressive;
the fifth main control module is used for predicting the forward distance of the instrument on line by adopting Gaussian mixture regression based on the updated distance primitive parameters, and correcting the forward direction of the instrument on line by adopting internal weighting of multiple motion primitives;
and the sixth main control module is used for fusing the predicted advancing distance and the advancing direction to acquire a predicted result of the advancing track of the instrument motion in the image.
9. A computer readable storage medium, characterized in that it has stored thereon a forward trajectory prediction program for an instrument operation in a minimally invasive surgery, which when executed by a processor, implements the steps of the forward trajectory prediction method for an instrument operation in a minimally invasive surgery according to any one of claims 1-7.
10. A terminal device, characterized in that it comprises a memory, a processor and a forward trajectory prediction program of an instrument operation in a minimally invasive surgery stored on the memory and executable on the processor, the forward trajectory prediction program of an instrument operation in a minimally invasive surgery being configured to implement the steps of the forward trajectory prediction method of an instrument operation in a minimally invasive surgery according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310501391.4A CN116597943A (en) | 2023-04-27 | 2023-04-27 | Forward track prediction method and equipment for instrument operation in minimally invasive surgery |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310501391.4A CN116597943A (en) | 2023-04-27 | 2023-04-27 | Forward track prediction method and equipment for instrument operation in minimally invasive surgery |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116597943A true CN116597943A (en) | 2023-08-15 |
Family
ID=87603699
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310501391.4A Pending CN116597943A (en) | 2023-04-27 | 2023-04-27 | Forward track prediction method and equipment for instrument operation in minimally invasive surgery |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116597943A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117338436A (en) * | 2023-12-06 | 2024-01-05 | 鸡西鸡矿医院有限公司 | Manipulator and control method thereof |
-
2023
- 2023-04-27 CN CN202310501391.4A patent/CN116597943A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117338436A (en) * | 2023-12-06 | 2024-01-05 | 鸡西鸡矿医院有限公司 | Manipulator and control method thereof |
| CN117338436B (en) * | 2023-12-06 | 2024-02-27 | 鸡西鸡矿医院有限公司 | Manipulator and control method thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11741605B2 (en) | Method and system for image registration using an intelligent artificial agent | |
| EP3370206B1 (en) | Spatially consistent multi-scale anatomical landmark detection in incomplete 3d-ct data | |
| van Amsterdam et al. | Multi-task recurrent neural network for surgical gesture recognition and progress prediction | |
| US10830584B2 (en) | Body posture tracking | |
| Qin et al. | Temporal segmentation of surgical sub-tasks through deep learning with multiple data sources | |
| da Costa Rocha et al. | Self-supervised surgical tool segmentation using kinematic information | |
| Fard et al. | Soft boundary approach for unsupervised gesture segmentation in robotic-assisted surgery | |
| JP7263216B2 (en) | Object Shape Regression Using Wasserstein Distance | |
| JP2009514111A (en) | Monocular tracking of 3D human motion using coordinated mixed factor analysis | |
| van Amsterdam et al. | Weakly supervised recognition of surgical gestures | |
| Qin et al. | davincinet: Joint prediction of motion and surgical state in robot-assisted surgery | |
| CN116597943A (en) | Forward track prediction method and equipment for instrument operation in minimally invasive surgery | |
| US20240156547A1 (en) | Generating augmented visualizations of surgical sites using semantic surgical representations | |
| Tschiatschek et al. | Variational inference for data-efficient model learning in pomdps | |
| Mici et al. | An incremental self-organizing architecture for sensorimotor learning and prediction | |
| US20230316545A1 (en) | Surgical task data derivation from surgical video data | |
| Hafez et al. | Efficient intrinsically motivated robotic grasping with learning-adaptive imagination in latent space | |
| Ren et al. | Insactor: Instruction-driven physics-based characters | |
| Husz et al. | Evaluation of a hierarchical partitioned particle filter with action primitives | |
| Sahni et al. | Addressing sample complexity in visual tasks using her and hallucinatory gans | |
| Darrell | Reinforcement learning of active recognition behaviors | |
| Ho et al. | An adaptive visual attentive tracker for human communicational behaviors using HMM-based TD learning with new state distinction capability | |
| CN116670726A (en) | System and method for surgical data classification | |
| Ghalamzan-E | Learning needle insertion from sample task executions | |
| CN116453701A (en) | Fine granularity component analysis modeling and identification method and equipment for operation flow |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |