WO2023097944A1

WO2023097944A1 - Bronchoscope position determination method and apparatus, system, device, and medium

Info

Publication number: WO2023097944A1
Application number: PCT/CN2022/086429
Authority: WO
Inventors: 李楠宇; 陈日清; 余坤璋; 刘润南; 徐宏; 苏晨晖
Original assignee: 杭州堃博生物科技有限公司
Priority date: 2021-12-03
Filing date: 2022-04-12
Publication date: 2023-06-08
Also published as: CN113855242B; CN113855242A

Abstract

The present invention provides a bronchoscope position determination method and apparatus, a system, a device, and a medium. The bronchoscope position determination method, comprising: obtaining a virtual bronchial tree of a target object; identifying bifurcated nodes of the virtual bronchial tree of the target object, and obtaining identification information of lung segments and the bifurcated nodes in the virtual bronchial tree of the target object on the basis of the identified bifurcated nodes; obtaining an intraoperative image of the target object; matching the intraoperative image with a virtual slice image of the virtual bronchial tree of the target object, and determining a target virtual slice image matched with the intraoperative image; and determining identification information of the corresponding lung segments and bifurcated nodes matched with the target virtual slice image in the virtual bronchial tree of the target object, the determined identification information being used for representing a current position of the bronchoscope in the target object.

Description

Method, device, system, equipment and medium for determining position of bronchoscope

technical field

The invention relates to the field of bronchoscopes, in particular to a method, device, system, equipment and medium for determining the position of a bronchoscope.

Background technique

Bronchoscope navigation refers to providing navigation guidance for actually taking video images during the operation by determining the position of the bronchoscope.

However, in the existing related technologies, when determining the position of the bronchoscope, it is not possible to accurately locate and describe which lung segment the bronchoscope is in. It can be seen that in the prior art, the information fed back during navigation and positioning is relatively limited, and it is difficult to meet the needs of the bronchoscope. mirror navigation requirements.

Contents of the invention

The invention provides a method, device, system, equipment and medium for determining the position of a bronchoscope, so as to solve the problem that the navigation result is difficult to meet the demand.

According to a first aspect of the present invention, a method for determining the position of a bronchoscope is provided, comprising:

Obtain the virtual bronchial tree of the target object;

Identifying the bifurcation nodes of the virtual bronchial tree of the target object, and obtaining identification information of each lung segment and bifurcation node in the virtual bronchial tree of the target object based on the identified bifurcation nodes;

acquiring an intraoperative image of the target object, the intraoperative image being captured when a bronchoscope travels within the target object;

determining a target virtual slice map matching the intraoperative image by matching the intraoperative image with a virtual slice map of a virtual bronchial tree of the target subject;

Determining the identification information of corresponding lung segments and bifurcation nodes in the virtual bronchial tree of the target object that match the target virtual slice map, and the determined identification information is used to characterize the bronchoscope in the target object the current location of .

Optionally, identifying the bifurcation nodes of the virtual bronchial tree of the target object includes:

Inputting the acquired virtual bronchial tree of the target object into the pre-trained node recognition neural network, and obtaining the positions of each branch node contained in the virtual bronchial tree of the target object output by the node recognition neural network.

Optionally, the node recognition neural network is trained in the following manner:

The sample features of each training sample in the training sample set are respectively extracted, and the virtual bronchial tree contained in the training sample is marked with a label, and the label is used to mark each branch node in the virtual bronchial tree contained in the training sample the actual location of

By inputting the extracted sample features into the node recognition neural network, the predicted position of each bifurcation node contained in the virtual bronchial tree contained in the training sample output by the node recognition neural network is obtained;

adjusting the node recognition neural network according to the difference information between the actual position and the predicted position, to obtain a trained node recognition neural network.

Optionally, based on the identified bifurcation node, the identification information of each lung segment and bifurcation node in the virtual bronchial tree of the target object is obtained, including:

The identification information is determined by matching the bifurcation nodes identified in the virtual bronchial tree of the target object with the knowledge map of the bronchial tree.

Optionally, the identification information is determined by matching the bifurcation nodes identified in the virtual bronchial tree of the target object with the knowledge map of the bronchial tree, including:

Constructing the third graph data with the identified bifurcation node as the vertex and the lung segment used to connect the identified bifurcation node in the virtual bronchial tree of the target object as the edge;

inputting the third graph data into the pre-trained second graph convolutional network, so as to use the second graph convolutional network to determine the corresponding relationship between bifurcation nodes and lung segments in the virtual bronchial tree of the target object ;

According to the corresponding relationship and the knowledge graph, identification information of each lung segment and a bifurcation node in the virtual bronchial tree of the target object is determined.

Optionally, the second graph convolutional network is configured to be able to calculate the probability that any lung segment in the third graph data corresponds to each bifurcation node, and the bifurcation node to which any lung segment belongs is the bifurcation node with the highest probability.

Optionally, determining the target virtual slice map matching the intraoperative image by matching the intraoperative image with the virtual slice map of the virtual bronchial tree of the target object includes:

Obtain the first map data to be matched corresponding to any virtual slice map, the first map data to be matched includes the number and distribution of lung segment openings in the virtual slice map;

Acquiring second image data to be matched corresponding to the intraoperative image, where the second image data to be matched includes the number and distribution of lung segment openings in the intraoperative image;

Comparing the first to-be-matched image data with the extracted second to-be-matched image data, and determining the target virtual slice image according to the comparison result.

Optionally, obtain the first image data to be matched corresponding to any virtual slice image, including:

Determining the virtual opening area in any of the virtual slice diagrams, each virtual opening area corresponding to a lung segment opening in the virtual slice diagram;

The center of the virtual opening area is used as a vertex, and the lines between the centers are used as edges to construct the first graph data; the first graph data includes the number, position, and vertices of any virtual slice graph. relative distance between

Using the first graph convolutional network, the first graph data is mapped to the first graph data to be matched of any virtual slice graph.

Optionally, acquiring second image data to be matched corresponding to the intraoperative image includes:

In the intraoperative image, determine an actual opening area; each actual opening area corresponds to a lung segment opening in the intraoperative image;

Constructing second graph data with the center of the actual opening area as the vertex and the connecting line between the centers as the edge; the second graph data includes the number and position of the vertices in the intraoperative image, and the relative distance between the vertices;

Using the first image convolutional network, the second image data is mapped to the second image data to be matched of the intraoperative image.

Optionally, the first graph convolutional network is an anti-deformation convolutional network;

The anti-deformation convolution network includes a spatial transformation layer and a convolution processing unit:

The space transformation layer is configured to: obtain the first image data and the second image data, perform space transformation on the first image data and/or the second image data, and obtain the first image The first graph data to be convolved corresponding to the data and the second graph data to be convolved corresponding to the second graph data;

The convolution processing unit is configured to convolve the first image data to be convolved to obtain the first image data to be matched of any virtual slice image, and perform convolution on the second image data to be convolved Convolve to obtain the second to-be-matched image data of the intraoperative image.

Acquiring historical matching information; the historical matching information represents: the position and slice angle of the virtual slice map matched by the historical intraoperative image in the virtual bronchial tree of the target object;

According to the historical matching information, determine the current matching range; wherein, the current matching range represents: the position range of the target virtual slice map in the virtual bronchial tree of the target object;

The target virtual slice map is determined by matching the intraoperative image with the virtual slice map corresponding to the current matching range.

Optionally, according to the historical matching information, determining the current matching range includes:

Converting the historical matching information and the shooting time of the historical intraoperative images into a vector to obtain a current vector, and inputting the current vector into a pre-trained long-short-term memory network, so as to use the long-short-term memory network to determine the Describe the current matching range.

Optionally, the target virtual slice map is determined by matching the intraoperative image with the corresponding virtual slice map within the current matching range, including:

By inputting the second map to be matched data corresponding to the intraoperative image into the long short-term memory network, the spliced map to be matched outputted by the long short-term memory network is obtained; wherein, the spliced map to be matched is Data refers to: the image data to be matched after splicing the second image data to be matched corresponding to the intraoperative image and the second image data to be matched corresponding to at least one historical intraoperative image;

The target virtual slice map is determined by matching the stitched map data to be matched with the first map data to be matched corresponding to the virtual slice map within the current matching range.

According to a second aspect of the present invention, a device for determining the position of a bronchoscope is provided, comprising:

The bronchial tree acquisition module is used to acquire the virtual bronchial tree of the target object;

An identification module, configured to identify a bifurcation node of the virtual bronchial tree of the target object, and obtain identification information of each lung segment and bifurcation node in the virtual bronchial tree of the target object based on the identified bifurcation node;

The intraoperative image acquisition module is used to acquire the intraoperative image of the target object, and the intraoperative image is captured when the bronchoscope travels in the human body;

An image matching module, configured to determine a target virtual slice map matching the intraoperative image by matching the intraoperative image with a virtual slice map of the virtual bronchial tree of the target object;

An identification matching module, configured to determine the identification information of corresponding lung segments and bifurcation nodes in the virtual bronchial tree of the target object that match the target virtual slice map, and the determined identification information is used to characterize the bronchoscope The current location within the target object.

According to a third aspect of the present invention, an electronic device is provided, including a processor and a memory,

The memory is used to store codes;

The processor is configured to execute the codes in the memory to implement the methods involved in the first aspect and alternative solutions thereof.

According to a fourth aspect of the present invention, a storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method involved in the first aspect and its optional solution is implemented.

According to a fifth aspect of the present invention, a bronchoscopic navigation system is provided, comprising: a bronchoscope and a data processing unit, the data processing unit is configured to implement the methods involved in the first aspect and its optional solutions.

In the method, device, system, equipment and medium for determining the position of the bronchoscope provided by the present invention, for the virtual bronchial tree of the target object, its bifurcation nodes are identified, and based on this, the lung segments, subdivisions, and branches in the virtual bronchial tree are determined. The identification information of the fork node, compared with the scheme of directly using the virtual bronchial tree for navigation and positioning without identifying and determining the fork node and identification information, the present invention can accurately and effectively locate which lung segment and intersection the bronchoscope reaches and display to meet the needs of bronchoscopic navigation. Furthermore, when identifying and determining based on the trained model, various virtual bronchial tree situations can be effectively taken into consideration.

In a further optional scheme, the current matching range is first determined according to the historical matching information, and then the matching target virtual slice map is searched for the current matching range. Compared with the global matching method in the prior art, this scheme can effectively reduce the cost of matching. The amount of data to be processed improves processing efficiency.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained according to these drawings without any creative effort.

Fig. 1 is a structural schematic diagram of a bronchial navigation system in an exemplary embodiment of the present invention;

Fig. 2 is a schematic flowchart of a method for determining the position of a bronchoscope in an exemplary embodiment of the present invention;

Fig. 3 is a schematic flow diagram of identifying fork nodes in an exemplary embodiment of the present invention;

Fig. 4 is a schematic diagram of the principle of a node recognition neural network in an exemplary embodiment of the present invention;

Fig. 5 is a schematic flowchart of determining identification information in an exemplary embodiment of the present invention;

Fig. 6 is a schematic diagram of a knowledge map in an exemplary embodiment of the present invention;

Fig. 7 is a schematic diagram of another knowledge graph in an exemplary embodiment of the present invention;

Fig. 8 is a schematic flowchart of determining identification information in another exemplary embodiment of the present invention;

Fig. 9 is a schematic diagram of the result after completion of partial naming of lung segments and bifurcation nodes of the virtual bronchial tree in an exemplary embodiment of the present invention;

Fig. 10 is a schematic flow chart of determining a target virtual slice map in an exemplary embodiment of the present invention;

Figure 11 is a schematic diagram of the opening of a lung segment in an exemplary embodiment of the present invention;

Fig. 12 is a schematic flowchart of determining the first graph data to be matched in an exemplary embodiment of the present invention;

Fig. 13 is a schematic diagram of a virtual opening area and an actual opening area in an exemplary embodiment of the present invention;

Fig. 14 is a schematic flow chart of determining the second graph data to be matched in an exemplary embodiment of the present invention;

Fig. 15 is a schematic diagram of the principle of determining a target virtual slice map in an exemplary embodiment of the present invention;

Fig. 16 is a schematic flow chart of determining a target virtual slice map in another exemplary embodiment of the present invention;

Fig. 17 is a schematic diagram of the program modules of the device for determining the position of the bronchoscope in an exemplary embodiment of the present invention;

Fig. 18 is a schematic structural diagram of an electronic device in an exemplary embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of the present invention and the above drawings are used to distinguish similar objects and not necessarily Describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

The technical solution of the present invention will be described in detail below with specific embodiments. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

Please refer to FIG. 1 , an embodiment of the present invention provides a bronchoscope navigation system 100 , including: a bronchoscope 101 and a data processing unit 102 .

The bronchoscope 101 may include an image acquisition unit, and the bronchoscope 101 may be understood as a device or a combination of devices that can use the image acquisition unit to acquire corresponding images after entering the trachea of a human body. Wherein, the bronchoscope 101 may also include a curved tube (such as an active curved tube and/or a passive curved tube), and the image acquisition unit may be located at one end of the curved tube. In addition, no matter what kind of bronchoscope is used, it does not depart from the embodiment of the present invention. range.

The data processing unit 102 can be understood as any device or a combination of devices with data processing capabilities. In the embodiment of the present invention, the data processing unit 102 can be used to implement the position determination method mentioned below. Furthermore, the data processing unit 102 can directly or indirectly perform data interaction with the image acquisition unit in the bronchoscope 101, so that the data acquisition unit 102 can receive intraoperative images.

Please refer to FIG. 2 , an embodiment of the present invention provides a method for determining the position of a bronchoscope, including:

S201: Obtain a virtual bronchial tree of the target object;

In an embodiment, the virtual bronchial tree is 3D, which can also be understood as a 3D virtual model of the bronchial tree. Wherein, the virtual bronchial tree may be a 3D virtual model of the bronchial tree obtained by reconstructing CT data. Of course, the virtual bronchial tree may also be obtained in other ways, which is not limited in this specification. Correspondingly, the target object can be understood as the human body that currently needs to be navigated in the body.

S202: Identify fork nodes of the virtual bronchial tree of the target object, and determine identification information of each lung segment and fork node in the virtual bronchial tree of the target object based on the identified fork nodes;

In an embodiment, a bifurcation node can be understood as any node capable of describing the position of a bifurcation of the virtual bronchial tree, and for each bifurcation, a bifurcation node can be formed to represent it. For example, the coordinates of the central position of the bifurcation in the virtual bronchial tree may be used as the coordinates of the bifurcation node. Furthermore, the process of identifying the bifurcation node can also be regarded as a process of marking the bifurcation opening in the virtual bronchial tree of the target object, and can also be regarded as a process of determining the position of the bifurcation node.

In an embodiment, the identification information can be understood as any information capable of identifying the bifurcation nodes and lung segments, so that different lung segments and different bifurcation nodes can be identified differently. In one example, the identification information includes the name of the lung segment and the identification of the bifurcation node.

S203: Acquiring an intraoperative image of the target object;

The intraoperative image is captured while the bronchoscope is traveling within the corresponding target object;

S204: Determine a target virtual slice map matching the intraoperative image by matching the intraoperative image with a virtual slice map of the virtual bronchial tree of the target object;

Wherein, the virtual slice map of a position in the virtual bronchial tree can be understood as: a map formed by slicing the position of the virtual bronchial tree. The virtual slice diagram may include a slice diagram and/or a cross-section diagram and the like.

S205: Determine the identification information of corresponding lung segments and bifurcation nodes in the virtual bronchial tree of the target object that match the target virtual slice map;

Wherein, the determined identification information is used to characterize the current position of the bronchoscope in the target object.

The current position matches the position of the target virtual slice map in the virtual bronchial tree of the target object. That is, the position of the target virtual slice map in the virtual bronchial tree of the target object may reflect the current position of the bronchoscope in the target object.

In one example, based on step S205, the complete virtual bronchial tree can be displayed, and the identification information and the current position of the bronchoscope in the target object can be displayed in the displayed virtual bronchial tree; another For example, based on step S205, a local virtual bronchus tree near the current position of the bronchoscope and corresponding identification information may also be displayed. In another example, based on step S205, a first interface can also be used to display a complete virtual bronchial tree, and another second interface can be used to display a partial virtual bronchial tree. At the same time, identification information can also be displayed in the two interfaces. The current position of the bronchoscope in the target object is displayed in the first interface;

Compared with the solution that only shows the current position, the above solution can provide richer information for the navigation of the bronchoscope.

In the above scheme, for the virtual bronchial tree of the target object, its bifurcation nodes are identified, and based on this, the identification information of each lung segment and bifurcation node in the virtual bronchial tree is determined. Compared with directly using the virtual bronchial tree for navigation Positioning does not identify and determine the identification information of bifurcation nodes and lung segments. The present invention can accurately and effectively locate and display which lung segment and intersection the bronchoscope reaches, and meet the needs of bronchoscopic navigation.

In one of the implementation manners, please refer to FIG. 3 , the process of identifying the bifurcation nodes of the virtual bronchial tree of the target object may include:

S301: Input the obtained virtual bronchial tree of the target object into a pre-trained node recognition neural network, and obtain the positions of each branch node contained in the virtual bronchial tree of the target object output by the node recognition neural network.

Step S301 can be understood as an implementation of the process of identifying the bifurcation nodes of the virtual bronchial tree of the target object in step S202 shown in FIG. Let me repeat.

The node identification neural network can be any neural network capable of identifying bifurcated nodes in the input virtual bronchial tree, for example, it can be a convolutional neural network, and in other examples, it can also be realized by using a perceptron neural network, a recurrent neural network, etc. . Taking the convolutional neural network as an example, during training, the weight value of each layer in the convolutional neural network can be updated based on the algorithm of forward propagation and backpropagation, and then, in the case of sufficient training samples, it can effectively guarantee the node identification neural network. Accuracy of network recognition.

The node recognition neural network is trained in the following way:

Wherein, the function value of the cost function used in the node identification neural network training matches the difference information, and the difference information represents: for the virtual bronchial tree contained in the training sample, the points marked in the label The error between the actual location of the fork node and the predicted location of the fork node predicted by the node recognition neural network. For example: the sum of the variances of the errors of the positions of all branch nodes of the virtual bronchial tree can be used as the function value of the cost function of the node recognition neural network.

At the same time, the virtual bronchial tree of normal people generally has 18 lung segments and 17 bifurcations, but because each patient has its own specificity, there are still many variations in the inspection. During the network, the training samples will cover various situations of the bronchial tree, and further, the node recognition results can effectively take into account various situations of the bronchial tree, for example, the situation of 17 bifurcations and 18 lung segments can also be taken into account. Variation, so that the trained node recognition neural network can recognize the bifurcation nodes and lung segments contained in various virtual bronchial trees, and improve the accuracy of the trained node recognition neural network output results.

The following uses a 3D convolutional neural network as an example of a node recognition neural network to describe a process of training and establishing the 3D convolutional neural network:

Step a: construct a deep learning data set, the data in the data set is: 3D bronchial tree (ie virtual bronchial tree included in the training sample) reconstructed and rendered based on the collected CT data of the patient. Labels can be formed after marking the bifurcation node positions of the virtual bronchial tree contained in the training samples, and then the virtual bronchial tree contained in the training samples and the corresponding labels can be used as training samples, thereby forming a set of training samples as a data set.

Step b: Take 50% of the data set as the training set, 10% as the validation set, and 40% as the test set. In other examples, data sets can also be allocated using other ratios.

Through the above steps a and b, the construction and distribution of the training sample set can be completed.

Step c: establish a 3D convolutional neural network, and use the Xavier initialization method to initialize the weight parameters of the 3D convolutional neural network.

After step c, for the input data X formed by each training sample, subsequent step d-step f may be cyclically implemented.

Step d: Perform maximum and minimum normalization on the input data X. The formula is as follows:

The input data X can be understood as: After the virtual bronchial tree is converted into a three-dimensional matrix, each point of the virtual bronchial tree is an element of the three-dimensional matrix, and the value of each element, for example, can represent color, grayscale, pixel The value of information such as value is the input data X here, and correspondingly, X' is the data after normalization.

Among them, min(X) can be understood as the smallest value among all points of the three-dimensional matrix of the virtual bronchial tree, and max(X) can be understood as the largest value among all points of the three-dimensional matrix of the virtual bronchial tree. Furthermore, through the above-mentioned maximum-minimum normalization process, the data can be mapped to the range of 0 to 1 for processing, which facilitates the speed and convenience of subsequent processing.

The above step d can be realized by the 3D convolutional neural network after the input data X is input into the 3D convolutional neural network, or normalization can be implemented before each input data X is input into the 3D convolutional neural network, and then the normalization The subsequent data is fed into a 3D convolutional neural network.

Step e: After the normalized data is input into the 3D convolutional neural network, after the 3D convolutional neural network predicts the bifurcation node, the position of the bifurcation node predicted by the 3D convolutional neural network can be calculated by the forward propagation algorithm The difference between the position of the bifurcation node and the fork node marked by the label (that is, the position error information), so as to calculate the value of the loss function, where the loss function can also be described as a cost function.

The forward propagation process uses the cost function to calculate the error between the position (such as coordinates) of the forked node marked by the label and the predicted position (such as coordinates) of the forked node. For the convenience of understanding, the difference between the corresponding positions can be used The mean square error is used as the function value C of the cost function, namely

If the number of input samples n=1, then there is

where j represents the jth fork node, y is the coordinate of the fork node marked in the label,

Represents the predicted value, that is, the coordinates of the identified bifurcation nodes, and L corresponds to the maximum number of layers of the neural network.

Step f: apply the function value of the cost function calculated in the forward propagation process to the error back propagation algorithm, thereby optimizing the weight parameters of the 3D convolutional neural network.

Repeat the above step d to step f until the set number of rounds (for example, 200 rounds) of training is completed. Each round of training is verified on the verification set. After 200 rounds of training, the 3D convolutional neural network with the best result of the verification set is tested on the test set, and the trained 3D convolutional neural network is used as the node recognition neural network.

Among them, the error backpropagation optimization algorithm is as follows:

Taking Figure 4 as an example, in the error backpropagation algorithm, use

Indicates the weight value between the jth neuron of the l-th layer and the kth neuron of the l-1th layer,

is the offset corresponding to the jth neuron in the l layer,

is the output of the jth neuron in layer l, and this value is available

Represents, where σ(·) is the activation function,

is the input value of the jth neuron in layer l, which can be used

express. In addition, the

Represents the j-th input data of the l-th layer;

When backpropagating, the ultimate goal is to obtain the gradient value of the overall loss value (that is, the function value of the cost function) relative to the parameters w and b. In order to calculate the gradient of these two parameters, it is necessary to first compare the loss value relative to the output value of the neuron and The output value of the neuron through the activation function is calculated first. if by

Indicates the gradient value of the overall loss value to the input value of the jth neuron in the last layer of the network L, because

for

The value obtained after activation, according to the chain rule:

Using matrix or vector form to consider all neurons in layer L at the same time, then,

where the symbol о represents the Hadamard product. Different from the last layer L, the input value of a single neuron in the hidden layer l comes from multiple neurons of the previous layer l-1, so there are:

Similarly, expressed in matrix or vector form, there is

Finally, the gradient values of w and b can be directly calculated according to the above results, where:

So far, the backpropagation process is completed, and then use the gradient descent algorithm to update the gradient values of the parameters w and b to form parameters w and b that conform to the updated gradient values, that is: w ^l →w ^l -η/nΣ _x δ ^{x ,l} (a ^x,l-1 ) ^T , b ^l →b ^l -η/nΣ _x δ ^x,l where η is the learning rate, which can be an artificially set value.

Through the above steps, the trained 3D convolutional neural network (which can also be understood as a virtual bronchial tree node segmentation model) is obtained, and the trained 3D convolutional neural network can be used to identify the fork nodes of various virtual bronchial trees .

In one of the implementation manners, please refer to FIG. 5, based on the identified bifurcation node, the process of determining the identification information of each lung segment and bifurcation node in the virtual bronchial tree of the target object may include:

S501: Determine the identification information by matching the bifurcation nodes identified in the virtual bronchial tree of the target object with the knowledge map of the bronchial tree.

Step S501 can be understood as an implementation of the process of determining the identification information of each lung segment and the bifurcation node in the virtual bronchial tree of the target object based on the identified bifurcation node in step S202 shown in FIG. The content that has been described in the embodiment shown in 2 will not be repeated here.

The knowledge graph of the bronchial tree can be any information carrier that can characterize the connection relationship between the intersection (or intersection node) and the lung segment to a certain extent. The number of knowledge graphs of the bronchial tree can be one or more. This paper The description does not limit this. For example, FIG. 6 shows an example of a knowledge graph, and FIG. 7 also shows an example of a knowledge graph.

Through the identified bifurcation nodes and the knowledge map, the actual bronchial situation and the objective knowledge of the bronchial tree can be effectively combined to accurately match the identification information of each lung segment and intersection in the virtual bronchial tree.

In one embodiment, based on the identified intersection nodes, start directly from the main airway, and match the name and identification of each bifurcation and lung segment in the knowledge map along each bifurcation node of the virtual bronchial tree to obtain the identification information;

In another embodiment, as shown in FIG. 8, before matching the identification information, the correspondence between the bifurcation node and the lung segment can be further confirmed based on the neural network (such as the second graph convolutional network), For example, if the positioning and recognition results of the bifurcation nodes deviate, it may cause errors and omissions in the corresponding relationship between the bifurcation node and the lung segment. Confirming the adjustment can ensure that the corresponding relationship between each bifurcation node and the lung segment can be adjusted and corrected, so that the identification information can be matched more accurately.

In the embodiment shown in FIG. 8, the process of determining the identification information by matching the bifurcation nodes identified in the virtual bronchial tree of the target object with the knowledge map of the bronchial tree may include:

S801: Taking the identified bifurcation node as a vertex, and taking the lung segment connected to the bifurcation node in the virtual bronchial tree of the target object as an edge, construct a third graph data;

The data in the third figure is a matrix that can characterize the connection relationship between vertices (that is, the identified fork nodes); it can be represented as G(V, E), and in the data in the third figure, V can be understood as a fork Or a bifurcation node, where E can be understood as a lung segment;

S802: Input the third graph data into the second graph convolutional network, so as to use the second graph convolutional network to determine the correspondence between bifurcation nodes and lung segments in the virtual bronchial tree of the target object relation;

The second graph convolution network can be understood as a neural network capable of processing the third graph data; specifically, the second graph convolution network can be configured to be able to calculate any A lung segment corresponds to the probability of each bifurcation node, and the bifurcation node with the highest probability is selected as the bifurcation node to which any lung segment belongs.

S803: According to the corresponding relationship and the knowledge map, determine the identification information of each lung segment and bifurcation node in the virtual bronchial tree of the target object.

The specific process of steps S801 to S803 can be, for example:

Step a: through the identification of the bifurcation node (characterizing the bifurcation mouth) (that is, node segmentation, such as the identification in step S301), since the bifurcation mouth of the bronchial tree is between adjacent lung segments, based on this, it is possible to construct The data in the third picture is: G(V, E), where V is a bifurcation, and E is a lung segment. This step a is an implementation of step S801.

Step b: Since the fork has been marked (that is, the fork node is identified), the next thing to solve is the lung segment. Unlike the common graph convolution for vertex classification, here you can use the second graph convolution network to do Edge classification, that is, by classifying the edges to the corresponding vertices, so as to determine the corresponding relationship between the edges and the vertices. The input of the second graph convolutional network is the third graph data G(V, E), and the output is the corresponding relationship between V and E , the graph convolutional network can play a smoothing role, and can better establish the relationship between nodes and edges; this step b is an implementation of step S802.

In other embodiments, the processing method of the commonly used graph convolutional network can also be applied to this to determine the corresponding relationship between edges and vertices, that is: the commonly used graph convolutional network is used as the second graph convolutional network; correspondingly, if using Commonly used graph convolutional networks, then: when determining the corresponding relationship, the corresponding relationship between edges and vertices can be determined by classifying vertices into edges. For example: when training this common graph convolutional network, the virtual bronchial tree contained in the training sample The graph data of can be marked with a label, which is used to mark the actual probability that any vertex (fork node) belongs to (that is, corresponds to) each edge (lung segment); after training, the common graph convolutional network can be used to output The actual probability that any vertex of the data in the third graph is classified (that is, corresponds to) any side.

Different from this scheme using the commonly used graph convolutional network, the second graph convolutional network can classify edges to corresponding vertices, thereby determining the corresponding relationship between edges and vertices. When training the second graph convolutional network, the training samples The graph data of the included virtual bronchial tree can be marked with labels, and the labels are used to mark the actual probability that any edge (lung segment) belongs to (that is, corresponds to) each vertex (fork node); then the second graph after training The convolutional network can be used to output the actual probability that any edge of the third graph data is classified (ie corresponds to) any vertex.

After the third image data is input to the second image convolutional network, the processing process in the second image convolutional network can be as shown in the following steps c-step e:

Step c: first solve the adjacency matrix A according to G (V, E), the formula is as follows:

Step d: Define the input of the lth layer of the second graph convolutional network as X _l+1 , and the output as X _l , the relationship is as follows:

Where σ is the nonlinear transformation layer, A is the adjacency matrix, D is the degree matrix of each edge, W _l , b _l are the weights of the lth layer of the second graph convolutional network.

Step e: Define the output of the second graph convolution and classify the edges. The formula used is as follows:

Z＝softmax(X _l W _z +b _z )

Where Z is the probability that each edge corresponds to each vertex (that is, the probability that any lung segment corresponds to each bifurcation node), and then, for each edge, the vertex with the highest probability can be selected as its corresponding vertex (also That is, the bifurcation node with the highest probability is selected as the bifurcation node to which any lung segment belongs); softmax is the activation function of the output layer, and W _z and b _z are the weights of the output layer, which can be updated using the backpropagation algorithm.

Through the above steps a-e, steps S801 and S802 can be realized.

Furthermore, a specific implementation manner of step S803 may be, for example, shown in step f below.

Step f: According to the knowledge map summarized by the doctor, traverse each fork in the knowledge map one by one, and match the identification information of the lung segment and the fork in the knowledge map to the fork of the virtual bronchial tree (that is, the fork node) and the lung segment, so as to complete the name of the edge and node of the virtual bronchial tree of the target object (that is, the identification information, which is the name of the lung segment and the identification of the intersection/intersection node), and the results can be referred to in Fig. 9, which shows the results of lung segment and bifurcation node local name completion of the virtual bronchial tree.

The specific matching naming process can be, for example:

Initialize the position, starting from the main airway, according to the knowledge map, the first one is the first bifurcation, which separates the left and right main bronchi;

Starting from the left main bronchus and going down in turn, complete the naming of each edge (that is, each lung segment of the virtual bronchial tree) and vertex (that is, each bifurcation or bifurcation node of the virtual bronchial tree) according to the knowledge map;

Starting from the right main bronchus and going down in turn, complete the naming of each edge and node according to the knowledge map.

That is: by traversing the knowledge graph, each edge and node in the virtual bronchial tree (that is, each lung segment, bifurcation or bifurcation node of the virtual bronchial tree) is matched to the identification information in the knowledge graph, and the order of traversal is not limited to the above example.

It can be seen that in the specific scheme of the above steps S801 to S803, usually the function of the convolutional network of the second graph classifies the vertex V (that is, the bifurcation or the bifurcation node) of the data G(V, E) of the third graph, and the virtual bronchus Each fork of the tree will aggregate several lung segments E, and then, through the second graph convolutional network to do edge aggregation, thus establishing the relationship between vertices and edges, that is, the fork (fork node) and lung segment corresponding relationship. On this basis, by introducing the knowledge graph, it is possible to complete the named edges and vertices (that is, to determine the identification information) for the classified edges and vertices.

In one of the implementations, in order to accurately and effectively realize the matching between the virtual slice map and the intraoperative image, the features of the map can be extracted in advance to form a form more suitable for matching. Of course, the embodiment of the present invention does not exclude A scheme that directly matches images. In the embodiment shown in FIG. 10, the image data to be matched will be formed based on the virtual slice image and the intraoperative image, which may include the following steps:

S1001: Obtain a virtual bronchial tree of the target object;

S1002: Acquiring the intraoperative image;

In one embodiment, the execution process of step S1001 is the same as that of step S201 in the above-mentioned embodiment shown in FIG. 2, and the execution process of step 1002 is the same as that of step S203 in the above-mentioned embodiment shown in FIG. repeat.

After step S1001, it may also include:

S1003: Obtain the first image data to be matched corresponding to any virtual slice image;

After step S1002, it may also include:

S1004: Acquire second image data to be matched corresponding to the intraoperative image.

Wherein, the number and distribution of lung segment openings in the corresponding image can be characterized by using the image data to be matched.

After step S1003 and step S1004, step S1005 may be executed: compare the first image to be matched with the extracted second image to be matched, and determine the target virtual slice image according to the comparison result;

For example, when the comparison result is that the first graph data to be matched is consistent with the second graph data to be matched, or the first graph data to be matched is similar to the second graph data to be matched, the corresponding first graph data to be matched can be correspondingly The virtual slice map of is determined as the target virtual slice map.

The execution process of the above step S1005 is the same as the step S204 in the embodiment shown in FIG. 2 , and will not be repeated here.

The opening of the lung segment can be understood as the entrance of the lung segment, which can be displayed as a closed-loop opening in the two-dimensional virtual slice or intraoperative image. For example: for the bifurcation of a first lung segment to the second lung segment and the third lung segment, the opening of the lung segment can be understood as the entrance of the second lung segment and the entrance of the third lung segment. Taking the intraoperative image shown in FIG. 11 (which can also be regarded as a virtual slice image) as an example, the lung segment openings can be, for example, the lung segment opening 1101 , the lung segment opening 1102 and the lung segment opening 1103 shown in FIG. 11 .

Furthermore, in part of the image data to be matched, information such as the shape and size of the opening of the lung segment can also be represented. Wherein, the first to-be-matched map data of the virtual slice map can at least represent the number and distribution of virtual lung segment openings in the virtual slice map, and the second to-be-matched map data of the intraoperative image can at least represent The number and distribution of the real lung segment openings in the intraoperative image are shown. The distribution mode therein refers to, for example, the relative position and distance between the openings of each two lung segments, and the distribution mode may also refer to the position of the center of the lung segment openings in the virtual slice map, etc., for example.

Compared with the scheme of directly matching the intraoperative image with the virtual slice map, the above method of matching the data of the first map to be matched with the data of the second map to be matched can avoid the intraoperative image, the virtual slice map and the lung segment opening. Irrelevant information (such as the color in the image, the texture of the trachea wall that has nothing to do with the opening of the lung segment, etc.) interferes with the matching results.

Both the first to-be-matched data and the second to-be-matched data can be represented in a matrix, and of course they can also be represented in other ways, without departing from the scope of the embodiments of the present invention.

Please refer to the embodiment shown in FIG. 12 , the process of forming the first image data to be matched of the virtual slice image may include:

S1201: Determine a virtual opening area in any virtual slice map;

Each virtual opening area corresponds to a lung segment opening in the virtual slice map;

S1202: Using the center of the virtual opening area as a node and the connection between the centers as an edge, construct the first graph data;

The first graph data is a matrix capable of characterizing the number and position characteristics of vertices in any virtual slice graph, and the relative distance between vertices;

S1203: Using the first graph convolutional network, map the first graph data to the first graph data to be matched of any virtual slice graph.

The above step S1201 to step S1203 can be regarded as an implementation of step S1003 in the embodiment shown in FIG. 10 , and the content already described in the embodiment shown in FIG. 10 will not be repeated here.

The virtual opening area (and the actual opening area hereinafter) can be understood with reference to the closed-loop graph shown on the right side of FIG. 13 . The identification of the virtual opening area (which can also be understood as delimitation and segmentation) can be identified by using an opening identification model. Of course, the determination of the virtual opening area and the actual opening area can also be achieved by extracting and screening lines in the image, and extracting closed lines, without resorting to an opening recognition model, which is not limited in this specification.

In one embodiment, step S1201 may include: using an opening identification model to identify an opening area in the virtual slice map, so as to determine the virtual opening area.

The opening recognition model can be understood as any model that can divide and delineate the corresponding lung segment opening. Since the opening is the salient content in the figure, the opening recognition model can also be understood as being able to The salient regions in the figure are segmented, which can also be described as a neural network for saliency detection. Each bifurcation contains one or more lung segment openings, and each lung segment opening corresponds to a lung segment. The texture information, size information, and shape information of the lung segment openings themselves or between them are unique.

The neural network for saliency detection (i.e., the opening recognition model) can be a convolutional neural network, which can be used for saliency segmentation of virtual slices and intraoperative images (i.e., the identification of opening regions). In some examples, it can also be used Two neural networks were used to perform saliency segmentation (that is, opening recognition) on the virtual slice map and intraoperative image respectively. A kind of establishment, training process of this convolutional neural network (being the opening recognition model) is given as an example below:

Step a, use the image labeling software with a graphical interface (such as label me software) to mark the salient areas of the virtual slice map and the intraoperative image (it can also be understood as marking the opening of the lung segment in the picture), and obtain the marked area The labeling result of the label can be used as a label, and then each virtual slice map and corresponding label can be used as a sample, and the intraoperative image and corresponding label can also be used as a sample, and then a set of samples can be constructed as a data set, taking 50% of the data set As a training set, 10% as a validation set, and 40% as a test set.

Step b, normalize the samples in the data set, so that the size of the graphs in the sample is unified to 500×500; that is, the size of each graph is unified;

Step c, establishing a convolutional neural network, and using the Xavier initialization method to initialize the weight parameters of the convolutional neural network;

Step d. Set the output matrix size of the convolutional neural network to 500×500×26, 26 means that in the case of variation, the human body will have up to 25 categories of lung segments, and there are 26 categories in total including the background.

After step d, the samples can be input to the convolutional neural network one by one, and the convolutional neural network can implement the following steps e and f.

Step e: The difference between the output matrix and the label obtained by the prediction of the convolutional neural network is the function value of the loss function;

Step f: apply the function value of the loss function to the error backpropagation algorithm, and optimize the weight optimization parameters of the convolutional neural network.

Step e and step f are repeated until the training of the set number of rounds (for example, 1000 rounds) is completed. Each round of training is verified on the verification set. After 1000 rounds of training, the convolutional neural network with the results of the verification set is tested on the test set, and the trained convolutional neural network is used as the opening recognition model.

Corresponding to the process of obtaining data corresponding to any one of the first image to be matched, please refer to FIG. 14 , the process of obtaining the data of the second image to be matched corresponding to the intraoperative image may include:

S1401: In the intraoperative image, determine an actual opening area;

Each actual opening area corresponds to a lung segment opening in the intraoperative image;

S1402: Using the center of the actual opening area as the vertex and the connection line between the centers as the side, construct the second graph data;

The second image data is a matrix capable of characterizing the number of vertices in the intraoperative image, their location features, and the relative distance between vertices; it can be understood with reference to the content of the first image data;

S1403: Using the first graph convolutional network, map the second graph data to the second image data to be matched of the intraoperative image.

In one embodiment, step S1301 may include: using an opening identification model to identify an opening area in the intraoperative image, so as to determine the actual opening area.

Steps S1401 to S1403 are similar to the embodiment shown in FIG. 12 , and will not be repeated here.

The opening area mentioned therein (such as the actual opening area, the virtual opening area) can also be a closed ellipse, circle or olive-shaped curve as shown in Figure 15, wherein the middle column shows a virtual slice diagram, where The area inside the closed ellipse, circle or olive-shaped curve is a virtual opening area, the center of which is used as the vertex when constructing the first graph data, and the connecting line at the center is used as the edge for constructing the first graph data; the column on the right shows is the intraoperative image, and the area inside the closed ellipse, circle or olive-shaped curve is the actual opening area, and its center is used as the vertex when constructing the second image data, and the connecting line of the center is used as the point for constructing the second image data side.

In one embodiment, the first graph convolutional network can adopt a non-resistant variable convolutional network (for example, without the space transformation layer mentioned later), and then, through the first graph convolutional network, only realize Convolution for the first image data and the second image data.

In another embodiment, it is considered that there may be some rigid transformation differences between the virtual slice image and the intraoperative image, such as left and right rotation, upside down and so on. Therefore, the first graph convolutional network used above can be an anti-deformation convolutional network;

Regardless of the implementation mode, through the convolution of the first image convolutional network, the features of the image data can be extracted, which can better reflect the characteristics of the first image data and the second image data, and make the matching result more accurate.

The anti-deformation convolutional network may include:

The space transformation layer is configured to: obtain the first image data and the second image data, transform the first image data and/or the second image data, and obtain the first image data corresponding to the first image data. Image data to be convoluted, and second image data to be convolved corresponding to the second image data;

in:

If the first image data is transformed and the second image data is not transformed, then the first image data to be convoluted is the image data after the first image data transformation, and the second image data to be convoluted is the second image data;

If the second image data is transformed and the first image data is not transformed, then the second graph data to be convoluted is the graph data after the transformation of the second graph data, and the first graph data to be convoluted is the first graph data;

If both the first image data and the second image data are transformed, the first image data to be convoluted is the image data transformed from the first image data, and the second image data to be convoluted is the image transformed from the second image data. data;

The transformation includes: alignment transformation;

The alignment transformation refers to: when the intraoperative image matches any of the virtual slice images, transforming the position of the vertex represented by the data of the first image and the position of the vertex represented by the data of the second image to be consistent or similar;

Taking the triangles and line segments formed by the connection of vertices in Figure 15 as an example, the effect represented by the transformation may include: rotating the triangle formed by the vertices, translating the triangle formed by the vertices, and transforming the line segment formed by the vertices Perform rotations, translations on line segments formed by vertices, etc.

When the intraoperative image does not match any of the virtual slice images, the spatial transformation layer can also transform the first image data and/or the second image data, because it will not affect the final intraoperative image and the virtual slice image. Therefore, no matter how the matching result of the slice map is transformed, it does not depart from the scope of the embodiments of the present invention.

In one of the implementation manners, the convolution processing unit includes:

The embedding layer is used to convert the data representing the relative distance between vertices in the first graph data to be convolved into a fixed vector to obtain the third graph data to be convoluted, and convert the data representing the vertices in the second graph data to be convoluted to The data of relative distance between is converted into described fixed vector, obtains the 4th to be convoluted image data;

A graph convolution layer, configured to convolve the third graph data to be convolved, obtain the first graph data to be matched of any virtual slice graph, and convolve the fourth graph data to be convolved , to obtain the second image data to be matched of the intraoperative image.

In another embodiment, the convolution processing unit may also include a space transformation layer and a graph convolution layer instead of an embedding layer, so that the first image data to be convolved and the second image data to be convolved can be directly convoluted. product.

Furthermore, through the above anti-deformation convolutional network, the alignment transformation between graph data can be realized, and the graph data to be matched obtained on this basis can accurately and effectively realize the matching of images.

Before using the anti-deformation convolutional network, the process that needs to be completed may include, for example:

Step a: Generate a 500×500 01 matrix based on the 500×500×26 output matrix of the convolutional neural network for saliency detection (that is, the opening recognition model), where 0 represents the background (that is, outside the virtual opening area and the actual opening area area), 1 represents the salient area (that is, the virtual opening area, the area inside the actual opening area).

Step b: After converting the virtual slice image and the intraoperative image into a matrix, multiply by the corresponding 01 matrix respectively to obtain the first matrix and the second matrix.

Step c: For the first matrix and the second matrix, take the lung segment opening as the connected domain, calculate its center position, take the center point of each lung segment opening as the vertex, and the line connecting the center points as the edge, construct the graph data (ie first graph data and second graph data).

Implementing the above steps a, b, and c for the virtual slice map is an implementation of step S1202, and implementing the above steps a, b, and c for intraoperative images is step S1402 A way of realizing .

Assume that the anti-deformation graph convolutional network contains three layers in total, where the first layer is a spatial transformation layer, the second layer is an embedding layer, and the third layer is a graph convolutional layer.

The first layer of spatial transformation layer can be used to perform spatial transformation on the first image data and/or the second image data, and its effect can be reflected in the structure of the center point of the lung segment opening in the corresponding image (such as intraoperative image, virtual slice image). 6D degrees of freedom transformation of graphics and line segments.

The role of the second embedding layer can be understood as: because there are differences in the size of the lung segments, they are unified into a fixed vector through the embedding layer, which is beneficial to the calculation of the convolutional layer of the following figure;

The function of the third layer graph convolution layer is as follows:

First solve the adjacency matrix A according to G(V,E), the formula is as follows:

The input of the lth layer of the definition graph convolution is X _l+1 , and the output is X _l . The relationship is as follows:

where σ is the nonlinear transformation layer, A is the adjacency matrix, D is the degree matrix of each edge, W _l , b _l are the graph convolution weights.

After the anti-deformation convolutional network, the alignment and feature smoothing of the to-be-matched map data of the virtual slice map and the to-be-matched map data of the intraoperative image are realized. The second to-be-matched image data) is matched, and then through the matching between the image data, the matching between the virtual slice image and the intraoperative image is realized.

Among them, because the map data of the virtual slice map (such as the first map data and the map data to be matched) all correspond to the specific position and slice angle of the virtual bronchial tree, so through the matching of the map data, the corresponding intraoperative image of the target object can be found The matched target virtual slice map, so as to judge the current position of the bronchoscope in the human body based on the position of the matched target virtual slice map in the virtual bronchial tree, and realize navigation.

In some schemes, after aligning and transforming the first image data and the second image data, the transformed first image data and the second image data can be directly used as the image data to be matched, so as to perform matching.

In addition, the graph data mentioned above can be represented by a matrix.

In one of the implementations, in order to effectively improve the matching effect, please refer to Figure 16, by matching the intraoperative image with the virtual slice map of the virtual bronchial tree of the target object, determine the The process of target virtual slice map can include:

S1601: Obtain historical matching information;

The historical matching information represents: the position and slice angle of the virtual slice map matched by the historical intraoperative image in the virtual bronchial tree of the target object;

S1602: Determine the current matching range according to the historical matching information;

Wherein, the current matching range represents: the position range of the target virtual slice map in the virtual bronchial tree;

S1603: Determine the target virtual slice map by matching the intraoperative image with a corresponding virtual slice map within the current matching range.

Step S1601 to step S1603 can be understood as an implementation of step S204 shown in FIG. 2 , and the content already described in the embodiment shown in FIG. 2 will not be repeated here.

Because vision-based bronchoscopic navigation has a spatio-temporal logic, for example, if the current bronchoscope is at the 15th bifurcation, then it may only go to the 16th bifurcation and the 13th bifurcation, adding multiple intraoperative images The inductive reasoning of the image can narrow the matching range of the virtual reality map, without matching the intraoperative image with all the virtual slices of the virtual bronchial tree of the target object, effectively reducing the amount of matching data, speeding up the matching, and eliminating some illogical solution.

It can be seen that, in the above scheme, the amount of data to be processed for matching can be effectively reduced, and the processing efficiency can be improved.

In an embodiment, the specific process of step S1602 may include:

converting the historical matching information and the shooting time of the historical intraoperative images into a vector to obtain a current vector; and inputting the current vector into a long-short-term memory network to determine the current matching by using the long-short-term memory network scope.

Specifically, the 6D degree of freedom information of the intraoperative image (representing its position and slice angle in the virtual bronchial tree) and the corresponding shooting time can be vectorized, for example, these information can be juxtaposed one by one to form a current vector.

The long-term and short-term memory network can use various training intraoperative images and training virtual slice images as materials to train the long-term and short-term memory network during training, and gradually update its matching range through the weight output by the long-term and short-term memory network. ability.

In step S1602, matching may be realized based on the image data to be matched determined in step S1003 and step S1004 in the embodiment shown in FIG. 10 , for example: in step S1603, may include:

The target virtual slice map is determined by matching the stitched map data to be matched with the first map data to be matched in the virtual slice map within the current matching range.

Since intraoperative images and historical intraoperative images are used for matching, the above scheme can effectively improve the accuracy of matching compared with only using intraoperative images for matching.

In other examples, splicing may not be performed, but the target virtual slice is determined by locally matching the second to-be-matched map data of the intraoperative image with the first to-be-matched map data of the virtual slice map within the current matching range picture.

Please refer to FIG. 17 , the embodiment of the present invention also provides a device 1700 for determining the position of a bronchoscope, including:

A bronchial tree acquiring module 1701, configured to acquire the virtual bronchial tree of the target object;

The identification module 1702 is configured to identify the bifurcation node of the virtual bronchial tree of the target object, and obtain the identification information of each lung segment and bifurcation node in the virtual bronchial tree of the target object based on the identified bifurcation node;

An intraoperative image acquisition module 1703, configured to acquire an intraoperative image of the target object, where the intraoperative image is captured when the bronchoscope travels in the human body;

An image matching module 1704, configured to determine a target virtual slice map matching the intraoperative image by matching the intraoperative image with the virtual slice map of the virtual bronchial tree of the target object;

The identification matching module 1705 is configured to determine the identification information of corresponding lung segments and bifurcation nodes in the virtual bronchial tree of the target object that match the target virtual slice map, and the determined identification information is used to characterize the bronchial The current position of the mirror within the target object.

Optionally, the identification module 1702 is specifically used for:

Optionally, the identification module 1702 is specifically used for;

Optionally, the image matching module 1704 is specifically used for:

Obtaining the first image data to be matched corresponding to any virtual slice image, the first image data to be matched includes the number and distribution of lung segment openings in the virtual slice image;

Optionally, the image matching module 1704 is specifically used for:

Please refer to FIG. 18 , an electronic device 1800 is provided, including:

processor 1801; and,

memory 1802, configured to store executable instructions of the processor;

Wherein, the processor 1801 is configured to execute the above-mentioned methods by executing the executable instructions.

The processor 1801 can communicate with the memory 1802 through the bus 1803 .

An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, and the above-mentioned method is implemented when the program is executed by a processor.

Those of ordinary skill in the art can understand that all or part of the steps for implementing the above method embodiments can be completed by program instructions and related hardware. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps including the above-mentioned method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other various media that can store program codes.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than limiting them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: It is still possible to modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the various embodiments of the present invention. scope.

Claims

A method for determining the position of a bronchoscope, comprising:

Obtain the virtual bronchial tree of the target object;

Identifying the bifurcation nodes of the virtual bronchial tree of the target object, and obtaining identification information of each lung segment and bifurcation node in the virtual bronchial tree of the target object based on the identified bifurcation nodes;

acquiring an intraoperative image of the target object, the intraoperative image being captured when a bronchoscope travels within the target object;

determining a target virtual slice map matching the intraoperative image by matching the intraoperative image with a virtual slice map of a virtual bronchial tree of the target object;

Determining the identification information of corresponding lung segments and bifurcation nodes in the virtual bronchial tree of the target object that match the target virtual slice map, and the determined identification information is used to characterize the bronchoscope in the target object the current location of .
The method for determining the position of a bronchoscope according to claim 1, wherein identifying the bifurcation node of the virtual bronchial tree of the target object comprises:

Inputting the acquired virtual bronchial tree of the target object into the pre-trained node recognition neural network, and obtaining the positions of each branch node contained in the virtual bronchial tree of the target object output by the node recognition neural network.
The method for determining the position of a bronchoscope according to claim 2, wherein the node recognition neural network is trained in the following manner:

The sample features of each training sample in the training sample set are respectively extracted, and the virtual bronchial tree contained in the training sample is marked with a label, and the label is used to mark each branch node in the virtual bronchial tree contained in the training sample the actual location of

By inputting the extracted sample features into the node recognition neural network, the predicted position of each bifurcation node contained in the virtual bronchial tree contained in the training sample output by the node recognition neural network is obtained;

adjusting the node recognition neural network according to the difference information between the actual position and the predicted position, to obtain a trained node recognition neural network.
The method for determining the position of a bronchoscope according to claim 1, wherein, based on the identified bifurcation node, obtaining identification information of each lung segment and bifurcation node in the virtual bronchial tree of the target object includes:

The identification information is determined by matching the bifurcation nodes identified in the virtual bronchial tree of the target object with the knowledge map of the bronchial tree.
The method for determining the position of a bronchoscope according to claim 4, wherein the identification information is determined by matching the bifurcation nodes identified in the virtual bronchial tree of the target object with the knowledge map of the bronchial tree, include:

Constructing the third graph data with the identified bifurcation node as the vertex and the lung segment used to connect the identified bifurcation node in the virtual bronchial tree of the target object as the edge;

inputting the third graph data into the pre-trained second graph convolutional network, so as to use the second graph convolutional network to determine the corresponding relationship between bifurcation nodes and lung segments in the virtual bronchial tree of the target object ;

According to the corresponding relationship and the knowledge graph, identification information of each lung segment and a bifurcation node in the virtual bronchial tree of the target object is determined.
The method for determining the position of a bronchoscope according to claim 5, wherein the second graph convolutional network is configured to be able to calculate the position of any lung segment corresponding to each bifurcation node in the third graph data. probability, and the bifurcation node to which any lung segment belongs is the bifurcation node with the highest probability.
The method for determining the position of a bronchoscope according to claim 1, characterized in that, by matching the intraoperative image with the virtual slice map of the virtual bronchial tree of the target object, the position matching the intraoperative image is determined. Target virtual slice map, including:

Obtaining the first image data to be matched corresponding to any virtual slice image, the first image data to be matched includes the number and distribution of lung segment openings in the virtual slice image;

Acquiring second image data to be matched corresponding to the intraoperative image, where the second image data to be matched includes the number and distribution of lung segment openings in the intraoperative image;

Comparing the first to-be-matched image data with the extracted second to-be-matched image data, and determining the target virtual slice image according to the comparison result.
The method for determining the position of a bronchoscope according to claim 7, wherein obtaining the first map data to be matched corresponding to any virtual slice map includes:

Determining the virtual opening area in any of the virtual slice diagrams, each virtual opening area corresponding to a lung segment opening in the virtual slice diagram;

The center of the virtual opening area is used as a vertex, and the lines between the centers are used as edges to construct the first graph data; the first graph data includes the number, position, and vertices of any virtual slice graph. relative distance between

Using the first graph convolutional network, the first graph data is mapped to the first graph data to be matched of any virtual slice graph.
The method for determining the position of a bronchoscope according to claim 8, wherein obtaining the second image data to be matched corresponding to the intraoperative image comprises:

In the intraoperative image, determine an actual opening area; each actual opening area corresponds to a lung segment opening in the intraoperative image;

Constructing second graph data with the center of the actual opening area as the vertex and the connecting line between the centers as the edge; the second graph data includes the number and position of the vertices in the intraoperative image, and the relative distance between the vertices;

Using the first image convolutional network, the second image data is mapped to the second image data to be matched of the intraoperative image.
The method for determining the position of a bronchoscope according to claim 9, wherein the first graph convolutional network is an anti-deformation convolutional network;

The anti-deformation convolution network includes a spatial transformation layer and a convolution processing unit:

The space transformation layer is configured to: obtain the first image data and the second image data, perform space transformation on the first image data and/or the second image data, and obtain the first image The first graph data to be convolved corresponding to the data and the second graph data to be convolved corresponding to the second graph data;

The convolution processing unit is configured to convolve the first image data to be convolved to obtain the first image data to be matched of any virtual slice image, and perform convolution on the second image data to be convolved Convolve to obtain the second to-be-matched image data of the intraoperative image.
The method for determining the position of a bronchoscope according to claim 1, characterized in that, by matching the intraoperative image with the virtual slice map of the virtual bronchial tree of the target object, the position matching the intraoperative image is determined. Target virtual slice map, including:

Acquiring historical matching information; the historical matching information represents: the position and slice angle of the virtual slice map matched by the historical intraoperative image in the virtual bronchial tree of the target object;

According to the historical matching information, determine the current matching range; wherein, the current matching range represents: the position range of the target virtual slice map in the virtual bronchial tree of the target object;

The target virtual slice map is determined by matching the intraoperative image with the virtual slice map corresponding to the current matching range.
The method for determining the position of a bronchoscope according to claim 11, wherein determining the current matching range according to the historical matching information includes:

Converting the historical matching information and the shooting time of the historical intraoperative images into a vector to obtain a current vector, and inputting the current vector into a pre-trained long-short-term memory network, so as to use the long-short-term memory network to determine the Describe the current matching range.
The method for determining the position of a bronchoscope according to claim 11, wherein the target virtual slice map is determined by matching the intraoperative image with the corresponding virtual slice map within the current matching range, comprising:

By inputting the second map data to be matched corresponding to the intraoperative image into the long short-term memory network, the spliced map data to be matched outputted by the long short-term memory network is obtained; wherein, the spliced map data to be matched refers to : the image data to be matched after splicing the second image data to be matched corresponding to the intraoperative image and the second image data to be matched corresponding to at least one historical intraoperative image;

The target virtual slice map is determined by matching the stitched map data to be matched with the first map data to be matched corresponding to the virtual slice map within the current matching range.
A device for determining the position of a bronchoscope, characterized in that it comprises:

The bronchial tree acquisition module is used to acquire the virtual bronchial tree of the target object;

An identification module, configured to identify a bifurcation node of the virtual bronchial tree of the target object, and obtain identification information of each lung segment and bifurcation node in the virtual bronchial tree of the target object based on the identified bifurcation node;

The intraoperative image acquisition module is used to acquire the intraoperative image of the target object, and the intraoperative image is captured when the bronchoscope travels in the human body;

An image matching module, configured to determine a target virtual slice map matching the intraoperative image by matching the intraoperative image with a virtual slice map of the virtual bronchial tree of the target object;

An identification matching module, configured to determine the identification information of corresponding lung segments and bifurcation nodes in the virtual bronchial tree of the target object that match the target virtual slice map, and the determined identification information is used to characterize the bronchoscope The current location within the target object.
An electronic device, characterized in that it includes a processor and a memory,

The memory is used to store codes;

The processor is configured to execute the codes in the memory to implement the method according to any one of claims 1 to 13.
A storage medium, on which a computer program is stored, and when the program is executed by a processor, the method described in any one of claims 1 to 13 is implemented.
A bronchoscopic navigation system, characterized by comprising: a bronchoscope and a data processing unit, the data processing unit being used to implement the method described in any one of claims 1 to 13.