CN111311541B

CN111311541B - Method, device and storage medium for determining movement displacement of lung fracture plane

Info

Publication number: CN111311541B
Application number: CN202010043475.4A
Authority: CN
Inventors: 魏征
Original assignee: WeBank Co Ltd
Current assignee: WeBank Co Ltd
Priority date: 2020-01-15
Filing date: 2020-01-15
Publication date: 2024-08-27
Anticipated expiration: 2040-01-15
Also published as: CN111311541A

Abstract

The invention discloses a method, a device and a storage medium for determining movement displacement of a lung fracture plane, and relates to the technical field of financial science and technology, wherein the method comprises the following steps: acquiring a first lung lobe segmentation image and a second lung lobe segmentation image of the lung during inspiration and/or expiration; respectively extracting fracture lines of the first lung lobe segmentation image and the second lung lobe segmentation image to obtain a plurality of first fracture lines and a plurality of second fracture lines; respectively calculating the distances between a plurality of corresponding first fracture lines and a plurality of corresponding second fracture lines to obtain a plurality of distances; determining the movement displacement of the plane of the lung fracture according to the plurality of distances; the first fracture lines and the second fracture lines are fracture lines for dividing the same two adjacent lung lobes.

Description

Method, device and storage medium for determining movement displacement of lung fracture plane

Technical Field

The invention relates to the technical field of financial science and technology (Fintech), in particular to a method and a device for determining movement displacement of a lung fracture plane and a storage medium.

Background

The plane of the lung cleft is the gap between adjacent lobes and is also the basis for lobe segmentation. Since the lung lobes are irregular organs, the calculation of the movement of the lung lobes is complex and difficult to realize, quantitative analysis is difficult, and the condition that the lung fracture plane reflects the relative movement of the lung lobes reflects the state of the lung lobes in the respiratory process in a certain sense, so that the research on how to determine the movement displacement of the lung fracture plane is a problem to be solved urgently.

The foregoing is provided merely for the purpose of facilitating understanding of the technical solutions of the present invention and is not intended to represent an admission that the foregoing is prior art.

Disclosure of Invention

In view of the above, the present invention provides a method, an apparatus and a storage medium for determining movement displacement of a lung fracture plane, so as to solve the problems that the current calculation of movement of lung lobes is complex and difficult to implement, and quantitative analysis cannot be performed.

In a first aspect, the present invention provides a method for determining the movement displacement of a plane of a lung fracture, comprising:

acquiring a first lung lobe segmentation image and a second lung lobe segmentation image of the lung during inspiration and/or expiration;

respectively extracting fracture lines of the first lung lobe segmentation image and the second lung lobe segmentation image to obtain a plurality of first fracture lines and a plurality of second fracture lines;

Respectively calculating the distances between a plurality of corresponding first fracture lines and a plurality of corresponding second fracture lines to obtain a plurality of distances;

determining the movement displacement of the plane of the lung fracture according to the plurality of distances;

the first fracture lines and the second fracture lines are fracture lines for dividing the same two adjacent lung lobes.

Optionally, before acquiring the first lung lobe segmentation image and the second lung lobe segmentation image of the lung during inspiration and/or expiration, the method further comprises:

acquiring a first lung image and a second lung image during inspiration and/or expiration;

And respectively carrying out lung lobe segmentation on the first lung image and the second lung image to obtain a first lung lobe segmentation image and a second lung lobe segmentation image.

Optionally, the method for performing lobe segmentation on the first lung image to obtain a first lobe segmented image and the method for performing lobe segmentation on the second lung image to obtain a second lobe segmented image are the same, and performing lobe segmentation on the first lung image and the second lung image respectively to obtain the first lobe segmented image and the second lobe segmented image includes:

Acquiring lung image features of the lung under sagittal plane, features of the lung under coronal plane and features of the lung under transverse plane;

Correcting the third lung lobe slit feature by utilizing the lung lobe slit features of any two of the sagittal plane, the coronal plane and the transverse plane;

and segmenting the first lung image and the second lung image by using the corrected lung lobe slit characteristics to obtain a first lung lobe segmented image and a second lung lobe segmented image.

Optionally, the calculating the distances between the corresponding first fracture lines and the corresponding second fracture lines, respectively, includes: respectively taking a first fracture line and a second fracture line which are in the same layer number;

And taking a plurality of first fracture points on the first fracture line, respectively calculating a plurality of distances from the plurality of first fracture points to the plurality of second fracture lines, and averaging the plurality of distances to obtain the distance between the first fracture line and the second fracture line under the same layer number.

Optionally, the calculating the distances from the first fracture points to the second fracture lines includes:

taking a plurality of second fracture points on the second fracture line;

calculating the distances from one point in the first fracture points to the second fracture points to obtain the point-to-point distances;

Obtaining the distance from the corresponding point to the second fracture line by taking the minimum value of the distances from the points to the points;

And calculating the distance from each point in the first fracture points to the second fracture line to obtain a plurality of distances.

Optionally, the first lobe segmentation image is represented in a first mask image form, the second lobe segmentation image is represented in a second mask image form, and the extracting the fracture lines of the first lobe segmentation image and the second lobe segmentation image respectively to obtain a plurality of first fracture lines and a plurality of second fracture lines includes:

determining whether mask values of adjacent pixels of the first mask image are the same, and forming a plurality of first pixel groups based on two pixels with different mask values; determining crack points corresponding to the pixel groups based on the position information of each pixel point in each first pixel group; forming a plurality of first fracture lines based on each fracture point in the first mask image;

Determining whether mask values of adjacent pixels of the second mask image are the same, and forming a plurality of second pixel groups based on two pixels with different mask values; determining crack points corresponding to the pixel groups based on the position information of each pixel point in each second pixel group; a number of second fracture lines are formed based on respective fracture points in the second mask image.

Optionally, the determining the motion displacement of the lung fracture plane from the plurality of distances comprises:

And if the first lung lobe segmentation image and/or the first lung lobe segmentation image comprises lung lobe segmentation results corresponding to multiple layers of sub-images, determining the movement displacement of the lung crack plane by utilizing the distance between crack lines corresponding to the sub-images of the same layer number.

In a second aspect, the present invention provides a motion displacement determination device for a plane of a lung fracture, comprising:

an acquisition unit for acquiring a first lung lobe segmentation image and a second lung lobe segmentation image of the lung during inspiration and/or expiration

The extraction unit is used for respectively extracting the fracture lines of the first lung lobe segmentation image and the second lung lobe segmentation image to obtain a plurality of first fracture lines and a plurality of second fracture lines;

the computing unit is used for respectively computing the distances between the corresponding first fracture lines and the corresponding second fracture lines to obtain a plurality of distances;

The determining unit is used for determining the movement displacement of the lung fracture plane according to the plurality of distances;

Optionally, the determining device further includes:

The segmentation unit is used for acquiring a first lung image and a second lung image in the inspiration and/or expiration process, and performing lung lobe segmentation on the first lung image and the second lung image respectively to obtain a first lung lobe segmentation image and a second lung lobe segmentation image.

Optionally, the segmentation unit is specifically configured to acquire a lung lobe slit feature of the lung image under the sagittal plane, a lung lobe slit feature under the coronal plane, and a lung lobe slit feature under the transverse plane; correcting the third lung lobe slit feature by utilizing the lung lobe slit features of any two of the sagittal plane, the coronal plane and the transverse plane; segmenting the lung image using the corrected lung lobe slit features; wherein the lung image is a first lung image or a second lung image.

Optionally, the computing unit is specifically configured to take a first fracture line and a second fracture line with the same number of layers respectively; and taking a plurality of first fracture points on the first fracture line, respectively calculating a plurality of distances from the plurality of first fracture points to the plurality of second fracture lines, and averaging the plurality of distances to obtain the distance between the first fracture line and the second fracture line under the same layer number.

Optionally, the computing unit is specifically configured to take a plurality of second fracture points on a second fracture line; calculating the distances from one point in the first fracture points to the second fracture points to obtain the point-to-point distances; obtaining the distance from the corresponding point to the second fracture line by taking the minimum value of the distances from the points to the points; and calculating the distance from each point in the first fracture points to the second fracture line to obtain a plurality of distances.

Optionally, the first lung lobe segmentation image is represented in a first mask image form, the second lung lobe segmentation image is represented in a second mask image form, and the calculating unit is specifically configured to determine whether mask values of adjacent pixels of the first mask image are the same, and form a plurality of first pixel groups based on two pixels with different mask values; determining crack points corresponding to the pixel groups based on the position information of each pixel point in each first pixel group; forming a plurality of first fracture lines based on each fracture point in the first mask image;

Optionally, the determining unit is specifically configured to determine, if the first lung lobe segmentation image and/or the first lung lobe segmentation image include lung lobe segmentation results corresponding to multiple layers of sub-images, a motion displacement of a lung crack plane by using a distance between crack lines corresponding to sub-images of the same layer number.

To achieve the above object, the present invention further provides a terminal device, the longitudinal federal learning system optimization device including: a memory, a processor and a program stored on the memory and executable on the processor, which when executed by the processor performs the steps of the method of determining the motion displacement of a plane of a lung fracture as described above.

In addition, to achieve the above object, the present invention also proposes a storage medium having stored thereon a program which, when executed by a processor, implements the steps of the method of determining a movement displacement of a plane of a lung fracture as described above.

The invention provides a method, a device and a storage medium for determining movement displacement of a lung fracture plane, which can acquire a first lung lobe segmentation image and a second lung lobe segmentation image of a lung in the process of inhaling and/or exhaling; respectively extracting fracture lines of the first lung lobe segmentation image and the second lung lobe segmentation image to obtain a plurality of first fracture lines and a plurality of second fracture lines; respectively calculating the distances between a plurality of corresponding first fracture lines and a plurality of corresponding second fracture lines to obtain a plurality of distances; the movement displacement of the lung fracture plane can be determined according to a plurality of distances; the corresponding first fracture lines and second fracture lines are fracture lines for dividing the same two adjacent lung lobes, so that the problems that the movement of the lung lobes is complex and difficult to realize and quantitative analysis cannot be performed are solved through the movement displacement of the lung fracture plane.

Drawings

The above and other objects, features and advantages of the present invention will become more apparent from the following description of embodiments of the present invention with reference to the accompanying drawings, in which:

FIG. 1 is a schematic diagram of a terminal device in a hardware operating environment according to an embodiment of the present invention;

FIG. 2 is a flow chart of a method for determining movement displacement of a plane of a lung fracture according to an embodiment of the present invention;

FIG. 3 is a flow chart of a lobe segmentation method based on multiple views according to an embodiment of the present invention;

fig. 4 is a schematic diagram of a network structure of a lung lobe segmentation method and/or apparatus based on multiple views according to an embodiment of the present invention.

Detailed Description

The present invention is described below based on examples, but it should be noted that the present invention is not limited to these examples. In the following detailed description of the present invention, certain specific details are set forth in detail. However, for the part not described in detail, the present invention is also fully understood by those skilled in the art.

Furthermore, those of ordinary skill in the art will appreciate that the drawings are provided solely for the purposes of illustrating the objects, features, and advantages of the invention and that the drawings are not necessarily drawn to scale.

Meanwhile, unless the context clearly requires otherwise, throughout the description and the claims, the words "comprise", "comprising", and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is, it is the meaning of "including but not limited to".

As shown in fig. 1, fig. 1 is a schematic structural diagram of a terminal device in a hardware running environment according to an embodiment of the present invention.

The terminal device in the embodiment of the present invention may be a PC, a central server of a distributed service system, etc., as shown in fig. 1, the terminal device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, a communication bus 1002. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display, an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also optionally be a storage device separate from the processor 1001 described above.

Optionally, the terminal device may further include a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like. Among other sensors, such as light sensors, motion sensors, and other sensors.

It will be appreciated by those skilled in the art that the terminal device structure shown in fig. 1 is not limiting of the terminal device and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.

As shown in fig. 1, an operating system, a network communication module, a user interface module, and a terminal program may be included in the memory 1005, which is a type of computer storage medium.

In the terminal device shown in fig. 1, the network interface 1004 is mainly used for connecting to a background server, and performing data communication with the background server; the user interface 1003 is mainly used for connecting a client (user side) and performing data communication with the client; and the processor 1001 may be used to invoke the terminal program stored in the memory 1005.

In this embodiment, the terminal device includes: the terminal device comprises a memory 1005, a processor 1001 and a terminal program stored in the memory 1005 and capable of running on the processor 1001, wherein the processor 1001 executes the operation in the terminal method when calling the terminal program stored in the memory 1005.

Fig. 2 is a flow chart of a method for determining movement displacement of a plane of a lung fracture according to an embodiment of the present invention. As shown in fig. 2, a method for determining the movement displacement of a lung fracture plane comprises the following steps:

Step S1001: a first lung lobe segmentation image and a second lung lobe segmentation image of the lung during inspiration and/or expiration are acquired.

In some possible embodiments, the first lung lobe segmentation image may be a lung lobe segmentation result of the first lung image and the second lung lobe segmentation image may be a lung lobe segmentation result of the second lung image. Wherein the first lung image and the second lung image may be two images of the lungs during inspiration and/or expiration, respectively. For example, the first and second lung images may be 2 (group) images during inspiration, or the first and second lung images may be 2 (group) images during expiration, or the first and second lung images may be 1 (group) image during inspiration and 1 (group) image during expiration, respectively. Correspondingly, the first lung lobe segmentation image and the second lung lobe segmentation image may represent the region division result of each lung in the corresponding lung image, respectively.

The lung image (the first lung image or the second lung image) in the embodiment of the present invention may include: the right lung includes 3 lobes of the upper right lobe, the middle right lobe and the lower right lobe, and the left lung includes 2 lobes of the upper left lobe and the lower left lobe, for a total of 5 lobe regions. In addition, the lung image in the embodiment of the invention can be a multi-layer sub-image, and the multi-layer sub-image can be a multi-layer lung image obtained when the image acquisition equipment such as CT acquires the lung. Accordingly, the segmented image (the first lung lobe segmented image or the second lung lobe segmented image) obtained by performing the segmentation processing on the lung image may also include the lung lobe segmentation result of each layer of sub-image in the lung image. Since a plurality of lung lobes may be included in each layer of the lung image, a slit line (parting line) may be formed between adjacent lung lobes, corresponding slit lines formed between identical lung lobes in each layer of the sub-image may form a slit plane, e.g., a right horizontal slit plane (right horizontal fissure plane, RHFP) may be formed from the slit line between the upper right lobe and the middle right lobe in each layer of the sub-image, a right diagonal slit plane (right oblique fissure plane, ROFP) may be formed from the slit line between the middle right lobe and the lower right lobe in each layer of the sub-image, and a left diagonal slit plane (left oblique fissure plane, LOFP) may be formed from the slit line between the upper left lobe and the lower left lobe in each layer of the sub-image.

It should be noted that if a lobe is missing, such as a lobe is resected, then the slit lobes or slit surfaces between the lobe and other lobes may disappear. For example, the upper left lobe is resected and the left oblique fracture plane disappears.

In some possible embodiments, performing the lobe segmentation process on the first and second lung images may be implemented using a neural network that enables lobe segmentation, which is trained to accurately segment each lobe region from the lung images, resulting in lobe segmented images (first and second lobe segmented images) that represent the segmentation results. For example, in the embodiment of the present invention, the lobe segmentation image may be a mask image, where the mask image includes a plurality of mask values, each of the lobe areas corresponds to one mask value, and in the lobe segmentation image, the same mask value forms one lobe area, for example, the embodiment of the present invention may assign the above five lobe areas (upper right lobe, middle right lobe, lower right lobe, upper left lobe, and lower left lobe) with unique corresponding mask values, such as 1, 2, 3, 4, and 5, respectively, where each of the mask values forms the location area where the corresponding lobe is located. The above mask values are merely exemplary, and other mask values may be configured in other embodiments.

Step S1002: respectively extracting fracture lines of the first lung lobe segmentation image and the second lung lobe segmentation image to obtain a plurality of first fracture lines and a plurality of second fracture lines;

In some possible embodiments, in case of obtaining a lobe segmented image, the fissures between different lobes may also be extracted from the lobe segmented image. As described in the above embodiments, the lobe segmentation image may include lobe segmentation results (hereinafter referred to as sub-lobe segmentation results) corresponding to the multi-layer sub-image, and the corresponding slit line between the lobes may be extracted from the sub-lobe segmentation results corresponding to each layer of sub-image. Wherein a slit line extracted from the plurality of sub-lobe segmented images of the first lobe segmented image is referred to as a first slit line, and a slit line extracted from the plurality of lobe segmented images of the second lobe segmented image is referred to as a second slit line. In addition, the first slit line and the second slit line in the embodiment of the present invention may be slit lines corresponding to the same at least two lung lobes, and may include at least one of a slit line between an upper right lobe and a middle right lobe, a slit line between a middle right lobe and a lower right lobe, and a slit line between a root upper left lobe and a lower left lobe, for example.

As in the above embodiments, the resulting lung lobe segmentation image may be represented as a mask image, and different mask values may correspond to different lung lobe areas. The embodiment of the invention can determine the fracture line in each layer of sub-lobe segmented image according to the mask value in each layer of sub-lobe segmented image in the lobe segmented image (the first lobe segmented image or the second lobe segmented image).

Wherein, if two adjacent (up-down or left-right adjacent) pixels have different mask values for each layer of sub-lung lobe divided image, such adjacent pixels may be referred to as a pixel group, according to the embodiment of the invention, the average value position among the positions of the pixel points in each pixel group can be determined as the crack point corresponding to the pixel group, and a crack line can be formed through the crack point corresponding to each pixel group. By the method, a plurality of first fracture lines corresponding to the first lung lobe segmentation image and a plurality of second fracture lines corresponding to the second lung lobe segmentation image can be obtained. Wherein the number of first and second slit lines is the same as the number of sub-segmented images in the first and second lung lobe segmented images, i.e. the same number of layers as the sub-lung images of the lung images.

Step S1003: respectively calculating the distances between a plurality of corresponding first fracture lines and a plurality of corresponding second fracture lines to obtain a plurality of distances, wherein the corresponding first fracture lines and second fracture lines are fracture lines for dividing the same adjacent two lung lobes;

In some possible embodiments, in a case where a plurality of first slit lines and second slit lines corresponding to the first lung segmentation image and the second lung lobe segmentation image, respectively, are obtained, a distance between the first slit lines and the second slit lines corresponding to the sub-segmentation images located in the same number of layers may be further obtained. For example, a lung lobe segmentation process is performed on an i-th layer sub-image in the first lung image to obtain a sub-segmented image of the i-th layer sub-image, and a first fracture line corresponding to the i-th layer sub-image can be obtained by using the sub-segmented image, so that a plurality of first fracture lines corresponding to a plurality of layers of sub-images can be obtained, wherein i is an integer greater than or equal to 1 and less than N, and N is the number of layers of the sub-images in the first lung image. And similarly, carrying out lobe segmentation processing on a j-th layer sub-image in the second lung image to obtain a sub-segmented image of the j-th layer sub-image, and utilizing the sub-segmented image to obtain a second fracture line corresponding to the j-th layer sub-image, thereby obtaining a plurality of second fracture lines corresponding to the multi-layer sub-image, wherein j is an integer which is greater than or equal to 1 and less than M, and M is the number of layers of the sub-image in the second lung image. In the embodiment of the invention, M and N can be equal.

Under the condition that the crack lines in each layer of sub-image are obtained, the distance between the first crack line corresponding to the ith layer of sub-image in the first lung image and the second crack line corresponding to the ith layer of sub-image in the second lung image can be further obtained, and then the distance between the crack lines of each layer of sub-image is obtained. Wherein the corresponding first and second fracture lines correspond to the same number of layers.

It should be noted that, as in the above embodiment, the lung image may include 5 lung lobe regions, and the corresponding first slit line and the second slit line may be at least one of the three slit lines, such as at least one of a right horizontal slit line a between the upper right lobe and the middle right lobe, a right diagonal slit line b between the middle right lobe and the lower right lobe, and a left diagonal slit line c between the upper left lobe and the lower left lobe. Correspondingly, the distances between sub-slit lines between identical and adjacent lobes, such as the distance between slit lines a, and the distance between slit lines b, and the distance between slit lines c, for the same number of layers, may be further determined. The first fracture line and the second fracture line are specific to which fracture line, and can be configured according to requirements, which is not particularly limited in the invention.

In some possible embodiments, a plurality of first fracture points may be selected from the first fracture lines, a distance between each first fracture point and a corresponding second fracture line is calculated, and a distance between the corresponding first fracture line and the corresponding second fracture line is obtained by an average value of the distances between each first fracture point and the corresponding second fracture line.

Step S1004: and determining the movement displacement of the lung fracture plane according to the plurality of distances.

In some possible embodiments, the distance between the fracture lines of the sub-images corresponding to the same number of layers may be used to determine the movement displacement of the lung fracture plane. Wherein the fracture plane may be formed by the same fracture line corresponding to each layer of sub-images in the lung image (first lung image or second lung image). For example, a right horizontal slit plane may be obtained through a sub-slit line a between an upper right lobe and a middle right lobe of each layer of sub-images, a right diagonal slit plane may be obtained through a sub-slit line b between a middle right lobe and a lower right lobe of each layer of sub-images, and a left diagonal slit plane may be obtained from a sub-slit line c between an upper left lobe and a lower left lobe of each layer of sub-images.

Based on the above embodiments, the distance between the lung fracture planes corresponding to the different lung images may be determined from the distance between the first fracture line and the second fracture line in the same lung fracture plane for the different lung images. That is, the relative displacement (motion displacement) between right horizontal slit planes may be determined using the distances between the first slit lines in the right horizontal slit planes corresponding to the first lung images and the second slit lines in the right horizontal slit planes corresponding to the second lung images, the relative displacement (motion displacement) between right diagonal slit planes may be determined using the distances between the first slit lines in the right diagonal slit planes corresponding to the first lung images and the second slit lines in the right diagonal slit planes corresponding to the second lung images, and the relative displacement (motion displacement) between left diagonal slit planes may be determined using the distances between the first slit lines in the left diagonal slit planes corresponding to the first lung images and the second slit lines in the left diagonal slit planes corresponding to the second lung images. It should be noted that, in the embodiment of the present invention, the distance between each first fracture line and the corresponding second fracture line is a vector distance, which includes both a distance value and a direction of relative movement, so that the relative movement displacement between the corresponding lung fracture surfaces can be conveniently determined.

In the embodiment of the invention, the distance between the first fracture line and the second fracture line under the same layer number can be utilized to form a matrix for representing the distance between the fracture lines of each layer, and the matrix can be used for representing the movement displacement between the corresponding lung fracture surfaces. Wherein the dimension of the matrix may be expressed as h×w, where H represents the length of the matrix, W represents the width of the matrix, where the value of H is the same as the number of layers of the lung image, and W represents the number of first slit lines or second slit lines of each layer. And element x _ij in the matrix represents a distance between the ith first slit line and the second slit line in the jth lung image, where i is an integer greater than or equal to 1 and less than 3.

In other possible embodiments, obtaining the distance between the planes of the lung fracture may further comprise determining an average of the distances between the corresponding first and second fracture lines, resulting in an average motion displacement between the planes of the lung fracture.

Based on the configuration, the embodiment of the invention can realize the motion change among the fracture lines in the lung images at different times or under different states, and further can obtain the relative motion among the planes of each lung fracture, so as to solve the problems that the calculation of the movement of the lung lobes is complex and difficult to realize and the quantitative analysis cannot be performed.

The following describes each procedure of the embodiment of the present invention in detail.

As described in the above embodiments, the embodiments of the present invention may first obtain the lung lobe segmentation results of two lung images (the first lung lobe image and the second lung lobe image). That is, before the first lung lobe segmentation image and the second lung lobe segmentation image of the lung during inspiration and/or expiration are acquired, the first lung image and the second lung image during inspiration and/or expiration may be acquired, and lung lobe segmentation is performed on the first lung image and the second lung image, respectively, to obtain the first lung lobe segmentation image and the second lung lobe segmentation image.

FIG. 3 is a flow chart of a lobe segmentation method based on multiple views according to an embodiment of the present invention; as shown in fig. 3, the method for obtaining the lung lobe segmentation image (the first lung lobe segmentation image or the second lung lobe segmentation image) by performing lung lobe segmentation on the lung image (the first lung lobe image or the second lung lobe image) is as follows:

s101: the features of the lung fissures in the sagittal plane, in the coronal plane, and in the transverse plane are obtained.

In some possible embodiments, the features of the lung fissures of the lung images at different perspectives may be extracted by means of a feature extraction process. The lung lobe slit feature is a feature for performing a segmentation process of each lung lobe region in a lung image.

According to the embodiment of the invention, feature extraction processing can be respectively carried out on the lung images under the sagittal plane, the coronal plane and the transverse section view angles to obtain the slit features of the lung images under the corresponding view angles, namely the lobe slit features of the lung images under the sagittal plane, the lobe slit features under the coronal plane and the lobe slit features under the transverse section can be respectively obtained. In the embodiment of the invention, the characteristics of the lung lobe cracks under each view angle can be expressed in a matrix or vector form, and the characteristics of the lung lobe cracks under the corresponding view angle can be expressed as characteristic values of the lung image at each pixel point.

In some possible implementations, the present embodiment may obtain lung images at different view angles by taking CT (Computed Tomography ) images. Correspondingly, a plurality of tomographic images can be obtained at each view angle, and the plurality of tomographic images constitute a lung image at that view angle.

In some possible implementations, the feature extraction process may be performed by a feature extraction neural network. For example, the neural network may be trained to achieve accurate extraction of the features of the lung lobular fissures of the lung image by the neural network, and perform lobular segmentation by the resulting features. Under the condition that the precision of the lung lobe segmentation exceeds a precision threshold, the precision of the lung lobe fracture characteristics obtained by the neural network is required, at the moment, a network layer for executing segmentation in the neural network can be removed, and the reserved network part can be used as the characteristic extraction neural network of the embodiment of the invention. The feature extraction neural network may be a convolutional neural network, such as a residual network, a pyramid feature network, or a U network, which are only exemplary, and are not specific limitations of the present invention.

S102: correcting a third lung lobe slit feature by utilizing the lung lobe slit features of any two of the sagittal plane, the coronal plane and the transverse plane;

In some possible embodiments, where the features of the lung lobal fissures at the three perspectives are obtained, the features of the lung lobal fissures at the third perspective may be used to correct the features of the lung lobal fissures at the third perspective, improving the accuracy of the features of the lung lobal fissures at the third perspective.

In one example, embodiments of the present invention may utilize the features of the lung lobal fissures at coronal and cross-sectional perspectives to correct for features of the lung lobal fissures at sagittal perspectives. In other embodiments, another lung lobe slit feature may also be corrected by any two of the three perspectives. For ease of description, the correction of the third lung lobe slit feature is described in the following examples by the first lung lobe slit feature and the second lung lobe slit feature. Wherein the first, second and third lung lobe slit features correspond to the lung lobe slit features at three perspectives of the embodiment of the invention, respectively.

In some possible embodiments, the first and second features of the lung lobe slit may be converted into the third feature of the lung lobe slit in a mapping manner, and feature fusion may be performed using the two features of the lung lobe slit mapped with the third feature of the lung lobe slit to obtain the corrected feature of the lung lobe slit.

S103: the lung image is segmented using the corrected lung lobe slit features.

In some possible embodiments, the lung lobe segmentation may be performed directly from the corrected lung lobe slit features, resulting in a segmentation result of the lung lobe slit. Or in other embodiments, feature fusion processing may be performed on the corrected features of the lung lobe slit and the third features of the lung lobe slit, and lung lobe segmentation may be performed based on the fusion result, so as to obtain a segmentation result of the lung lobe slit, and either the first lung lobe segmentation image or the second lung lobe segmentation image may be obtained. The segmentation result may include location information corresponding to each partition in the identified lung image, among other things. For example, the lung image may include five lung lobe regions, which are respectively an upper right lobe, a middle right lobe, a lower right lobe, an upper left lobe and a lower left lobe, and the obtained segmentation result may include position information of the five lung lobes in the lung image, respectively. The embodiment of the present invention may represent the segmentation result by means of a mask feature, that is, the segmentation result obtained by the embodiment of the present invention may be a feature represented as a mask, for example, the embodiment of the present invention may allocate unique corresponding mask values, such as 1,2,3,4 and 5, to the five lung lobe regions respectively, where each mask value forms a region that is a location region where a corresponding lung lobe is located. The above mask values are merely exemplary, and other mask values may be configured in other embodiments.

Based on the embodiment, the lung lobe slit characteristics under three visual angles can be fully fused, the information content and the accuracy of the corrected slit characteristics are improved, and the accuracy of a lung lobe segmentation result is further improved.

In order to describe the embodiments of the present invention in detail, the respective processes of the embodiments of the present invention are described below.

In the embodiment of the invention, the method for acquiring the lung lobe slit characteristics of the lung image under the sagittal plane, the coronary plane and the cross section comprises the following steps:

Obtaining a multi-sequence lung image under the sagittal, coronal, and transverse planes; and respectively extracting the features of the lung lobe cracks of the multi-sequence lung images under the sagittal plane, the coronal plane and the cross section to obtain the features of the lung lobe cracks under the sagittal plane, the features of the lung lobe cracks under the coronal plane and the features of the lung lobe cracks under the cross section.

The embodiment of the invention can firstly acquire the multi-sequence lung images under three visual angles, and can acquire the multi-layer lung images (multi-sequence images) of the lung images under different visual angles by a CT imaging mode as described in the embodiment.

In the case of obtaining lung images at three view angles, feature extraction processing may be performed on each lung image, for example, feature extraction processing may be performed on the lung images at the respective view angles through the above-described feature extraction neural network, to obtain the features of the lung lobal fissures, such as the features of the lung lobal fissures in the sagittal plane, the features of the lung lobal fissures in the coronal plane, and the features of the lung lobal fissures in the transverse plane, for each lung image at the three view angles. The embodiment of the invention can execute the characteristic extraction processing of the multilayer image in parallel through a plurality of characteristic extraction neural networks, thereby improving the characteristic extraction efficiency. Fig. 3 is a schematic diagram of a network structure of a lung lobe segmentation method and/or apparatus based on multiple views according to an embodiment of the present invention. As shown in fig. 3, the network for performing the feature extraction processing in the embodiment of the present invention may be a U network (U-net), or may be another convolutional neural network capable of performing feature extraction.

In the case of obtaining the features of the lung lobal fissures of the lung image at each view angle, the third feature of the lung lobal fissures may be corrected using the features of the lung lobal fissures of any two of the sagittal, coronal, and transverse planes, and the process may include:

Mapping the features of the any two to the view angle at which the third pair of features of the lung lobes resides; and correcting the third lung lobe crack characteristic by using the mapped lung lobe crack characteristics of any two.

For ease of description, the correction of the third lung lobe slit feature will be described below by taking the first lung lobe slit feature and the second lung lobe slit feature as examples.

Because the extracted features of the lung lobe fissures are different under different viewing angles, the embodiment of the invention can map and convert the features of the lung lobe fissures under three viewing angles to one viewing angle. Wherein the mapping of the features of any two of the lung lobes to the view angle at which the third pair of features of the lung lobes is located comprises: and mapping the features of the lung lobe fissures of the multi-sequence lung images of any two of the sagittal plane, the coronal plane and the transverse plane to the viewing angles of the features of the third lung lobe fissures. That is, the first and second lung lobe slit features may be mapped into view angles at which the third lung lobe slit feature is located. The feature information of the visual angle before mapping can be fused in the lung lobe fracture features obtained after mapping through mapping conversion of the visual angle.

As described in the above embodiments, the embodiments of the present invention may obtain a plurality of lung images at each view angle, where the plurality of lung images have a plurality of lung lobe slit features. And each characteristic value in the lung lobe crack characteristic corresponds to each pixel point of the corresponding lung image one by one.

According to the embodiment of the invention, the position mapping relation among all pixel points in the lung image when the visual angle is converted to another visual angle can be determined according to the three-dimensional vertical lung image formed by a plurality of lung images under one visual angle, for example, a certain pixel point is switched from a first position of the first visual angle to a second position of the second visual angle, and at the moment, the characteristic value corresponding to the first position under the first visual angle is mapped to the second position. By the embodiment, mapping conversion between the lung lobe crack characteristics of each lung image under different visual angles can be realized.

In some possible embodiments, where the three perspectives of the lung-lobe-crack-feature are mapped to the same perspectives, a correction process may be performed on the third lung-lobe-crack-feature using the two mapped lung-lobe-crack-features, improving the information content and accuracy of the third lung-lobe-crack-feature.

In the embodiment of the present invention, the method for correcting the third lung lobe crack feature by using the mapped lung lobe crack features of any two of the lung lobe crack features includes:

Respectively carrying out spatial attention feature fusion by using the mapped lung lobe slit features of any two and the third lung lobe slit features to obtain a first fusion feature and a second fusion feature; and obtaining the corrected third lung lobe slit feature according to the first fusion feature and the second fusion feature.

The embodiment of the invention can refer to the feature after the first lung lobe crack feature is mapped as a first mapping feature and the feature after the second lung lobe crack feature is mapped as a second mapping feature. Under the condition that the first mapping feature and the second mapping feature are obtained, spatial attention feature fusion between the first mapping feature and the third lung lobe slit feature can be performed to obtain a first fusion feature, and spatial attention feature fusion between the second mapping feature and the third lung lobe slit feature can be performed to obtain a second fusion feature.

The method for obtaining the first fusion feature and the second fusion feature by respectively utilizing the mapped lung lobe slit features of any two and the third lung lobe slit feature to perform spatial attention feature fusion comprises the following steps:

Respectively connecting the lung lobe slit features of any two with the third lung lobe slit feature to obtain a first connection feature and a second connection feature; performing a first convolution operation on the first connection feature to obtain a first convolution feature, and performing a first convolution operation on the second connection feature to obtain a second convolution feature; performing a second convolution operation on the first convolution feature to obtain a first attention coefficient, and performing a second convolution operation on the second convolution feature to obtain a second attention coefficient; the first fused feature is obtained using a first convolution feature and a first attention coefficient, and the second fused feature is obtained using a second convolution feature and a second attention coefficient.

In some possible embodiments, as shown in fig. 3, the above-mentioned spatial attention feature fusion process may be performed by a network module of a spatial attention mechanism, which is employed by the embodiments of the present invention in consideration of the importance of the lung lobe crack features in different locations. The convolution processing based on the attention mechanism can be realized through a spatial attention neural network (attention), and important characteristics are further highlighted in the obtained fusion characteristics. The importance of each position of the spatial feature can be adaptively learned in the training process of the spatial attention neural network, and the attention coefficient of the feature object corresponding to each position is formed, for example, the coefficient can represent the coefficient value of the [0,1] interval, and the larger the coefficient is, the more important the feature of the corresponding position is.

In the process of performing the spatial attention fusion process, a connection process may be performed on the first mapping feature and the third lung lobe slit feature to obtain a first connection feature, and a connection process may be performed on the second mapping feature and the third lung lobe slit feature to obtain a second connection feature, where the connection process may be performing connection (connection) in a channel direction. In the embodiment of the invention, the dimensions of the first mapping feature, the second mapping feature and the third lung lobe crack feature can be marked as (C/2, H, W), wherein C represents the channel number of each feature, H represents the height of the feature, and W represents the width of the feature. Correspondingly, the dimensions of the first connection feature and the second connection feature obtained by the connection process may be expressed as (C, H, W).

In the case of obtaining the first connection feature and the second connection feature, a first convolution operation may be performed on each of the first connection feature and the second connection feature, for example, by using a convolution layer a to perform the first convolution operation through a convolution kernel of 3*3, and then batch normalization (bn) and activation function (relu) processing may also be performed to obtain a first convolution feature corresponding to the first connection feature and a second convolution feature corresponding to the second connection feature. The scale of the first convolution feature and the second convolution feature can be expressed as (C/2, H, W), and parameters in the feature map can be reduced through the first convolution operation, so that subsequent calculation cost is reduced.

In some possible embodiments, in the case of obtaining the first convolution feature and the second convolution feature, a second convolution operation and sigmoid function processing may be performed on the first convolution feature and the second convolution feature, respectively, to obtain the corresponding first attention coefficient and the second attention coefficient, respectively. Wherein the first attention coefficient may represent the importance of the features of the individual elements of the first convolution feature and the second attention coefficient may represent the importance of the features of the elements of the second convolution feature.

As shown in fig. 4, for either the first convolution feature or the second convolution feature, the second convolution operation may be performed using two convolution layers B and C, where after the convolution layer B is processed by the convolution kernel of 1*1, a batch normalization (bn) and an activation function (relu) process is performed to obtain a first intermediate feature, where the scale of the first intermediate feature map may be denoted as (C/8, h, w), and then a convolution operation of the convolution kernel of 1*1 is performed on the first intermediate feature map by the second convolution layer C to obtain a second intermediate feature map of (1, h, w). Further, an activation function process may be performed on the second intermediate feature map using a sigmoid function, so as to obtain an attention coefficient corresponding to the first convolution feature or the second performance feature, where the coefficient value of the attention coefficient may be a value in the range of [0,1 ].

The second convolution operation can execute dimension reduction processing on the first connection feature and the second connection feature to obtain the attention coefficient of the single channel.

In some possible embodiments, in the case of obtaining the first attention coefficient corresponding to the first convolution feature and the second attention coefficient corresponding to the second convolution feature, product processing may be performed on the first convolution feature and the first attention coefficient, and the product result may be added to the first convolution feature to obtain the first fusion feature. And performing product processing on the second convolution feature and the second attention coefficient matrix, and adding the product result and the second convolution feature to obtain a second fusion feature. Where the product process (mul) may multiply for the corresponding elements and the feature addition (add) may add for the corresponding elements. By the mode, the characteristics under three visual angles can be effectively fused.

Or in other embodiments, the features obtained by multiplying the first convolution feature by the first attention coefficient may be added to the first convolution feature, and a plurality of convolution operations are performed on the added features to obtain the first fusion feature; and adding the characteristic multiplied by the second attention coefficient by the second convolution characteristic with the second convolution characteristic, and carrying out a plurality of convolution operations on the added characteristic to obtain the second fusion characteristic. By the method, the accuracy of fusion characteristics can be further improved, and the information content of fusion can be improved.

In the case of obtaining the first fusion feature and the second fusion feature, a corrected third lung lobe slit feature may be obtained using the first fusion feature and the second fusion feature.

In some possible embodiments, since the first fusion feature and the second fusion feature respectively include feature information under three views, the first fusion feature and the second fusion feature may be directly connected, and a third convolution operation is performed on the connected features, so as to obtain a corrected third lung lobe slit feature. Or the first fusion feature, the second fusion feature and the third lung lobe crack feature can be connected, and a third convolution operation is performed on the connected features to obtain a corrected third lung lobe crack feature.

Wherein the third convolution operation may include a packet convolution process. Further fusion of feature information in each feature may be further achieved by a third convolution operation. As shown in fig. 2, the third convolution operation of the embodiment of the present invention may include a group convolution D (DEPTH WISE conv), where the group convolution may increase the convolution speed while increasing the accuracy of the convolution feature.

In the case where the corrected third lung lobe slit feature is obtained by the third convolution operation, the lung image may be segmented using the corrected lung lobe slit feature. The embodiment of the invention can obtain the segmentation result corresponding to the corrected lung lobe fracture characteristics in a convolution mode. As shown in fig. 3, the embodiment of the present invention may input the corrected lung lobe slit feature into the convolution layer E, and perform standard convolution through the convolution kernel 1*1 to obtain the segmentation result of the lung image. As described in the above embodiment, the location areas where the five lung lobes in the lung image are respectively located may be represented in the segmentation result. As shown in fig. 2, the lung lobe areas in the lung image are distinguished by means of a filled-in color.

Based on the configuration, the lung lobe segmentation method based on multiple visual angles provided by the embodiment of the invention can solve the technical problems that information is lost and lung lobes cannot be accurately segmented due to the fact that the lung lobes are segmented by not fully utilizing information of other visual angles.

As described in the above embodiments, the embodiments of the present invention may be implemented by a neural network, and as shown in fig. 3, the neural network performing the lobe segmentation method under multiple views of the embodiments of the present invention may include a feature extraction neural network, a spatial attention neural network, and a segmentation network (including convolution layers D and E).

The embodiment of the invention can comprise three feature extraction neural networks which are respectively used for extracting the characteristics of the lung lobe fissures under different visual angles. Among these, three feature extraction networks may be referred to as a first branch network, a second branch network, and a third branch network. The three branch networks in the embodiment of the invention have the same structure, and the input images of each branch network are different. For example, a sagittal plane lung image sample is input to the first branch network, a coronal plane lung image sample is input to the second branch network, and a transverse plane lung image sample is input to the third branch network, for performing feature extraction processing of the lung image samples at respective perspectives, respectively.

Specifically, in the embodiment of the present invention, the process of training the feature extraction neural network includes:

Acquiring training samples under sagittal plane, coronal plane and cross section, wherein the training samples are lung image samples with marked lung lobe crack characteristics; performing feature extraction on a sagittal plane lung image sample by using the first branch network to obtain a first predicted lung lobe slit feature; performing feature extraction on the lung image sample under the coronal plane by using the second branch network to obtain second predicted lung lobe slit features; performing feature extraction on the lung image sample under the cross section by using the third branch network to obtain a third predicted lung lobe crack feature; and obtaining network losses of the first branch network, the second branch network and the third branch network by using the first predicted lung lobe slit characteristic, the second predicted lung lobe slit characteristic and the third predicted lung lobe slit characteristic and the corresponding marked lung lobe slit characteristic respectively, and adjusting parameters of the first branch network, the second branch network and the third branch network by using the network losses.

As described in the above embodiment, the feature extraction processing of the lung image samples under the sagittal plane, coronal plane, and transverse view angles is performed by using the first branch network, the second branch network, and the third branch network, respectively, so that the predicted features, that is, the first predicted lung lobe slit feature, the second predicted lung lobe slit feature, and the third predicted lung lobe slit feature, can be obtained correspondingly.

In case of obtaining each predicted lung lobe slit feature, network losses of the first branch network, the second branch network and the third branch network may be obtained by using the first predicted lung lobe slit feature, the second predicted lung lobe slit feature and the third predicted lung lobe slit feature, respectively, and the corresponding labeled lung lobe slit features. For example, the loss function of the embodiment of the present invention may be a logarithmic loss function, the network loss of the first branch network may be obtained by the first predicted and labeled real lung lobe slit features, the network loss of the second branch network may be obtained by the second predicted and labeled real lung lobe slit features, and the network loss of the third branch network may be obtained by the third predicted and labeled real lung lobe slit features.

In the case of obtaining the network loss of each branch network, parameters of the first branch network, the second branch network, and the third branch network may be adjusted according to the network loss of each network until a termination condition is satisfied. The embodiment of the invention can respectively and simultaneously adjust network parameters, such as convolution parameters and the like, of the first branch network, the second branch network and the third branch network by utilizing the network loss of any branch of the first branch network, the second branch network and the third branch network. Therefore, the network parameters at any view angle can be related to the features at the other two view angles, the correlation between the extracted features of the lung lobe cracks and the features of the lung lobe cracks at the other two view angles can be improved, and the preliminary fusion of the features of the lung lobe cracks at each view angle can be realized.

In addition, the training termination condition of the feature extraction neural network is that the network loss of each branch network is smaller than a first loss threshold value, and at the moment, the feature extraction neural network indicates that each branch network of the feature extraction neural network can accurately extract the lung lobe fissure feature of the lung image under the corresponding visual angle.

Under the condition that the feature extraction neural network is trained, the feature extraction neural network, the spatial attention neural network and the segmentation network can be used for training simultaneously, and the segmentation result output by the segmentation network and the corresponding marking result in the marked lung lobe fissure feature are used for determining the network loss of the whole neural network. And further feeding back and adjusting network parameters of the characteristic extraction neural network, the spatial attention neural network and the segmentation network by utilizing the network loss of the whole neural network until the network loss of the whole neural network is smaller than a second loss threshold value. The first loss threshold value is larger than or equal to the second loss threshold value in the embodiment of the invention, so that the network precision of the network can be improved.

When the neural network of the embodiment of the invention is applied to execute the lobe segmentation based on multiple views, the lung images under different views of the same lung can be respectively and correspondingly input into the three branch networks, and finally the final segmentation result of the lung images, namely the lobe segmentation images, is obtained through the neural network.

Based on the configuration, the method and the device for segmenting the lung lobes based on the multiple visual angles provided by the embodiment of the invention can fuse the characteristic information of the multiple visual angles, execute the lung lobe segmentation of the lung image, and solve the problems that the lung lobes are segmented by not fully utilizing the information of other visual angles, so that the information is lost and the lung lobes cannot be segmented accurately.

By the above embodiment, a first lobe segmented image of the first lung image and a second lobe segmented image of the second lung image may be obtained. Then, a first fracture line in the first segmented image and a second fracture line in the second segmented image may be extracted. Since the first segmented image and the second segmented image may be represented in the form of mask images, the process of extracting the slit line (the first slit line or the second slit line) from the lobe segmented image (the first lobe segmented image or the second lobe segmented image) in the embodiment of the present invention may include:

Determining whether mask values of adjacent pixel points in the mask image are the same, and forming a pixel group based on two pixel points with different mask values; determining a crack point corresponding to each pixel group based on the position information of each pixel point in each pixel group; a fracture line is formed based on all fracture points in the mask image.

Wherein the above-described operation can be performed on the lung lobe segmented image obtained for each layer of the image, as described in the above-described embodiment, if mask values corresponding to two adjacent (up-down adjacent, left-right adjacent) pixels are different for each layer of the lung lobe segmented image, such adjacent pixel points may be referred to as pixel groups, and in the embodiment of the present invention, the average position between the positions of the pixel points in each pixel group may be determined as a slit point corresponding to the pixel group, and a slit line may be formed by the slit point corresponding to each pixel group. The position of the pixel point may be in the form of (x, y), that is, a coordinate value represented in the lung lobe segmentation image. The location of the corresponding fracture line may then be determined based on the average location between two pixel points in the pixel set. By the method, a plurality of first fracture lines corresponding to the first lung lobe segmentation image and a plurality of second fracture lines corresponding to the second lung lobe segmentation image can be obtained. Wherein the number of first and second slit lines is the same as the number of sub-segmented images in the first and second lung lobe segmented images, i.e. the same number of layers as the sub-lung images of the lung images.

Further, in the case where the first fracture line and the second fracture line corresponding to each layer of image are obtained, the distance between the first fracture line and the second fracture line under the same layer may be obtained. In the embodiment of the invention, the specific method for respectively calculating the distances between the corresponding first fracture lines and the corresponding second fracture lines to obtain the distances is as follows:

respectively taking a first fracture line and a second fracture line which are in the same layer number; and taking a plurality of first fracture points on the first fracture line, respectively calculating a plurality of distances from the plurality of first fracture points to the plurality of second fracture lines, and averaging the plurality of distances to obtain the distance between the first fracture line and the second fracture line under the same layer number.

The first fracture line and the second fracture line under the same layer number can be obtained first, wherein a plurality of first fracture points are selected on the first fracture line, and the first fracture points can be selected randomly or at preset distance intervals. The number of the first fracture points may be preconfigured, which is not particularly limited in the present invention.

In the case of acquiring a plurality of first fracture points on a first fracture line, a distance of each first fracture to a second fracture line may be calculated. The specific method for respectively calculating the distances from the first fracture points to the second fracture lines in the embodiment of the invention comprises the following steps: taking a plurality of second fracture points on the second fracture line; step 1: calculating the distances from one point of the first fracture points to all the second fracture points to obtain the point-to-point distances; step 2: obtaining the distance from the point to the second fracture line by taking the minimum value of the distances from the points to the point; and (3) executing the methods of the step 1 and the step 2 at all points in the first fracture points to obtain a plurality of distances.

In some possible embodiments, a plurality of second fracture points may be selected on the second fracture line, or a plurality of second fracture points may be randomly selected, or a plurality of second fracture points may be obtained according to a preset distance interval. The number of the second slit points may be the same as or different from the number of the first slit points, and the present invention is not particularly limited thereto.

Further, a first distance (the above-described point-to-point distance) between each first fracture point and each second fracture point may be obtained, and, for each first fracture point, a minimum distance of the first distance between the first fracture point and each second fracture point may be determined as a distance between the first fracture point and the second fracture line. By the method, the average value of the distance between each first fracture point and each second fracture line can be obtained, the distance between the first fracture line and each second fracture line is determined, and then the distance between the first fracture line and each second fracture line obtained under the same layer number in the first lung lobe segmentation image and the second segmentation image can be obtained.

In the case of obtaining the distance between the first fracture line and the second fracture line corresponding to each layer of image in the lung image, the motion displacement of the lung fracture plane can be determined according to the obtained distances. Wherein the motion displacement may be a matrix representation of the distance between the corresponding first and second fracture lines of each layer of the image, as described in the above embodiments. Alternatively, an average value of distances between a first slit line and a second slit line in the same lung slit plane corresponding to the first lung image and the second lung image may be determined as a movement displacement between the same lung slit plane in the first lung image and the second lung image.

Based on the above, in the embodiment of the present invention, if the first lung lobe segmentation image and the second lung lobe segmentation image may be 2 segmentation images in the exhalation process, the embodiment of the present invention may determine the motion displacement between the planes of the lung fissures in the exhalation process. If the first lung lobe segmentation image and the second lung lobe segmentation image can be 2 segmentation images in the inspiration process, the motion displacement between the planes of each lung crack in the inspiration process can be determined according to the embodiment of the invention. If the first lung lobe segmentation image and the second lung lobe segmentation image are respectively 1 segmentation image in the air process and 1 segmentation image in the expiration process, the embodiment of the invention can determine the motion displacement between the lung fracture planes in the inspiration process and the expiration process.

The computing unit is specifically configured to determine whether mask values of adjacent pixels of the first mask image are the same, and form a plurality of first pixel groups based on two pixels with different mask values; determining crack points corresponding to the pixel groups based on the position information of each pixel point in each first pixel group; forming a plurality of first fracture lines based on each fracture point in the first mask image;

The determining unit is specifically configured to determine, if the first lung lobe segmentation image and/or the first lung lobe segmentation image include lung lobe segmentation results corresponding to multiple layers of sub-images, a motion displacement of a lung crack plane by using a distance between crack lines corresponding to sub-images of the same layer number.

In summary, the embodiment of the invention can solve the problems that the movement of the lung lobes is complex and difficult to realize and quantitative analysis cannot be performed.

It will be appreciated that the above-mentioned method embodiments of the present invention can be combined with each other to form a combined embodiment without departing from the principle logic, and the present invention is not repeated herein.

In addition, the embodiment of the invention also provides a device for determining the movement displacement of the lung fracture plane, wherein the device comprises: the device comprises an acquisition unit, an extraction unit, a calculation unit and a determination unit;

The acquisition unit is connected with the extraction unit, the extraction unit is also connected with the calculation unit, and the calculation unit is also connected with the determination unit;

Optionally, the apparatus further comprises: a dividing unit;

the segmentation unit is connected with the acquisition unit, and before the acquisition unit acquires a first lung lobe segmentation image and a second lung lobe segmentation image of the lung in the inspiration and/or expiration process, the segmentation unit acquires the first lung image and the second lung image in the inspiration and/or expiration process, and performs lung lobe segmentation on the first lung image and the second lung image respectively to acquire the first lung lobe segmentation image and the second lung lobe segmentation image.

Preferably, the method of performing lobe segmentation on the first lung image by the segmentation unit to obtain a first lobe segmented image is the same as the method of performing lobe segmentation on the second lung image to obtain a second lobe segmented image, and the method is as follows:

Optionally, the calculating unit calculates distances between the first fracture lines and the second fracture lines at corresponding positions, so as to obtain the distances specifically by:

respectively taking a first fracture line and a second fracture line which are in the same layer number;

Optionally, the specific method for calculating the distances from the first fracture points to the second fracture lines is as follows:

a plurality of second fracture points are also taken on the second fracture line;

Step 1: calculating the distances from one point of the first fracture points to all the second fracture points to obtain the point-to-point distances;

step 2: obtaining the distance from the point to the second fracture line by taking the minimum value of the distances from the points to the point;

and (3) executing the methods of the step 1 and the step 2 at all points in the first fracture points to obtain a plurality of distances.

In addition, the embodiment of the invention also provides a device for determining the movement displacement of the lung fracture plane, electronic equipment, a computer readable storage medium and a program, which can be used for realizing any one of the methods for determining the movement displacement of the lung fracture plane provided by the invention, and the corresponding technical scheme and description are omitted from description.

In some embodiments, the functions or modules included in the apparatus provided by the embodiments of the present invention may be used to perform the methods described in the foregoing method embodiments, and specific implementations thereof may refer to descriptions of the foregoing method embodiments, which are not repeated herein for brevity.

The above examples are merely illustrative embodiments of the present invention and are described in more detail and are not to be construed as limiting the scope of the invention. It should be noted that modifications, equivalent substitutions, improvements, etc. can be made by those skilled in the art without departing from the spirit of the present invention, and these are all within the scope of the present invention. Accordingly, the scope of protection of the present invention is to be determined by the appended claims.

Claims

1.A method of determining movement displacement of a plane of a lung fracture, comprising:

The first fracture lines and the second fracture lines are fracture lines for dividing two adjacent lung lobes under the same layer number; the first lung lobe segmentation image and the second lung lobe segmentation image are lung lobe segmentation results corresponding to a plurality of layers of sub-images, each layer of sub-image comprises a plurality of lung lobes, a slit line is formed between every two adjacent lung lobes, and the slit line formed between the same lung lobes in each layer of sub-image forms a lung slit plane;

before acquiring the first lung lobe segmentation image and the second lung lobe segmentation image of the lung during inspiration and/or expiration, the method further comprises:

respectively carrying out lung lobe segmentation on the first lung image and the second lung image to obtain a first lung lobe segmentation image and a second lung lobe segmentation image;

The method for obtaining a first lung lobe segmentation image by lung lobe segmentation of the first lung image is the same as the method for obtaining a second lung lobe segmentation image by lung lobe segmentation of the second lung image, and the steps of respectively performing lung lobe segmentation on the first lung image and the second lung image to obtain the first lung lobe segmentation image and the second lung lobe segmentation image include:

dividing the first lung image and the second lung image by using the corrected lung lobe slit characteristics to obtain a first lung lobe divided image and a second lung lobe divided image;

wherein correcting the third lung lobe slit feature using the lung lobe slit features of any two of the sagittal plane, the coronal plane, and the transverse plane comprises:

Mapping the features of the any two to the view angle at which the third feature of the lung is located;

correcting the third lung lobe slit feature by using the mapped lung lobe slit features of any two;

wherein the method for correcting the third lung lobe slit feature using the mapped lung lobe slit features of any two comprises:

respectively carrying out spatial attention feature fusion by using the mapped lung lobe slit features of any two and the third lung lobe slit features to obtain a first fusion feature and a second fusion feature;

obtaining the corrected third lung lobe slit feature according to the first fusion feature and the second fusion feature;

Respectively connecting the lung lobe slit features of any two with the third lung lobe slit feature to obtain a first connection feature and a second connection feature;

performing a first convolution operation on the first connection feature to obtain a first convolution feature, and performing a first convolution operation on the second connection feature to obtain a second convolution feature;

performing a second convolution operation on the first convolution feature to obtain a first attention coefficient, and performing a second convolution operation on the second convolution feature to obtain a second attention coefficient;

obtaining the first fusion feature using a first convolution feature and a first attention coefficient, and obtaining the second fusion feature using a second convolution feature and a second attention coefficient;

Wherein the method of obtaining the first fusion feature using a first convolution feature and a first attention coefficient and obtaining the second fusion feature using a second convolution feature and a second attention coefficient comprises:

adding the characteristic multiplied by the first attention coefficient by the first convolution characteristic to obtain the first fusion characteristic; adding the characteristic multiplied by the second attention coefficient by the second convolution characteristic to obtain a second fusion characteristic; or alternatively, the first and second heat exchangers may be,

Adding the characteristic multiplied by the first convolution characteristic and the first attention coefficient with the first convolution characteristic, and carrying out a plurality of convolution operations on the added characteristic to obtain the first fusion characteristic; and adding the characteristic multiplied by the second attention coefficient by the second convolution characteristic to the second convolution characteristic, and carrying out a plurality of convolution operations on the added characteristic to obtain the second fusion characteristic.

2. The method of determining according to claim 1, wherein calculating distances between the corresponding first plurality of fracture lines and the corresponding second plurality of fracture lines, respectively, includes: respectively taking a first fracture line and a second fracture line which are in the same layer number;

3. The method of determining of claim 2, wherein calculating the respective distances of the first plurality of fracture points to the second plurality of fracture lines comprises:

taking a plurality of second fracture points on the second fracture line;

4. The method of determining according to claim 1, wherein the first lobe segmentation image is represented in a first mask image form and the second lobe segmentation image is represented in a second mask image form, and wherein the extracting the crack lines of the first lobe segmentation image and the second lobe segmentation image, respectively, results in a number of first crack lines and a number of second crack lines, comprises:

5. The method of determining of claim 1, wherein determining the movement displacement of the plane of the lung fracture based on the plurality of distances comprises:

6. A motion displacement determination device for a plane of a lung fracture, comprising:

an acquisition unit for acquiring a first lung lobe segmentation image and a second lung lobe segmentation image of the lung during inspiration and/or expiration;

The computing unit is used for respectively computing the distances between the corresponding first fracture lines and the corresponding second fracture lines to obtain a plurality of distances, wherein the corresponding first fracture lines and second fracture lines are fracture lines for dividing two adjacent lung lobes under the same layer number; the first lung lobe segmentation image and the second lung lobe segmentation image are lung lobe segmentation results corresponding to a plurality of layers of sub-images, each layer of sub-image comprises a plurality of lung lobes, a slit line is formed between every two adjacent lung lobes, and the slit line formed between the same lung lobes in each layer of sub-image forms a lung slit plane;

the first fracture lines and the second fracture lines are fracture lines for dividing the same two adjacent lung lobes;

The apparatus further comprises: a dividing unit; the segmentation unit is connected with the acquisition unit, and before the acquisition unit acquires a first lung lobe segmentation image and a second lung lobe segmentation image of the lung in the inspiration and/or expiration process, the segmentation unit acquires the first lung image and the second lung image in the inspiration and/or expiration process, and performs lung lobe segmentation on the first lung image and the second lung image respectively to acquire a first lung lobe segmentation image and a second lung lobe segmentation image;

The method for obtaining the first lung lobe segmentation image by the segmentation unit through lung lobe segmentation on the first lung image is the same as the method for obtaining the second lung lobe segmentation image by lung lobe segmentation on the second lung image, and the method comprises the following steps: acquiring lung image features of the lung under sagittal plane, features of the lung under coronal plane and features of the lung under transverse plane; correcting the third lung lobe slit feature by utilizing the lung lobe slit features of any two of the sagittal plane, the coronal plane and the transverse plane; dividing the first lung image and the second lung image by using the corrected lung lobe slit characteristics to obtain a first lung lobe divided image and a second lung lobe divided image;

7. A terminal device, characterized in that the terminal device comprises: a memory, a processor and a program stored on the memory and executable on the processor, the movement displacement determination program of the lung fracture plane, when executed by the processor, implementing the steps of movement displacement determination of the lung fracture plane as claimed in any one of claims 1 to 5.

8. A storage medium having stored thereon a terminal program which when executed by a processor performs the steps of the method for determining the movement displacement of a plane of a lung fracture according to any one of claims 1 to 5.