CN113705807B

CN113705807B - Neural network training device and method, ablation needle arrangement planning device and method

Info

Publication number: CN113705807B
Application number: CN202110988204.0A
Authority: CN
Inventors: 罗中宝; 唐章源; 张朕华
Original assignee: Shanghai Remedicine Co ltd
Current assignee: Shanghai Ruidao Medical Technology Co ltd
Priority date: 2021-08-26
Filing date: 2021-08-26
Publication date: 2022-06-10
Anticipated expiration: 2041-08-26
Also published as: CN113705807A

Abstract

A neural network training device and method, an ablation needle arrangement planning device and method, electronic equipment and a storage medium are provided. The training device of the neural network comprises: an image acquisition module configured to acquire a training image; the reference information acquisition module is configured to acquire reference needle arrangement information corresponding to the training image; the processing module is configured to process the training image by using the neural network to obtain the predicted needle arrangement information corresponding to the training image; and the correction module is configured to calculate a loss value corresponding to the neural network according to the reference needle arrangement information, the target area and the predicted needle arrangement information, and correct parameters of the neural network based on the loss value when the loss value does not meet a preset convergence condition. The training device calculates the loss value based on the reference needle arrangement information, the predicted needle arrangement information and the target area, and the obtained trained neural network can obtain an ablation needle arrangement scheme capable of realizing puncture at any position within a few seconds, so that the execution efficiency is high.

Description

Neural network training device and method, ablation needle arrangement planning device and method

Technical Field

Embodiments of the present disclosure relate to a training device of a neural network, an ablation needle arrangement planning device, a training method of a neural network, an ablation needle arrangement planning method, an electronic device, and a non-transitory computer-readable storage medium.

Background

At present, the medical field can adopt the pulse electric field ablation technology to carry out ablation treatment on a focus area, for example, a group of electrode needles are inserted into the focus area through puncture, high-frequency current is released in the focus area, high temperature is generated in a small range, and water in local tissues is evaporated, dried and necrotized through thermal efficiency to achieve the treatment purpose. When ablation treatment is carried out, if a focus area is large, complete ablation cannot be achieved only by adopting two electrode needles (namely a group of electrode needles), at the moment, a plurality of electrode needles are required to be adopted for combined ablation, namely a plurality of electrode needles are inserted into the focus area, and then ablation is carried out by taking the two electrode needles as a group.

Disclosure of Invention

At least one embodiment of the present disclosure provides a training apparatus for a neural network, the training apparatus including: an image acquisition module configured to acquire a training image, wherein the training image includes a target region; a reference information acquisition module configured to acquire reference needle arrangement information corresponding to the training image, wherein the reference needle arrangement information is used for providing an ablation needle arrangement scheme as a standard; the processing module is configured to process the training image by using the neural network to obtain predicted needle arrangement information corresponding to the training image; and the correction module is configured to calculate a loss value corresponding to the neural network according to the reference needle arrangement information, the target area and the predicted needle arrangement information, and correct parameters of the neural network based on the loss value when the loss value does not meet a preset convergence condition.

For example, in at least one embodiment of the present disclosure, a training device for a neural network is provided, according to claim 1, wherein the modification module includes: the region determining submodule is configured to determine a predicted ablation region according to the predicted needle arrangement information; an evaluation index determination submodule configured to determine an evaluation index according to the predicted ablation region and the target region, wherein the evaluation index is used for representing the scheme effectiveness of the predicted needle arrangement information; the first calculation submodule is configured to calculate a middle loss value according to the reference needle arrangement information and the predicted needle arrangement information; and the second calculation submodule is configured to calculate a loss value corresponding to the neural network according to the intermediate loss value and the evaluation index.

For example, in at least one embodiment of the present disclosure, the evaluation index determination sub-module performs, when determining an evaluation index according to the predicted ablation region and the target region, the following operations: determining ablation accuracy according to the predicted ablation region and the target region, wherein the ablation accuracy is used for representing the ratio of the coincidence region of the predicted ablation region and the target region to the predicted ablation region; determining an ablation recall ratio according to the predicted ablation region and the target region, wherein the ablation recall ratio is used for representing the ratio of the coincidence region of the predicted ablation region and the target region to the target region; and determining the evaluation index according to the ablation accuracy and the ablation recall rate.

For example, in a training apparatus for a neural network provided in at least one embodiment of the present disclosure, the predicted needle arrangement information includes predicted confidence degrees of N candidate needle arrangement points, predicted coordinate information of the N candidate needle arrangement points, and needle group probabilities corresponding to any two candidate needle arrangement points, where N is a positive integer and is greater than or equal to 2, and the determining, by the region determining sub-module, a predicted ablation region according to the predicted needle arrangement information includes: determining an ablation region corresponding to every two different candidate needle laying points in the N candidate needle laying points to obtain a plurality of ablation regions; and superposing the plurality of ablation regions to obtain the predicted ablation region.

For example, in the training apparatus for neural networks, when the region determination sub-module performs the determination of the ablation region corresponding to each of the two different candidate needle placement points, the following operations are performed: for an ith candidate card layout point and a jth candidate card layout point of the N candidate card layout points: calculating effective probability of a needle group corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point, wherein the effective probability of the needle group is used for representing the probability that the ith candidate needle arrangement point and the jth candidate needle arrangement point form an effective ablation needle group; determining an actual coordinate value of the ith candidate card distribution point and an actual coordinate value of the jth candidate card distribution point according to the predicted coordinate information of the ith candidate card distribution point and the predicted coordinate information of the jth candidate card distribution point; determining electric field distribution corresponding to the ith candidate needle laying point and the jth candidate needle laying point according to the actual coordinate value of the ith candidate needle laying point and the actual coordinate value of the jth candidate needle laying point; determining the maximum value of the electric field intensity according to the electric field distribution corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point; determining electric field intensity ablation thresholds corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point according to the needle group effective probability corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point, the electric field intensity maximum value and a preset electric field intensity ablation threshold corresponding to the target area; and determining an ablation region corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point according to the electric field intensity ablation threshold, wherein i and j are positive integers and are less than or equal to N, and i is not equal to j.

For example, in a training apparatus for a neural network, when the region determining sub-module performs the calculation of the effective probability of the needle group corresponding to the ith candidate needle layout point and the jth candidate needle layout point, the method includes the following operations: determining two needle group probabilities corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point according to the predicted needle arrangement information; and calculating the effective probability of the needle group corresponding to the ith candidate needle layout point and the jth candidate needle layout point according to the prediction confidence of the ith candidate needle layout point, the prediction confidence of the jth candidate needle layout point and the maximum value of the two needle group probabilities.

For example, in a training apparatus for a neural network provided in at least one embodiment of the present disclosure, when the region determining sub-module determines an electric field strength ablation threshold corresponding to the ith candidate needle placement point and the jth candidate needle placement point according to a needle group effective probability corresponding to the ith candidate needle placement point and the jth candidate needle placement point, the electric field strength maximum value, and a preset electric field strength ablation threshold corresponding to the target region, the following operations are performed: the electric field intensity ablation threshold value corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point is closer to the preset electric field intensity ablation threshold value in response to the fact that the effective probability of the needle group corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point is larger, and the electric field intensity ablation threshold value corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point is closer to the maximum electric field intensity in response to the fact that the effective probability of the needle group corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point is smaller.

For example, in a training apparatus for a neural network provided in at least one embodiment of the present disclosure, when the region determining sub-module determines an electric field strength ablation threshold corresponding to the ith candidate needle placement point and the jth candidate needle placement point according to a needle group effective probability corresponding to the ith candidate needle placement point and the jth candidate needle placement point, the electric field strength maximum value, and a preset electric field strength ablation threshold corresponding to the target region, the following operations are performed: the calculation formula for determining the electric field ablation threshold corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point is as follows:

Eth′＝p*e_th+(1-p)*e_max

wherein Eth' represents an electric field ablation threshold corresponding to the ith candidate needle laying point and the jth candidate needle laying point, p represents a needle group effective probability corresponding to the ith candidate needle laying point and the jth candidate needle laying point, and e_maxRepresents the maximum value of the electric field intensity, e_thRepresenting the preset electric field intensity ablation threshold.

For example, in at least one embodiment of the present disclosure, in a training apparatus for a neural network, the reference fitting information includes reference confidences of N candidate fitting points, reference coordinate information of the N candidate fitting points and at least one reference fitting combination formed by the N candidate fitting points, the predicted fitting information includes prediction confidences of the N candidate fitting points, predicted coordinate information of the N candidate fitting points, and a pin group probability corresponding to any two candidate fitting points, N is a positive integer and is greater than or equal to 2, and the first calculating sub-module performs the following steps when calculating an intermediate loss value according to the reference fitting information and the predicted fitting information: calculating a pin number loss value based on the reference confidence degrees of the N candidate needle placement points and the prediction confidence degrees of the N candidate needle placement points; calculating a coordinate loss value based on the reference coordinate information of the N candidate needle layout points and the predicted coordinate information of the N candidate needle layout points; calculating a needle group loss value based on the effective probability of the needle groups corresponding to the at least one reference needle arrangement combination and the any two candidate needle arrangement points; and obtaining a middle loss value corresponding to the neural network based on the needle number loss value, the coordinate loss value and the needle group loss value.

For example, in at least one embodiment of the present disclosure, a training apparatus for a neural network is provided, where a trained neural network is obtained when a loss value corresponding to the neural network converges.

For example, at least one embodiment of the present disclosure provides a training apparatus for a neural network, further comprising a testing module configured to perform the following operations: acquiring a plurality of test images, wherein each test image comprises the target area; acquiring a plurality of reference needle arrangement information which corresponds to the plurality of test images one by one; processing the plurality of test images by using the trained neural network respectively to obtain a plurality of pieces of predicted needle arrangement information; determining a prediction accuracy rate based on the plurality of predicted needle arrangement information and the plurality of reference needle arrangement information; and in response to the prediction accuracy being less than a preset accuracy threshold, retraining the neural network.

For example, in a training apparatus for a neural network, the test module, when performing the determination of the prediction accuracy based on the plurality of predicted needle arrangement information and the plurality of reference needle arrangement information, performs the following operations: determining a plurality of candidate first thresholds and a plurality of candidate second thresholds; determining a plurality of candidate threshold combinations based on the plurality of candidate first thresholds and the plurality of candidate second thresholds, wherein each candidate threshold combination comprises one candidate first threshold and one candidate second threshold; processing the plurality of pieces of predicted needle arrangement information based on each candidate threshold combination, and determining a plurality of ablation needle arrangement schemes corresponding to each candidate threshold combination; comparing the plurality of ablation needle distribution schemes corresponding to each candidate threshold combination with a plurality of ablation needle distribution schemes determined based on the plurality of reference needle distribution information, and determining a plurality of accuracy rates corresponding to the plurality of candidate threshold combinations; taking a maximum value of the plurality of accuracy rates as the prediction accuracy rate.

For example, in at least one embodiment of the present disclosure, a training apparatus for a neural network is provided, where the test module is further configured to: determining a first threshold and a second threshold, wherein the first threshold and the second threshold are used for converting the predicted needle arrangement information into an ablation needle arrangement scheme; determining a first threshold and a second threshold, comprising: and taking a candidate first threshold and a candidate second threshold in the candidate threshold combination corresponding to the prediction accuracy as the first threshold and the second threshold.

For example, in a training apparatus for a neural network, the test module, when performing the determination of the prediction accuracy based on the plurality of predicted needle arrangement information and the plurality of reference needle arrangement information, performs the following operations: determining a first threshold and a second threshold; processing the plurality of predicted needle distribution information according to the first threshold and the second threshold to obtain a plurality of predicted needle distribution schemes; comparing the plurality of predicted needle arrangement schemes with a plurality of ablation needle arrangement schemes determined based on the plurality of reference needle arrangement information, respectively, and determining the prediction accuracy.

For example, in at least one embodiment of the present disclosure, a training apparatus of a neural network is provided, where the neural network includes a feature extraction sub-network and a feature fitting sub-network, the feature extraction sub-network is configured to extract image features of the training image, the feature fitting sub-network is configured to process the training image features to obtain predicted pin placement information corresponding to the training image, the feature fitting sub-network includes at least one fully-connected layer, the at least one fully-connected layer is configured to process the image features to obtain a plurality of feature information, and the feature fitting sub-network is further configured to obtain the predicted pin placement information based on the plurality of feature information.

For example, in at least one embodiment of the present disclosure, when the image acquisition module performs acquisition of a training image, the method includes the following steps: acquiring a medical image, wherein the medical image comprises a lesion area to be segmented; performing region segmentation processing on the medical image to obtain the training image, wherein the training image has a target region corresponding to the lesion region in the medical image.

At least one embodiment of the present disclosure further provides an ablation needle planning device, including: an image acquisition module configured to acquire an input image, wherein the input image includes a target region; the processing module is configured to process the input image by using a neural network so as to obtain predicted needle arrangement information corresponding to the input image; a planning module configured to obtain an ablation needle distribution scheme based on the predicted needle distribution information, wherein the neural network is obtained at least in part by training according to the training apparatus of at least one embodiment of the present disclosure.

For example, in at least one embodiment of the present disclosure, an ablation needle arrangement planning apparatus is provided, where the predicted needle arrangement information includes predicted confidence degrees of N candidate needle arrangement points, predicted coordinate information of the N candidate needle arrangement points, and needle group probabilities corresponding to any two candidate needle arrangement points, where N is a positive integer and is greater than or equal to 2, and when the planning module performs obtaining an ablation needle arrangement scheme based on the predicted needle arrangement information, the planning module performs the following steps: acquiring a first threshold value and a second threshold value; taking the candidate card distribution points with the prediction confidence degrees larger than the first threshold value as effective pins to obtain a plurality of effective pins; acquiring size information of an input image, and determining actual position coordinates of the effective needles according to the size information of the input image and the predicted coordinate information of the effective needles; and regarding the needle group with the needle group probability larger than the second threshold value as an effective needle group to obtain at least one effective needle group, wherein the ablation needle arrangement scheme comprises the effective needles, the actual position coordinates of the effective needles and the at least one effective needle group.

At least one embodiment of the present disclosure provides a training method for a neural network, including: acquiring a training image, wherein the training image comprises a target area; acquiring reference needle arrangement information corresponding to the training image, wherein the reference needle arrangement information is used for providing an ablation needle arrangement scheme serving as a standard; processing the training image by using the neural network to obtain predicted needle arrangement information corresponding to the training image; and calculating a loss value corresponding to the neural network according to the reference needle arrangement information, the target area and the predicted needle arrangement information, and correcting parameters of the neural network based on the loss value when the loss value does not meet a preset convergence condition.

For example, in a training method of a neural network provided in at least one embodiment of the present disclosure, calculating a loss value corresponding to the neural network according to the reference needle arrangement information, the target area, and the predicted needle arrangement information, and correcting a parameter of the neural network based on the loss value when the loss value does not satisfy a predetermined convergence condition includes: determining a predicted ablation region according to the predicted needle arrangement information; determining an evaluation index according to the predicted ablation region and the target region, wherein the evaluation index is used for representing the scheme effectiveness of the predicted needle arrangement information; calculating a middle loss value according to the reference needle arrangement information and the predicted needle arrangement information; and calculating the loss value corresponding to the neural network according to the intermediate loss value and the evaluation index.

For example, in a training method of a neural network provided in at least one embodiment of the present disclosure, determining an evaluation index according to the predicted ablation region and the target region includes: determining ablation accuracy according to the predicted ablation region and the target region, wherein the ablation accuracy is used for representing the ratio of the coincidence region of the predicted ablation region and the target region to the predicted ablation region; determining an ablation recall ratio according to the predicted ablation region and the target region, wherein the ablation recall ratio is used for representing the ratio of the coincidence region of the predicted ablation region and the target region to the target region; and determining the evaluation index according to the ablation accuracy and the ablation recall rate.

For example, in a training method of a neural network provided in at least one embodiment of the present disclosure, the predicted needle arrangement information includes prediction confidence degrees of N candidate needle arrangement points, predicted coordinate information of the N candidate needle arrangement points, and needle group probabilities corresponding to any two candidate needle arrangement points, where N is a positive integer and is greater than or equal to 2, and determining a predicted ablation region according to the predicted needle arrangement information includes: determining an ablation region corresponding to every two different candidate needle laying points in the N candidate needle laying points to obtain a plurality of ablation regions; and superposing the plurality of ablation regions to obtain the predicted ablation region.

For example, in a training method of a neural network provided by at least one embodiment of the present disclosure, determining an ablation region corresponding to each two different candidate needle placement points includes: for an ith candidate card layout point and a jth candidate card layout point of the N candidate card layout points: calculating effective probability of a needle group corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point, wherein the effective probability of the needle group is used for representing the probability that the ith candidate needle arrangement point and the jth candidate needle arrangement point form an effective ablation needle group; determining an actual coordinate value of the ith candidate needle distribution point and an actual coordinate value of the jth candidate needle distribution point according to the predicted coordinate information of the ith candidate needle distribution point and the predicted coordinate information of the jth candidate needle distribution point; determining electric field distribution corresponding to the ith candidate needle laying point and the jth candidate needle laying point according to the actual coordinate value of the ith candidate needle laying point and the actual coordinate value of the jth candidate needle laying point; determining the maximum value of the electric field intensity according to the electric field distribution corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point; determining electric field intensity ablation thresholds corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point according to the needle group effective probability corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point, the electric field intensity maximum value and a preset electric field intensity ablation threshold corresponding to the target area; and determining an ablation region corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point according to the electric field intensity ablation threshold, wherein i and j are positive integers and are less than or equal to N, and i is not equal to j.

For example, in a training method of a neural network provided in at least one embodiment of the present disclosure, calculating effective probabilities of needle groups corresponding to the ith candidate needle placement point and the jth candidate needle placement point includes: determining two needle group probabilities corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point according to the predicted needle arrangement information; and calculating the effective probability of the needle group corresponding to the ith candidate needle layout point and the jth candidate needle layout point according to the prediction confidence of the ith candidate needle layout point, the prediction confidence of the jth candidate needle layout point and the maximum value of the two needle group probabilities.

For example, in a training method of a neural network provided in at least one embodiment of the present disclosure, determining an electric field strength ablation threshold corresponding to an ith candidate needle placement point and a jth candidate needle placement point according to a needle group effective probability corresponding to the ith candidate needle placement point and the jth candidate needle placement point, the electric field strength maximum value, and a preset electric field strength ablation threshold corresponding to the target area includes: the electric field intensity ablation threshold value corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point is closer to the preset electric field intensity ablation threshold value in response to the fact that the effective probability of the needle group corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point is larger, and the electric field intensity ablation threshold value corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point is closer to the maximum electric field intensity in response to the fact that the effective probability of the needle group corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point is smaller.

For example, in a training method of a neural network provided in at least one embodiment of the present disclosure, determining an electric field strength ablation threshold corresponding to an ith candidate needle placement point and a jth candidate needle placement point according to a needle group effective probability corresponding to the ith candidate needle placement point and the jth candidate needle placement point, the electric field strength maximum value, and a preset electric field strength ablation threshold corresponding to the target area includes: the calculation formula for determining the electric field ablation threshold corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point is as follows:

Eth′＝p*e_th+(1-p)*e_max

For example, in a training method of a neural network provided in at least one embodiment of the present disclosure, the reference needle arrangement information includes reference confidence degrees of N candidate needle arrangement points, reference coordinate information of the N candidate needle arrangement points, and at least one reference needle arrangement combination formed by the N candidate needle arrangement points, the predicted needle arrangement information includes prediction confidence degrees of the N candidate needle arrangement points, predicted coordinate information of the N candidate needle arrangement points, and needle group probabilities corresponding to any two candidate needle arrangement points, where N is a positive integer and is greater than or equal to 2, and a median loss value is calculated according to the reference needle arrangement information and the predicted needle arrangement information, including: calculating a pin number loss value based on the reference confidence levels of the N candidate card layout points and the prediction confidence levels of the N candidate card layout points; calculating a coordinate loss value based on the reference coordinate information of the N candidate needle layout points and the predicted coordinate information of the N candidate needle layout points; calculating a needle group loss value based on the effective probability of the needle groups corresponding to the at least one reference needle arrangement combination and the any two candidate needle arrangement points; and obtaining an intermediate loss value corresponding to the neural network based on the needle number loss value, the coordinate loss value and the needle group loss value.

For example, in a training method for a neural network provided in at least one embodiment of the present disclosure, when a loss value corresponding to the neural network converges, the trained neural network is obtained.

For example, at least one embodiment of the present disclosure provides a training method of a neural network, further including: acquiring a plurality of test images, wherein each test image comprises the target area; acquiring a plurality of reference needle arrangement information which corresponds to the plurality of test images one by one; processing the plurality of test images by using the trained neural network respectively to obtain a plurality of pieces of predicted needle arrangement information; determining a prediction accuracy rate based on the plurality of predicted needle arrangement information and the plurality of reference needle arrangement information; and in response to the prediction accuracy being less than a preset accuracy threshold, retraining the neural network.

For example, in a training method of a neural network provided in at least one embodiment of the present disclosure, determining a prediction accuracy based on the plurality of pieces of predicted needle arrangement information and the plurality of pieces of reference needle arrangement information includes: determining a plurality of candidate first thresholds and a plurality of candidate second thresholds; determining a plurality of candidate threshold combinations based on the plurality of candidate first thresholds and the plurality of candidate second thresholds, wherein each candidate threshold combination comprises one candidate first threshold and one candidate second threshold; processing the plurality of pieces of predicted needle arrangement information based on each candidate threshold combination, and determining a plurality of ablation needle arrangement schemes corresponding to each candidate threshold combination; comparing the plurality of ablation needle distribution schemes corresponding to each candidate threshold combination with a plurality of ablation needle distribution schemes determined based on the plurality of reference needle distribution information, and determining a plurality of accuracy rates corresponding to the plurality of candidate threshold combinations respectively; taking a maximum value of the plurality of accuracy rates as the prediction accuracy rate.

For example, at least one embodiment of the present disclosure provides a training method of a neural network, further including: determining a first threshold and a second threshold, wherein the first threshold and the second threshold are used for converting the predicted needle arrangement information into an ablation needle arrangement scheme; determining a first threshold and a second threshold, comprising: and taking a candidate first threshold and a candidate second threshold in the candidate threshold combination corresponding to the prediction accuracy as the first threshold and the second threshold.

For example, in a training method of a neural network provided in at least one embodiment of the present disclosure, determining a prediction accuracy based on the plurality of pieces of predicted needle distribution information and the plurality of pieces of reference needle distribution information includes: determining a first threshold and a second threshold; processing the plurality of predicted needle distribution information according to the first threshold and the second threshold to obtain a plurality of predicted needle distribution schemes; comparing the plurality of predicted needle arrangement schemes with a plurality of ablation needle arrangement schemes determined based on the plurality of reference needle arrangement information, respectively, and determining the prediction accuracy.

For example, in a training method of a neural network provided in at least one embodiment of the present disclosure, the neural network includes a feature extraction sub-network and a feature fitting sub-network, the feature extraction sub-network is configured to extract image features of the training image, the feature fitting sub-network is configured to process the training image features to obtain predicted pin placement information corresponding to the training image, the feature fitting sub-network includes at least one fully-connected layer, the at least one fully-connected layer is configured to process the image features to obtain a plurality of feature information, and the feature fitting sub-network is further configured to obtain the predicted pin placement information based on the plurality of feature information.

For example, in a training method of a neural network provided in at least one embodiment of the present disclosure, acquiring a training image includes: acquiring a medical image, wherein the medical image comprises a lesion area to be segmented; performing region segmentation processing on the medical image to obtain the training image, wherein the training image has a target region corresponding to the lesion region in the medical image.

At least one embodiment of the present disclosure further provides an ablation needle planning method, including: acquiring an input image, wherein the input image comprises a target area; processing the input image by using a neural network to obtain predicted needle arrangement information corresponding to the input image; obtaining an ablation needle arrangement scheme based on the predicted needle arrangement information; wherein the neural network is trained, at least in part, by the training apparatus according to at least one embodiment of the present disclosure.

For example, in an ablation needle arrangement planning method provided in at least one embodiment of the present disclosure, the predicted needle arrangement information includes predicted confidence degrees of N candidate needle arrangement points, predicted coordinate information of the N candidate needle arrangement points, and needle group probabilities corresponding to any two candidate needle arrangement points, where N is a positive integer and is greater than or equal to 2, and an ablation needle arrangement scheme is obtained based on the predicted needle arrangement information, including: acquiring a first threshold value and a second threshold value; taking the candidate card distribution points with the prediction confidence degrees larger than the first threshold value as effective pins to obtain a plurality of effective pins; acquiring size information of an input image, and determining actual position coordinates of the effective needles according to the size information of the input image and the predicted coordinate information of the effective needles; and regarding the needle group with the needle group probability larger than the second threshold value as an effective needle group to obtain at least one effective needle group, wherein the ablation needle distribution scheme comprises the effective needles, the actual position coordinates of the effective needles and the at least one effective needle group.

At least one embodiment of the present disclosure further provides an electronic device, including: a memory non-transiently storing computer executable instructions; a processor configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement a neural network training method according to at least one embodiment of the present disclosure or an ablation needle planning method according to at least one embodiment of the present disclosure.

At least one embodiment of the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, which when executed by a processor, implement a neural network training method according to at least one embodiment of the present disclosure or an ablation needle placement planning method according to at least one embodiment of the present disclosure.

Drawings

To more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below, and it is apparent that the drawings in the following description relate only to some embodiments of the present disclosure and are not limiting to the present disclosure.

Fig. 1 is a schematic structural diagram of a training apparatus for a neural network according to at least one embodiment of the present disclosure;

fig. 2 is a schematic diagram of a training image provided in at least one embodiment of the present disclosure;

FIG. 3 is a block diagram of a neural network provided in at least one embodiment of the present disclosure;

FIG. 4 is a schematic illustration of an electric field distribution provided by at least one embodiment of the present disclosure;

fig. 5 is a schematic diagram of a training process of a neural network according to at least one embodiment of the present disclosure;

fig. 6 is a flow chart of a training process of a neural network according to at least one embodiment of the present disclosure;

fig. 7 is a schematic block diagram of an ablation needle placement planning apparatus provided in at least one embodiment of the present disclosure;

fig. 8A is a schematic flow chart of a training method of a neural network provided in at least one embodiment of the present disclosure;

FIG. 8B is a schematic flow chart of step S304 in the training method of the neural network shown in FIG. 8A;

fig. 8C is a schematic flow chart of a training method of a neural network provided in at least one embodiment of the present disclosure;

fig. 9 is a schematic flow chart of an ablation needle planning method provided in at least one embodiment of the present disclosure;

fig. 10 is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure;

fig. 11 is a schematic diagram of a non-transitory computer-readable storage medium according to at least one embodiment of the disclosure.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be described below clearly and completely with reference to the accompanying drawings of the embodiments of the present disclosure. It is to be understood that the described embodiments are only a few embodiments of the present disclosure, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the described embodiments of the disclosure without any inventive step, are within the scope of protection of the disclosure.

Unless otherwise defined, technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this disclosure belongs. The use of "first," "second," and similar terms in this disclosure is not intended to indicate any order, quantity, or importance, but rather is used to distinguish one element from another. The word "comprising" or "comprises", and the like, means that the element or item listed before the word covers the element or item listed after the word and its equivalents, but does not exclude other elements or items. The terms "connected" or "coupled" and the like are not restricted to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "upper", "lower", "left", "right", and the like are used merely to indicate relative positional relationships, and when the absolute position of the object being described is changed, the relative positional relationships may also be changed accordingly. To maintain the following description of the embodiments of the present disclosure clear and concise, a detailed description of some known functions and components may be omitted from the present disclosure.

The mainstream ablation needle distribution method at present adopts a positioning template, and the minimum distance of puncture holes of the positioning template is 5mm, so that the design restricts that a doctor can only insert a needle in a limited puncture hole. Although this approach simplifies the needle placement process, it also results in a needle ablation that does not achieve precise needle ablation.

In order to realize precise puncture ablation, an ablation puncture mode without a puncture template or a puncture template without puncture holes is needed to realize puncture at any position. The problem with this method of puncture is that it is difficult for the physician to provide a needle-laying plan with optimal ablation, and planning a needle-laying plan is extremely time-consuming. For example, some current automatic needle arrangement schemes require traversal of all pixel points in a medical image with a lesion area to obtain a needle arrangement scheme with an optimal ablation effect, which is performed with low efficiency.

At least one embodiment of the present disclosure provides a training apparatus of a neural network, an ablation needle arrangement planning apparatus, a training method of a neural network, an ablation needle arrangement planning method, an electronic device, and a non-transitory computer-readable storage medium, the training apparatus of a neural network including: an image acquisition module configured to acquire a training image, wherein the training image includes a target region; the reference information acquisition module is configured to acquire reference needle arrangement information corresponding to the training image, wherein the reference needle arrangement information is used for providing an ablation needle arrangement scheme serving as a standard; the processing module is configured to process the training image by using the neural network to obtain the predicted needle arrangement information corresponding to the training image; and the correction module is configured to calculate a loss value corresponding to the neural network according to the reference needle arrangement information, the target area and the predicted needle arrangement information, and correct parameters of the neural network based on the loss value when the loss value does not meet a preset convergence condition.

The training device for the neural network provided by at least one embodiment of the present disclosure calculates a loss value based on reference needle arrangement information, predicted needle arrangement information, and a target region (e.g., a lesion region) to correct parameters of the neural network, and the obtained trained neural network can obtain an ablation needle arrangement scheme capable of achieving puncture at any position within several seconds, so that the execution efficiency is high.

The training method of the neural network provided by the embodiment of the disclosure can be applied to the training device of the neural network provided by the embodiment of the disclosure, and the ablation needle arrangement planning method provided by the embodiment of the disclosure can be applied to the ablation needle arrangement planning device provided by the embodiment of the disclosure. The electronic device may be a personal computer, a mobile terminal, and the like, and the mobile terminal may be a hardware device such as a mobile phone and a tablet computer.

Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, but the present disclosure is not limited to these specific embodiments.

Fig. 1 is a schematic structural diagram of a training apparatus for a neural network according to at least one embodiment of the present disclosure.

As shown in fig. 1, the training apparatus 10 for neural networks provided in at least one embodiment of the present disclosure includes an image acquisition module 101, a reference information acquisition module 102, a processing module 103, and a modification module 104, each of which may be implemented by software, hardware, firmware, or any combination thereof, and may communicate, such as transmit data and/or instructions, etc., as required during operation, for example, they may be interconnected by a bus system and/or other form of connection mechanism (not shown) at a hardware level, and may be interconnected by a communication process or thread at a software level. It should be noted that the components and configuration of the exercise device 10 shown in FIG. 1 are exemplary only, and not limiting, and that the exercise device 10 may have other components and configurations as desired.

For example, the image acquisition module 101 is configured to acquire a training image, e.g., the training image includes a target region.

For example, the reference information acquiring module 102 is configured to acquire reference needle arrangement information corresponding to the training image, for example, the reference needle arrangement information is used to provide an ablation needle arrangement scheme as a standard.

For example, the processing module 103 is configured to process the training image by using a neural network, and obtain the predicted needle arrangement information corresponding to the training image.

For example, the modification module 104 is configured to calculate a loss value corresponding to the neural network according to the reference needle arrangement information, the target area and the predicted needle arrangement information, and modify a parameter of the neural network based on the loss value when the loss value does not satisfy a predetermined convergence condition.

For example, when the image obtaining module 101 performs obtaining the training image, it includes the following steps: acquiring a medical image, for example, the medical image including a lesion region to be segmented; a region segmentation process is performed on the medical image to obtain a training image, for example, a training image having a target region corresponding to a lesion region in the medical image.

For example, the lesion region is a region that needs medical judgment and is obtained by a doctor based on a diagnostic image and diagnostic data of a patient, and a diagnostic result of the patient can be obtained by analyzing the lesion region. For example, the lesion region may be acquired by detection means such as ultrasound, magnetic resonance, needle biopsy, and the like. For example, the lesion area is determined by combining the magnetic resonance data with the results of the needle biopsy by ultrasound imaging.

For example, when the medical image is an endoscopic image, the focal region may be a polyp region.

For example, the imaging modality of the medical image may be any medical imaging modality. For example, the Imaging mode of the medical image may be NBI (Narrow Band Imaging), white light, CT (Computed Tomography), MRI (Magnetic Resonance Imaging), and the like.

For example, the medical image may be an image obtained by performing medical examination on an arbitrary examination region by using an arbitrary medical examination means, the medical image requires a real pathology image of a patient, and the images may be preprocessed with respect to size, resolution, gray scale, and the like. For example, the medical detection means may include endoscopy, angiography, ultrasound, and the like. For example, the detection region may include a human body region such as colon, stomach, lung, heart, liver, prostate, etc.

For example, the medical image may be an image taken from a medical inspection video, e.g., the medical inspection video may be an endoscopic video, a cardiac ultrasound video, or the like.

For example, the medical image may be obtained by scanning, shooting, or the like the medical detection image by using the image acquisition device, and the training image may be an image directly acquired by the image acquisition device, or an image obtained by preprocessing the acquired image.

For example, medical images need to meet acquisition area requirements and quality requirements.

For example, the clarity of medical images should be sufficient to ensure that a clear lesion area is obtained, thereby facilitating accurate diagnosis results for the physician.

For example, the medical image may be subjected to a region segmentation process using a region segmentation model or the like to obtain a training image. For example, the region segmentation model may be a neural network model of any structure capable of implementing the region segmentation function, such as a U-NET network model, and the like, which is not limited by the present disclosure.

Fig. 2 is a schematic diagram of a training image according to at least one embodiment of the present disclosure.

For example, in the training image of fig. 2, a white portion represents a target region, i.e., a lesion region in the medical image; the black parts represent the background region, i.e. the part of the training image other than the target region, i.e. normal tissue.

For example, the reference information obtaining module 102 may obtain an ablation needle arrangement scheme with the best ablation effect by using an exhaustion method, and use the ablation needle arrangement scheme as reference needle arrangement information; for example, the reference information acquiring module 102 may also determine the reference needle arrangement information based on the needle arrangement combination determined in patent ZL202011345756.1, and of course, the reference information acquiring module 102 may also obtain the reference needle arrangement information by using any other method capable of finding an ablation needle arrangement scheme with the best ablation effect, which is not limited in this disclosure.

For example, before training the neural network, a maximum number N of electrode needles that can be used in the ablation needle arrangement scheme is predefined, for example, in some embodiments, N is 7, that is, the ablation needle arrangement scheme can use 7 electrode needles at most. For example, the reference needle arrangement information and the predicted needle arrangement information are obtained on the basis of the maximum number N of electrode needles defined in advance. Of course, in different embodiments, the maximum number N of electrode needles that can be used may also be selected as needed, for example, N may also be 5, 6, 8, etc., which is not limited by the present disclosure.

For example, the reference needling information may include reference confidences of N candidate needling points, reference coordinate information of the N candidate needling points, and at least one reference needling combination composed of the N candidate needling points. The N candidate needle distribution points are the N electrode needles that can be used at most in the aforementioned predefined ablation needle distribution scheme.

For example, the reference confidence of the candidate needle layout point indicates whether the candidate needle layout point is a valid needle, and since the reference needle layout information is a predetermined ablation needle layout scheme with the best ablation effect, the reference confidence of the candidate needle layout point may be 1 or 0, where 1 indicates that the candidate needle layout point is a valid needle, and 0 indicates that the candidate needle layout point is not a valid needle.

For example, the reference coordinate information of the candidate needle disposing point includes a reference abscissa value and a reference ordinate value of the candidate needle disposing point, and the reference abscissa value and the reference ordinate value are coordinate values obtained by mapping the lesion area into the reference coordinate system in which the needle disposing point is labeled, and the reference coordinate information is 0 if the candidate needle disposing point is not a valid needle.

For example, each reference needle placement combination includes two different candidate needle placement points that may constitute an active needle set for achieving ablation. According to the actual situation, the reference cloth needle combination can be in various combinations.

For example, the reference needle distribution information may be encoded in a matrix form of N × N +3, so as to be stored and operated. For example, a row of elements in the standard matrix M1 obtained by encoding the reference card distribution information represents the relevant information of a candidate card distribution point, for example, the relevant information includes the reference confidence of the candidate card distribution point, the abscissa and ordinate scale values and N reference card group probabilities obtained based on the reference coordinate information of the candidate card distribution point, where the N reference card group probabilities represent the reference card group probabilities respectively corresponding to the candidate card distribution point and the N candidate card distribution points.

For example, if the candidate needle layout point is not a valid needle in the ablation needle layout scheme as a reference, all relevant information thereof is filled with 0 or empty.

For example, the abscissa scale value of the candidate card-laying point is obtained by dividing the reference abscissa value of the candidate card-laying point by the transverse dimension of the training image, and the ordinate scale value is obtained by dividing the reference ordinate value of the candidate card-laying point by the longitudinal dimension of the training image. For example, the training image has a size H × W, H denotes a lateral size, W denotes a longitudinal size, and H and W are both in pixels.

For example, the reference needle arrangement probability is used to indicate whether two candidate needle arrangement points constitute an ablation needle group, for example, according to the reference needle arrangement information, if two candidate needle arrangement points constitute the ablation needle group, the reference needle group probability corresponding to the two candidate needle arrangement points is 1, and if two candidate needle arrangement points do not constitute the ablation needle group, the corresponding reference needle group probability is 0. Since the candidate needle layout points and the candidate needle layout points themselves cannot constitute an ablation needle group, the probability of the corresponding reference needle group in the matrix is 0.

For the content of the information related to the standard matrix M1 and the candidate needle layout points, reference may be made to the following specific descriptions in tables 2 and 3, which are not repeated herein.

At least one embodiment of the present disclosure provides a neural network that is a convolutional neural network to fit predicted needle placement information.

For example, the neural network comprises a feature extraction sub-network and a feature fitting sub-network, wherein the feature extraction sub-network is used for extracting image features of a training image, and the feature fitting sub-network is used for processing the image features to obtain predicted needle arrangement information corresponding to the training image.

For example, the feature extraction sub-network may adopt a network structure such as U-NET (U-type network), ResNet (residual error network), VGG (Visual Geometry Group), and the like, which is not limited by the present disclosure.

For example, the feature fitting sub-network comprises at least one fully connected layer configured to process the image feature to obtain a plurality of feature information, and the feature fitting sub-network is further configured to obtain the predicted needle placement information based on the plurality of feature information.

For example, deriving the predicted needle placement information based on the plurality of feature information may include: the plurality of feature information are arranged in the output order and are equally distributed into N parts, which are respectively used as the relevant prediction information of the N candidate needle distribution points, for example, the predicted needle distribution information includes the relevant prediction information of the N candidate needle distribution points.

For example, the feature fitting sub-network processes the obtained image features into a plurality of feature information layer by layer through a plurality of fully connected layers, so that the problem of feature loss caused by direct processing of one fully connected layer into a plurality of feature information is avoided.

Fig. 3 is a block diagram of an exemplary neural network provided in at least one embodiment of the present disclosure.

As shown in fig. 3, the neural network 200 includes a feature extraction subnetwork 201 and a feature fitting subnetwork 202.

For example, feature extraction subnetwork 201 includes 5 convolutional layers, convolutional layers CONV1 through CONV5, respectively, and 5 pooling layers, pooling layer PL1 through pooling layer PL5, respectively, interleaved between adjacent convolutional layers.

For example, feature fitting subnetwork 202 includes 3 fully connected layers, fully connected layer CL1 through fully connected layer CL3, respectively.

For example, the convolution kernels of convolutional layers CONV1 to CONV5 are all 3 × 3 in size, and the pooling layers PL1 to PL5 are all maximum pooling.

For example, the maximum number of electrode needles that can be used in the predefined ablation needle arrangement is 7, that is, N is 7, and the size of the training image is 224 × 224, in this case, the specific structure of the feature extraction sub-network and the feature fitting sub-network may refer to the network structure described below.

For example, the training image is a binarized image obtained by dividing the region shown in fig. 2, the size of the training image is 224 × 224, and the number of input channels of the convolutional layer CONV1 is 3.

In one example, for example, the number of output channels of the convolutional layer CONV1 is 64, which means that after the convolutional layer is convolved with training images input by using 64 convolution kernels, the convolutional layer CONV1 outputs 64 feature maps with the size of 224 × 224, that is, the convolutional layer CONV1 outputs 64 feature maps with the size of 224 × 224 to the pooling layer PL 1; the pooling layer PL1 performs maximum pooling on the 64 feature maps and outputs 64 feature maps with the size of 112 × 112 to the convolutional layer CONV 2; the number of output channels of the convolutional layer CONV2 is 128, which indicates that the convolutional layer CONV2 outputs 128 feature maps with the size of 112 × 112 after convolution by using 128 convolutional cores to perform convolution on input feature maps, that is, the convolutional layer CONV2 outputs 128 feature maps with the size of 112 × 112 to the pooling layer PL2, and the pooling layer PL2 performs maximum pooling on the 128 feature maps and outputs 128 feature maps with the size of 56 × 56 to the convolutional layer CONV 3; the number of output channels of the convolutional layer CONV3 is 256, which indicates that the convolutional layer CONV3 outputs 256 feature maps with the size of 56 × 56 after convolution by using 256 convolutional cores to perform convolution on input feature maps, that is, the convolutional layer CONV3 outputs 256 feature maps with the size of 56 × 56 to the pooling layer PL3, and the pooling layer PL3 performs maximum pooling on the 256 feature maps and outputs 256 feature maps with the size of 28 × 28 to the convolutional layer CONV 4; the number of output channels of the convolutional layer CONV4 is 512, which indicates that the convolutional layer CONV4 outputs 512 feature maps with the size of 28 × 28 after convolution by using 512 convolutional cores to input feature maps, that is, the convolutional layer CONV4 outputs 512 feature maps with the size of 28 × 28 to the pooling layer PL4, and the pooling layer PL4 performs maximum pooling on the 512 feature maps and outputs 512 feature maps with the size of 14 × 14 to the convolutional layer CONV 5; the number of output channels of the convolutional layer CONV5 is 512, and this shows that the convolutional layer CONV4 outputs 512 feature maps with the size of 14 × 14 after convolution by using 512 convolutional cores to input feature maps, that is, 512 feature maps with the size of 14 × 14 are output to the pooling layer PL5 by the convolutional layer CONV5, and 512 feature maps with the size of 7 × 7 are output after maximum pooling processing is performed on the 512 feature maps by the pooling layer PL 5.

For example, the full-link layer CL1 receives the image features output by the pooling layer PL5, that is, 512 feature maps with the size of 7 × 7, and then the full-link layer CL1 processes the image features by using a parameter matrix with (7 × 7 × 512) rows and 4096 columns, and outputs 4096 pieces of feature information to the full-link layer CL 2; the full link layer CL2 processes the 4096 pieces of feature information by using a parameter matrix of 4096 rows and 1024 columns, and outputs 1024 pieces of feature information to the full link layer CL 3; the full connection layer CL3 processes the 1024 pieces of characteristic information by using 1024 rows and 70 columns of parameter matrixes, and outputs 70 pieces of characteristic information; and then, arranging the 70 pieces of characteristic information according to an output sequence, averagely distributing the characteristic information into 7 pieces of characteristic information, and respectively using the characteristic information as the relevant prediction information of 7 candidate needle distribution points, wherein the predicted needle distribution information comprises the relevant prediction information of the 7 candidate needle distribution points.

For example, the relevant prediction information of each candidate card distribution point includes the prediction confidence of the candidate card distribution point, the prediction coordinate information of the candidate card distribution point, and N card group probabilities, where the N card group probabilities include probabilities that the candidate card distribution point and the N candidate card distribution points respectively form an ablation card group, and the N card group probabilities also include card group probabilities that the candidate card distribution point and the candidate card distribution point themselves correspond to.

For example, the prediction confidence of a candidate card layout point is used to indicate whether the candidate card layout point is a valid card, and the larger the prediction confidence of the candidate card layout point is, the more likely the candidate card layout point belongs to the valid card, whereas the smaller the prediction confidence of the candidate card layout point is, the less likely the candidate card layout point belongs to the valid card, that is, the invalid card.

For example, the predicted coordinate information of the candidate needle layout point includes a predicted abscissa and a predicted ordinate, and the predicted abscissa may be obtained by multiplying the predicted abscissa by the lateral size of the training image, and the predicted ordinate may be obtained by multiplying the predicted ordinate by the longitudinal size of the training image.

For example, the probability of the needle group corresponding to any two candidate needle arrangement points represents the probability that the two candidate needle arrangement points constitute one ablation needle group, the higher the probability of the needle group corresponding to any two candidate needle arrangement points is, the more likely the two candidate needle arrangement points constitute the ablation needle group, and conversely, the lower the probability of the needle group is, the less likely the two candidate needle arrangement points constitute the ablation needle group. For example, the predicted needle arrangement information includes N × N needle group probabilities in common, for example, the needle group probabilities corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point may be different from the needle group probabilities corresponding to the jth candidate needle arrangement point and the ith candidate needle arrangement point, and i and j are both positive integers and are less than or equal to N.

For example, the predicted pin distribution information may be encoded in a matrix form of N × N +3 for storage and operation.

For example, the prediction matrix M2 obtained by encoding the predicted needle placement information is shown as an N × (N +3) region defined by a bold line frame in the following table 1.

TABLE 1

Needle numbering	Confidence of prediction	Predicted abscissa	Predicted ordinate	1	2	3	...	N-2	N-1	N
											1
2
											3
...
											N-2
N-1
											N

For example, a row of elements of the matrix represents the relevant prediction information of one candidate card distribution point, for example, the relevant prediction information includes the prediction confidence of the candidate card distribution point, the prediction abscissa and the prediction ordinate obtained based on the prediction coordinate information of the candidate card distribution point, and the card group probabilities respectively corresponding to the candidate card distribution point and the N candidate card distribution points.

For example, the prediction confidence, the prediction coordinate information, and the pin group probability may all be floating point numbers between 0 and 1.

For example, after obtaining the predicted needle arrangement information, the modification module 104 calculates a loss value corresponding to the neural network according to the reference needle arrangement information, the target area and the predicted needle arrangement information, so as to modify parameters of the neural network.

For example, the modification module 104 includes a region determination sub-module, an evaluation index determination sub-module, a first calculation sub-module, and a second calculation sub-module.

For example, the region determination sub-module is configured to determine a predicted ablation region based on the predicted needle placement information.

For example, the evaluation index determination sub-module is configured to determine an evaluation index according to the predicted ablation region and the target region, wherein the evaluation index is used for representing the scheme effectiveness of the predicted needle arrangement information.

For example, the first calculation sub-module is configured to calculate the intermediate loss value based on the reference needle arrangement information and the predicted needle arrangement information.

For example, the second calculating submodule is configured to calculate a loss value corresponding to the neural network according to the intermediate loss value and the evaluation index.

For example, the region determining sub-module performs the following operations when determining the predicted ablation region according to the predicted needle arrangement information: determining ablation areas corresponding to every two different candidate needle distribution points in the N candidate needle distribution points to obtain a plurality of ablation areas; and superposing the plurality of ablation regions to obtain a predicted ablation region. For example, here ablation region overlap is to obtain a union of multiple ablation regions.

For example, when the region determination sub-module performs the determination of the ablation regions corresponding to each two different candidate needle placement points, the following operations are performed: for an ith candidate card layout point and a jth candidate card layout point of the N candidate card layout points: calculating the effective probability of a needle group corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point, wherein the effective probability of the needle group is used for representing the probability that the ith candidate needle arrangement point and the jth candidate needle arrangement point form an effective ablation needle group; determining the actual coordinate value of the ith candidate card distribution point and the actual coordinate value of the jth candidate card distribution point according to the predicted coordinate information of the ith candidate card distribution point and the predicted coordinate information of the jth candidate card distribution point; determining electric field distribution corresponding to the ith candidate card distribution point and the jth candidate card distribution point according to the actual coordinate value of the ith candidate card distribution point and the actual coordinate value of the jth candidate card distribution point; determining the maximum value of the electric field intensity according to the electric field distribution corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point; determining an electric field intensity ablation threshold corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point according to the effective probability of the needle group corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point, the maximum electric field intensity value and a preset electric field intensity ablation threshold corresponding to the target area; and determining an ablation region corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point according to the electric field intensity ablation threshold, wherein i and j are positive integers and are less than or equal to N, and i is not equal to j.

For example, when the region determining sub-module performs the calculation of the effective probability of the needle group corresponding to the ith candidate needle layout point and the jth candidate needle layout point, the following operations are performed: determining two needle group probabilities corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point according to the predicted needle arrangement information; and calculating the effective probability of the needle group corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point according to the prediction confidence of the ith candidate needle distribution point, the prediction confidence of the jth candidate needle distribution point and the maximum value of the two needle group probabilities.

For example, the calculation formula of the effective probability of the needle group is shown in formula 1:

p＝c_i*c_j*max(g_ij,g_ji) (formula 1)

Wherein p represents the effective probability of the needle group corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point, c_iRepresenting the prediction confidence of the ith candidate card layout point, c_jRepresents the prediction confidence of the jth candidate card layout point, max (x) represents the maximum function, g_ijRepresenting the probability of a group of pins corresponding to the ith candidate pin layout point and the jth candidate pin layout point, g_jiAnd representing the probability of the needle group corresponding to the jth candidate needle layout point and the ith candidate needle layout point.

For example, in the prediction matrix M2 shown in Table 1, g_ijThe values of the elements 1, g representing the ith row, the jth +3 column in the prediction matrix M2_jiThe element values 2, c representing the j-th row and i + 3-th column of the prediction matrix M2_iThe element values 3, c representing the 1 st column of the ith row of the prediction matrix M2_jRepresenting the element value 4 in row j and column 1 of the prediction matrix M2, the effective probability of the needle group corresponding to the ith candidate card distribution point and the jth candidate card distribution point is the product of the element value 1 and the maximum value of the element value 2 multiplied by the element value 3 and the element value 4.

And then, when the actual coordinate value of the ith candidate card distribution point and the actual coordinate value of the jth candidate card distribution point are determined according to the predicted coordinate information of the ith candidate card distribution point and the predicted coordinate information of the jth candidate card distribution point, multiplying the predicted abscissa in the predicted coordinate information of the ith candidate card distribution point by the transverse dimension of the training image to obtain the actual abscissa of the ith candidate card distribution point, and multiplying the predicted ordinate in the predicted coordinate information of the ith candidate card distribution point by the longitudinal dimension of the training image to obtain the actual ordinate of the ith candidate card distribution point. Similarly, the actual coordinate value of the jth candidate needle distribution point is calculated, which is not described herein again.

For example, after the actual coordinate value of the ith candidate needle laying point and the actual coordinate value of the jth candidate needle laying point are obtained, based on the actual coordinate value of the ith candidate needle laying point and the actual coordinate value of the jth candidate needle laying point, the electric pulse ablation region corresponding to the ith candidate needle laying point and the jth candidate needle laying point is calculated or predicted in any feasible manner, and the electric field distribution corresponding to the ith candidate needle laying point and the jth candidate needle laying point is determined.

Fig. 4 is a schematic diagram of an electric field distribution provided by at least one embodiment of the present disclosure.

As shown in fig. 4, point a indicates a position where the ith needle dispensing candidate is located, point B indicates a position where the jth needle dispensing candidate is located, e1, e2, e3 and e4 are electric field strength values corresponding to four electric field contour lines in an electric field formed by applying a certain voltage difference between a and B through the electrode needles, respectively, and e1< e2< e3< e 4.

For example, the contour corresponding to the electric field intensity value e4 is a circular contour of the needle circumference close to the candidate needle layout point, and the electric field intensity value e4 is also the maximum value e of the electric field intensity in the electric field distribution_max. For example, the maximum value e of the electric field intensity can be determined according to the electric field distribution corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point_max。

For example, the electric field intensity value e1 corresponds to the contour line as the preset ablation threshold e_thField intensity contour line of (a), preset ablation threshold e_thA minimum value representing the electric field strength applied to the biological tissue to cause cell death of the biological tissue, for example, a preset ablation threshold is simulated based on the biological tissue to be ablated, and different biological tissues correspond to different preset ablation thresholds, for example, when the liver is ablated, the preset ablation threshold is 480.

For example, according to the needle group probability corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point, the maximum value e of the electric field intensity_maxAnd a preset electric field intensity ablation threshold e corresponding to the target area_thAn electric field strength ablation threshold is determined to determine a predicted ablation zone based on the electric field strength ablation threshold.

For example, when the region determining sub-module determines the electric field intensity ablation threshold corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point according to the effective probability of the needle group corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point, the electric field intensity maximum value and the preset electric field intensity ablation threshold corresponding to the target region, the region determining sub-module performs the following operations: the electric field intensity ablation threshold corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point is closer to the preset electric field intensity ablation threshold in response to the larger the effective probability of the needle group corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point is, and the electric field intensity ablation threshold corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point is closer to the maximum electric field intensity in response to the smaller the effective probability of the needle group corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point is.

That is, when the effective probability of the needle group corresponding to the ith candidate needle arranging point and the jth candidate needle arranging point is larger, it indicates that the ith candidate needle arranging point and the jth candidate needle arranging point are more likely to form the effective needle group, and the actual coordinate values of the ith candidate needle arranging point and the jth candidate needle arranging point are more likely to be more accurate, so that the electric field intensity ablation threshold corresponding to the ith candidate needle arranging point and the jth candidate needle arranging point is closer to the preset electric field intensity ablation threshold. On the contrary, when the effective probability of the needle group corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point is smaller, the fact that the ith candidate needle distribution point and the jth candidate needle distribution point are more unlikely to form the effective needle group is shown, and the calculated actual coordinate values of the ith candidate needle distribution point and the jth candidate needle distribution point deviate from accurate coordinate values, so that the ablation threshold values corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point are closer to the preset electric field intensity maximum value.

For example, the calculation formula for determining the electric field ablation threshold corresponding to the ith candidate needle layout point and the jth candidate needle layout point is as follows:

Eth′＝p*e_th+(1-p)*e_max(formula 2)

Here, Eth' represents an electric field ablation threshold corresponding to the ith candidate needle laying point and the jth candidate needle laying point, p represents a needle group effective probability corresponding to the ith candidate needle laying point and the jth candidate needle laying point, and e_maxRepresents the maximum value of the electric field intensity, e_thRepresenting a preset electric field strength ablation threshold.

In this way, the electric field intensity ablation threshold is corrected by using the effective probability of the needle group corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point, so as to obtain a more accurate ablation region.

For example, when determining the ablation regions corresponding to the ith candidate needle placement point and the jth candidate needle placement point according to the electric field intensity ablation threshold, any feasible technique may be used to determine the ablation regions, which is not limited by the present disclosure.

After the predicted ablation region is obtained, an evaluation index is determined based on the predicted ablation region and the target region.

For example, the evaluation index determination sub-module performs the following operations when determining the evaluation index according to the predicted ablation region and the target region: determining ablation accuracy according to the predicted ablation region and the target region, wherein the ablation accuracy is used for representing the ratio of the coincidence region of the predicted ablation region and the target region to the predicted ablation region; determining an ablation recall rate according to the predicted ablation region and the target region, wherein the ablation recall rate is used for representing the ratio of the overlapping region of the predicted ablation region and the target region to the target region; and determining an evaluation index according to the ablation accuracy and the ablation recall rate.

The calculation formula of the ablation accuracy is as follows:

wherein precision represents ablation accuracy, TP represents the number of pixel points in a coincidence region formed by the target region and the predicted ablation region, and FP represents the number of pixel points in a coincidence region formed by the non-target region (i.e., the background region) and the predicted ablation region.

The ablation accuracy is used for representing the accuracy of the predicted ablation region, namely the proportion of an accurate part in a predicted result, and the higher the ablation accuracy is, the closer the predicted ablation region is to the target region is.

The calculation formula of the ablation recall rate is as follows:

wherein recall represents the ablation recall rate, TP represents the number of pixel points in a superposition region formed by the target region and the predicted ablation region, and FN represents the number of pixel points in a part which is not superposed with the predicted ablation region in the target region.

The ablation recall rate is used to indicate whether the predicted ablation region completely covers the target region, i.e., whether the lesion region is completely ablated.

The calculation formula of the evaluation index is as follows:

where β is a predetermined scaling parameter, for example, it is generally of greater concern in practice whether the lesion area is completely ablated, i.e., it is desirable that a change in FN will cause a large change in F, and therefore, β is greater than 1.

In practice, the ablation region is predicted as accurately as possible, so that the normal body function of a patient is prevented from being influenced by excessive ablation of biological tissues which do not belong to a focus part, and the focus is completely ablated completely to achieve the purpose of treatment. Therefore, the training device for the neural network provided by the disclosure corrects the parameters of the neural network by comprehensively considering the ablation accuracy and the ablation recall ratio, so that the ablation needle distribution scheme provided by the neural network can have better ablation accuracy and ablation recall ratio, and not only can the ablation needle distribution scheme meet the accuracy requirement of an ablation region, but also can meet the requirement that a focus region is completely ablated.

For example, as mentioned above, the reference needle arrangement information includes reference confidence degrees of N candidate needle arrangement points, reference coordinate information of the N candidate needle arrangement points, and at least one reference needle arrangement combination formed by the N candidate needle arrangement points, the predicted needle arrangement information includes prediction confidence degrees of the N candidate needle arrangement points, predicted coordinate information of the N candidate needle arrangement points, and needle group probabilities corresponding to any two candidate needle arrangement points, and the first calculation sub-module performs the following steps when calculating the intermediate loss value according to the reference needle arrangement information and the predicted needle arrangement information: calculating a pin number loss value based on the reference confidence degrees of the N candidate card layout points and the prediction confidence degrees of the N candidate card layout points; calculating a coordinate loss value based on the reference coordinate information of the N candidate card distribution points and the predicted coordinate information of the N candidate card distribution points; calculating a needle group loss value based on the effective probability of the needle groups corresponding to at least one reference needle arrangement combination and any two candidate needle arrangement points; and obtaining a loss value corresponding to the neural network based on the needle number loss value, the coordinate loss value and the needle group loss value.

For example, the needle number loss value is calculated as follows:

wherein l₁Representing a loss value of the pin number, c_iRepresenting the prediction confidence of the ith candidate card layout point, c_i' denotes the reference confidence of the ith candidate card layout point.

For example, the coordinate loss value is calculated as follows:

wherein l₂Representing the value of coordinate loss, x_iRepresenting the predicted abscissa, y, of the ith candidate card layout point_iRepresenting the predicted ordinate, x, of the ith candidate card layout point_i' represents an abscissa scale value, y, obtained based on the reference coordinate information of the ith candidate needle layout point_i' denotes a vertical coordinate ratio value obtained based on the reference coordinate information of the ith candidate needle arranging point.

For example, the calculation formula for the needle set loss value is as follows:

here, |₃Representing the value of the loss of the needle set, c_iRepresenting the prediction confidence of the ith candidate card layout point, c_jTo representConfidence of prediction of jth candidate card layout point, c_i' denotes the reference confidence of the ith candidate card layout point, c_j' denotes the reference confidence of the jth candidate card layout point, g_ijRepresenting the probability of a group of pins corresponding to the ith candidate pin layout point and the jth candidate pin layout point, i.e., g_ijThe value of the element, g, representing the ith row, the j +3 column of the prediction matrix M2_i′_jIndicating the probability of the reference needle group corresponding to the ith candidate needle laying point and the jth candidate needle laying point, e.g. if the ith candidate needle laying point and the jth candidate needle laying point form an ablation needle group, g_i′_jIs 1, otherwise is 0.

For example, the calculation formula of the corresponding intermediate loss value of the neural network is as follows:

c1 ═ l1+ l2+ l3 (formula 9)

Wherein, C1 represents the middle loss value corresponding to the neural network, that is, the middle loss value corresponding to the neural network is the sum of the needle number loss value, the coordinate loss value and the needle group loss value.

And after the intermediate loss value and the evaluation index are obtained, calculating the loss value corresponding to the neural network according to the intermediate loss value and the evaluation index.

For example, the calculation formula of the loss value corresponding to the neural network is as follows:

c2 ═ (1-F) (l1+ l2+ l3) (equation 10)

Wherein C2 represents a loss value corresponding to the neural network, and other parameters are defined as described above and are not described herein again.

For example, after obtaining the loss value C2, if the loss value C2 does not satisfy the predetermined convergence condition, that is, if the loss value C2 corresponding to the neural network does not converge, the parameters of the feature extraction sub-network and the feature fitting sub-network are corrected according to the loss value C2, and then the training process is continuously performed, that is, the training image is input again to obtain the predicted needle arrangement information, the loss value C2 is calculated according to the predicted needle arrangement information, the target region, and the reference needle arrangement information, and the parameters of the neural network are corrected according to the loss value C2, so that the parameters of the feature extraction sub-network and the feature fitting sub-network are continuously updated in an iterative manner.

For example, when the loss value C2 corresponding to the neural network converges, the trained neural network is obtained. That is, when C2 approaches 0 indefinitely, the parameters of the feature extraction sub-network and the feature fitting sub-network are no longer adjusted, resulting in a neural network for the target area.

When the C2 is infinitely close to 0, the predicted needle arrangement information obtained through the neural network is infinitely close to the reference needle arrangement information, and due to the fact that the accuracy of the reference needle arrangement information is high, after the predicted needle arrangement information is infinitely close to the reference needle arrangement information, the training parameters in the neural network can be considered to be preliminarily trained.

Fig. 5 is a schematic diagram of a training process of a neural network according to at least one embodiment of the present disclosure.

For example, the maximum number of electrode needles that can be used in the ablation needle arrangement is predetermined to be 7, that is, N is 7, and the 7 candidate needle arrangement points are respectively numbered as N1, N2.

The following describes a training method of a neural network provided in an embodiment of the present disclosure in detail with reference to fig. 5.

For example, training images are shown in fig. 2, with training images having a size of 224 x 224 and a resolution of 0.05 mm.

For example, as shown in fig. 5, after the training image is obtained, the criteria matrix M1 is obtained from the training image.

For example, as mentioned above, any method of obtaining an ablation needle distribution scheme with the best ablation effect may be adopted to obtain the reference ablation needle distribution scheme, for example, the reference ablation needle distribution scheme may be determined based on the needle distribution combination determined in patent ZL202011345756.1, and the reference ablation needle distribution scheme provided by an embodiment of the present disclosure is shown in table 2 below:

TABLE 2

As shown in table 2, the ablation needle arrangement scheme requires 5 electrode needles in total, namely candidate needle arrangement points N1 to N5; the abscissa and ordinate reference values in the second and third columns of Table 2 represent Cartesian coordinate values of the candidate card distribution point, i.e., coordinate values obtained in a reference coordinate system; the fourth column to the seventh column in table 2 indicate the information on the ablation needle group, and for example, for the candidate needle layout point N1 (as indicated by the second row in table 2), the candidate needle layout point N1 and the candidate needle layout point N2 constitute the ablation needle group 1, the candidate needle layout point N1 and the candidate needle layout point N5 constitute the ablation needle group 2, and for the candidate needle layout point N3 (as indicated by the fourth row in table 2), the candidate needle layout point N3 and the candidate needle layout point N2 constitute the ablation needle group 1, and the candidate needle layout point N3 and the candidate needle layout point N4 constitute the ablation needle group 2.

For example, an ablation needle set composed of the same two candidate needle layout points is regarded as one reference needle layout combination, and for example, an ablation needle set 2 composed of the candidate needle layout point N1 and the candidate needle layout point N5 obtained in the second row of table 2 and an ablation needle set 1 composed of the candidate needle layout point N5 and the candidate needle layout point N1 obtained in the sixth row of table 2 are one reference needle layout combination, and thus, there are 5 reference needle layout combinations provided in table 2.

Encoding the reference needle distribution scheme shown in table 2 to obtain a standard matrix M1 as reference needle distribution information, wherein the standard matrix M1 is schematically shown in table 3 below:

TABLE 3

Reference confidence	Proportional value of abscissa	Proportional value of ordinate	N1	N2	N3	N4	N5	N6	N7
										1	0.205	0.464	0	1	0	0	1	0	0
1	0.393	0.607	1	0	1	0	1	0	0
										1	0.653	0.634	0	1	0	1	0	0	0
1	0.821	0.482	0	0	1	0	1	0	0
										1	0.652	0.411	1	1	0	1	0	0	0
0	0	0	0	0	0	0	0	0	0
										0	0	0	0	0	0	0	0	0	0

For example, the contents of the second to eighth rows in table 3 (regions defined by black bold line boxes) represent a standard matrix M1, which standard matrix M1 includes 7 rows and 10 columns.

For example, the first row to the seventh row in the criteria matrix M1 are used to represent the relevant information of the candidate needle layout point N1 to the reference needle layout point N7, respectively.

For example, taking the candidate card layout point N1 as an example, the candidate card layout point N1 is a valid needle, and the reference confidence is 1; the abscissa proportion value of the candidate card clothing point N1 is the ratio of the reference abscissa value of the candidate card clothing point N1 to the transverse dimension of the training image, i.e., 46/224, and similarly, the ordinate proportion value of the candidate card clothing point N1 is the ratio of the reference ordinate value of the candidate card clothing point N1 to the longitudinal dimension of the training image, i.e., 104/224; for example, as can be seen from table 2, when the candidate needle layout point N1 and the candidate needle layout point N2 form a ablation needle group, and the candidate needle layout point N1 and the candidate needle layout point N5 form an ablation needle group, in the first row of the standard matrix M1, the fifth column and the eighth column are 1, and the other elements in the fourth column to the tenth column are all 0.

For example, since neither the candidate card layout point N6 nor the candidate card layout point N7 is a valid needle, the related information thereof is 0.

For example, as shown in fig. 5, a training image is input to the neural network, and the training image is processed by the processing module 103 to obtain the prediction needle arrangement information, that is, to obtain the prediction matrix M2, and the specific process may refer to the relevant content of the processing module 103, which is not described herein again.

For example, the prediction matrix M2 may be as shown in table 4 below.

TABLE 4

For example, the area defined by the black bold frame in table 4 represents the prediction matrix M2, which prediction matrix M2 includes 7 rows and 10 columns.

For example, taking the candidate card distribution point N2 as an example, the prediction confidence of the candidate card distribution point N2 is 0.82, which indicates that the probability of the candidate card distribution point N2 being a valid needle is high; the predicted abscissa of the candidate needle distribution point N2 is 0.39, the predicted ordinate is 0.61, the predicted abscissa is multiplied by the lateral dimension of the training image to obtain the predicted abscissa (224 × 0.39 ═ 87) of the candidate needle distribution point N2, and the predicted ordinate is multiplied by the longitudinal dimension of the training image to obtain the predicted ordinate (224 × 0.61 ═ 137) of the candidate needle distribution point N2; the needle group probabilities of the candidate needle layout point N2 and the 7 candidate needle layout points are, for example, 0.96 for the candidate needle layout point N2 and the candidate needle layout point N1, a high probability for the candidate needle layout point N2 and the candidate needle layout point N1 to form an effective ablation needle group, 0.04 for the candidate needle layout point N2 and the candidate needle layout point N2, and a failure of the candidate needle layout point N2 and the candidate needle layout point N2 to form an effective ablation needle group, as shown in the fourth column of the second row to the tenth column of the second row in the prediction matrix M2.

After the predicted ablation region is obtained according to the prediction matrix M2, the specific process may refer to the relevant content of the region determination sub-module, which is not described herein again.

Then, an evaluation index F is calculated according to the predicted ablation region and the target region, and the specific process may refer to the evaluation index to determine the relevant content of the sub-module, which is not described herein again.

Then, an intermediate loss value is calculated according to the prediction matrix M2 and the standard matrix M1, and a loss value corresponding to the neural network is calculated according to the intermediate loss value and the evaluation index, and the specific process may refer to the relevant contents of the first calculation submodule and the second calculation submodule, which is not described herein again.

And then, correcting parameters of the neural network according to the loss values until the loss values corresponding to the neural network are converged, obtaining the trained neural network, and storing the parameters of the feature extraction sub-network and the feature fitting sub-network.

After the trained neural network is obtained, the neural network needs to be tested to judge whether the neural network is qualified.

In addition, in order to obtain an ablation needle arrangement scheme according to the predicted needle arrangement information, a first threshold and a second threshold are further determined, the first threshold and the second threshold are used for converting the predicted needle arrangement information into the ablation needle arrangement scheme, for example, the first threshold is used for being compared with the prediction confidence to judge whether the candidate needle arrangement point is an effective needle, and the second threshold is used for being compared with the needle group probability to judge whether the two candidate needle arrangement points form an effective ablation needle group.

For example, the candidate card distribution points with the prediction confidence degrees larger than the first threshold value are used as effective pins to obtain a plurality of effective pins; and regarding the needle group with the needle group probability larger than the second threshold value as an effective needle group to obtain at least one effective needle group, for example, an ablation needle arrangement scheme obtained based on the predicted needle arrangement information at least comprises a plurality of effective needles and the at least one effective needle group.

For example, as shown in fig. 1, the training apparatus 10 further comprises a test module 105, the test module 105 being configured to perform the following operations: acquiring a plurality of test images, wherein each test image comprises a target area; acquiring a plurality of reference needle arrangement information which corresponds to a plurality of test images one by one; respectively processing the plurality of test images by using the trained neural network to obtain a plurality of pieces of predicted needle arrangement information; determining prediction accuracy based on the plurality of predicted needle arrangement information and the plurality of reference needle arrangement information; and in response to the prediction accuracy rate being smaller than the preset accuracy rate threshold value, retraining the neural network. Likewise, the test module 105 may be implemented in software, hardware, firmware, or any combination of these ways.

For example, in some embodiments, the test module 105 when performing the determination of the prediction accuracy rate based on the plurality of predicted needle placement information and the plurality of reference needle placement information comprises performing the following operations: determining a plurality of candidate first thresholds and a plurality of candidate second thresholds; determining a plurality of candidate threshold combinations based on the plurality of candidate first thresholds and the plurality of candidate second thresholds, wherein each candidate threshold combination comprises one candidate first threshold and one candidate second threshold; processing the plurality of pieces of predicted needle arrangement information based on each candidate threshold combination, and determining a plurality of ablation needle arrangement schemes corresponding to each candidate threshold combination; comparing the plurality of ablation needle distribution schemes corresponding to each candidate threshold combination with the plurality of ablation needle distribution schemes determined based on the plurality of reference needle distribution information respectively, and determining a plurality of accuracy rates corresponding to the plurality of candidate threshold combinations respectively; and taking the maximum value of the accuracy rates as the prediction accuracy rate.

For example, the test module may be further configured to perform the following operations: determining a first threshold and a second threshold, wherein the first threshold and the second threshold are used for converting the predicted needle arrangement information into an ablation needle arrangement scheme; determining a first threshold and a second threshold, comprising: and taking a candidate first threshold and a candidate second threshold in the candidate threshold combination corresponding to the prediction accuracy as a first threshold and a second threshold.

For example, M test images with target regions are used to test a neural network, M pieces of predicted needle arrangement information corresponding to the test images one to one are obtained, and M pieces of reference needle arrangement information are determined based on the M test images, so as to determine M pieces of reference ablation needle arrangement schemes corresponding to the M test images, respectively, for example, the reference ablation needle arrangement schemes may be as shown in table 2; then, the M pieces of predicted needle distribution information are respectively processed based on P preset candidate threshold combinations to obtain M ablation needle distribution schemes corresponding to each candidate threshold combination, for example, the predicted needle distribution information is converted into the corresponding ablation needle distribution schemes according to the candidate first threshold and the candidate second threshold in each candidate threshold combination; then, comparing the M ablation needle arrangement schemes corresponding to each candidate threshold combination with the M reference ablation needle arrangement schemes, and determining the accuracy corresponding to each candidate threshold combination, for example, if Q ablation needle arrangement schemes in the M ablation needle arrangement schemes are consistent with the reference ablation needle arrangement schemes, the accuracy corresponding to the candidate threshold combination is Q/M; then, the maximum value of the P accuracy rates corresponding to the P candidate threshold combinations is used as the prediction accuracy rate, for example, if the prediction accuracy rate is greater than or equal to a preset accuracy rate threshold, the neural network training is considered to be qualified, otherwise, the neural network training is considered to be unqualified, and the neural network needs to be trained again. Here, M, P, and Q are positive integers, and Q is equal to or less than M.

For example, if the neural network is qualified in training, that is, the prediction accuracy is greater than or equal to the preset accuracy threshold, the candidate first threshold and the candidate second threshold in the candidate threshold combination corresponding to the prediction accuracy are used as the first threshold and the second threshold, for example, when the candidate first threshold is 0.4 and the candidate second threshold is 0.5, the prediction accuracy is 90%, for example, if the preset accuracy threshold is 80%, the neural network is considered to be qualified at this time, and the first threshold is 0.4 and the second threshold is 0.5, and then, in the process of using the neural network, the predicted needle arrangement information is processed by using the first threshold and the second threshold to obtain the ablation needle arrangement scheme.

For example, in other embodiments, the test module 105, when determining the prediction accuracy based on the plurality of predicted needle placement information and the plurality of reference needle placement information, performs the following operations: determining a first threshold and a second threshold; processing the plurality of pieces of predicted needle arrangement information according to the first threshold value and the second threshold value to obtain a plurality of predicted needle arrangement schemes; and comparing the plurality of predicted needle arrangement schemes with a plurality of ablation needle arrangement schemes determined based on the plurality of reference needle arrangement information respectively to determine prediction accuracy.

That is, in this embodiment, the first threshold value and the second threshold value are determined first, and for example, one of a plurality of candidate combinations may be selected as the first threshold value and the second threshold value, or may be determined empirically; and then, according to a first threshold and a second threshold, carrying out a series of processing on the plurality of pieces of predicted needle arrangement information to obtain a predicted accuracy, if the predicted accuracy is greater than or equal to a preset accuracy threshold, considering that the neural network is qualified, and in the process of using the neural network, processing the predicted needle arrangement information by using the first threshold and the second threshold to obtain an ablation needle arrangement scheme.

Fig. 6 is a flowchart illustrating a training process of a neural network according to at least one embodiment of the present disclosure.

As shown in fig. 6, firstly, training data (a plurality of training images (e.g. 100 or more) with target areas) are obtained to train the neural network, and the training process is as described above and will not be described herein again.

And then, when the loss value corresponding to the neural network is converged, obtaining the trained neural network.

Thereafter, test data (a plurality of test images (e.g., 100 or more) with the target area) is obtained to test the neural network, and the testing process is as described above and will not be described herein again.

And then, acquiring the prediction accuracy, judging whether the neural network is qualified according to the prediction accuracy, if the neural network is qualified, putting the neural network into use, and otherwise, acquiring the training data again to train the neural network.

For example, the training process of the neural network further includes a process of obtaining a first threshold and a second threshold (not shown), for example, when the neural network is determined to be qualified, the candidate first threshold and the candidate second threshold in the candidate threshold combination corresponding to the prediction accuracy may be respectively used as the first threshold and the second threshold, or the first threshold and the second threshold may be determined first, the prediction accuracy is determined according to the first threshold and the second threshold, and when the neural network is determined to be qualified and put into use, the first threshold and the second threshold are used to determine the ablation needle arrangement scheme.

At least one embodiment of the present disclosure also provides an ablation needle planning device. Fig. 7 is a schematic block diagram of an ablation needle arrangement planning apparatus according to at least one embodiment of the present disclosure.

As shown in fig. 7, the ablation needle arrangement planning apparatus 20 includes an image acquisition module 21, a processing module 22, and a planning module 23, each of which may be implemented by software, hardware, firmware, or any combination thereof, and may communicate, such as transmit data and/or instructions, etc., as needed during operation, for example, they may be interconnected by a bus system and/or other form of connection mechanism (not shown) at a hardware level, and may be interconnected by a communication process or thread at a software level. It should be noted that the components and configuration of the ablation needle planning device 20 shown in fig. 7 are exemplary only, and not limiting, and that the ablation needle planning device 20 may have other components and configurations as desired.

For example, the image acquisition module 21 is configured to acquire an input image. For example, the input image includes a target region, the meaning of which is as described above.

The processing module 22 is configured to process the input image using a neural network to obtain predicted needle placement information corresponding to the input image.

And the planning module 23 is configured to obtain an ablation needle arrangement scheme based on the predicted needle arrangement information.

For example, the neural network is trained, at least in part, by a training apparatus provided in accordance with at least one embodiment of the present disclosure.

For example, when the planning module 23 performs the obtaining of the ablation needle distribution scheme based on the predicted needle distribution information, the following steps are performed: acquiring a first threshold value and a second threshold value; taking the candidate card distribution points with the prediction confidence degrees larger than the first threshold value as effective pins to obtain a plurality of effective pins; acquiring size information of an input image, and determining actual position coordinates of a plurality of effective needles according to the size information of the input image and the predicted coordinate information of the plurality of effective needles; and taking the needle group with the needle group probability larger than the second threshold value as an effective needle group to obtain at least one effective needle group, wherein the ablation needle arrangement scheme comprises a plurality of effective needles, actual position coordinates of the plurality of effective needles and the at least one effective needle group.

For example, the size of the input image is 224 × 224, and the input image is processed by the processing module 22, and the obtained predicted needle distribution information is shown as the prediction matrix M2 in table 4. For example, the training apparatus provided according to at least one embodiment of the present disclosure trains the neural network, and the obtained first threshold is 0.4, and the obtained second threshold is 0.5, and the following describes the implementation process of the planning module 23 in detail by combining the foregoing parameters.

For example, since the first threshold is 0.4, the candidate card distribution points in the prediction matrix M2 with the prediction confidence greater than the first threshold are regarded as valid pins, that is, the valid pins are the candidate card distribution points N1 to N5.

The input image size is 224 x 224, the predicted abscissa is multiplied by the input image lateral size (i.e., 224) to obtain the predicted abscissa value, and the predicted ordinate is multiplied by the input image longitudinal size (i.e., 224) to obtain the predicted ordinate value, for example, the effective needle coordinate values are shown in table 5:

TABLE 5

Needle numbering	N1	N2	N3	N4	N5
						Coordinate (mm)	[47,103]	[87,137]	[146,141]	[184,108]	[146,92]

In Table 5, [ x, y ] indicates the coordinate values of the effective pins, x indicates the predicted abscissa value, and y indicates the predicted ordinate value.

For example, since the second threshold value is 0.5, the needle group in the prediction matrix M2, of which the needle group probability is greater than the second threshold value, is regarded as the valid needle group. For example, the needle group probability of the needle group region is binarized based on the second threshold value, the needle group probability equal to or higher than the second threshold value is set to 1, the needle group probability smaller than the second threshold value is set to 0, and the analysis results after the processing are shown in table 6 below:

TABLE 6

If the results shown in table 6 are subjected to deduplication processing, for example, if a needle group with the same needle number is considered to be the same needle group, the effective needle groups obtained from the analysis results include the following six types:

candidate card layout point N1 and candidate card layout point N2,

candidate card layout point N1 and candidate card layout point N5,

candidate card layout point N2 and candidate card layout point N3,

candidate card layout point N2 and candidate card layout point N5,

candidate stitching point N3 and candidate stitching point N4, an

Candidate card layout point N4 and candidate card layout point N5.

For example, the resulting ablation needle layout scheme based on the prediction matrix M2 includes 5 effective needles, i.e., the candidate needle layout point N1 to the candidate needle layout point N5, six effective needle groups, as described above, and the actual position coordinate values of the 5 candidate needle layout points, as shown in table 6.

According to the ablation needle arrangement planning device provided by at least one embodiment of the disclosure, after the input image including the target area is input, an ablation needle arrangement planning scheme can be obtained within a few seconds, the globally optimal needle arrangement scheme is efficiently and quickly provided, and the ablation needle arrangement planning scheme obtained by the ablation needle arrangement planning device provided by the disclosure does not need a positioning template, so that accurate puncture ablation at any position can be realized.

Fig. 8A is a schematic flow chart of a training method of a neural network according to at least one embodiment of the present disclosure.

As shown in fig. 8A, at least one embodiment of the present disclosure further provides a training method of a neural network, including steps S301 to S304.

Step S301: a training image is acquired.

For example, the training image includes a target region, the meaning of which is as described above.

Step S302: and acquiring reference needle arrangement information corresponding to the training image.

For example, the reference needle arrangement information is used to provide an ablation needle arrangement as a standard.

Step S303: and processing the training image by using the neural network to obtain the predicted needle arrangement information corresponding to the training image.

Step S304: and calculating a loss value corresponding to the neural network according to the reference needle arrangement information, the target area and the predicted needle arrangement information, and correcting parameters of the neural network based on the loss value when the loss value does not meet a preset convergence condition.

It should be noted that the neural network has the same structure and function as the neural network 200 in the embodiment of the training apparatus of the neural network, and is not described herein again.

For example, step S01 may include: acquiring a medical image, wherein the medical image comprises a lesion area to be segmented; the medical image is subjected to region segmentation processing to obtain a training image, wherein the training image has a target region corresponding to a lesion region in the medical image.

For the related content of the medical image, the training image, the target region, and the lesion region, reference may be made to the related content of the image obtaining module 101 shown in fig. 1, which is not described herein again.

For example, the process of acquiring the reference needling information in step S302 and the related description of the reference needling information may refer to the related content of the reference information acquiring module 102 shown in fig. 1, and are not described herein again.

For example, the process of acquiring the predicted needle arrangement information in step S303 and the related description of the predicted needle arrangement information may refer to the related content of the processing module 103 shown in fig. 1, and are not described herein again.

Fig. 8B is a schematic flowchart of step S304 in the training method of the neural network shown in fig. 8A. As shown in fig. 8B, the step S304 of the training method according to at least one embodiment of the present disclosure at least includes steps S3041 to S3044.

Step S3041, a predicted ablation region is determined according to the predicted needle placement information.

Step S3042, determining an evaluation index according to the predicted ablation region and the target region, where the evaluation index is used to represent the validity of the scheme of the predicted needle arrangement information.

Step S3043, a middle loss value is calculated based on the reference needle arrangement information and the predicted needle arrangement information.

Step S3044, a loss value corresponding to the neural network is calculated according to the intermediate loss value and the evaluation index.

For example, the predicted needle arrangement information includes the prediction confidence degrees of N candidate needle arrangement points, the predicted coordinate information of the N candidate needle arrangement points, and the needle group probabilities corresponding to any two candidate needle arrangement points, where N is a positive integer and is greater than or equal to 2, and the related content of the predicted needle arrangement information may refer to the training device of the neural network, and repeated details are not repeated.

For example, one example of step S3041 may include: determining ablation areas corresponding to every two different candidate needle distribution points in the N candidate needle distribution points to obtain a plurality of ablation areas; and superposing the plurality of ablation regions to obtain a predicted ablation region.

For example, determining the ablation regions corresponding to each of the two different candidate needle distribution points may include: for an ith candidate card layout point and a jth candidate card layout point of the N candidate card layout points: calculating effective probabilities of needle groups corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point, wherein the effective probabilities of the needle groups are used for representing the probability that the ith candidate needle arrangement point and the jth candidate needle arrangement point form an effective ablation needle group; determining the actual coordinate value of the ith candidate card distribution point and the actual coordinate value of the jth candidate card distribution point according to the predicted coordinate information of the ith candidate card distribution point and the predicted coordinate information of the jth candidate card distribution point; determining electric field distribution corresponding to the ith candidate card distribution point and the jth candidate card distribution point according to the actual coordinate value of the ith candidate card distribution point and the actual coordinate value of the jth candidate card distribution point; determining the maximum value of the electric field intensity according to the electric field distribution corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point; determining an electric field intensity ablation threshold corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point according to the effective probability of the needle group corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point, the maximum electric field intensity value and a preset electric field intensity ablation threshold corresponding to the target area; and determining an ablation region corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point according to the electric field intensity ablation threshold, wherein i and j are positive integers and are less than or equal to N, and i is not equal to j.

For example, calculating the effective probability of the needle group corresponding to the ith candidate needle layout point and the jth candidate needle layout point may include: determining two needle group probabilities corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point according to the predicted needle arrangement information; and calculating the effective probability of the needle group corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point according to the prediction confidence of the ith candidate needle distribution point, the prediction confidence of the jth candidate needle distribution point and the maximum value of the two needle group probabilities.

For example, determining the electric field intensity ablation threshold corresponding to the ith candidate needle layout point and the jth candidate needle layout point according to the effective probability of the needle group corresponding to the ith candidate needle layout point and the jth candidate needle layout point, the electric field intensity maximum value and the preset electric field intensity ablation threshold corresponding to the target area may include: the electric field intensity ablation threshold value corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point is closer to the preset electric field intensity ablation threshold value in response to the fact that the effective probability of the needle group corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point is larger, and the electric field intensity ablation threshold value corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point is closer to the maximum electric field intensity in response to the fact that the effective probability of the needle group corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point is smaller.

For example, the calculation formula for determining the electric field ablation threshold corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point may be as shown in the foregoing formula (2), and details are not repeated here.

For example, the specific content of the predicted ablation region obtained in step S3041 may refer to the related content of the region determination sub-module, and the repeated description is omitted here.

For example, step S3042 may include: determining ablation accuracy according to the predicted ablation region and the target region, wherein the ablation accuracy is used for representing the ratio of the coincidence region of the predicted ablation region and the target region to the predicted ablation region; determining an ablation recall rate according to the predicted ablation region and the target region, wherein the ablation recall rate is used for representing the ratio of the overlapping region of the predicted ablation region and the target region to the target region; and determining an evaluation index according to the ablation accuracy and the ablation recall rate.

The specific calculation process of the ablation accuracy, the ablation recall rate and the evaluation index may refer to the relevant content of the evaluation index determination sub-module, and specifically, the ablation accuracy, the ablation recall rate and the evaluation index may be calculated by referring to formula (3) -formula (5), and repeated parts are not described again.

For example, step S3043 may include: calculating a pin number loss value based on the reference confidence degrees of the N candidate card layout points and the prediction confidence degrees of the N candidate card layout points; calculating a coordinate loss value based on the reference coordinate information of the N candidate needle distribution points and the predicted coordinate information of the N candidate needle distribution points; calculating a needle group loss value based on the effective probability of the needle groups corresponding to at least one reference needle arrangement combination and any two candidate needle arrangement points; and obtaining a middle loss value corresponding to the neural network based on the needle number loss value, the coordinate loss value and the needle group loss value.

The specific calculation process of the needle number loss value, the coordinate loss value, the needle group loss value, and the intermediate loss value may refer to the related content of the first calculation sub-module, and specifically, the needle number loss value, the coordinate loss value, the needle group loss value, and the intermediate loss value may be calculated by referring to formulas (6) to (9), and repeated parts are not described again.

For example, the process of calculating the loss value corresponding to the neural network in step S3044 may be obtained by calculating the related content, the specific address, and the loss value corresponding to the neural network with reference to formula 10, which is not described herein again.

And circularly executing the step S301 to the step S304, continuously carrying out iterative updating on the parameters of the feature extraction sub-network and the feature fitting sub-network, and obtaining the trained neural network when the loss value C2 corresponding to the neural network is converged.

Fig. 8C is a schematic flow chart of a training method of a neural network according to at least one embodiment of the present disclosure.

As shown in fig. 8C, the training method of the neural network further includes steps S305 to S309.

In step S305, a plurality of test images are acquired.

For example, each test image includes a target region.

In step S306, a plurality of pieces of reference needle distribution information corresponding one-to-one to the plurality of test images are acquired.

In step S307, the trained neural network is used to process each of the plurality of test images to obtain a plurality of predicted needle arrangement information.

In step S308, a prediction accuracy is determined based on the plurality of pieces of predicted needle arrangement information and the plurality of pieces of reference needle arrangement information;

in step S309, in response to the prediction accuracy being less than the preset accuracy threshold, the neural network is retrained.

For example, in some embodiments, step S308 may include: determining a plurality of candidate first thresholds and a plurality of candidate second thresholds; determining a plurality of candidate threshold combinations based on the plurality of candidate first thresholds and the plurality of candidate second thresholds, wherein each candidate threshold combination comprises one candidate first threshold and one candidate second threshold; processing the plurality of pieces of predicted needle arrangement information based on each candidate threshold combination, and determining a plurality of ablation needle arrangement schemes corresponding to each candidate threshold combination; comparing the plurality of ablation needle distribution schemes corresponding to each candidate threshold combination with the plurality of ablation needle distribution schemes determined based on the plurality of reference needle distribution information respectively, and determining a plurality of accuracy rates corresponding to the plurality of candidate threshold combinations respectively; and taking the maximum value of the accuracy rates as the prediction accuracy rate.

For example, when the prediction accuracy is greater than or equal to the prediction accuracy threshold, the neural network is considered to be qualified, and at this time, the candidate first threshold and the candidate second threshold in the candidate threshold combination corresponding to the prediction accuracy may be used as the first threshold and the second threshold for subsequent use of the neural network, that is, the predicted needle arrangement information is converted into the ablation needle arrangement scheme by using the first threshold and the second threshold.

For example, in other embodiments, step S308 may include: determining a first threshold and a second threshold; processing the plurality of pieces of predicted needle arrangement information according to the first threshold value and the second threshold value to obtain a plurality of predicted needle arrangement schemes; and comparing the plurality of predicted needle arrangement schemes with a plurality of ablation needle arrangement schemes determined based on the plurality of reference needle arrangement information respectively to determine prediction accuracy.

Specifically, for the related contents of steps S305 to S309, reference may be made to the related contents of the test module 105 in the training apparatus of the neural network shown in fig. 1, and repeated details are not repeated.

It should be noted that the training method of the neural network can achieve similar technical effects to the aforementioned training device 10 of the neural network, and will not be described herein again.

Fig. 9 is a schematic flow chart of an ablation needle planning method according to at least one embodiment of the present disclosure.

As shown in fig. 9, at least one embodiment of the present disclosure further provides a training method of a neural network, including steps S401 to S403.

In step S401, an input image is acquired.

For example, the input image includes a target region, the meaning of which is as described above.

In step S402, the input image is processed by using the neural network to obtain predicted needle distribution information corresponding to the input image.

In step S403, an ablation needle arrangement scheme is obtained based on the predicted needle arrangement information.

For example, the relevant content of step S401 may refer to the description of the image obtaining module 21 shown in fig. 7, and is not described herein again.

For example, the relevant content of step S402 can refer to the description of the processing module 22 shown in fig. 7, and is not described here again.

For example, step S403 may include: acquiring a first threshold value and a second threshold value; taking the candidate card distribution points with the prediction confidence degrees larger than the first threshold value as effective pins to obtain a plurality of effective pins; acquiring size information of an input image, and determining actual position coordinates of a plurality of effective needles according to the size information of the input image and the predicted coordinate information of the plurality of effective needles; and taking the needle group with the needle group probability larger than the second threshold value as an effective needle group to obtain at least one effective needle group, wherein the ablation needle arrangement scheme comprises a plurality of effective needles, actual position coordinates of the plurality of effective needles and the at least one effective needle group.

The specific relevant content of step S403 may refer to the relevant description for implementing the processing module 22 shown in fig. 7, and repeated details are not repeated.

It should be noted that the ablation needle arrangement planning method can achieve similar technical effects to the ablation needle arrangement planning apparatus 20, and will not be described herein again.

Fig. 10 is a schematic block diagram of an electronic device according to an embodiment of the present disclosure. As shown in fig. 10, the electronic device 400 is suitable for implementing a neural network training method or an ablation needle planning method provided by the embodiments of the present disclosure, for example. It should be noted that the components of the electronic device 400 shown in fig. 10 are only exemplary and not limiting, and the electronic device 400 may have other components according to the actual application.

As shown in fig. 10, electronic device 400 may include a processing means (e.g., central processing unit, graphics processor, etc.) 401 that may perform various appropriate actions and processes in accordance with non-transitory computer-readable instructions stored in a memory to implement various functions.

For example, the computer readable instructions when executed by the processing device 401 may perform one or more steps of a neural network training method or an ablation needle planning method according to any of the above embodiments. It should be noted that, for the detailed description of the processing procedure of the neural network training method or the ablation needle arrangement planning method, reference may be made to the related description in the above embodiments of the neural network training method or the ablation needle arrangement planning method, and repeated parts are not repeated.

For example, the memory may comprise any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. Volatile memory can include, for example, a Random Access Memory (RAM)403, and/or a cache memory (cache), etc., where, for example, computer-readable instructions can be loaded from storage 408 into Random Access Memory (RAM)403 for execution. The non-volatile memory may include, for example, Read Only Memory (ROM)402, a hard disk, an Erasable Programmable Read Only Memory (EPROM), a portable compact disc read only memory (CD-ROM), a USB memory, a flash memory, and so forth. Various applications and various data, such as style images, and various data used and/or generated by the applications, may also be stored in the computer-readable storage medium.

For example, the processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input/output (I/O) interface 405 is also connected to bus 404.

Generally, the following devices may be connected to the I/O interface 405: input devices 406 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 407 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 408 including, for example, tape, hard disk, flash memory, etc.; and a communication device 409. The communication means 409 may allow the electronic device 400 to communicate wirelessly or by wire with other electronic devices to exchange data. While fig. 10 illustrates an electronic device 400 having various means, it is to be understood that not all illustrated means are required to be implemented or provided, and that the electronic device 400 may alternatively be implemented or provided with more or less means. For example, the processing device 401 may control other components in the electronic device 400 to perform desired functions. The processing device 401 may be a device having data processing capability and/or program execution capability, such as a Central Processing Unit (CPU), Tensor Processor (TPU), or Graphics Processor (GPU). The Central Processing Unit (CPU) may be an X86 or ARM architecture, etc. The GPU may be separately integrated directly onto the motherboard or built into the north bridge chip of the motherboard. The GPU may also be built into the Central Processing Unit (CPU).

Fig. 11 is a schematic diagram of a non-transitory computer-readable storage medium according to at least one embodiment of the disclosure. For example, as shown in fig. 11, the storage medium 500 may be a non-transitory computer-readable storage medium, on which one or more computer-readable instructions 501 may be non-temporarily stored on the storage medium 500. For example, the computer readable instructions 501, when executed by a processor, may perform one or more steps of a training method or an ablation needle planning method according to a neural network as described above.

For example, the storage medium 500 may be applied to the electronic device described above, and for example, the storage medium 500 may include a memory in the electronic device.

For example, the storage medium may include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a Random Access Memory (RAM), a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM), a portable compact disc read only memory (CD-ROM), a flash memory, or any combination of the above, as well as other suitable storage media.

For example, the description of the storage medium 500 may refer to the description of the memory in the embodiment of the electronic device, and repeated descriptions are omitted.

For the present disclosure, there are also the following points to be explained:

(1) the drawings of the embodiments of the disclosure only relate to the structures related to the embodiments of the disclosure, and other structures can refer to the common design.

(2) Thicknesses and dimensions of layers or structures may be exaggerated in the drawings used to describe embodiments of the present invention for clarity. It will be understood that when an element such as a layer, film, region, or substrate is referred to as being "on" or "under" another element, it can be "directly on" or "under" the other element or intervening elements may be present.

(3) Without conflict, embodiments of the present disclosure and features of the embodiments may be combined with each other to arrive at new embodiments.

The above description is only for the specific embodiments of the present disclosure, but the scope of the present disclosure is not limited thereto, and the scope of the present disclosure should be subject to the scope of the claims.

Claims

1. An apparatus for training a neural network, the apparatus comprising:

an image acquisition module configured to acquire a training image, wherein the training image includes a target region;

a reference information acquisition module configured to acquire reference needle arrangement information corresponding to the training image, wherein the reference needle arrangement information is used for providing an ablation needle arrangement scheme as a standard;

the processing module is configured to process the training image by using the neural network to obtain predicted needle arrangement information corresponding to the training image;

the correction module is configured to calculate a loss value corresponding to the neural network according to the reference needle arrangement information, the target area and the predicted needle arrangement information, and correct parameters of the neural network based on the loss value when the loss value does not meet a preset convergence condition;

wherein the correction module comprises:

the region determining submodule is configured to determine a predicted ablation region according to the predicted needle arrangement information;

an evaluation index determination submodule configured to determine an evaluation index according to the predicted ablation region and the target region, wherein the evaluation index is used for representing the scheme effectiveness of the predicted needle arrangement information;

the first calculation submodule is configured to calculate a middle loss value according to the reference needle arrangement information and the predicted needle arrangement information;

the second calculation submodule is configured to calculate a loss value corresponding to the neural network according to the intermediate loss value and the evaluation index;

wherein the evaluation index determining sub-module executes the following operations when determining the evaluation index according to the predicted ablation region and the target region:

determining ablation accuracy according to the predicted ablation region and the target region, wherein the ablation accuracy is used for representing the ratio of the coincidence region of the predicted ablation region and the target region to the predicted ablation region;

determining an ablation recall ratio according to the predicted ablation region and the target region, wherein the ablation recall ratio is used for representing the ratio of the coincidence region of the predicted ablation region and the target region to the target region;

and determining the evaluation index according to the ablation accuracy and the ablation recall rate.

2. The training device as claimed in claim 1, wherein the predicted needle layout information comprises the predicted confidence degrees of N candidate needle layout points, the predicted coordinate information of the N candidate needle layout points and the needle group probabilities corresponding to any two candidate needle layout points, N is a positive integer and is greater than or equal to 2,

the region determining sub-module executes the following operations when determining the predicted ablation region according to the predicted needle arrangement information:

determining an ablation region corresponding to every two different candidate needle laying points in the N candidate needle laying points to obtain a plurality of ablation regions;

and superposing the plurality of ablation regions to obtain the predicted ablation region.

3. The training device of claim 2, wherein the region determining sub-module, when performing the determination of the ablation region corresponding to each of the two different candidate needle placement points, comprises performing the following operations:

for an ith candidate card layout point and a jth candidate card layout point of the N candidate card layout points:

calculating effective probability of a needle group corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point, wherein the effective probability of the needle group is used for representing the probability that the ith candidate needle arrangement point and the jth candidate needle arrangement point form an effective ablation needle group;

determining an actual coordinate value of the ith candidate card distribution point and an actual coordinate value of the jth candidate card distribution point according to the predicted coordinate information of the ith candidate card distribution point and the predicted coordinate information of the jth candidate card distribution point;

determining electric field distribution corresponding to the ith candidate needle laying point and the jth candidate needle laying point according to the actual coordinate value of the ith candidate needle laying point and the actual coordinate value of the jth candidate needle laying point;

determining the maximum value of the electric field intensity according to the electric field distribution corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point;

determining electric field intensity ablation thresholds corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point according to the needle group effective probability corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point, the electric field intensity maximum value and a preset electric field intensity ablation threshold corresponding to the target area;

determining an ablation region corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point according to the electric field intensity ablation threshold value,

wherein i and j are both positive integers and are less than or equal to N, and i is not equal to j.

4. The training apparatus as claimed in claim 3, wherein the region determining sub-module when performing the calculation of the effective probability of the needle group corresponding to the ith candidate needle layout point and the jth candidate needle layout point comprises performing the following operations:

determining two needle group probabilities corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point according to the predicted needle arrangement information;

and calculating the effective probability of the needle group corresponding to the ith candidate needle layout point and the jth candidate needle layout point according to the prediction confidence of the ith candidate needle layout point, the prediction confidence of the jth candidate needle layout point and the maximum value of the two needle group probabilities.

5. The training device according to claim 3, wherein the region determining sub-module performs the following operations when determining the electric field strength ablation threshold corresponding to the ith candidate card distribution point and the jth candidate card distribution point according to the effective probability of the needle group corresponding to the ith candidate card distribution point and the jth candidate card distribution point, the maximum electric field strength value and the preset electric field strength ablation threshold corresponding to the target region:

in response to the higher effective probability of the needle group corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point, the closer the electric field intensity ablation threshold corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point is to the preset electric field intensity ablation threshold,

in response to the smaller effective probability of the needle group corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point, the closer the electric field intensity ablation threshold value corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point is to the maximum electric field intensity value.

6. The training device according to claim 3, wherein the region determining sub-module performs the following operations when determining the electric field strength ablation threshold corresponding to the ith candidate card distribution point and the ith candidate card distribution point according to the effective probability of the needle group corresponding to the ith candidate card distribution point and the jth candidate card distribution point, the maximum electric field strength value and the preset electric field strength ablation threshold corresponding to the target region:

the calculation formula for determining the electric field ablation threshold corresponding to the ith candidate needle distribution point and the jth candidate needle distribution point is as follows:

Eth′＝p*e_th+(1-p)*e_max

wherein Eth' represents the i-th candidate needle distribution point and the electric field ablation threshold corresponding to the i-th candidate needle distribution point, and p represents theThe effective probability of the needle group corresponding to the ith candidate needle arrangement point and the jth candidate needle arrangement point, e_maxRepresents the maximum value of the electric field intensity, e_thRepresenting the preset electric field intensity ablation threshold.

7. The training device according to claim 1, wherein the reference clothing information includes reference confidence degrees of N candidate clothing points, reference coordinate information of the N candidate clothing points, and at least one reference clothing combination of the N candidate clothing points,

the predicted needle arrangement information comprises the prediction confidence degrees of the N candidate needle arrangement points, the prediction coordinate information of the N candidate needle arrangement points and the needle group probability corresponding to any two candidate needle arrangement points, N is a positive integer and is more than or equal to 2,

the first calculation sub-module executes the following steps when calculating the middle loss value according to the reference needle arrangement information and the predicted needle arrangement information:

calculating a pin number loss value based on the reference confidence levels of the N candidate card layout points and the prediction confidence levels of the N candidate card layout points;

calculating a coordinate loss value based on the reference coordinate information of the N candidate needle layout points and the predicted coordinate information of the N candidate needle layout points;

calculating a needle group loss value based on the effective probability of the needle groups corresponding to the at least one reference needle arrangement combination and the any two candidate needle arrangement points;

and obtaining a middle loss value corresponding to the neural network based on the needle number loss value, the coordinate loss value and the needle group loss value.

8. The training apparatus according to claim 1, wherein the trained neural network is obtained when a loss value corresponding to the neural network converges.

9. The training device of claim 8, further comprising a testing module configured to:

acquiring a plurality of test images, wherein each test image comprises the target area;

acquiring a plurality of reference needle arrangement information which corresponds to the plurality of test images one by one;

processing the plurality of test images by using the trained neural network respectively to obtain a plurality of pieces of predicted needle arrangement information;

determining a prediction accuracy rate based on the plurality of predicted needle arrangement information and the plurality of reference needle arrangement information;

and in response to the prediction accuracy being less than a preset accuracy threshold, retraining the neural network.

10. The training device of claim 9, wherein the testing module, when performing the determination of the prediction accuracy based on the plurality of predicted needle placement information and the plurality of reference needle placement information, comprises performing the following operations:

determining a plurality of candidate first thresholds and a plurality of candidate second thresholds;

determining a plurality of candidate threshold combinations based on the plurality of candidate first thresholds and the plurality of candidate second thresholds, wherein each candidate threshold combination comprises one candidate first threshold and one candidate second threshold;

processing the plurality of pieces of predicted needle arrangement information based on each candidate threshold combination, and determining a plurality of ablation needle arrangement schemes corresponding to each candidate threshold combination;

comparing the plurality of ablation needle distribution schemes corresponding to each candidate threshold combination with a plurality of ablation needle distribution schemes determined based on the plurality of reference needle distribution information, and determining a plurality of accuracy rates corresponding to the plurality of candidate threshold combinations respectively;

taking a maximum value of the plurality of accuracy rates as the prediction accuracy rate.

11. The training apparatus of claim 10, wherein the testing module is further configured to:

determining a first threshold and a second threshold, wherein the first threshold and the second threshold are used for converting the predicted needle arrangement information into an ablation needle arrangement scheme;

determining a first threshold and a second threshold, comprising:

and taking a candidate first threshold and a candidate second threshold in the candidate threshold combination corresponding to the prediction accuracy as the first threshold and the second threshold.

12. The training device of claim 9, wherein the testing module, when performing the determination of the prediction accuracy based on the plurality of predicted needle placement information and the plurality of reference needle placement information, comprises performing the following operations:

determining a first threshold and a second threshold;

processing the plurality of predicted needle distribution information according to the first threshold and the second threshold to obtain a plurality of predicted needle distribution schemes;

comparing the plurality of predicted needle arrangement schemes with a plurality of ablation needle arrangement schemes determined based on the plurality of reference needle arrangement information, respectively, and determining the prediction accuracy.

13. The training device of claim 1, wherein the neural network comprises a feature extraction sub-network and a feature fitting sub-network, the feature extraction sub-network is configured to extract image features of the training image, the feature fitting sub-network is configured to process the training image features to obtain predicted needle placement information corresponding to the training image,

the feature fitting sub-network comprises at least one fully connected layer,

the at least one fully-connected layer is configured to process the image features to obtain a plurality of feature information, and

the feature fitting subnetwork is further configured to derive the predicted needle placement information based on the plurality of feature information.

14. The training device of claim 1, wherein the image acquisition module, when executing acquiring the training image, comprises executing the following steps:

acquiring a medical image, wherein the medical image comprises a lesion area to be segmented;

performing region segmentation processing on the medical image to obtain the training image, wherein the training image has a target region corresponding to the lesion region in the medical image.

15. An ablation needle planning device comprising:

an image acquisition module configured to acquire an input image, wherein the input image includes a target region;

the processing module is configured to process the input image by using a neural network so as to obtain predicted needle arrangement information corresponding to the input image;

a planning module configured to derive an ablation needle arrangement plan based on the predicted needle arrangement information,

wherein the neural network is at least partially trained according to the training apparatus of any one of claims 1-14.

16. The ablation needle placement planning device of claim 15, wherein the predicted needle placement information comprises predicted confidence levels of N candidate needle placement points, predicted coordinate information of the N candidate needle placement points, and needle group probabilities corresponding to any two candidate needle placement points, N being a positive integer and greater than or equal to 2,

when the planning module executes the ablation needle arrangement scheme obtained based on the predicted needle arrangement information, the planning module executes the following steps:

acquiring a first threshold value and a second threshold value;

taking the candidate card distribution points with the prediction confidence degrees larger than the first threshold value as effective pins to obtain a plurality of effective pins;

acquiring size information of an input image, and determining actual position coordinates of the effective needles according to the size information of the input image and the predicted coordinate information of the effective needles;

taking the needle group with the probability greater than the second threshold value as an effective needle group to obtain at least one effective needle group,

wherein the ablation needle distribution scheme comprises the plurality of effective needles, the actual position coordinates of the plurality of effective needles and at least one effective needle group.

17. A method of training a neural network, comprising:

acquiring a training image, wherein the training image comprises a target area;

acquiring reference needle arrangement information corresponding to the training image, wherein the reference needle arrangement information is used for providing an ablation needle arrangement scheme serving as a standard;

processing the training image by using the neural network to obtain predicted needle arrangement information corresponding to the training image;

calculating a loss value corresponding to the neural network according to the reference needle arrangement information, the target area and the predicted needle arrangement information, and correcting parameters of the neural network based on the loss value when the loss value does not meet a preset convergence condition;

calculating a loss value corresponding to the neural network according to the reference needle arrangement information, the target area and the predicted needle arrangement information, and correcting parameters of the neural network based on the loss value when the loss value does not meet a preset convergence condition, wherein the method comprises the following steps of:

determining a predicted ablation region according to the predicted needle arrangement information;

determining an evaluation index according to the predicted ablation region and the target region, wherein the evaluation index is used for representing the scheme effectiveness of the predicted needle arrangement information;

calculating a middle loss value according to the reference needle arrangement information and the predicted needle arrangement information;

calculating a loss value corresponding to the neural network according to the intermediate loss value and the evaluation index;

wherein determining an evaluation index based on the predicted ablation region and the target region comprises:

18. An ablation needle placement planning method, comprising:

acquiring an input image, wherein the input image comprises a target area;

processing the input image by using a neural network to obtain predicted needle arrangement information corresponding to the input image;

obtaining an ablation needle arrangement scheme based on the predicted needle arrangement information;

19. An electronic device, comprising:

a memory non-transiently storing computer executable instructions;

a processor configured to execute the computer-executable instructions,

wherein the computer executable instructions, when executed by the processor, implement the training method of a neural network of claim 17 or the ablation needle planning method of claim 18.

20. A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions,

the computer executable instructions, when executed by a processor, implement the training method of a neural network of claim 17 or the ablation needle planning method of claim 18.