CN113298854B - Image registration method based on mark points - Google Patents

Image registration method based on mark points Download PDF

Info

Publication number
CN113298854B
CN113298854B CN202110585049.8A CN202110585049A CN113298854B CN 113298854 B CN113298854 B CN 113298854B CN 202110585049 A CN202110585049 A CN 202110585049A CN 113298854 B CN113298854 B CN 113298854B
Authority
CN
China
Prior art keywords
image
matrix
points
matching
dimensional
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110585049.8A
Other languages
Chinese (zh)
Other versions
CN113298854A (en
Inventor
朱德明
魏军
沈烁
田孟秋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Perception Vision Medical Technology Co ltd
Original Assignee
Perception Vision Medical Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Perception Vision Medical Technology Co ltd filed Critical Perception Vision Medical Technology Co ltd
Priority to CN202110585049.8A priority Critical patent/CN113298854B/en
Publication of CN113298854A publication Critical patent/CN113298854A/en
Priority to PCT/CN2022/070425 priority patent/WO2022247296A1/en
Application granted granted Critical
Publication of CN113298854B publication Critical patent/CN113298854B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G06T7/337—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G06T7/0014—Biomedical image inspection using an image reference approach
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G06T2207/10084—Hybrid tomography; Concurrent acquisition with multiple different tomographic modalities
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Radiology & Medical Imaging (AREA)
  • Quality & Reliability (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Medical Informatics (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Evolutionary Computation (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses an image registration method based on mark points, which mainly comprises the following steps: inputting medical images of two arbitrary modalities; extracting pyramid characteristics of two input images by adopting a pre-trained neural network, wherein the training process of the network comprises a plurality of different tasks and relates to the plurality of different input modes; extracting pyramid features by using the neural network, and obtaining a plurality of matching point pairs representing certain semantics between two images through the processes of searching, screening, matching and the like; and fitting a transformation matrix and a displacement vector of rigid registration by minimizing the sum of the distances of all the matched points to the points so as to obtain the medical image subjected to rigid registration. And on the basis of rigid registration, obtaining a non-rigid registered displacement field three-dimensional matrix by an interpolation method based on a radial basis, thereby obtaining a non-rigid registered medical image. Therefore, the problem of the lack of the standard of the mark point gold can be effectively solved.

Description

Image registration method based on mark points
Technical Field
The invention relates to the field of image processing, deep learning and medical treatment, in particular to an image registration method based on mark points.
Background
Image registration has numerous applications of practical value in medical image processing and analysis. With the advancement of medical imaging equipment, images of a variety of different modalities, such as CT, CBCT, MRI, PET, etc., containing accurate anatomical information can be acquired for the same patient. However, diagnosis by observing different images requires a spatial imagination and a subjective experience of a doctor. By adopting a correct image registration method, various information can be accurately fused into the same image, so that doctors can observe the focus and the structure from various angles more conveniently and more accurately. Meanwhile, the change conditions of the focus and the organ can be quantitatively analyzed by registering the dynamic images acquired at different moments, so that the medical diagnosis, the operation plan formulation and the radiotherapy plan are more accurate and reliable.
The traditional image registration method is based on the optimization solving problem of the similarity objective function, is easy to converge to a local minimum value, has poor registration effect on images of different modes, and consumes long time in the iterative solving process. The image registration method based on the mark points can solve the problems, but the acquisition of the gold standard of the mark points needs to consume a lot of time of doctors and experts, and the cost is high. In recent years, there has been a great interest in exploring diagnoses using artificial intelligence, and mathematical models that perform better than human medical experts have been established in some fields using AI algorithms. Therefore, it is reasonable to believe that the effect of image registration can be effectively improved by improving the traditional image registration method by using the AI algorithm.
The information disclosed in this background section is only for enhancement of understanding of the general background of the invention and should not be taken as an acknowledgement or any form of suggestion that this information forms the prior art already known to a person skilled in the art.
Disclosure of Invention
The invention aims to provide an image registration method based on mark points, which can improve the traditional image registration method by utilizing an AI algorithm and can effectively improve the image registration effect.
In order to achieve the above object, the present invention provides an image registration method based on a mark point, which mainly comprises the following steps: inputting medical images of two arbitrary modalities (CT, CBCT, MRI, PET, etc.), one as a fixed image and the other as a moving image; extracting pyramid characteristics of two input images by adopting a pre-trained neural network, wherein the training process of the network comprises a plurality of different tasks and relates to the plurality of different input modes; extracting pyramid features by using the neural network, and obtaining a plurality of matching point pairs representing certain semantics between two images through the processes of searching, screening, matching and the like; and fitting a transformation matrix and a displacement vector of rigid registration by minimizing the sum of the distances of all the matched points to the points so as to obtain the medical image subjected to rigid registration. And on the basis of rigid registration, obtaining a non-rigid registered displacement field three-dimensional matrix by an interpolation method based on a radial basis, thereby obtaining a non-rigid registered medical image.
In a preferred embodiment, extracting the pyramid features of two input images by using a pre-trained neural network comprises: the structure of the neural network is divided into a backbone network and a plurality of subsequent branch networks. The backbone network is shared among different tasks, and each branch network corresponds to one task. Finally, a backbone network is used for extracting the image features. The training process of neural networks involves a variety of different tasks and involves a variety of different input modalities, including but not limited to: CT-based segmentation of primary tumors of nasopharyngeal carcinoma (GTV), MRI-based segmentation of primary tumors of nasopharyngeal carcinoma, CT-based segmentation of primary tumors of cervical carcinoma, PET-based segmentation of primary tumors of lung, CT-based segmentation of Organs At Risk (OAR), MRI-based segmentation of organs at risk, CBCT-based segmentation of organs at risk, CT-based target detection of lung nodules, etc. And firstly training the neural network by using one task, simultaneously training the other input mode tasks, independently training the rest tasks, fixing the parameters of the backbone network during training, and finally training and finely adjusting all the parameters by all the tasks simultaneously.
In a preferred embodiment, the marker point-based image registration method further includes: the method comprises the following steps of extracting pyramid features by utilizing the neural network, and obtaining a plurality of matching point pairs representing certain semantics between two images through the processes of searching, screening, matching and the like: handle If(fixed image) and Im(moving image) inputting the pre-trained neural network, extracting pyramid feature maps (feature maps) of the two input images
Figure GDA0003343457220000031
And
Figure GDA0003343457220000032
wherein l is belonged to {1,2,3,4,5} represents the l-th level characteristic, and the larger the number is, the larger the tableThe deeper the number of layers shown, i.e. the smaller the feature size but with more high level semantics implied. The search for a matching point needs to be generated within a specific search range, which is set starting from l-5:
S5＝{(P5,Q5)}
wherein:
Figure GDA0003343457220000033
Figure GDA0003343457220000034
is IfThe nth search range of the l-th stage of (1), correspondingly
Figure GDA0003343457220000035
Is 1mN search range of the l stage, NlNumber of search ranges, S, of level llA set of multiple search range pairs for the l-th level. When l is 5, the search range is
Figure GDA0003343457220000036
And
Figure GDA0003343457220000037
of (2), i.e. N5＝1。
Search range is matched by
Figure GDA0003343457220000038
And
Figure GDA0003343457220000039
characteristic diagram of
Figure GDA00033434572200000310
And
Figure GDA00033434572200000311
is transformed to obtain
Figure GDA00033434572200000312
And
Figure GDA00033434572200000313
Figure GDA00033434572200000314
Figure GDA00033434572200000315
Figure GDA00033434572200000316
Figure GDA00033434572200000317
wherein the content of the first and second substances,
Figure GDA00033434572200000318
is shown in
Figure GDA00033434572200000319
A local feature map within the range of,
Figure GDA00033434572200000320
in order to obtain the transformed feature map,
Figure GDA00033434572200000321
is composed of
Figure GDA00033434572200000322
The average value of (a) of (b),
Figure GDA00033434572200000323
is composed of
Figure GDA00033434572200000324
Standard deviation of (D) and (D)
Figure GDA00033434572200000325
Figure GDA00033434572200000326
In the search area
Figure GDA00033434572200000327
And
Figure GDA00033434572200000328
searching matching point pair in the interior, when the following conditions are satisfied, two points plAnd q islFor matching point pairs:
Figure GDA0003343457220000041
Figure GDA0003343457220000042
Figure GDA0003343457220000043
that is to say if
Figure GDA0003343457220000044
Inner point plIn that
Figure GDA0003343457220000045
The point with the highest similarity when searching within the range is qlOtherwise, p is also truelAnd q islAre pairs of matching points. The similarity is calculated by the following formula:
Figure GDA0003343457220000046
wherein, epsilon (p)l) Is a point plA set of points of a neighborhood within the specified range.
All N of class llThe step of searching the matching points is respectively executed in each searching range, and the set Lambda of all matching point pairs is obtainedl：
Figure GDA0003343457220000047
For the matching point pair searched in the above step, the following filtering condition must be passed, i.e. the value of the point in the feature map (feature map) must be sufficiently large:
Figure GDA0003343457220000048
Figure GDA0003343457220000049
wherein the content of the first and second substances,
Figure GDA00033434572200000410
and gamma is a self-defined threshold value for the finally obtained matching point pair set.
To obtain
Figure GDA00033434572200000411
Then, the search range set of the upper level is obtained by the following formula,
Figure GDA0003343457220000051
Figure GDA0003343457220000052
wherein the content of the first and second substances,
Figure GDA0003343457220000053
is composed of
Figure GDA0003343457220000054
The number of the (c) component(s),
Figure GDA0003343457220000055
the coordinate of the point p is (p) relative to the neural network receptive field of level l-1 and level lx,py,pz). And after the search range set of the previous stage is obtained, repeating the steps to obtain a final output result
Figure GDA0003343457220000056
I.e. a set of matching point pairs for both images.
In a preferred embodiment, the marker point-based image registration method further includes: fitting a transformation matrix and a displacement vector of rigid registration by minimizing the sum of distances of all matching points to obtain a rigidly registered medical image, wherein the method comprises the following steps: after all the matching point pairs are obtained, the optimal solution of the transformation matrix and the displacement vector of the rigid registration is obtained by minimizing the following formula:
Figure GDA0003343457220000057
the optimal solution is as follows:
R＝(PTP)-1PTQ
A＝R[0:3,0:3]
b＝R[0:3,3]
wherein N is the number of matching point pairs, pnIs the n-th matching point of fixed image, qnThe image is a pixel point in the corresponding moving image. P is a matrix formed by all matched points of the fixed image and has the size of [ N,4 ]]That is, a matrix composed of N four-dimensional row vectors, the first three dimensions of the four dimensions are physical coordinates of the pixel points, and the fourth dimension is a fixed value of 1. Q is a matrix formed by all matching points of the moving image and has the size of [ N,4 ]]. The size of the matrix R is [4,4 ]]，R[0:3,0:3]The first 3 rows and the first 3 columns of the matrix R are taken as the size of [3,3 ]]A matrix of (1), R0: 3,3]A three-dimensional column vector is taken for the first 3 rows and column 3 of the matrix R. A and b are respectively the optimal solutions of transformation matrix and displacement vector. And finally obtaining the rigidly registered medical image transmitted image through A and b.
In a preferred embodiment, the marker point-based image registration method further includes: on the basis of rigid registration, obtaining a non-rigid registered displacement field three-dimensional matrix by an interpolation method based on a radial basis, thereby obtaining a non-rigid registered medical image, comprising the following steps: the size of the displacement field three-dimensional matrix is the same as the fixed image. After N matching point pairs are obtained, the value of the residual pixel point of the displacement field matrix is obtained by adopting the following interpolation method:
Figure GDA0003343457220000061
A＝(a1,a2,a3)
G(r)＝r2lnr
where p is the coordinate in the three-dimensional matrix of the displacement field of (x)p,yp,zp) Pixel of (b), pnIs the nth matching point in the fixed image. G () is a radial basis function. A. b, wnThe value of (d) is solved in the following way:
setting:
Figure GDA0003343457220000062
rij＝||pi-pj||
V＝(v1,v2,…,vN,0,0,0,0)
vn＝(qn-pn)[k]k∈(0,1,2)
Figure GDA0003343457220000063
Ω＝(w1,w2,…,wN,b,a1,a2,a3)
wherein P is a matrix formed by all matching points of the fixed image,size of [ N,4 ]]That is, a matrix composed of n four-dimensional row vectors, the rear three-dimensional of the four-dimensional is the physical coordinate of the pixel point, and the first dimension is a fixed value of 1. q. q.snIs pnThe corresponding matching point in the moving image. Since the displacement value is a three-dimensional vector (x, y, z direction), the above solving process takes the x axis when k is 0, the y axis when k is 1, and the z axis when k is 2.
By vn＝f(pn) Comprises the following steps:
V＝LΩT
further, the values of all parameters to be solved are obtained:
Ω＝(L-1V)T
and the displacement values of the pixel points except the matching points in the displacement field can be obtained through f () fitting. Since the displacement values are three-dimensional vectors, i.e. x, y, z directions, the above interpolation process needs to be repeated 3 times, i.e. once for each direction. And finally obtaining a displacement field three-dimensional matrix so as to obtain the non-rigid registered medical image.
Compared with the prior art, the image registration method based on the mark points has the following beneficial effects: (1) the invention can register images of any two modalities. (2) The invention adopts a pre-trained neural network to extract the image characteristics, the training process of the network comprises a plurality of different tasks and relates to a plurality of different input modes, and the effectiveness and the universality of the characteristics can be effectively improved. (3) The invention utilizes a pre-trained neural network to extract image characteristics, obtains a plurality of matching point pairs representing certain semantics between two images through the processes of searching, screening, matching and the like, and can effectively solve the problem of gold standard shortage of the marking points. (4) The invention solves the transformation matrix and the displacement vector by minimizing the sum of the distances between all the matching points, thereby realizing rigid registration. (5) The method solves the displacement field three-dimensional matrix through an interpolation method based on the radial basis to realize non-rigid registration.
Drawings
Fig. 1 is a flowchart illustrating an image registration method based on a marker according to an embodiment of the present invention.
Detailed Description
The following detailed description of the present invention is provided in conjunction with the accompanying drawings, but it should be understood that the scope of the present invention is not limited to the specific embodiments.
Throughout the specification and claims, unless explicitly stated otherwise, the word "comprise", or variations such as "comprises" or "comprising", will be understood to imply the inclusion of a stated element or component but not the exclusion of any other element or component.
As shown in fig. 1, an image registration method based on a marker point according to a preferred embodiment of the present invention includes the following steps:
inputting medical images of two arbitrary modalities (CT, CBCT, MRI, PET, etc.), one as a fixed image and the other as a moving image;
extracting pyramid characteristics of two input images by adopting a pre-trained neural network, wherein the training process of the network comprises a plurality of different tasks and relates to the plurality of different input modes;
extracting pyramid features by using the neural network, and obtaining a plurality of matching point pairs representing certain semantics between two images through the processes of searching, screening, matching and the like;
and fitting a transformation matrix and a displacement vector of rigid registration by minimizing the sum of the distances of all the matched points to the points so as to obtain the medical image subjected to rigid registration.
On the basis of rigid registration, a non-rigid registered displacement field three-dimensional matrix is obtained through an interpolation method based on a radial basis, and thus a non-rigid registered medical image is obtained.
The work flow of the specific implementation of the image registration method based on the mark points comprises the following steps:
in some embodiments, step S1, constructing a pre-trained neural network to extract pyramid features of two input images;
step S1 specifically includes the following steps:
and S11, dividing the structure of the neural network into a backbone network and a plurality of subsequent branch networks. The backbone network is shared among different tasks, and each branch network corresponds to one task. Finally, a backbone network is used for extracting the image features.
S12, the training process of the neural network includes a variety of different tasks and involves a variety of different input modalities, including but not limited to: CT-based segmentation of primary tumors of nasopharyngeal carcinoma (GTV), MRI-based segmentation of primary tumors of nasopharyngeal carcinoma, CT-based segmentation of primary tumors of cervical carcinoma, PET-based segmentation of primary tumors of lung, CT-based segmentation of Organs At Risk (OAR), MRI-based segmentation of organs at risk, CBCT-based segmentation of organs at risk, CT-based target detection of lung nodules, etc.
And S13, firstly, training the neural network by using one task, simultaneously training the other input mode tasks, then independently training the rest tasks, fixing the parameters of the backbone network during training, and finally, simultaneously training and finely adjusting all the parameters by all the tasks.
In some embodiments, the marker point-based image registration method further includes:
s2, extracting pyramid features by using the neural network, and obtaining a plurality of matching point pairs representing certain semantics between two images through the processes of searching, screening, matching and the like;
step S2 specifically includes the following steps:
s21, If(fixed image) and Im(moving image) inputting the pre-trained neural network, extracting pyramid feature maps (feature maps) of the two input images
Figure GDA0003343457220000091
And
Figure GDA0003343457220000092
wherein l is belonged to {1,2,3,4,5} represents the l-th level feature, and the larger the number is, the deeper the layer number is, namely, the smaller the feature size is, but the higher level semantic meaning is included.
S22, the search for the matching point needs to be generated within a specific search range, and the search range is set from l to 5:
S5＝{(P5,Q5)}
wherein:
Figure GDA0003343457220000093
Figure GDA0003343457220000094
is IfThe nth search range of the l-th stage of (1), correspondingly
Figure GDA0003343457220000095
Is 1mN search range of the l stage, NlNumber of search ranges, S, of level llA set of multiple search range pairs for the l-th level. When l is 5, the search range is
Figure GDA0003343457220000096
And
Figure GDA0003343457220000097
of (2), i.e. N5＝1。
S23, search range is matched by the following formula
Figure GDA0003343457220000101
And
Figure GDA0003343457220000102
characteristic diagram of
Figure GDA0003343457220000103
And
Figure GDA0003343457220000104
is transformed to obtain
Figure GDA0003343457220000105
And
Figure GDA0003343457220000106
Figure GDA0003343457220000107
Figure GDA0003343457220000108
Figure GDA0003343457220000109
Figure GDA00033434572200001010
wherein the content of the first and second substances,
Figure GDA00033434572200001011
is shown in
Figure GDA00033434572200001012
A local feature map within the range of,
Figure GDA00033434572200001013
in order to obtain the transformed feature map,
Figure GDA00033434572200001014
is composed of
Figure GDA00033434572200001015
The average value of (a) of (b),
Figure GDA00033434572200001016
is composed of
Figure GDA00033434572200001017
Standard deviation of (D) and (D)
Figure GDA00033434572200001018
Figure GDA00033434572200001019
S24, search for the scope
Figure GDA00033434572200001020
And
Figure GDA00033434572200001021
searching matching point pair in the interior, when the following conditions are satisfied, two points plAnd q islFor matching point pairs:
Figure GDA00033434572200001022
Figure GDA00033434572200001023
Figure GDA00033434572200001024
that is to say if
Figure GDA00033434572200001025
Inner point plIn that
Figure GDA00033434572200001026
The point with the highest similarity when searching within the range is qlOtherwise, p is also truelAnd q islAre pairs of matching points. The similarity is calculated by the following formula:
Figure GDA00033434572200001027
wherein, epsilon (p)l) Is a point plA set of points of a neighborhood within the specified range.
S25, all N of the l-th stagelThe above steps S23-S24 of searching the matching points are performed for each search range, respectivelySet Λ to all matching point pairsl：
Figure GDA0003343457220000111
S26, for the matching point pair searched in the above step, the value of the point in the feature map (feature map) must be large enough to pass the following filtering condition:
Figure GDA0003343457220000112
Figure GDA0003343457220000113
wherein the content of the first and second substances,
Figure GDA0003343457220000114
and gamma is a custom threshold value which is 0.05.
S27, obtaining
Figure GDA0003343457220000115
Then, the search range set of the upper level is obtained by the following formula,
Figure GDA0003343457220000116
Figure GDA0003343457220000117
wherein the content of the first and second substances,
Figure GDA0003343457220000118
is composed of
Figure GDA0003343457220000119
The number of the (c) component(s),
Figure GDA00033434572200001110
the coordinate of the point p is (p) relative to the neural network receptive field of level l-1 and level lx,py,pz)。
S28, after the search range set of the previous level is obtained, for each search range, jumping to the step S23 and repeating the steps S23-S28 to obtain the final output result
Figure GDA00033434572200001111
I.e. a set of matching point pairs for both images.
In some embodiments, the marker point-based image registration method further includes:
s3, fitting a transformation matrix and a displacement vector of rigid registration by minimizing the sum of distances between all matching points, so as to obtain a medical image subjected to rigid registration;
step S3 specifically includes the following steps:
s31, obtaining the optimal solution of the transformation matrix and the displacement vector of the rigid registration by minimizing the following formula after obtaining all the matching point pairs:
Figure GDA0003343457220000121
the optimal solution is as follows:
R＝(PTP)-1PTQ
A＝R[0:3,0:3]
b＝R[0:3,3]
wherein N is the number of matching point pairs, pnIs the n-th matching point of fixed image, qnThe image is a pixel point in the corresponding moving image. P is a matrix formed by all matched points of the fixed image and has the size of [ N,4 ]]That is, a matrix composed of N four-dimensional row vectors, the first three dimensions of the four dimensions are physical coordinates of the pixel points, and the fourth dimension is a fixed value of 1. Q is a matrix formed by all matching points of the moving image and has the size of [ N,4 ]]. The size of the matrix R is [4,4 ]]，R[0:3,0:3]The first 3 rows and the first 3 columns of the matrix R are taken as the size of [3,3 ]]The matrix of (a) is,R[0:3,3]a three-dimensional column vector is taken for the first 3 rows and column 3 of the matrix R. A and b are the optimal solutions of the transformation matrix and the displacement vector, respectively.
And S32, obtaining a rigidly registered medical image warp image through A and b.
In some embodiments, the marker point-based image registration method further includes:
s4, obtaining a non-rigid registration displacement field three-dimensional matrix through an interpolation method based on a radial basis on the basis of rigid registration, so as to obtain a non-rigid registration medical image;
step S4 specifically includes the following steps:
s41, the size of the displacement field three-dimensional matrix is the same as the fixed image. After N matching point pairs are obtained, the value of the residual pixel point of the displacement field matrix is obtained by adopting the following interpolation method:
Figure GDA0003343457220000122
A＝(a1,a2,a3)
G(r)＝r2lnr
where p is the coordinate in the three-dimensional matrix of the displacement field of (x)p,yp,zp) Pixel of (b), pnIs the nth matching point in the fixed image. G () is a radial basis function. A. b, wnThe value of (d) is solved in the following way:
setting:
Figure GDA0003343457220000131
rij＝||pi-pj||
V＝(v1,v2,…,vN,0,0,0,0)
vn＝(qn-pn)[k]k∈(0,1,2)
Figure GDA0003343457220000132
Ω＝(w1,w2,…,wN,b,a1,a2,a3)
wherein P is a matrix formed by all matched points of the fixed image and has the size of [ N,4 ]]That is, a matrix composed of n four-dimensional row vectors, the rear three-dimensional of the four-dimensional is the physical coordinate of the pixel point, and the first dimension is a fixed value of 1. q. q.snIs pnThe corresponding matching point in the moving image. Since the displacement value is a three-dimensional vector (x, y, z direction), the above solving process takes the x axis when k is 0, the y axis when k is 1, and the z axis when k is 2.
By vn＝f(pn) Comprises the following steps:
V＝LΩT
further, the values of all parameters to be solved are obtained:
Ω＝(L-1V)T
and the displacement values of the pixel points except the matching points in the displacement field can be obtained through f () fitting.
S42, since the displacement values are three-dimensional vectors, i.e. x, y, and z directions, the interpolation process needs to be repeated 3 times, i.e. each direction is performed once in step S41.
And S43, finally obtaining a displacement field three-dimensional matrix, thereby obtaining the non-rigid registered medical image.
In summary, the image registration method based on the mark points has the following advantages: (1) the invention can register images of any two modalities. (2) The invention adopts a pre-trained neural network to extract the image characteristics, the training process of the network comprises a plurality of different tasks and relates to a plurality of different input modes, and the effectiveness and the universality of the characteristics can be effectively improved. (3) The invention utilizes a pre-trained neural network to extract image characteristics, obtains a plurality of matching point pairs representing certain semantics between two images through the processes of searching, screening, matching and the like, and can effectively solve the problem of gold standard shortage of the marking points. (4) The invention solves the transformation matrix and the displacement vector by minimizing the sum of the distances between all the matching points, thereby realizing rigid registration. (5) The method solves the displacement field three-dimensional matrix through an interpolation method based on the radial basis to realize non-rigid registration.
The foregoing descriptions of specific exemplary embodiments of the present invention have been presented for purposes of illustration and description. It is not intended to limit the invention to the precise form disclosed, and obviously many modifications and variations are possible in light of the above teaching. The exemplary embodiments were chosen and described in order to explain certain principles of the invention and its practical application to enable one skilled in the art to make and use various exemplary embodiments of the invention and various alternatives and modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the claims and their equivalents.

Claims (5)

1. An image registration method based on a mark point is characterized by comprising the following steps:
inputting two medical images of any modality, wherein one medical image is used as a fixed image, and the other medical image is used as a moving image;
extracting pyramid characteristics of the medical image of two input arbitrary modes by adopting a pre-trained neural network, wherein the training process of the neural network comprises a plurality of tasks and relates to a plurality of input modes;
obtaining a plurality of matching point pairs representing certain semantics between two images by utilizing the pyramid characteristics extracted by the neural network through searching, screening and matching processes;
fitting a transformation matrix and a displacement vector of rigid registration by minimizing the sum of distances of all matching points to obtain a rigidly registered medical image; and
on the basis of rigid registration, obtaining a non-rigid registered displacement field three-dimensional matrix by an interpolation method based on a radial basis, thereby obtaining a non-rigid registered medical image;
wherein the extracting the pyramid features of the medical image of two input arbitrary modalities using one pre-trained neural network comprises: the structure of the neural network is divided into a backbone network and a plurality of subsequent branch networks; the backbone network is shared among different tasks, and each branch network corresponds to one task; finally, the backbone network is used for extracting image features; and
firstly, the neural network trained by one task and the other input mode tasks are trained simultaneously, then each of the rest tasks is trained independently, the parameters of the backbone network are fixed during training, and finally all the tasks are trained simultaneously to fine tune all the parameters.
2. The method of claim 1, wherein the extracting the pyramid features of the medical image of two input arbitrary modalities with one pre-trained neural network further comprises:
the training process of the neural network involves a variety of different tasks and involves a variety of different input modalities, including: CT-based segmentation of primary tumors of nasopharyngeal carcinoma, MRI-based segmentation of primary tumors of nasopharyngeal carcinoma, CT-based segmentation of primary tumors of cervical carcinoma, PET-based segmentation of primary tumors of lung, CT-based segmentation of organs-at-risk, MRI-based segmentation of organs-at-risk, CBCT-based segmentation of organs-at-risk, and CT-based target detection of lung nodules.
3. The method for registering images based on labeled points as claimed in claim 1, wherein said pyramid features extracted by said neural network are used to obtain a plurality of matching point pairs representing a certain semantic meaning between two images through a searching, screening and matching process, comprising the following steps:
handle IfAnd ImInputting the pre-trained neural network, and extracting pyramid feature maps of two input images
Figure FDA0003368699270000021
And
Figure FDA0003368699270000022
wherein l belongs to {1,2,3,4,5} represents the l-th level feature, the larger the number is, the deeper the layer number is, namely, the smaller the feature size is, but more high-level semantics are contained; wherein IfIs fixed image, ImNamely, the moving image;
the search for a matching point needs to be generated within a specific search range, which is set starting from l-5:
S5＝{(P5,Q5)}
wherein:
Figure FDA0003368699270000023
Figure FDA0003368699270000024
is IfThe nth search range of the l-th stage of (1), correspondingly
Figure FDA0003368699270000025
Is ImN search range of the l stage, NlNumber of search ranges, S, of level llA set of a plurality of search range pairs for the l-th level; when l is 5, the search range is
Figure FDA0003368699270000026
And
Figure FDA0003368699270000027
of (2), i.e. N5＝1；
Search range is matched by
Figure FDA0003368699270000028
And
Figure FDA0003368699270000029
characteristic diagram of
Figure FDA00033686992700000210
And
Figure FDA00033686992700000211
is transformed to obtain
Figure FDA00033686992700000212
And
Figure FDA00033686992700000213
Figure FDA00033686992700000214
Figure FDA00033686992700000215
Figure FDA00033686992700000216
Figure FDA00033686992700000217
wherein the content of the first and second substances,
Figure FDA0003368699270000031
is shown in
Figure FDA0003368699270000032
A local feature map within the range of,
Figure FDA0003368699270000033
in order to obtain the transformed feature map,
Figure FDA0003368699270000034
is composed of
Figure FDA0003368699270000035
The average value of (a) of (b),
Figure FDA0003368699270000036
is composed of
Figure FDA0003368699270000037
Standard deviation of (D) and (D)
Figure FDA0003368699270000038
Figure FDA0003368699270000039
In the search area
Figure FDA00033686992700000310
And
Figure FDA00033686992700000311
searching matching point pair in the interior, when the following conditions are satisfied, two points plAnd q islFor matching point pairs:
Figure FDA00033686992700000312
Figure FDA00033686992700000313
Figure FDA00033686992700000314
that is to say if
Figure FDA00033686992700000315
Inner point plIn that
Figure FDA00033686992700000316
The point with the highest similarity when searching within the range is qlOtherwise, p is also truelAnd q islIs a matching point pair; the similarity is calculated by the following formula:
Figure FDA00033686992700000317
wherein, epsilon (p)l) Is a point plA set of points of a neighborhood within the specified range of (a);
all N of class llThe step of searching the matching points is respectively executed in each searching range, and the set Lambda of all matching point pairs is obtainedl：
Figure FDA00033686992700000318
For the matching point pair searched in the above steps, the following screening condition must be passed, i.e. the value of the point in the feature map must be large enough:
Figure FDA00033686992700000319
Figure FDA00033686992700000320
wherein the content of the first and second substances,
Figure FDA0003368699270000041
gamma is a self-defined threshold value for the finally obtained matching point pair set;
to obtain
Figure FDA0003368699270000042
Then, the search range set of the previous level is obtained by the following formula:
Figure FDA0003368699270000043
Figure FDA0003368699270000044
wherein the content of the first and second substances,
Figure FDA0003368699270000045
is composed of
Figure FDA0003368699270000046
The number of the (c) component(s),
Figure FDA0003368699270000047
the coordinate of the point p is (p) relative to the neural network receptive field of level l-1 and level lx,py,pz)，ε(ql) Is a point qlOf the neighborhood within the specified range, epsilon (q)l) And epsilon (p)l) To correspond to a concept, plAnd q islMatching point pairs of the two graphs at the l level; and
after the search range set of the previous level is obtained, the steps are repeated to obtain the final output result
Figure FDA0003368699270000048
I.e. a set of matching point pairs for both images.
4. The image registration method based on the marker points as claimed in claim 1, wherein the rigidly registered medical image forward image is obtained by fitting a transformation matrix and a displacement vector of the rigid registration by minimizing the sum of the distances between all the matching points, comprising the following steps:
after all the matching point pairs are obtained, the optimal solution of the transformation matrix and the displacement vector of the rigid registration is obtained by minimizing the following formula:
Figure FDA0003368699270000049
the optimal solution is as follows:
R＝(PTP)-1PTQ
A＝R[0:3,0:3]
b＝R[0:3,3]
wherein N is the number of matching point pairs, pnIs the n-th matching point of fixed image, qnThe pixel points in the corresponding moving image are obtained; p is a matrix formed by all matched points of the fixed image and has the size of [ N,4 ]]The matrix is composed of N four-dimensional row vectors, the front three-dimensional of the four-dimensional is the physical coordinate of the pixel point, and the fourth dimension is a fixed value 1; q is a matrix formed by all matching points of the moving image and has the size of [ N,4 ]](ii) a The size of the matrix R is [4,4 ]]，R[0:3,0:3]The first 3 rows and the first 3 columns of the matrix R are taken as the size of [3,3 ]]A matrix of (1), R0: 3,3]The three-dimensional column vector of the 3 rd column of the first 3 rows of the matrix R is taken; a and b are respectively the optimal solutions of the transformation matrix and the displacement vector; and
and finally, obtaining a rigidly registered medical image transmitted image through A and b.
5. The image registration method based on the marker points as claimed in claim 1, wherein based on the rigid registration, a non-rigid registered displacement field three-dimensional matrix is obtained by interpolation based on radial basis, so as to obtain a non-rigid registered medical image, comprising the following steps:
the size of the displacement field three-dimensional matrix is the same as the fixed image; after N matching point pairs are obtained, the values of the residual pixel points of the displacement field matrix are obtained by adopting the following interpolation method:
Figure FDA0003368699270000051
A＝(a1,a2,a3)
G(r)＝r2lnr
wherein p is the coordinate in the three-dimensional matrix of the displacement field of (x)p,yp,zp) Pixel of (b), pnThe nth matching point in the fixed image; g () is a radial basis function; A. b, wnThe value of (d) is solved in the following way:
setting:
Figure FDA0003368699270000061
rij＝||pi-pj||
V＝(v1,v2,…,vN,0,0,0,0)
vn＝(qn-pn)[k]k∈(0,1,2)
Figure FDA0003368699270000062
Ω＝(w1,w2,…,wN,b,a1,a2,a3)
wherein P is a matrix formed by all matched points of the fixed image and has the size of [ N,4 ]]The matrix is composed of n four-dimensional row vectors, the back three-dimensional of the four-dimensional is the physical coordinate of the pixel point, and the first dimension is a fixed value 1; q. q.snIs pnThe corresponding matching point in the moving image; k is an expression dimension, that is, since the displacement value is a three-dimensional vector (x, y, z direction), the solving process only aims at one dimension, so the x axis is taken when k is 0, the y axis is taken when k is 1, and the z axis is taken when k is 2;
by vn＝f(pn) Comprises the following steps:
V＝LΩT
further, the values of all parameters to be solved are obtained:
Ω＝(L-1V)T
the displacement values of the pixel points except the matching points in the displacement field can be obtained through f () fitting;
since the displacement values are three-dimensional vectors, i.e. x, y, z directions, the above interpolation process needs to be repeated 3 times, i.e. each direction is performed once; and
and finally, obtaining a displacement field three-dimensional matrix so as to obtain a non-rigid registered medical image.
CN202110585049.8A 2021-05-27 2021-05-27 Image registration method based on mark points Active CN113298854B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202110585049.8A CN113298854B (en) 2021-05-27 2021-05-27 Image registration method based on mark points
PCT/CN2022/070425 WO2022247296A1 (en) 2021-05-27 2022-01-06 Mark point-based image registration method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110585049.8A CN113298854B (en) 2021-05-27 2021-05-27 Image registration method based on mark points

Publications (2)

Publication Number Publication Date
CN113298854A CN113298854A (en) 2021-08-24
CN113298854B true CN113298854B (en) 2022-02-01

Family

ID=77325584

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110585049.8A Active CN113298854B (en) 2021-05-27 2021-05-27 Image registration method based on mark points

Country Status (2)

Country Link
CN (1) CN113298854B (en)
WO (1) WO2022247296A1 (en)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113298854B (en) * 2021-05-27 2022-02-01 广州柏视医疗科技有限公司 Image registration method based on mark points
CN113744328B (en) * 2021-11-05 2022-02-15 极限人工智能有限公司 Medical image mark point identification method and device, electronic equipment and storage medium
CN113920178B (en) * 2021-11-09 2022-04-12 广州柏视医疗科技有限公司 Mark point-based multi-vision 2D-3D image registration method and system
CN113920179B (en) * 2021-11-09 2022-04-29 广州柏视医疗科技有限公司 Mark point-based multi-vision 2D-3D image non-rigid registration method and system
CN114404039B (en) * 2021-12-30 2023-05-05 华科精准（北京）医疗科技有限公司 Tissue drift correction method and device for three-dimensional model, electronic equipment and storage medium
CN114241077B (en) * 2022-02-23 2022-07-15 南昌睿度医疗科技有限公司 CT image resolution optimization method and device

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104021547A (en) * 2014-05-17 2014-09-03 清华大学深圳研究生院 Three dimensional matching method for lung CT
CN106097347A (en) * 2016-06-14 2016-11-09 福州大学 A kind of multimodal medical image registration and method for visualizing
CN111931929A (en) * 2020-07-29 2020-11-13 深圳地平线机器人科技有限公司 Training method and device of multi-task model and storage medium

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170337682A1 (en) * 2016-05-18 2017-11-23 Siemens Healthcare Gmbh Method and System for Image Registration Using an Intelligent Artificial Agent
CN107818564B (en) * 2017-10-27 2020-10-30 深圳市一图智能科技有限公司 Liver 3D medical image segmentation method
CN109064502B (en) * 2018-07-11 2021-10-22 西北工业大学 Multi-source image registration method based on combination of deep learning and artificial design features
CN111091589B (en) * 2019-11-25 2023-11-17 北京理工大学 Ultrasonic and nuclear magnetic image registration method and device based on multi-scale supervised learning
CN111640143B (en) * 2020-04-12 2023-05-30 复旦大学 Neural navigation rapid surface registration method and system based on PointNet
CN113298854B (en) * 2021-05-27 2022-02-01 广州柏视医疗科技有限公司 Image registration method based on mark points

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104021547A (en) * 2014-05-17 2014-09-03 清华大学深圳研究生院 Three dimensional matching method for lung CT
CN106097347A (en) * 2016-06-14 2016-11-09 福州大学 A kind of multimodal medical image registration and method for visualizing
CN111931929A (en) * 2020-07-29 2020-11-13 深圳地平线机器人科技有限公司 Training method and device of multi-task model and storage medium

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Feature-Based Retinal Image Registration Using D-Saddle Feature;Pamli 等;《JOURNAL OF HEALTHCARE ENGINEERING》;20171231;第1-15页 *
医学图像非刚性配准方法及系统研究;王伟;《中国优秀硕士学位论文全文数据库 (医药卫生科技辑)》;20200615;正文第2-5章 *

Also Published As

Publication number Publication date
CN113298854A (en) 2021-08-24
WO2022247296A1 (en) 2022-12-01

Similar Documents

Publication Publication Date Title
CN113298854B (en) Image registration method based on mark points
Jiao et al. Learning with limited annotations: a survey on deep semi-supervised learning for medical image segmentation
Cao et al. Deformable image registration using a cue-aware deep regression network
Liu et al. MS-Net: multi-site network for improving prostate segmentation with heterogeneous MRI data
Fan et al. Adversarial learning for mono-or multi-modal registration
Yu et al. Segmentation of fetal left ventricle in echocardiographic sequences based on dynamic convolutional neural networks
Cao et al. Region-adaptive deformable registration of CT/MRI pelvic images via learning-based image synthesis
Zhu et al. Exploiting interslice correlation for MRI prostate image segmentation, from recursive neural networks aspect
CN113298855B (en) Image registration method based on automatic delineation
CN113826143A (en) Feature point detection
Turan et al. A deep learning based 6 degree-of-freedom localization method for endoscopic capsule robots
WO2023030281A1 (en) Training method and apparatus, image processing method, electronic device and storage medium
Jin et al. Object recognition in medical images via anatomy-guided deep learning
Lyu et al. Attention to fine-grained information: hierarchical multi-scale network for retinal vessel segmentation
Zhu et al. Similarity attention-based CNN for robust 3D medical image registration
Zhao et al. MSKD: Structured knowledge distillation for efficient medical image segmentation
Song et al. Learning 3d features with 2d cnns via surface projection for ct volume segmentation
Dinh A novel approach using the local energy function and its variations for medical image fusion
Zhang et al. Multimodal medical volumes translation and segmentation with generative adversarial network
CN116883462A (en) Medical image registration method based on LOFTR network model and improved particle swarm optimization
Schwab et al. Multimodal medical image registration using particle swarm optimization with influence of the data's initial orientation
Dickson et al. An integrated 3D-sparse deep belief network with enriched seagull optimization algorithm for liver segmentation
Wang et al. Automatic carotid artery detection using attention layer region-based convolution neural network
Fan et al. Deep feature descriptor based hierarchical dense matching for X-ray angiographic images
Law et al. Quasiconformal model with CNN features for large deformation image registration

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant