CN113298854A - Image registration method based on mark points - Google Patents

Image registration method based on mark points Download PDF

Info

Publication number
CN113298854A
CN113298854A CN202110585049.8A CN202110585049A CN113298854A CN 113298854 A CN113298854 A CN 113298854A CN 202110585049 A CN202110585049 A CN 202110585049A CN 113298854 A CN113298854 A CN 113298854A
Authority
CN
China
Prior art keywords
image
matrix
points
matching
dimensional
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110585049.8A
Other languages
Chinese (zh)
Other versions
CN113298854B (en
Inventor
朱德明
魏军
沈烁
田孟秋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangzhou Boshi Medical Technology Co ltd
Original Assignee
Guangzhou Boshi Medical Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangzhou Boshi Medical Technology Co ltd filed Critical Guangzhou Boshi Medical Technology Co ltd
Priority to CN202110585049.8A priority Critical patent/CN113298854B/en
Publication of CN113298854A publication Critical patent/CN113298854A/en
Priority to PCT/CN2022/070425 priority patent/WO2022247296A1/en
Application granted granted Critical
Publication of CN113298854B publication Critical patent/CN113298854B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G06T7/337—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G06T7/0014—Biomedical image inspection using an image reference approach
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G06T2207/10084—Hybrid tomography; Concurrent acquisition with multiple different tomographic modalities
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Radiology & Medical Imaging (AREA)
  • Quality & Reliability (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Medical Informatics (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Evolutionary Computation (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses an image registration method based on mark points, which mainly comprises the following steps: inputting medical images of two arbitrary modalities; extracting pyramid characteristics of two input images by adopting a pre-trained neural network, wherein the training process of the network comprises a plurality of different tasks and relates to the plurality of different input modes; extracting pyramid features by using the neural network, and obtaining a plurality of matching point pairs representing certain semantics between two images through the processes of searching, screening, matching and the like; and fitting a transformation matrix and a displacement vector of rigid registration by minimizing the sum of the distances of all the matched points to the points so as to obtain the medical image subjected to rigid registration. And on the basis of rigid registration, obtaining a non-rigid registered displacement field three-dimensional matrix by an interpolation method based on a radial basis, thereby obtaining a non-rigid registered medical image. Therefore, the problem of the lack of the standard of the mark point gold can be effectively solved.

Description

Image registration method based on mark points
Technical Field
The invention relates to the field of image processing, deep learning and medical treatment, in particular to an image registration method based on mark points.
Background
Image registration has numerous applications of practical value in medical image processing and analysis. With the advancement of medical imaging equipment, images of a variety of different modalities, such as CT, CBCT, MRI, PET, etc., containing accurate anatomical information can be acquired for the same patient. However, diagnosis by observing different images requires a spatial imagination and a subjective experience of a doctor. By adopting a correct image registration method, various information can be accurately fused into the same image, so that doctors can observe the focus and the structure from various angles more conveniently and more accurately. Meanwhile, the change conditions of the focus and the organ can be quantitatively analyzed by registering the dynamic images acquired at different moments, so that the medical diagnosis, the operation plan formulation and the radiotherapy plan are more accurate and reliable.
The traditional image registration method is based on the optimization solving problem of the similarity objective function, is easy to converge to a local minimum value, has poor registration effect on images of different modes, and consumes long time in the iterative solving process. The image registration method based on the mark points can solve the problems, but the acquisition of the gold standard of the mark points needs to consume a lot of time of doctors and experts, and the cost is high. In recent years, there has been a great interest in exploring diagnoses using artificial intelligence, and mathematical models that perform better than human medical experts have been established in some fields using AI algorithms. Therefore, it is reasonable to believe that the effect of image registration can be effectively improved by improving the traditional image registration method by using the AI algorithm.
The information disclosed in this background section is only for enhancement of understanding of the general background of the invention and should not be taken as an acknowledgement or any form of suggestion that this information forms the prior art already known to a person skilled in the art.
Disclosure of Invention
The invention aims to provide an image registration method based on mark points, which can improve the traditional image registration method by utilizing an AI algorithm and can effectively improve the image registration effect.
In order to achieve the above object, the present invention provides an image registration method based on a mark point, which mainly comprises the following steps: inputting medical images of two arbitrary modalities (CT, CBCT, MRI, PET, etc.), one as a fixed image and the other as a moving image; extracting pyramid characteristics of two input images by adopting a pre-trained neural network, wherein the training process of the network comprises a plurality of different tasks and relates to the plurality of different input modes; extracting pyramid features by using the neural network, and obtaining a plurality of matching point pairs representing certain semantics between two images through the processes of searching, screening, matching and the like; and fitting a transformation matrix and a displacement vector of rigid registration by minimizing the sum of the distances of all the matched points to the points so as to obtain the medical image subjected to rigid registration. And on the basis of rigid registration, obtaining a non-rigid registered displacement field three-dimensional matrix by an interpolation method based on a radial basis, thereby obtaining a non-rigid registered medical image.
In a preferred embodiment, extracting the pyramid features of two input images by using a pre-trained neural network comprises: the structure of the neural network is divided into a backbone network and a plurality of subsequent branch networks. The backbone network is shared among different tasks, and each branch network corresponds to one task. Finally, a backbone network is used for extracting the image features. The training process of neural networks involves a variety of different tasks and involves a variety of different input modalities, including but not limited to: CT-based segmentation of primary tumors of nasopharyngeal carcinoma (GTV), MRI-based segmentation of primary tumors of nasopharyngeal carcinoma, CT-based segmentation of primary tumors of cervical carcinoma, PET-based segmentation of primary tumors of lung, CT-based segmentation of Organs At Risk (OAR), MRI-based segmentation of organs at risk, CBCT-based segmentation of organs at risk, CT-based target detection of lung nodules, etc. And firstly training the neural network by using one task, simultaneously training the other input mode tasks, independently training the rest tasks, fixing the parameters of the backbone network during training, and finally training and finely adjusting all the parameters by all the tasks simultaneously.
In a preferred embodiment, the image registration method based on the mark points further comprisesComprises the following steps: the method comprises the following steps of extracting pyramid features by utilizing the neural network, and obtaining a plurality of matching point pairs representing certain semantics between two images through the processes of searching, screening, matching and the like: handle If(fixed image) and Im(moving image) inputting the pre-trained neural network, extracting pyramid feature maps (feature maps) of the two input images
Figure BDA0003087816210000031
And
Figure BDA0003087816210000032
wherein l is belonged to {1,2,3,4,5} represents the l-th level feature, and the larger the number is, the deeper the layer number is, namely, the smaller the feature size is, but the higher level semantic meaning is included. The search for a matching point needs to be generated within a specific search range, which is set starting from l-5:
S5＝{(P5，Q5)}
wherein:
Figure BDA0003087816210000033
Figure BDA0003087816210000034
is IfThe nth search range of the l-th stage of (1), correspondingly
Figure BDA0003087816210000035
Is ImN search range of the l stage, NlNumber of search ranges, S, of level llA set of multiple search range pairs for the l-th level. When l is 5, the search range is
Figure BDA0003087816210000036
And
Figure BDA0003087816210000037
of (2), i.e. N5＝1。
Search range is matched by
Figure BDA0003087816210000038
And
Figure BDA0003087816210000039
characteristic diagram of
Figure BDA00030878162100000310
And
Figure BDA00030878162100000311
is transformed to obtain
Figure BDA00030878162100000312
And
Figure BDA00030878162100000313
Figure BDA00030878162100000314
Figure BDA00030878162100000315
Figure BDA00030878162100000316
Figure BDA00030878162100000317
wherein,
Figure BDA00030878162100000318
is shown in
Figure BDA00030878162100000319
A local feature map within the range of,
Figure BDA00030878162100000320
in order to obtain the transformed feature map,
Figure BDA00030878162100000321
is composed of
Figure BDA00030878162100000322
The average value of (a) of (b),
Figure BDA00030878162100000323
is composed of
Figure BDA00030878162100000324
Standard deviation of (D) and (D)
Figure BDA00030878162100000325
Figure BDA00030878162100000326
In the search area
Figure BDA00030878162100000327
And
Figure BDA00030878162100000328
searching matching point pair in the interior, when the following conditions are satisfied, two points plAnd q islFor matching point pairs:
Figure BDA0003087816210000041
Figure BDA0003087816210000042
Figure BDA0003087816210000043
that is to say if
Figure BDA0003087816210000044
Inner point plIn that
Figure BDA0003087816210000045
The point with the highest similarity when searching within the range is qlOtherwise, p is also truelAnd q islAre pairs of matching points. The similarity is calculated by the following formula:
Figure BDA0003087816210000046
wherein, epsilon (p)l) Is a point plA set of points of a neighborhood within the specified range.
All N of class llThe step of searching the matching points is respectively executed in each searching range, and the set Lambda of all matching point pairs is obtainedl：
Figure BDA0003087816210000047
For the matching point pair searched in the above step, the following filtering condition must be passed, i.e. the value of the point in the feature map (feature map) must be sufficiently large:
Figure BDA0003087816210000048
Figure BDA0003087816210000049
wherein,
Figure BDA00030878162100000410
and gamma is a self-defined threshold value for the finally obtained matching point pair set.
To obtain
Figure BDA00030878162100000411
Then, the search of the previous stage is obtained by the following formulaThe set of ranges is such that,
Figure BDA0003087816210000051
Figure BDA0003087816210000052
wherein,
Figure BDA0003087816210000053
is composed of
Figure BDA0003087816210000054
The number of the (c) component(s),
Figure BDA0003087816210000055
the coordinate of the point p is (p) relative to the neural network receptive field of level l-1 and level lx，py，pz). And after the search range set of the previous stage is obtained, repeating the steps to obtain a final output result
Figure BDA0003087816210000056
I.e. a set of matching point pairs for both images.
In a preferred embodiment, the marker point-based image registration method further includes: fitting a transformation matrix and a displacement vector of rigid registration by minimizing the sum of distances of all matching points to obtain a rigidly registered medical image, wherein the method comprises the following steps: after all the matching point pairs are obtained, the optimal solution of the transformation matrix and the displacement vector of the rigid registration is obtained by minimizing the following formula:
Figure BDA0003087816210000057
the optimal solution is as follows:
R＝(PTP)-1PTQ
A＝R[0:3，0:3]
b＝R[0:3，3]
wherein N is the number of matching point pairs, pnIs the n-th matching point of fixed image, qnThe image is a pixel point in the corresponding moving image. P is a matrix formed by all matched points of the fixed image and has the size of [ N,4 ]]That is, a matrix composed of N four-dimensional row vectors, the first three dimensions of the four dimensions are physical coordinates of the pixel points, and the fourth dimension is a fixed value of 1. Q is a matrix formed by all matching points of the moving image and has the size of [ N,4 ]]. The size of the matrix R is [4,4 ]]，R[0:3，0:3]The first 3 rows and the first 3 columns of the matrix R are taken as the size of [3,3 ]]A matrix of (1), R0: 3,3]A three-dimensional column vector is taken for the first 3 rows and column 3 of the matrix R. A and b are the optimal solutions of the transformation matrix and the displacement vector, respectively. And finally obtaining the rigidly registered medical image transmitted image through A and b.
In a preferred embodiment, the marker point-based image registration method further includes: on the basis of rigid registration, obtaining a non-rigid registered displacement field three-dimensional matrix by an interpolation method based on a radial basis, thereby obtaining a non-rigid registered medical image, comprising the following steps: the size of the displacement field three-dimensional matrix is the same as the fixed image. After N matching point pairs are obtained, the value of the residual pixel point of the displacement field matrix is obtained by adopting the following interpolation method:
Figure BDA0003087816210000061
A＝(a1，a2，a3)
G(r)＝r2 lnr
where p is the coordinate in the three-dimensional matrix of the displacement field of (x)p，yp，zp) Pixel of (b), pnIs the nth matching point in the fixed image. G () is a radial basis function. A. b, wnThe value of (d) is solved in the following way:
setting:
Figure BDA0003087816210000062
rij＝‖pi-pj‖
V＝(v1，v2，…，vN，0，0，0，0)
vn＝(qn-pn)[k]k∈(0，1，2)
Figure BDA0003087816210000063
Ω＝(w1，w2，…，wN，b，a1，a2，a3)
wherein P is a matrix formed by all matched points of the fixed image and has the size of [ N,4 ]]That is, a matrix composed of n four-dimensional row vectors, the rear three-dimensional of the four-dimensional is the physical coordinate of the pixel point, and the first dimension is a fixed value of 1. q. q.snIs pnThe corresponding matching point in the moving image. Since the displacement value is a three-dimensional vector (x, y, z direction), the above solving process takes the x axis when k is 0, the y axis when k is 1, and the z axis when k is 2.
By vn＝f(pn) Comprises the following steps:
V＝LΩT
further, the values of all parameters to be solved are obtained:
Ω＝(L-1V)T
and the displacement values of the pixel points except the matching points in the displacement field can be obtained through f () fitting. Since the displacement values are three-dimensional vectors, i.e. x, y, z directions, the above interpolation process needs to be repeated 3 times, i.e. once for each direction. And finally obtaining a displacement field three-dimensional matrix so as to obtain the non-rigid registered medical image.
Compared with the prior art, the image registration method based on the mark points has the following beneficial effects: (1) the invention can register images of any two modalities. (2) The invention adopts a pre-trained neural network to extract the image characteristics, the training process of the network comprises a plurality of different tasks and relates to a plurality of different input modes, and the effectiveness and the universality of the characteristics can be effectively improved. (3) The invention utilizes a pre-trained neural network to extract image characteristics, obtains a plurality of matching point pairs representing certain semantics between two images through the processes of searching, screening, matching and the like, and can effectively solve the problem of gold standard shortage of the marking points. (4) The invention solves the transformation matrix and the displacement vector by minimizing the sum of the distances between all the matching points, thereby realizing rigid registration. (5) The method solves the displacement field three-dimensional matrix through an interpolation method based on the radial basis to realize non-rigid registration.
Drawings
Fig. 1 is a flowchart illustrating an image registration method based on a marker according to an embodiment of the present invention.
Detailed Description
The following detailed description of the present invention is provided in conjunction with the accompanying drawings, but it should be understood that the scope of the present invention is not limited to the specific embodiments.
Throughout the specification and claims, unless explicitly stated otherwise, the word "comprise", or variations such as "comprises" or "comprising", will be understood to imply the inclusion of a stated element or component but not the exclusion of any other element or component.
As shown in fig. 1, an image registration method based on a marker point according to a preferred embodiment of the present invention includes the following steps:
inputting medical images of two arbitrary modalities (CT, CBCT, MRI, PET, etc.), one as a fixed image and the other as a moving image;
extracting pyramid characteristics of two input images by adopting a pre-trained neural network, wherein the training process of the network comprises a plurality of different tasks and relates to the plurality of different input modes;
extracting pyramid features by using the neural network, and obtaining a plurality of matching point pairs representing certain semantics between two images through the processes of searching, screening, matching and the like;
and fitting a transformation matrix and a displacement vector of rigid registration by minimizing the sum of the distances of all the matched points to the points so as to obtain the medical image subjected to rigid registration.
On the basis of rigid registration, a non-rigid registered displacement field three-dimensional matrix is obtained through an interpolation method based on a radial basis, and thus a non-rigid registered medical image is obtained.
The work flow of the specific implementation of the image registration method based on the mark points comprises the following steps:
in some embodiments, step S1, constructing a pre-trained neural network to extract pyramid features of two input images;
step S1 specifically includes the following steps:
and S11, dividing the structure of the neural network into a backbone network and a plurality of subsequent branch networks. The backbone network is shared among different tasks, and each branch network corresponds to one task. Finally, a backbone network is used for extracting the image features.
S12, the training process of the neural network includes a variety of different tasks and involves a variety of different input modalities, including but not limited to: CT-based segmentation of primary tumors of nasopharyngeal carcinoma (GTV), MRI-based segmentation of primary tumors of nasopharyngeal carcinoma, CT-based segmentation of primary tumors of cervical carcinoma, PET-based segmentation of primary tumors of lung, CT-based segmentation of Organs At Risk (OAR), MRI-based segmentation of organs at risk, CBCT-based segmentation of organs at risk, CT-based target detection of lung nodules, etc.
And S13, firstly, training the neural network by using one task, simultaneously training the other input mode tasks, then independently training the rest tasks, fixing the parameters of the backbone network during training, and finally, simultaneously training and finely adjusting all the parameters by all the tasks.
In some embodiments, the marker point-based image registration method further includes:
s2, extracting pyramid features by using the neural network, and obtaining a plurality of matching point pairs representing certain semantics between two images through the processes of searching, screening, matching and the like;
step S2 specifically includes the following steps:
s21, If(fixed image) and Im(moving image) inputting the pre-trained neural network, extracting pyramid feature maps (feature maps) of the two input images
Figure BDA0003087816210000091
And
Figure BDA0003087816210000092
wherein l is belonged to {1,2,3,4,5} represents the l-th level feature, and the larger the number is, the deeper the layer number is, namely, the smaller the feature size is, but the higher level semantic meaning is included.
S22, the search for the matching point needs to be generated within a specific search range, and the search range is set from l to 5:
S5＝{(P5，Q5)}
wherein:
Figure BDA0003087816210000093
Figure BDA0003087816210000094
is IfThe nth search range of the l-th stage of (1), correspondingly
Figure BDA0003087816210000095
Is ImN search range of the l stage, NlNumber of search ranges, S, of level llA set of multiple search range pairs for the l-th level. When l is 5, the search range is
Figure BDA0003087816210000096
And
Figure BDA0003087816210000097
of (2), i.e. N5＝1。
S23, search range is matched by the following formula
Figure BDA0003087816210000101
And
Figure BDA0003087816210000102
characteristic diagram of
Figure BDA0003087816210000103
And
Figure BDA0003087816210000104
is transformed to obtain
Figure BDA0003087816210000105
And
Figure BDA0003087816210000106
Figure BDA0003087816210000107
Figure BDA0003087816210000108
Figure BDA0003087816210000109
Figure BDA00030878162100001010
wherein,
Figure BDA00030878162100001011
is shown in
Figure BDA00030878162100001012
A local feature map within the range of,
Figure BDA00030878162100001013
for transformed featuresIn the figure, the figure shows that,
Figure BDA00030878162100001014
is composed of
Figure BDA00030878162100001015
The average value of (a) of (b),
Figure BDA00030878162100001016
is composed of
Figure BDA00030878162100001017
Standard deviation of (D) and (D)
Figure BDA00030878162100001018
Figure BDA00030878162100001019
S24, search for the scope
Figure BDA00030878162100001020
And
Figure BDA00030878162100001021
searching matching point pair in the interior, when the following conditions are satisfied, two points plAnd q islFor matching point pairs:
Figure BDA00030878162100001022
Figure BDA00030878162100001027
Figure BDA00030878162100001023
that is to say if
Figure BDA00030878162100001024
Inner point plIn that
Figure BDA00030878162100001025
The point with the highest similarity when searching within the range is qlOtherwise, p is also truelAnd q islAre pairs of matching points. The similarity is calculated by the following formula:
Figure BDA00030878162100001026
wherein, epsilon (p)l) Is a point plA set of points of a neighborhood within the specified range.
S25, all N of the l-th stagelThe search ranges respectively execute the steps S23-S24 of searching the matching points, so as to obtain the set Lambda of all matching point pairsl：
Figure BDA0003087816210000111
S26, for the matching point pair searched in the above step, the value of the point in the feature map (feature map) must be large enough to pass the following filtering condition:
Figure BDA0003087816210000112
Figure BDA0003087816210000113
wherein,
Figure BDA0003087816210000114
and gamma is a custom threshold value which is 0.05.
S27, obtaining
Figure BDA0003087816210000115
Then, the search range set of the upper level is obtained by the following formula,
Figure BDA0003087816210000116
Figure BDA0003087816210000117
wherein,
Figure BDA0003087816210000118
is composed of
Figure BDA0003087816210000119
The number of the (c) component(s),
Figure BDA00030878162100001110
the coordinate of the point p is (p) relative to the neural network receptive field of level l-1 and level lx，py，pz)。
S28, after the search range set of the previous level is obtained, for each search range, jumping to the step S23 and repeating the steps S23-S28 to obtain the final output result
Figure BDA00030878162100001111
I.e. a set of matching point pairs for both images.
In some embodiments, the marker point-based image registration method further includes:
s3, fitting a transformation matrix and a displacement vector of rigid registration by minimizing the sum of distances between all matching points, so as to obtain a medical image subjected to rigid registration;
step S3 specifically includes the following steps:
s31, obtaining the optimal solution of the transformation matrix and the displacement vector of the rigid registration by minimizing the following formula after obtaining all the matching point pairs:
Figure BDA0003087816210000121
the optimal solution is as follows:
R＝(PTP)-1PTQ
A＝R[0:3，0:3]
b＝R[0:3，3]
wherein N is the number of matching point pairs, pnIs the n-th matching point of fixed image, qnThe image is a pixel point in the corresponding moving image. P is a matrix formed by all matched points of the fixed image and has the size of [ N,4 ]]That is, a matrix composed of N four-dimensional row vectors, the first three dimensions of the four dimensions are physical coordinates of the pixel points, and the fourth dimension is a fixed value of 1. Q is a matrix formed by all matching points of the moving image and has the size of [ N,4 ]]. The size of the matrix R is [4,4 ]]，R[0:3，0:3]The first 3 rows and the first 3 columns of the matrix R are taken as the size of [3,3 ]]A matrix of (1), R0: 3,3]A three-dimensional column vector is taken for the first 3 rows and column 3 of the matrix R. A and b are the optimal solutions of the transformation matrix and the displacement vector, respectively.
And S32, obtaining a rigidly registered medical image warp image through A and b.
In some embodiments, the marker point-based image registration method further includes:
s4, obtaining a non-rigid registration displacement field three-dimensional matrix through an interpolation method based on a radial basis on the basis of rigid registration, so as to obtain a non-rigid registration medical image;
step S4 specifically includes the following steps:
s41, the size of the displacement field three-dimensional matrix is the same as the fixed image. After N matching point pairs are obtained, the value of the residual pixel point of the displacement field matrix is obtained by adopting the following interpolation method:
Figure BDA0003087816210000122
A＝(a1，a2，a3)
G(r)＝r2 lnr
where p is the coordinate in the three-dimensional matrix of the displacement field of (x)p，yp，zp) Image ofElement, pnIs the nth matching point in the fixed image. G () is a radial basis function. A. b, wnIs solved in the following manner
Setting:
Figure BDA0003087816210000131
rij＝‖pi-pj‖
V＝(v1，v2，…，vN，0，0，0，0)
vn＝(qn-pn)[k]k∈(0，1，2)
Figure BDA0003087816210000132
Ω＝(w1，w2，…，wN，b，a1，a2，a3)
wherein P is a matrix formed by all matched points of the fixed image and has the size of [ N,4 ]]That is, a matrix composed of n four-dimensional row vectors, the rear three-dimensional of the four-dimensional is the physical coordinate of the pixel point, and the first dimension is a fixed value of 1. q. q.snIs pnThe corresponding matching point in the moving image. Since the displacement value is a three-dimensional vector (x, y, z direction), the above solving process takes the x axis when k is 0, the y axis when k is 1, and the z axis when k is 2.
By vn＝f(pn) Comprises the following steps:
V＝LΩT
further, the values of all parameters to be solved are obtained:
Ω＝(L-1V)T
and the displacement values of the pixel points except the matching points in the displacement field can be obtained through f () fitting.
S42, since the displacement values are three-dimensional vectors, i.e. x, y, and z directions, the interpolation process needs to be repeated 3 times, i.e. each direction is performed once in step S41.
And S43, finally obtaining a displacement field three-dimensional matrix, thereby obtaining the non-rigid registered medical image.
In summary, the image registration method based on the mark points has the following advantages: (1) the invention can register images of any two modalities. (2) The invention adopts a pre-trained neural network to extract the image characteristics, the training process of the network comprises a plurality of different tasks and relates to a plurality of different input modes, and the effectiveness and the universality of the characteristics can be effectively improved. (3) The invention utilizes a pre-trained neural network to extract image characteristics, obtains a plurality of matching point pairs representing certain semantics between two images through the processes of searching, screening, matching and the like, and can effectively solve the problem of gold standard shortage of the marking points. (4) The invention solves the transformation matrix and the displacement vector by minimizing the sum of the distances between all the matching points, thereby realizing rigid registration. (5) The method solves the displacement field three-dimensional matrix through an interpolation method based on the radial basis to realize non-rigid registration.
The foregoing descriptions of specific exemplary embodiments of the present invention have been presented for purposes of illustration and description. It is not intended to limit the invention to the precise form disclosed, and obviously many modifications and variations are possible in light of the above teaching. The exemplary embodiments were chosen and described in order to explain certain principles of the invention and its practical application to enable one skilled in the art to make and use various exemplary embodiments of the invention and various alternatives and modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the claims and their equivalents.

Claims (5)

1. An image registration method based on a mark point is characterized by comprising the following steps:
inputting two medical images of any modality, wherein one medical image is used as a fixed image, and the other medical image is used as a moving image;
extracting pyramid characteristics of the medical image of two input arbitrary modes by adopting a pre-trained neural network, wherein the training process of the neural network comprises a plurality of different tasks and relates to the plurality of different input modes;
obtaining a plurality of matching point pairs representing certain semantics between two images by utilizing the pyramid characteristics extracted by the neural network through searching, screening and matching processes;
fitting a transformation matrix and a displacement vector of rigid registration by minimizing the sum of distances of all matching points to obtain a rigidly registered medical image; and
on the basis of rigid registration, a non-rigid registered displacement field three-dimensional matrix is obtained through an interpolation method based on a radial basis, and thus a non-rigid registered medical image is obtained.
2. The method according to claim 1, wherein the extracting the pyramid features of the medical image of two input arbitrary modalities by using a pre-trained neural network comprises:
the structure of the neural network is divided into a backbone network and a plurality of subsequent branch networks; the backbone network is shared among different tasks, and each branch network corresponds to one task; finally, the backbone network is used for extracting image features;
the training process of the neural network involves a variety of different tasks and involves a variety of different input modalities, including: CT-based segmentation of primary tumors of nasopharyngeal carcinoma, MRI-based segmentation of primary tumors of nasopharyngeal carcinoma, CT-based segmentation of primary tumors of cervical carcinoma, PET-based segmentation of primary tumors of lung, CT-based segmentation of organs-at-risk, MRI-based segmentation of organs-at-risk, CBCT-based segmentation of organs-at-risk, and CT-based detection of pulmonary nodule targets; and
firstly, the neural network trained by one task and the other input mode tasks are trained simultaneously, then each of the rest tasks is trained independently, the parameters of the backbone network are fixed during training, and finally all the tasks are trained simultaneously to fine tune all the parameters.
3. The method for registering images based on labeled points as claimed in claim 1, wherein said pyramid features extracted by said neural network are used to obtain a plurality of matching point pairs representing a certain semantic meaning between two images through a searching, screening and matching process, comprising the following steps:
handle If(fixed image) and Im(moving image) inputting the pre-trained neural network, extracting pyramid feature maps (feature maps) of the two input images
Figure FDA0003087816200000021
And
Figure FDA0003087816200000022
wherein l belongs to {1,2,3,4,5} represents the l-th level feature, the larger the number is, the deeper the layer number is, namely, the smaller the feature size is, but more high-level semantics are contained;
the search for a matching point needs to be generated within a specific search range, which is set starting from l-5:
S5＝{(P5,Q5)}
wherein:
Figure FDA0003087816200000023
Figure FDA0003087816200000024
is IfThe nth search range of the l-th stage of (1), correspondingly
Figure FDA0003087816200000025
Is ImN search range of the l stage, NlNumber of search ranges, S, of level llA set of a plurality of search range pairs for the l-th level; when l is 5, the search range is
Figure FDA0003087816200000026
And
Figure FDA0003087816200000027
of (2), i.e. N5＝1；
Search range is matched by
Figure FDA0003087816200000028
And
Figure FDA0003087816200000029
characteristic diagram of
Figure FDA00030878162000000210
And
Figure FDA00030878162000000211
is transformed to obtain
Figure FDA00030878162000000212
And
Figure FDA00030878162000000213
Figure FDA00030878162000000214
Figure FDA00030878162000000215
Figure FDA00030878162000000216
Figure FDA00030878162000000217
wherein,
Figure FDA0003087816200000031
is shown in
Figure FDA0003087816200000032
A local feature map within the range of,
Figure FDA0003087816200000033
in order to obtain the transformed feature map,
Figure FDA0003087816200000034
is composed of
Figure FDA0003087816200000035
The average value of (a) of (b),
Figure FDA0003087816200000036
is composed of
Figure FDA0003087816200000037
Standard deviation of (D) and (D)
Figure FDA0003087816200000038
Figure FDA0003087816200000039
In the search area
Figure FDA00030878162000000310
And
Figure FDA00030878162000000311
searching matching point pair in the interior, when the following conditions are satisfied, two points plAnd q islFor matching point pairs:
Figure FDA00030878162000000312
Figure FDA00030878162000000313
Figure FDA00030878162000000314
that is to say if
Figure FDA00030878162000000315
Inner point plIn that
Figure FDA00030878162000000316
The point with the highest similarity when searching within the range is qlOtherwise, p is also truelAnd q islIs a matching point pair; the similarity is calculated by the following formula:
Figure FDA00030878162000000317
wherein, epsilon (p)l) Is a point plA set of points of a neighborhood within the specified range of (a);
all N of class llThe step of searching the matching points is respectively executed in each searching range, and the set Lambda of all matching point pairs is obtainedl：
Figure FDA00030878162000000318
For the matching point pair searched in the above step, the following filtering condition must be passed, i.e. the value of the point in the feature map (feature map) must be sufficiently large:
Figure FDA00030878162000000319
Figure FDA00030878162000000320
wherein,
Figure FDA0003087816200000041
gamma is a self-defined threshold value for the finally obtained matching point pair set;
to obtain
Figure FDA0003087816200000042
Then, the search range set of the previous level is obtained by the following formula:
Figure FDA0003087816200000043
Figure FDA0003087816200000044
wherein,
Figure FDA0003087816200000045
is composed of
Figure FDA0003087816200000046
The number of the (c) component(s),
Figure FDA0003087816200000047
the coordinate of the point p is (p) relative to the neural network receptive field of level l-1 and level lx,py,pz) (ii) a And
after the search range set of the previous level is obtained, the steps are repeated to obtain the final output result
Figure FDA0003087816200000049
I.e. two imagesThe set of matching point pairs.
4. The image registration method based on the marker points as claimed in claim 1, wherein the rigidly registered medical image forward image is obtained by fitting a transformation matrix and a displacement vector of the rigid registration by minimizing the sum of the distances between all the matching points, comprising the following steps:
after all the matching point pairs are obtained, the optimal solution of the transformation matrix and the displacement vector of the rigid registration is obtained by minimizing the following formula:
Figure FDA0003087816200000048
the optimal solution is as follows:
R＝(PTP)-1PTQ
A＝R[0:3,0:3]
b＝R[0:3,3]
wherein N is the number of matching point pairs, pnIs the n-th matching point of fixed image, qnThe pixel points in the corresponding moving image are obtained; p is a matrix formed by all matched points of the fixed image and has the size of [ N,4 ]]The matrix is composed of N four-dimensional row vectors, the front three-dimensional of the four-dimensional is the physical coordinate of the pixel point, and the fourth dimension is a fixed value 1; q is a matrix formed by all matching points of the moving image and has the size of [ N,4 ]](ii) a The size of the matrix R is [4,4 ]]，R[0:3,0:3]The first 3 rows and the first 3 columns of the matrix R are taken as the size of [3,3 ]]A matrix of (1), R0: 3,3]The three-dimensional column vector of the 3 rd column of the first 3 rows of the matrix R is taken; a and b are respectively the optimal solutions of the transformation matrix and the displacement vector; and
and finally, obtaining a rigidly registered medical image transmitted image through A and b.
5. The image registration method based on the marker points as claimed in claim 1, wherein based on the rigid registration, a non-rigid registered displacement field three-dimensional matrix is obtained by interpolation based on radial basis, so as to obtain a non-rigid registered medical image, comprising the following steps:
the size of the displacement field three-dimensional matrix is the same as the fixed image; after N matching point pairs are obtained, the values of the residual pixel points of the displacement field matrix are obtained by adopting the following interpolation method:
Figure FDA0003087816200000051
A＝(a1,a2,a3)
G(r)＝r2 ln r
wherein p is the coordinate in the three-dimensional matrix of the displacement field of (x)p,yp,zp) Pixel of (b), pnThe nth matching point in the fixed image; g () is a radial basis function; A. b, wnThe value of (d) is solved in the following way:
setting:
Figure FDA0003087816200000052
rij＝‖pi-pj‖
V＝(v1，v2,…,vN,0,0,0,0)
vn＝(qn-pn)[k]k∈(0,1,2)
Figure FDA0003087816200000061
Ω＝(w1,w2,…,wN,b,a1，a2，a3)
wherein P is a matrix formed by all matched points of the fixed image and has the size of [ N,4 ]]The matrix is composed of n four-dimensional row vectors, the back three-dimensional of the four-dimensional is the physical coordinate of the pixel point, and the first dimension is a fixed value 1; q. q.snIs pnThe corresponding matching point in the moving image; k is the expression dimension, i.e., since the displacement values are three-dimensional vectors (x, y,z direction), the above solving process only aims at one dimension, so the x axis is taken when k is 0, the y axis is taken when k is 1, and the z axis is taken when k is 2;
by vn＝f(pn) Comprises the following steps:
V＝LΩT
further, the values of all parameters to be solved are obtained:
Ω＝(L-1V)T
the displacement values of the pixel points except the matching points in the displacement field can be obtained through f () fitting;
since the displacement values are three-dimensional vectors, i.e. x, y, z directions, the above interpolation process needs to be repeated 3 times, i.e. each direction is performed once; and
and finally, obtaining a displacement field three-dimensional matrix so as to obtain a non-rigid registered medical image.
CN202110585049.8A 2021-05-27 2021-05-27 Image registration method based on mark points Active CN113298854B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202110585049.8A CN113298854B (en) 2021-05-27 2021-05-27 Image registration method based on mark points
PCT/CN2022/070425 WO2022247296A1 (en) 2021-05-27 2022-01-06 Mark point-based image registration method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110585049.8A CN113298854B (en) 2021-05-27 2021-05-27 Image registration method based on mark points

Publications (2)

Publication Number Publication Date
CN113298854A true CN113298854A (en) 2021-08-24
CN113298854B CN113298854B (en) 2022-02-01

Family

ID=77325584

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110585049.8A Active CN113298854B (en) 2021-05-27 2021-05-27 Image registration method based on mark points

Country Status (2)

Country Link
CN (1) CN113298854B (en)
WO (1) WO2022247296A1 (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113744328A (en) * 2021-11-05 2021-12-03 极限人工智能有限公司 Medical image mark point identification method and device, electronic equipment and storage medium
CN113920179A (en) * 2021-11-09 2022-01-11 广州柏视医疗科技有限公司 Mark point-based multi-vision 2D-3D image non-rigid registration method and system
CN113920178A (en) * 2021-11-09 2022-01-11 广州柏视医疗科技有限公司 Mark point-based multi-vision 2D-3D image registration method and system
CN114241077A (en) * 2022-02-23 2022-03-25 南昌睿度医疗科技有限公司 CT image resolution optimization method and device
CN114404039A (en) * 2021-12-30 2022-04-29 华科精准（北京）医疗科技有限公司 Tissue drift correction method and device for three-dimensional model, electronic equipment and storage medium
WO2022247296A1 (en) * 2021-05-27 2022-12-01 广州柏视医疗科技有限公司 Mark point-based image registration method

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117474993B (en) * 2023-10-27 2024-05-24 哈尔滨工程大学 Underwater image feature point sub-pixel position estimation method and device
CN118314215A (en) * 2024-04-02 2024-07-09 上海栎元医疗科技有限公司 Three-dimensional medical image mark point ordering method, system, electronic equipment and storage medium
CN118644819B (en) * 2024-08-14 2024-10-22 摸鱼科技(大连)有限公司 Video monitoring management method and system for face care

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104021547A (en) * 2014-05-17 2014-09-03 清华大学深圳研究生院 Three dimensional matching method for lung CT
CN106097347A (en) * 2016-06-14 2016-11-09 福州大学 A kind of multimodal medical image registration and method for visualizing
CN107818564A (en) * 2017-10-27 2018-03-20 深圳市图智能科技有限公司 A kind of liver 3D medical image segmentation methods
CN109064502A (en) * 2018-07-11 2018-12-21 西北工业大学 The multi-source image method for registering combined based on deep learning and artificial design features
CN111640143A (en) * 2020-04-12 2020-09-08 复旦大学 Nerve navigation rapid surface registration method and system based on PointNet
CN111931929A (en) * 2020-07-29 2020-11-13 深圳地平线机器人科技有限公司 Training method and device of multi-task model and storage medium

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170337682A1 (en) * 2016-05-18 2017-11-23 Siemens Healthcare Gmbh Method and System for Image Registration Using an Intelligent Artificial Agent
CN111091589B (en) * 2019-11-25 2023-11-17 北京理工大学 Ultrasonic and nuclear magnetic image registration method and device based on multi-scale supervised learning
CN113298854B (en) * 2021-05-27 2022-02-01 广州柏视医疗科技有限公司 Image registration method based on mark points

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104021547A (en) * 2014-05-17 2014-09-03 清华大学深圳研究生院 Three dimensional matching method for lung CT
CN106097347A (en) * 2016-06-14 2016-11-09 福州大学 A kind of multimodal medical image registration and method for visualizing
CN107818564A (en) * 2017-10-27 2018-03-20 深圳市图智能科技有限公司 A kind of liver 3D medical image segmentation methods
CN109064502A (en) * 2018-07-11 2018-12-21 西北工业大学 The multi-source image method for registering combined based on deep learning and artificial design features
CN111640143A (en) * 2020-04-12 2020-09-08 复旦大学 Nerve navigation rapid surface registration method and system based on PointNet
CN111931929A (en) * 2020-07-29 2020-11-13 深圳地平线机器人科技有限公司 Training method and device of multi-task model and storage medium

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
PAMLI 等: "Feature-Based Retinal Image Registration Using D-Saddle Feature", 《JOURNAL OF HEALTHCARE ENGINEERING》 *
王伟: "医学图像非刚性配准方法及系统研究", 《中国优秀硕士学位论文全文数据库 (医药卫生科技辑)》 *

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022247296A1 (en) * 2021-05-27 2022-12-01 广州柏视医疗科技有限公司 Mark point-based image registration method
CN113744328A (en) * 2021-11-05 2021-12-03 极限人工智能有限公司 Medical image mark point identification method and device, electronic equipment and storage medium
CN113920179A (en) * 2021-11-09 2022-01-11 广州柏视医疗科技有限公司 Mark point-based multi-vision 2D-3D image non-rigid registration method and system
CN113920178A (en) * 2021-11-09 2022-01-11 广州柏视医疗科技有限公司 Mark point-based multi-vision 2D-3D image registration method and system
CN113920178B (en) * 2021-11-09 2022-04-12 广州柏视医疗科技有限公司 Mark point-based multi-vision 2D-3D image registration method and system
CN113920179B (en) * 2021-11-09 2022-04-29 广州柏视医疗科技有限公司 Mark point-based multi-vision 2D-3D image non-rigid registration method and system
CN114404039A (en) * 2021-12-30 2022-04-29 华科精准（北京）医疗科技有限公司 Tissue drift correction method and device for three-dimensional model, electronic equipment and storage medium
CN114241077A (en) * 2022-02-23 2022-03-25 南昌睿度医疗科技有限公司 CT image resolution optimization method and device
CN114241077B (en) * 2022-02-23 2022-07-15 南昌睿度医疗科技有限公司 CT image resolution optimization method and device

Also Published As

Publication number Publication date
WO2022247296A1 (en) 2022-12-01
CN113298854B (en) 2022-02-01

Similar Documents

Publication Publication Date Title
CN113298854B (en) Image registration method based on mark points
Jiao et al. Learning with limited annotations: a survey on deep semi-supervised learning for medical image segmentation
Liu et al. MS-Net: multi-site network for improving prostate segmentation with heterogeneous MRI data
Zhou et al. One-pass multi-task networks with cross-task guided attention for brain tumor segmentation
Yu et al. Segmentation of fetal left ventricle in echocardiographic sequences based on dynamic convolutional neural networks
Cao et al. Deformable image registration using a cue-aware deep regression network
Cao et al. Region-adaptive deformable registration of CT/MRI pelvic images via learning-based image synthesis
JP7558243B2 (en) Feature Point Detection
CN113298855B (en) Image registration method based on automatic delineation
CN108416802A (en) A kind of multi modal medical image non-rigid registration method and system based on deep learning
Turan et al. A deep learning based 6 degree-of-freedom localization method for endoscopic capsule robots
Liu et al. Rotation-invariant siamese network for low-altitude remote-sensing image registration
Yang et al. Registration of pathological images
Hsu Automatic left ventricle recognition, segmentation and tracking in cardiac ultrasound image sequences
Jin et al. Object recognition in medical images via anatomy-guided deep learning
Ayatollahi et al. A new hybrid particle swarm optimization for multimodal brain image registration
Liu et al. Reducing domain gap in frequency and spatial domain for cross-modality domain adaptation on medical image segmentation
Song et al. Classifying tongue images using deep transfer learning
Zhao et al. MSKD: Structured knowledge distillation for efficient medical image segmentation
CN111080676A (en) Method for tracking endoscope image sequence feature points through online classification
Zhang et al. Multimodal medical volumes translation and segmentation with generative adversarial network
Song et al. Learning 3d features with 2d cnns via surface projection for ct volume segmentation
Schwab et al. Multimodal medical image registration using particle swarm optimization with influence of the data's initial orientation
Fan et al. Deep feature descriptor based hierarchical dense matching for X-ray angiographic images
Wang et al. Automatic carotid artery detection using attention layer region-based convolution neural network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant