CN114581340A - Image correction method and device - Google Patents
Image correction method and device Download PDFInfo
- Publication number
- CN114581340A CN114581340A CN202210285649.7A CN202210285649A CN114581340A CN 114581340 A CN114581340 A CN 114581340A CN 202210285649 A CN202210285649 A CN 202210285649A CN 114581340 A CN114581340 A CN 114581340A
- Authority
- CN
- China
- Prior art keywords
- image
- template
- target
- target part
- parameter matrix
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000003702 image correction Methods 0.000 title claims abstract description 30
- 230000009466 transformation Effects 0.000 claims abstract description 121
- 239000011159 matrix material Substances 0.000 claims abstract description 100
- 238000006243 chemical reaction Methods 0.000 claims abstract description 30
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims abstract description 16
- 239000013598 vector Substances 0.000 claims description 147
- 238000012549 training Methods 0.000 claims description 60
- 238000000605 extraction Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 12
- 238000003860 storage Methods 0.000 claims description 6
- 230000001575 pathological effect Effects 0.000 abstract description 7
- 230000006870 function Effects 0.000 description 14
- 238000010586 diagram Methods 0.000 description 10
- 238000012545 processing Methods 0.000 description 8
- 210000001835 viscera Anatomy 0.000 description 8
- 238000013519 translation Methods 0.000 description 7
- 238000004458 analytical method Methods 0.000 description 4
- 210000000056 organ Anatomy 0.000 description 4
- 238000010827 pathological analysis Methods 0.000 description 4
- 238000012937 correction Methods 0.000 description 3
- 238000005314 correlation function Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000005457 optimization Methods 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 241001465754 Metazoa Species 0.000 description 1
- 241001494479 Pecora Species 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Images
Classifications
-
- G06T5/80—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/088—Non-supervised learning, e.g. competitive learning
-
- G06T3/147—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/60—Rotation of a whole image or part thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
Abstract
The invention discloses an image correction method and equipment, which are suitable for scenes without obvious contours or key points, such as endoscopes and the like, and improve the efficiency of analyzing pathological pictures. The method comprises the following steps: acquiring a target part image of an object to be detected and a template image matched with the target part image, wherein the template image represents an image conforming to a preset image specification; determining a transformation parameter matrix of the target part image and the template image, wherein the transformation parameter matrix represents the mutual conversion relation of the target part image and the template image, and elements in the transformation parameter matrix comprise affine transformation parameters representing the rotation angle; and correcting the target part image according to the transformation parameter matrix.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to an image correction method and an image correction apparatus.
Background
Due to the complexity of the internal organ structure of the human body, when a doctor uses a medical endoscope to inspect and photograph the human body, the stored pictures may have the problems of wrong shooting, blurring and the like, so that the cost of analyzing pathological parts by the doctor is increased due to the problems of image quality, angle and the like when the stored pictures are analyzed at a later stage.
At present, the quality of the saved picture can be improved by an image correction method, for example, a method of matching key points is adopted for calibration, but the calibration method needs an endoscope image with obvious outlines or key points, and is not suitable for complex endoscope scenes.
Therefore, how to calibrate the picture of the complex endoscope scene by the image correction method, which improves the difficulty of analyzing the pathological picture by the doctor and improves the analysis efficiency, has become a technical problem to be solved urgently at present.
Disclosure of Invention
The invention provides an image correction method and equipment, which are used for correcting a target part image according to a transformation parameter matrix representing the interconversion relationship between the target part image and a template image, are suitable for scenes without obvious contours or key points, such as an endoscope and the like, and improve the efficiency of analyzing pathological pictures.
In a first aspect, an embodiment of the present invention provides an image correction method, including:
acquiring a target part image of an object to be detected and a template image matched with the target part image, wherein the template image represents an image conforming to a preset image specification;
determining a transformation parameter matrix of the target part image and the template image, wherein the transformation parameter matrix represents the mutual conversion relation of the target part image and the template image, and elements in the transformation parameter matrix comprise affine transformation parameters representing the rotation angle;
and correcting the target part image according to the transformation parameter matrix.
The image correction method in this embodiment performs template image registration on the target portion image by using the registration network model, corrects the target portion image according to the output transformation parameter matrix, and corrects the rotation angle of the target portion image by using the interconversion relationship between the target portion image and the matched template image, so that the method is applicable to scenes without an obvious contour or a key point, such as an endoscope, and can effectively improve the efficiency of analyzing pathological pictures.
As an optional implementation, the determining a transformation parameter matrix of the target portion image and the template image includes:
extracting a target feature vector of a target part image and a template feature vector of the template image according to a trained feature extraction layer of the registration network model; the registration network model is obtained by performing unsupervised training by using a training part image and a template image matched with the training part image;
and determining the transformation parameter matrix according to the target characteristic vector and the template characteristic vector.
As an optional implementation, the determining the transformation parameter matrix according to the target feature vector and the template feature vector includes:
performing feature matching on the target feature vector and the template feature vector according to a feature matching layer of the registration network model to obtain a matching feature map; wherein the matching feature map characterizes a matching degree of each vector component contained in the target feature vector and the template feature vector;
and determining the transformation parameter matrix according to the matching feature map.
As an optional implementation manner, the determining the transformation parameter matrix according to the matching feature map includes:
and performing parameter regression on the matching feature map according to a parameter regression layer of the registration network model to determine the transformation parameter matrix, wherein the parameter regression is used for predicting the feature matching modes of the target feature vector and the template feature vector contained in the matching feature map.
As an optional implementation, the acquiring a template image matching the target portion image includes:
acquiring a template image set, wherein the template image set comprises template images of all parts of an object to be detected;
and determining a template image matched with the target part image according to the similarity between the target part image and the template image.
As an optional implementation manner, the determining, according to the similarity between the target portion image and the template image, a template image matching the target portion image includes:
inputting the target part image into an image retrieval model, and determining a template image matched with the target part image from a retrieval library of the image retrieval model, wherein the retrieval library comprises a template image set, and the image retrieval model is obtained by utilizing a training part image and a template image corresponding to the training part image in the template image set for training.
As an alternative embodiment, the determining a template image matching the target part image from a search library of image search models includes:
inputting the target part image into the image retrieval model to obtain a first feature vector of the target part image;
and determining the template image corresponding to the highest similarity as the template image matched with the target part image according to the similarity of the first feature vector and the second feature vector of each template image in the template image set.
As an optional implementation, the correcting the target portion image according to the transformation parameter matrix includes:
determining the rotation angle of the target part image according to the conversion relation between the transformation parameter matrix and the rotation angle, and correcting the target part image by using the rotation angle; and/or the presence of a gas in the gas,
and determining the scaling scale of the target part image according to the conversion relation between the transformation parameter matrix and the scaling scale, and correcting the target part image by using the scaling scale.
In a second aspect, an embodiment of the present invention provides an image correction apparatus, including a processor and a memory, where the memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and execute the following steps:
acquiring a target part image of an object to be detected and a template image matched with the target part image, wherein the template image represents an image conforming to a preset image specification;
determining a transformation parameter matrix of the target part image and the template image, wherein the transformation parameter matrix represents the mutual conversion relation of the target part image and the template image, and elements in the transformation parameter matrix comprise affine transformation parameters representing the rotation angle;
and correcting the target part image according to the transformation parameter matrix.
As an alternative embodiment, the processor is configured to perform:
the determining a transformation parameter matrix of the target portion image and the template image includes:
extracting a target feature vector of a target part image and a template feature vector of the template image according to a trained feature extraction layer of the registration network model; the registration network model is obtained by performing unsupervised training by using a training part image and a template image matched with the training part image;
and determining the transformation parameter matrix according to the target characteristic vector and the template characteristic vector.
As an alternative embodiment, the processor is configured to perform:
performing feature matching on the target feature vector and the template feature vector according to a feature matching layer of the registration network model to obtain a matching feature map; wherein the matching feature map characterizes a matching degree of each vector component contained in the target feature vector and the template feature vector;
and determining the transformation parameter matrix according to the matching feature map.
As an alternative embodiment, the processor is configured to perform:
and performing parameter regression on the matching feature map according to a parameter regression layer of the registration network model to determine the transformation parameter matrix, wherein the parameter regression is used for predicting the feature matching modes of the target feature vector and the template feature vector contained in the matching feature map.
As an alternative embodiment, the processor is configured to perform:
acquiring a template image set, wherein the template image set comprises template images of all parts of an object to be detected;
and determining a template image matched with the target part image according to the similarity between the target part image and the template image.
As an alternative embodiment, the processor is configured to perform:
inputting the target part image into an image retrieval model, and determining a template image matched with the target part image from a retrieval library of the image retrieval model, wherein the retrieval library comprises a template image set, and the image retrieval model is obtained by utilizing a training part image and a template image corresponding to the training part image in the template image set for training.
As an alternative embodiment, the processor is configured to perform:
inputting the target part image into the image retrieval model to obtain a first feature vector of the target part image;
and determining the template image corresponding to the highest similarity as the template image matched with the target part image according to the similarity of the first feature vector and the second feature vector of each template image in the template image set.
As an alternative embodiment, the processor is configured to perform:
determining the rotation angle of the target site image according to the conversion relation between the transformation parameter matrix and the rotation angle, and correcting the target site image by using the rotation angle; and/or the presence of a gas in the gas,
and determining the scaling scale of the target part image according to the conversion relation between the transformation parameter matrix and the scaling scale, and correcting the target part image by using the scaling scale.
In a third aspect, an embodiment of the present invention further provides an image correction apparatus, including:
the image acquisition unit is used for acquiring a target part image of an object to be detected and a template image matched with the target part image, wherein the template image represents an image which accords with a preset image specification;
an image registration unit, configured to determine a transformation parameter matrix of the target part image and the template image, where the transformation parameter matrix represents a mutual transformation relationship between the target part image and the template image, and an element in the transformation parameter matrix includes an affine transformation parameter representing a rotation angle;
and the image correction unit is used for correcting the target part image according to the transformation parameter matrix.
As an alternative implementation, the image registration unit is specifically configured to:
extracting a target feature vector of a target part image and a template feature vector of the template image according to a trained feature extraction layer of the registration network model; the registration network model is obtained by performing unsupervised training by using a training part image and a template image matched with the training part image;
and determining the transformation parameter matrix according to the target characteristic vector and the template characteristic vector.
As an alternative implementation, the image registration unit is specifically configured to:
performing feature matching on the target feature vector and the template feature vector according to a feature matching layer of the registration network model to obtain a matching feature map; wherein the matching feature map characterizes a matching degree of each vector component contained in the target feature vector and the template feature vector;
and determining the transformation parameter matrix according to the matching feature map.
As an alternative implementation, the image registration unit is specifically configured to:
and performing parameter regression on the matching feature map according to a parameter regression layer of the registration network model to determine the transformation parameter matrix, wherein the parameter regression is used for predicting the feature matching modes of the target feature vector and the template feature vector contained in the matching feature map.
As an optional implementation manner, the image acquiring unit is specifically configured to:
acquiring a template image set, wherein the template image set comprises template images of all parts of an object to be detected;
and determining a template image matched with the target part image according to the similarity between the target part image and the template image.
As an optional implementation manner, the image acquiring unit is specifically configured to:
inputting the target part image into an image retrieval model, and determining a template image matched with the target part image from a retrieval library of the image retrieval model, wherein the retrieval library comprises a template image set, and the image retrieval model is obtained by utilizing a training part image and a template image corresponding to the training part image in the template image set for training.
As an optional implementation manner, the image acquiring unit is specifically configured to:
inputting the target part image into the image retrieval model to obtain a first feature vector of the target part image;
and determining the template image corresponding to the highest similarity as the template image matched with the target part image according to the similarity of the first feature vector and the second feature vector of each template image in the template image set.
As an optional implementation manner, the image correction unit is specifically configured to:
determining the rotation angle of the target site image according to the conversion relation between the transformation parameter matrix and the rotation angle, and correcting the target site image by using the rotation angle; and/or the presence of a gas in the gas,
and determining the scaling scale of the target part image according to the conversion relation between the transformation parameter matrix and the scaling scale, and correcting the target part image by using the scaling scale.
In a fourth aspect, an embodiment of the present invention further provides a computer storage medium, on which a computer program is stored, where the computer program is used to implement the steps of the method in the first aspect when the computer program is executed by a processor.
These and other aspects of the present application will be more readily apparent from the following description of the embodiments.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
Fig. 1 is a flowchart illustrating an implementation of an image correction method according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating an embodiment of image correction according to the present invention;
FIG. 3 is a schematic diagram of an image correction apparatus according to an embodiment of the present invention;
fig. 4 is a schematic diagram of an image correction apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the present invention will be described in further detail with reference to the accompanying drawings, and it is apparent that the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The term "and/or" in the embodiments of the present invention describes an association relationship of associated objects, and indicates that three relationships may exist, for example, a and/or B may indicate: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.
The application scenario described in the embodiment of the present invention is for more clearly illustrating the technical solution of the embodiment of the present invention, and does not form a limitation on the technical solution provided in the embodiment of the present invention, and it can be known by a person skilled in the art that with the occurrence of a new application scenario, the technical solution provided in the embodiment of the present invention is also applicable to similar technical problems. In the description of the present invention, the term "plurality" means two or more unless otherwise specified.
Example 1, due to the complexity of the internal organ structure of the human body, when a doctor uses a medical endoscope to examine and photograph the human body, the stored pictures may have problems of skew shooting, blurring and the like, so that when the stored pictures are analyzed at a later stage, the cost of analyzing pathological parts by the doctor is increased due to the problems of the quality, the angle and the like of the images. At present, the quality of the stored picture can be improved by an image correction method, and if a key point matching method is adopted for calibration, the calibration method needs an endoscope image with an obvious outline or key points and is not suitable for a complex endoscope scene; if the endoscope image registration method based on the multi-scale context is adopted, the generated registration field can not calculate the overall deflection angle of the picture, and the correction accuracy is limited and is not accurate enough. Therefore, how to calibrate the picture of the complex endoscope scene by the image correction method, which improves the difficulty of analyzing the pathological picture by the doctor and improves the analysis efficiency, has become a technical problem to be solved urgently at present.
The image correction method provided by the embodiment has the core idea that a transformation parameter matrix is determined through a registration network model by utilizing a mutual transformation relation between a target part image to be corrected and a template image, so that the rotation angle of the target part image is corrected by utilizing affine transformation parameters representing the rotation angle in the transformation parameter matrix, the target part image is enabled to be infinitely close to the template image, and the analysis efficiency and the accuracy are improved.
As shown in fig. 1, the image correction method provided in this embodiment can be applied to the correction of an image of an internal organ of a human body captured by a medical apparatus (such as an endoscope, etc.), and the specific implementation flow is as follows:
100, acquiring a target part image of an object to be detected and a template image matched with the target part image, wherein the template image represents an image conforming to a preset image specification;
in some embodiments, the object to be detected in this embodiment refers to a living body with biological characteristics, including but not limited to a human body, an animal such as a pig, a cow, a sheep, and the like, taking the object to be detected as the human body as an example, the part in this embodiment specifically includes an organ of the object to be detected or a certain part of the organ, taking the human body as an example, the target part image of the object to be detected in this embodiment represents an image of an internal organ of the human body or a certain part of the organ.
In some embodiments, the template image in this embodiment is an image of a human internal organ that meets a preset specification and is predefined by a user based on an analysis requirement, where different pathological analysis requirements may define different image specifications, and the preset image specification may correspond to the pathological analysis requirement of the human internal organ. The preset image specification in this embodiment includes a specification definition of image quality and a specification definition of an angle of an organ portion in an image, so that when a target portion image of an internal organ of a human body has problems such as angle deflection and image blurring, the target portion image can still be corrected based on a matched template image.
in some embodiments, the elements in the transformation parameter matrix further include translation transformation parameters characterizing a translation scale, so that the target portion image is displacement-corrected using the translation transformation parameters.
And 102, correcting the target part image according to the transformation parameter matrix.
In some embodiments, the transformation parameter matrix in the present embodiment represents a mutual transformation relationship between the target part image and the template image, and elements in the transformation parameter matrix include affine transformation parameters representing rotation angles, so that the rotation angles of the target part image can be corrected based on the mutual transformation relationship and the affine transformation parameters to make the target part image approach the template image infinitely, so that the corrected target part image conforms to a preset image specification.
In the embodiment, the rotation angle of the whole rotation of the target part image is obtained by converting affine transformation parameters contained in the parameter matrix, so that the image correction is more convenient for pathological analysis.
In some embodiments, the present embodiment inputs the target site image and the template image into a registration network model, and outputs a corresponding transformation parameter matrix, where the registration network model uses a normalized cross-correlation function as a loss function, and the registration network model is obtained by performing unsupervised training using a training site image and a template image matched with the training site image.
The training samples in the embodiment comprise training part images and template images, the normalized cross-correlation function is used as a loss function to perform unsupervised training, training parameters in the initial model are adjusted, and finally a trained registration network model is obtained. The method comprises the steps of taking a training part image and a template image as input, calculating a loss function value by utilizing the feature vectors of the training part image and the template image, continuously adjusting training parameters in an initial model according to the loss function value, and finally obtaining a registration network model.
In some embodiments, the loss function used by the registration network model in this embodiment is a normalized cross-correlation function, which is specifically shown as follows:
where NCC denotes a loss function, f (x, y) denotes a feature vector of the target site image, t (x, y) denotes a feature vector of the template image, σ denotesfStandard deviation, σ, of a feature vector representing an image of a target sitetStandard deviation, mu, of a feature vector representing a template imagefMean, mu, of feature vectors representing an image of the target sitetThe average value of the feature vectors of the template image is represented, and n represents the dimension of the feature vector of the target region image and also represents the dimension of the feature vector of the template image.
In some embodiments, the registration network model in this embodiment includes a feature extraction layer, the target portion image and the template image are input to the feature extraction layer, and a target feature vector of the target portion image and a template feature vector of the template image are extracted; and determining and outputting a transformation parameter matrix according to the target characteristic vector and the template characteristic vector.
The registration network model in the embodiment can extract the depth features of the target part picture, does not need the target part picture to have obvious contours or key points, and is suitable for complex endoscope scenes.
In implementation, the target part image is input into a feature extraction layer for feature extraction to obtain a corresponding target feature vector FAIn which F isA∈Rh×w×dInputting the template image into the feature extraction hierarchyExtracting line characteristics to obtain corresponding template characteristic vector FBIn which F isB∈Rh×w×dH denotes the width of the target feature vector and also denotes the width of the template feature vector, w denotes the height of the target feature vector and also denotes the height of the template feature vector, d denotes the depth of the target feature vector and also denotes the depth of the template feature vector.
In some embodiments, the registration network model in this embodiment further includes a feature matching layer, where the target feature vector and the template feature vector are input to the feature matching layer, and feature matching is performed on the target feature vector and the template feature vector to obtain a matching feature map; and determining and outputting a transformation parameter matrix according to the matching characteristic graph.
Wherein the matching feature map characterizes a matching degree of each vector component included in the target feature vector and the template feature vector.
Optionally, the registration network model in this embodiment is a deep learning model, and a correspondence between feature vectors can be established by using a feature matching layer, so as to solve the problem of large-scale rotation.
In implementation, the target feature vector F isAAnd a template feature vector FBInputting the feature data into a feature matching layer for feature matching to obtain a matching feature graph, wherein the feature matching is carried out through the following formula:
CAB(i,j,k)＝FB(i,j)TFA(ik,jk) Formula (2);
k＝h(jk-1)+ikformula (3);
wherein, CAB(i, j, k) denotes a matching feature map, FB(i, j) represents a template feature vector, wherein (i, j) represents that the width of the template feature vector is i and the height is j, i belongs to [0, h) j belongs to [0, w); (i)k,jk) Width of the target feature vector is ikHeight jk，ik∈[0，h)，jkE [0, w); k is [0, w ] h), and i, j, ik，jkAnd k is an integer.
In some embodiments, the registration network model in this embodiment further includes a parametric regression layer, the matching feature map is input to the parametric regression layer, the parametric regression is performed on the matching feature map, and a transformation parameter matrix is determined and output.
In the implementation, parameter regression can be performed through convolution and dimension reduction processing, and after the convolution processing is performed on the matching characteristic graph, dimension reduction processing is performed, so that a transformation parameter matrix is obtained.
Wherein the parameter regression in this embodiment is used to predict the feature matching pattern of the target feature vector and the template feature vector included in the matching feature map. The feature matching pattern can be learned by the registration network model, so that two feature vectors (i.e. the target feature vector and the template feature vector) with similar or identical feature matching patterns are converted into each other.
In practice, the feature map C will be matchedAB(i, j, k) is input into the parameter regression layer for parameter regression to obtain a transformation parameter matrix H2×3Including affine transformation parameters and translation transformation parameters, wherein:
a11,a12,a21,a22respectively representing affine transformation parameters, tx,tyRespectively represent translation transformation parameters which respectively represent translation proportions in the horizontal direction and the vertical direction.
In some embodiments, a transformation parameter matrix H is derived2×3Then, the target part image is corrected by:
1) determining the rotation angle of the target part image according to the conversion relation between the transformation parameter matrix and the rotation angle, and correcting the target part image by using the rotation angle;
2) determining the scaling scale of the target part image according to the conversion relation between the transformation parameter matrix and the scaling scale, and correcting the target part image by using the scaling scale;
3) and determining the rotation angle of the target part image according to the conversion relation between the conversion parameter matrix and the rotation angle, determining the scaling scale of the target part image according to the conversion relation between the conversion parameter matrix and the scaling scale, and correcting the target part image by using the rotation angle and the scaling scale.
In implementation, the conversion relationship between the transformation parameter matrix and the rotation angle and the scaling scale is expressed by the following formula:
wherein H2×3Representing a transformation parameter matrix, a11,a12,a21,a22Respectively representing affine transformation parameters, tx,tyRespectively representing translation transformation parameters, | λ x | represents the scaling of the coordinate system (new coordinate system) of the target part image relative to the coordinate system (original coordinate system) of the template image on the x coordinate axis, | λ y | represents the scaling of the coordinate system (new coordinate system) of the target part image relative to the coordinate system (original coordinate system) of the template image on the y coordinate axis, and the scaling of the target part image is obtained through the formula (5); θ represents the rotation angle, and since λ x and λ y both represent the zoom ratio and are both greater than zero, | λ x | and | λ y | in equation (5) can be substituted into equation (4) to obtain θ.
In implementation, the obtained rotation angle is used to perform rotation correction on the target part image, so that the angle of the target part image is in accordance with the preset specification, for example, the angle of the target part image is always forward, which facilitates more accurate pathological analysis.
In some embodiments, the present embodiment may further acquire a template image matching the target portion image by:
step 1) acquiring a template image set, wherein the template image set comprises template images of all parts of an object to be detected;
in implementation, template images of all parts of the object to be detected can be collected in advance, the template images of all parts are in accordance with preset image specifications, for example, the template images are all forward angles, and the image quality meets preset requirements. Taking a medical endoscope as an example, the medical endoscope is used for collecting template images of various parts of internal organs of a human body to be shot in advance, and all the shot template images are used as a template image set for subsequent retrieval.
And 2) determining a template image matched with the target part image according to the similarity between the target part image and the template image.
In implementation, the template image with the similarity to the target part image satisfying a preset condition in the template image set is determined as the template image matched with the target part image.
In some embodiments, the template image with the highest similarity to the target part image in the template image set is determined as the template image matching the target part image.
In some embodiments, this embodiment provides a process of retrieving template images in a model-based manner, and specifically determines a template image in the template image set, whose similarity to the target part image satisfies a preset condition, by the following means:
in implementation, the target part image is input into an image retrieval model, and a template image matched with the target part image is determined and output from a retrieval library of the image retrieval model, wherein the retrieval library comprises a template image set, and the image retrieval model is obtained by utilizing a training part image and a template image corresponding to the training part image in the template image set to train an initial retrieval model. Optionally, the image retrieval model is obtained by training a training part image and a template image corresponding to the training part image in the template image set by using Circle loss as a loss function.
In some embodiments, the training samples of the image retrieval model in this embodiment include training part images, the training part images are used as input, template images matched with the training part images are used as output, the initial model is trained based on Circle loss, and parameters for model optimization are determined adaptively.
In implementation, the template image set may be registered in advance in the image search model as a search library of the image search model. Optionally, the model structure of the image retrieval model in this embodiment is ResNet18, and the loss function used is Circle loss, where the loss function can be represented by the following formula:
where Lcircle represents the loss function, snRepresenting the inter-class similarity, s, of the input training part image and the template images in the search corpuspRepresenting the degree of intra-similarity, alpha, of the input training part image and the template images in the search corpusnAnd alphapRespectively represents the weight of the optimized parameters, can be continuously updated in the training process, and can ensure that snAnd spThe step length of optimization is determined in a self-adaptive mode, and gamma represents a scale parameter and is a predefined hyperparameter. K denotes an intra-class similarity score of the input training part image and the template image in the search library, and L denotes an inter-class similarity score of the input training part image and the template image in the search library.
In some embodiments, the present embodiment may further determine the template image with the highest similarity to the target portion image by:
inputting the target part image into an image retrieval model to obtain a first feature vector of the target part image;
and according to the similarity of the first feature vector and the second feature vector of each template image in the template image set, determining the template image corresponding to the highest similarity as the template image matched with the target part image and outputting the template image.
In implementation, according to the cosine similarity between the first feature vector and the second feature vector of each template image in the template image set, the template image corresponding to the highest cosine similarity may be determined as the template image matching the target portion image.
The embodiment can automatically select the matched template image from the search library by using the image search model, and is more effective and accurate compared with a mode of manually selecting registration.
In implementation, the second feature vector of each template image in the template image set is obtained by performing feature extraction operation by using an image retrieval model, that is, each template image in the template image set is input to the image retrieval model in advance for feature extraction, so as to obtain the second feature vector of each template image;
then, inputting the target part image into an image retrieval model to obtain a first feature vector of the target part image; and calculating cosine similarity of the first characteristic vector and each second characteristic vector to obtain the cosine similarity of the first characteristic vector and each second characteristic vector, and finally selecting a template image corresponding to the highest cosine similarity, and taking the template image as an output result corresponding to the target site image.
In implementation, the cosine similarity between the first eigenvector and any one of the second eigenvectors is calculated by the following formula:
wherein similarity represents cosine similarity, Ai represents a first feature vector, Bi represents a second feature vector, and n represents the dimension of the first feature vector and also represents the dimension of the second feature vector.
As shown in fig. 2, this embodiment further provides a specific implementation flow of image correction, as follows:
and step 204, inputting the matching characteristic graph into a parameter regression layer, and outputting a transformation parameter matrix.
And step 205, determining the rotation angle of the target part image according to the conversion relation between the conversion parameter matrix and the rotation angle, determining the scaling scale of the target part image according to the conversion relation between the conversion parameter matrix and the scaling scale, and correcting the target part image by using the rotation angle and the scaling scale.
Embodiment 2, based on the same inventive concept, an embodiment of the present invention further provides an image correction apparatus, and since the apparatus is an apparatus in the method in the embodiment of the present invention, and the principle of the apparatus to solve the problem is similar to the method, the implementation of the apparatus may refer to the implementation of the method, and repeated details are omitted.
As shown in fig. 3, the apparatus comprises a processor 300 and a memory 301, wherein the memory 301 is used for storing programs executable by the processor 300, and the processor 300 is used for reading the programs in the memory 301 and executing the following steps:
acquiring a target part image of an object to be detected and a template image matched with the target part image, wherein the template image represents an image conforming to a preset image specification;
determining a transformation parameter matrix of the target part image and the template image, wherein the transformation parameter matrix represents the mutual conversion relation of the target part image and the template image, and elements in the transformation parameter matrix comprise affine transformation parameters representing the rotation angle;
and correcting the target part image according to the transformation parameter matrix.
As an alternative implementation, the processor 300 is specifically configured to perform:
the determining a transformation parameter matrix of the target portion image and the template image includes:
extracting a target feature vector of a target part image and a template feature vector of the template image according to a trained feature extraction layer of the registration network model; the registration network model is obtained by performing unsupervised training by using a training part image and a template image matched with the training part image;
and determining the transformation parameter matrix according to the target characteristic vector and the template characteristic vector.
As an alternative implementation, the processor 300 is specifically configured to perform:
performing feature matching on the target feature vector and the template feature vector according to a feature matching layer of the registration network model to obtain a matching feature map; wherein the matching feature map characterizes a matching degree of each vector component contained in the target feature vector and the template feature vector;
and determining the transformation parameter matrix according to the matching feature map.
As an alternative implementation, the processor 300 is specifically configured to perform:
and performing parameter regression on the matching feature map according to a parameter regression layer of the registration network model to determine the transformation parameter matrix, wherein the parameter regression is used for predicting the feature matching modes of the target feature vector and the template feature vector contained in the matching feature map.
As an alternative implementation, the processor 300 is specifically configured to perform:
acquiring a template image set, wherein the template image set comprises template images of all parts of an object to be detected;
and determining a template image matched with the target part image according to the similarity between the target part image and the template image.
As an alternative implementation, the processor 300 is specifically configured to perform:
inputting the target part image into an image retrieval model, and determining a template image matched with the target part image from a retrieval library of the image retrieval model, wherein the retrieval library comprises a template image set, and the image retrieval model is obtained by utilizing a training part image and a template image corresponding to the training part image in the template image set for training.
As an alternative implementation, the processor 300 is specifically configured to perform:
inputting the target part image into the image retrieval model to obtain a first feature vector of the target part image;
and determining the template image corresponding to the highest similarity as the template image matched with the target part image according to the similarity of the first feature vector and the second feature vector of each template image in the template image set.
As an alternative implementation, the processor 300 is specifically configured to perform:
determining the rotation angle of the target part image according to the conversion relation between the transformation parameter matrix and the rotation angle, and correcting the target part image by using the rotation angle; and/or the presence of a gas in the gas,
and determining the scaling scale of the target part image according to the conversion relation between the transformation parameter matrix and the scaling scale, and correcting the target part image by using the scaling scale.
Embodiment 3, based on the same inventive concept, an embodiment of the present invention further provides an image correction apparatus, and since the apparatus is an apparatus in the method in the embodiment of the present invention, and the principle of the apparatus to solve the problem is similar to the method, the implementation of the apparatus may refer to the implementation of the method, and repeated details are not repeated.
As shown in fig. 4, the apparatus includes:
an image obtaining unit 400, configured to obtain a target portion image of an object to be detected and a template image matched with the target portion image, where the template image represents an image that meets a preset image specification;
an image registration unit 401, configured to determine a transformation parameter matrix of the target part image and the template image, where the transformation parameter matrix represents a mutual transformation relationship between the target part image and the template image, and an element in the transformation parameter matrix includes an affine transformation parameter representing a rotation angle;
an image correction unit 402, configured to correct the target portion image according to the transformation parameter matrix.
As an optional implementation, the image registration unit 401 is specifically configured to:
extracting a target feature vector of a target part image and a template feature vector of the template image according to a trained feature extraction layer of the registration network model; the registration network model is obtained by performing unsupervised training by using a training part image and a template image matched with the training part image;
and determining the transformation parameter matrix according to the target characteristic vector and the template characteristic vector.
As an optional implementation, the image registration unit 401 is specifically configured to:
performing feature matching on the target feature vector and the template feature vector according to a feature matching layer of the registration network model to obtain a matching feature map; wherein the matching feature map characterizes a matching degree of each vector component contained in the target feature vector and the template feature vector;
and determining the transformation parameter matrix according to the matching feature map.
As an optional implementation, the image registration unit 401 is specifically configured to:
and performing parameter regression on the matching feature map according to a parameter regression layer of the registration network model to determine the transformation parameter matrix, wherein the parameter regression is used for predicting the feature matching modes of the target feature vector and the template feature vector contained in the matching feature map.
As an optional implementation manner, the acquire image unit 400 is specifically configured to:
acquiring a template image set, wherein the template image set comprises template images of all parts of an object to be detected;
and determining a template image matched with the target part image according to the similarity of the target part image and the template image.
As an optional implementation manner, the image acquiring unit 400 is specifically configured to:
inputting the target part image into an image retrieval model, and determining a template image matched with the target part image from a retrieval library of the image retrieval model, wherein the retrieval library comprises a template image set, and the image retrieval model is obtained by utilizing a training part image and a template image corresponding to the training part image in the template image set for training.
As an optional implementation manner, the image acquiring unit 400 is specifically configured to:
inputting the target part image into the image retrieval model to obtain a first feature vector of the target part image;
and determining the template image corresponding to the highest similarity as the template image matched with the target part image according to the similarity of the first feature vector and the second feature vector of each template image in the template image set.
As an optional implementation manner, the image correction unit 402 is specifically configured to:
determining the rotation angle of the target part image according to the conversion relation between the transformation parameter matrix and the rotation angle, and correcting the target part image by using the rotation angle; and/or the presence of a gas in the gas,
and determining the scaling scale of the target part image according to the conversion relation between the transformation parameter matrix and the scaling scale, and correcting the target part image by using the scaling scale.
Based on the same inventive concept, an embodiment of the present invention further provides a computer storage medium, on which a computer program is stored, which when executed by a processor implements the following steps:
acquiring a target part image of an object to be detected and a template image matched with the target part image, wherein the template image represents an image conforming to a preset image specification;
determining a transformation parameter matrix of the target part image and the template image, wherein the transformation parameter matrix represents the mutual conversion relation of the target part image and the template image, and elements in the transformation parameter matrix comprise affine transformation parameters representing the rotation angle;
and correcting the target part image according to the transformation parameter matrix.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (10)
1. An image correction method, characterized in that the method comprises:
acquiring a target part image of an object to be detected and a template image matched with the target part image, wherein the template image represents an image conforming to a preset image specification;
determining a transformation parameter matrix of the target part image and the template image, wherein the transformation parameter matrix represents the mutual conversion relation of the target part image and the template image, and elements in the transformation parameter matrix comprise affine transformation parameters representing the rotation angle;
and correcting the target part image according to the transformation parameter matrix.
2. The method of claim 1, wherein determining a transformation parameter matrix for the target site image and the template image comprises:
extracting a target characteristic vector of a target part image and a template characteristic vector of the template image according to a trained characteristic extraction layer of the registration network model; the registration network model is obtained by performing unsupervised training by using a training part image and a template image matched with the training part image;
and determining the transformation parameter matrix according to the target characteristic vector and the template characteristic vector.
3. The method of claim 2, wherein determining the transformation parameter matrix based on the target feature vector and the template feature vector comprises:
performing feature matching on the target feature vector and the template feature vector according to a feature matching layer of the registration network model to obtain a matching feature map; wherein the matching feature map characterizes a matching degree of each vector component contained in the target feature vector and the template feature vector;
and determining the transformation parameter matrix according to the matching feature map.
4. The method of claim 3, wherein determining the transformation parameter matrix from the matching feature map comprises:
and performing parameter regression on the matching feature map according to a parameter regression layer of the registration network model to determine the transformation parameter matrix, wherein the parameter regression is used for predicting the feature matching modes of the target feature vector and the template feature vector contained in the matching feature map.
5. The method of claim 1, wherein the obtaining a template image that matches the target site image comprises:
acquiring a template image set, wherein the template image set comprises template images of all parts of an object to be detected;
and determining a template image matched with the target part image according to the similarity between the target part image and the template image.
6. The method according to claim 5, wherein determining a template image matching the target portion image according to the similarity between the target portion image and the template image comprises:
inputting the target part image into an image retrieval model, and determining a template image matched with the target part image from a retrieval library of the image retrieval model, wherein the retrieval library comprises a template image set, and the image retrieval model is obtained by utilizing a training part image and a template image corresponding to the training part image in the template image set for training.
7. The method of claim 6, wherein determining the template image matching the target site image from a search library of image search models comprises:
inputting the target part image into the image retrieval model to obtain a first feature vector of the target part image;
and determining the template image corresponding to the highest similarity as the template image matched with the target part image according to the similarity of the first feature vector and the second feature vector of each template image in the template image set.
8. The method according to any one of claims 1 to 7, wherein correcting the target site image according to the transformation parameter matrix comprises:
determining the rotation angle of the target part image according to the conversion relation between the transformation parameter matrix and the rotation angle, and correcting the target part image by using the rotation angle; and/or the presence of a gas in the gas,
and determining the scaling scale of the target part image according to the conversion relation between the transformation parameter matrix and the scaling scale, and correcting the target part image by using the scaling scale.
9. An image correction apparatus, characterized in that the apparatus comprises a processor and a memory for storing a program executable by the processor, the processor being adapted to read the program in the memory and to perform the steps of the method according to any of claims 1 to 8.
10. A computer storage medium having a computer program stored thereon, the program, when executed by a processor, implementing the steps of the method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210285649.7A CN114581340A (en) | 2022-03-22 | 2022-03-22 | Image correction method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210285649.7A CN114581340A (en) | 2022-03-22 | 2022-03-22 | Image correction method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114581340A true CN114581340A (en) | 2022-06-03 |
Family
ID=81775974
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210285649.7A Pending CN114581340A (en) | 2022-03-22 | 2022-03-22 | Image correction method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114581340A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116503913A (en) * | 2023-06-25 | 2023-07-28 | 浙江华诺康科技有限公司 | Medical image recognition method, device, system and storage medium |
-
2022
- 2022-03-22 CN CN202210285649.7A patent/CN114581340A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116503913A (en) * | 2023-06-25 | 2023-07-28 | 浙江华诺康科技有限公司 | Medical image recognition method, device, system and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| AU2017292642B2 (en) | System and method for automatic detection, localization, and semantic segmentation of anatomical objects | |
| US11557391B2 (en) | Systems and methods for human pose and shape recovery | |
| CN110599528A (en) | Unsupervised three-dimensional medical image registration method and system based on neural network | |
| US20220157047A1 (en) | Feature Point Detection | |
| Aranguren et al. | Improving the segmentation of magnetic resonance brain images using the LSHADE optimization algorithm | |
| CN111881926A (en) | Image generation method, image generation model training method, image generation device, image generation equipment and image generation medium | |
| US11963741B2 (en) | Systems and methods for human pose and shape recovery | |
| US20080310760A1 (en) | Method, a System and a Computer Program for Volumetric Registration | |
| CN111640145B (en) | Image registration method and related model training method, equipment and device thereof | |
| CN107767358B (en) | Method and device for determining ambiguity of object in image | |
| US20190392552A1 (en) | Spine image registration method | |
| Tang et al. | Retinal image registration based on robust non-rigid point matching method | |
| US11941738B2 (en) | Systems and methods for personalized patient body modeling | |
| CN114581340A (en) | Image correction method and device | |
| CN112070181B (en) | Image stream-based cooperative detection method and device and storage medium | |
| US8831301B2 (en) | Identifying image abnormalities using an appearance model | |
| CN111724371B (en) | Data processing method and device and electronic equipment | |
| CN113850794A (en) | Image processing method and device | |
| CN114038060A (en) | Detection method, terminal and storage medium for gait of leg joint | |
| CN114519729A (en) | Image registration quality evaluation model training method and device and computer equipment | |
| CN111681270A (en) | Method, device and storage medium for realizing registration between image frames | |
| CN113487659B (en) | Image registration method, device, equipment and storage medium | |
| CN116503453B (en) | Image registration method, image registration device, computer-readable storage medium and electronic device | |
| Bednarek et al. | Simulated Local Deformation & Focal Length Optimisation For Improved Template-Based 3D Reconstruction of Non-Rigid Objects | |
| CN117974472A (en) | Image stitching method and system based on depth point line characteristics |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |