CN110473241A - Method for registering images, storage medium and computer equipment - Google Patents

Method for registering images, storage medium and computer equipment Download PDF

Info

Publication number
CN110473241A
CN110473241A CN201910698059.5A CN201910698059A CN110473241A CN 110473241 A CN110473241 A CN 110473241A CN 201910698059 A CN201910698059 A CN 201910698059A CN 110473241 A CN110473241 A CN 110473241A
Authority
CN
China
Prior art keywords
image
transformation matrix
original floating
reference image
spatial transformation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910698059.5A
Other languages
Chinese (zh)
Other versions
CN110473241B (en
Inventor
高菲菲
曹晓欢
薛忠
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai United Imaging Intelligent Healthcare Co Ltd
Original Assignee
Shanghai United Imaging Intelligent Healthcare Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai United Imaging Intelligent Healthcare Co Ltd filed Critical Shanghai United Imaging Intelligent Healthcare Co Ltd
Priority to CN201910698059.5A priority Critical patent/CN110473241B/en
Priority to CN202210284241.8A priority patent/CN114820740A/en
Publication of CN110473241A publication Critical patent/CN110473241A/en
Application granted granted Critical
Publication of CN110473241B publication Critical patent/CN110473241B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/30Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/37Determination of transform parameters for the alignment of images, i.e. image registration using transform domain methods
    • GPHYSICS
    • G16INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16HHEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00ICT specially adapted for the handling or processing of medical images
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/30Subject of image; Context of image processing
    • G06T2207/30004Biomedical image processing

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Radiology & Medical Imaging (AREA)
  • Epidemiology (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Primary Health Care (AREA)
  • Public Health (AREA)
  • Image Processing (AREA)

Abstract

This application involves a kind of method for registering images, storage medium and computer equipments, when the position direction of original floating image and reference picture is inconsistent, it is not direct progress image registration, but directional correction first is carried out to original floating image by correction for direction image identical with reference picture position direction, then image registration is carried out again, it thereby may be ensured that the orientation consistency of registration image, registration accuracy improved, so that the robustness of method for registering images is more preferable.

Description

Image registration method, storage medium and computer device
Technical Field
The present application relates to the field of image processing technologies, and in particular, to an image registration method, a storage medium, and a computer device.
Background
With the continuous development of medical imaging, medical image analysis becomes an important content of clinical diagnosis, and the problem of medical image registration is gradually raised and becomes one of the hot topics in the field of medical image research. Medical image registration refers to finding some kind of spatial transformation to make the corresponding points of two images completely consistent in spatial position and anatomical structure, and the result of registration is required to make all anatomical points on the two images, or at least all points with diagnostic significance and operation region, match.
In the prior art, in the process of image registration, for image data in a group of image data sets, a spatial transformation is searched to map one image to another image, so that the purpose of information fusion is achieved. However, when the two images have a large posture deviation, the registration accuracy is reduced by performing the registration through the above process, and even the registration fails, so the existing image registration method has poor robustness.
Disclosure of Invention
Based on this, it is necessary to provide an image registration method, a storage medium, and a computer device with better robustness to solve the problems in the prior art.
An image registration method, comprising:
acquiring an original floating image to be registered and a reference image;
when the body position directions of the original floating image and the reference image are not consistent, acquiring a direction correction image, wherein the body position directions of the direction correction image and the reference image are consistent;
obtaining a first spatial transformation matrix according to the original floating image and the direction correction image, and performing body position direction transformation on the original floating image according to the first spatial transformation matrix to obtain a first transformation image, wherein the body position direction of the first transformation image is consistent with the body position direction of the reference image;
and after the original floating image is subjected to body position direction transformation to obtain the first transformation image, carrying out image registration processing on the original floating image and the reference image to obtain a corresponding image registration result.
A computer device comprising a memory storing a computer program and a processor implementing the steps of the above method when executing the computer program.
A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method.
The image registration method, the storage medium and the computer equipment acquire an original floating image to be registered and a reference image; when the body position directions of the original floating image and the reference image are not consistent, acquiring a direction correction image, wherein the body position directions of the direction correction image and the reference image are consistent; obtaining a first spatial transformation matrix according to the original floating image and the direction correction image, and carrying out body position direction transformation on the original floating image according to the first spatial transformation matrix to obtain a first transformation image, wherein the body position direction of the first transformation image is consistent with the body position direction of the reference image; after the body position direction transformation is carried out on the original floating image to obtain a first transformation image, image registration processing of the original floating image and the reference image is carried out to obtain a corresponding image registration result. When the body position directions of the original floating image and the reference image are not consistent, the image registration is not directly carried out, but the original floating image is subjected to orientation correction by means of the direction correction image with the same body position direction as the reference image, and then the image registration is carried out, so that the orientation consistency of the registered image can be ensured, the registration precision is improved, and the robustness of the image registration method is better.
Drawings
FIG. 1 is a flow diagram illustrating an image registration method in one embodiment;
FIG. 2 is a flowchart illustrating step S300 according to an embodiment;
FIG. 3 is a flow chart illustrating step 410 according to one embodiment;
FIG. 4 is a flowchart illustrating step 420 according to one embodiment;
FIG. 5 is a schematic flow chart of step 430 in one embodiment;
FIG. 6 is a schematic diagram of an embodiment of an image registration apparatus;
FIG. 7 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
In one embodiment, as shown in fig. 1, an image registration method is provided, which is explained by taking the method as an example applied to a processor capable of image registration, and the method comprises the following steps:
and step S100, acquiring an original floating image to be registered and a reference image.
In the image registration process, an image subjected to image transformation processing is called a floating image, and an image which is kept unchanged is called a reference image. Because the floating image and the reference image are respectively positioned in different coordinate systems, and the spatial positions of corresponding points contained in the two images are different, image registration of the floating image and the reference image is required, namely, the corresponding points of the two images are completely consistent in spatial position by searching for a spatial transformation, and the registration result is to enable all anatomical points on the two images, or at least all points with diagnostic significance and an operation area to be matched, so that the image data of the two images are combined to the same image, and the purpose of information fusion is achieved.
Specifically, the processor may acquire the image to be registered in real time, that is, perform image reconstruction and correction on data of the object to be detected acquired by the medical scanning device, so as to obtain the original floating image and the reference image to be registered. Of course, the image to be registered may also be reconstructed and corrected in advance, and stored in a memory connected to the processor, and when the image to be registered needs to be subjected to the registration processing, the processor directly reads the image to be registered from the memory. Of course, the processor may also obtain the image to be registered from an external device. For example, the image to be registered is stored in the cloud, and when the registration processing operation needs to be performed, the processor acquires the image to be registered from the cloud. The embodiment does not limit the specific way in which the processor acquires the image to be registered.
Optionally, after obtaining the original floating image and the reference image, the processor may first perform image preprocessing, such as resizing processing, noise removal processing, image segmentation processing, and the like, before performing image registration, and specifically, when the pixel sizes of the images to be registered are different, the resizing processing may facilitate feature correspondence. If the image to be registered is noisy, the noise removal process may be performed by an operation that can remove noise through a smoothing process or the like. Image segmentation refers to the segmentation of an image into portions from which features can be extracted. In the actual processing process, the preprocessing operation to be executed may be selected according to the actual situation, and is not specifically limited herein.
And step S200, when the body position directions of the original floating image and the reference image are not consistent, acquiring a direction correction image, wherein the body position directions of the direction correction image and the reference image are consistent.
After the processor acquires the original floating image and the reference image, firstly, the body position direction information of the two images is acquired, and whether the body position directions of the two images are consistent or not is judged. When the body positions are consistent in direction, the registration can be directly carried out through a registration algorithm in the prior art. The existing registration algorithm can match corresponding structures in two images to be registered, but the existing registration algorithm can be effectively realized on the premise that the posture information of the images to be registered is basically consistent. The following situations are often encountered in clinical practice: (1) patient coordinate system definitions are inconsistent, for example, in the itk (insight Segmentation and registration toolkit) toolkit, and the patient coordinate system definitions of the 3D Slicer are exactly opposite; (2) the scanning instrument, the examination part and the examination content are different, for example, three-dimensional volume data obtained by reconstruction along the directions of the coronal plane, the sagittal plane and the transverse plane of a patient according to the fracture condition during fracture CT scanning are different; (3) during scanning, the body position is inconsistent, such as whether the patient enters the scanning bed with the head advanced or the feet advanced, and the like. The above situation may cause a large deviation of the posture of the image before registration, which is difficult to be estimated and corrected effectively by an algorithm, and the use of such an image for direct registration may reduce the registration accuracy, even cause registration failure, and thus may be difficult to be applied effectively in clinical practice.
It can be understood that the body position directions indicated in this embodiment are the same, and may be the case that the body position directions are completely the same, or the errors of the body position directions of the two images are within a preset range, and specifically, the body position directions may be flexibly selected according to the actual situation.
The method and the device mainly aim at the condition that the body position directions of the two images are not consistent, and provide a workflow for firstly carrying out body position direction correction on an original floating image through a direction correction image and then carrying out image registration, so that the accuracy of a registration result is ensured. Therefore, in this step, when the processor determines that the body position directions of the original floating image to be registered and the reference image are not consistent by performing body position direction comparison, a direction correction image consistent with the body position direction of the reference image is acquired.
Alternatively, the orientation correction image may be constructed in real time according to the posture orientation of the reference image; or an image set consisting of a plurality of images with different body position directions may be constructed in advance, and then an image with the body position direction consistent with that of the reference image is screened from the image set as a direction correction image, which is not specifically limited herein.
And step S300, obtaining a first spatial transformation matrix according to the original floating image and the direction correction image, and performing body position direction transformation on the original floating image according to the first spatial transformation matrix to obtain a first transformation image, wherein the body position direction of the first transformation image is consistent with the body position direction of the reference image.
After the processor acquires the direction correction image, because the body position directions of the original floating image and the reference image are not consistent, if the image registration is directly carried out, the accuracy of the image registration result is greatly reduced, and therefore an ideal registration result cannot be obtained.
Step S400, after the body position direction of the original floating image is changed to obtain a first changed image, image registration processing of the original floating image and the reference image is carried out to obtain a corresponding image registration result.
The processor can obtain a first transformation image with the body position direction consistent with that of the reference image after the body position direction transformation is carried out on the original floating image, so that under the condition that the body position directions of the two images are consistent, the image registration processing of the original floating image and the reference image can be carried out, and a corresponding image registration result is obtained.
It is understood that in this embodiment, the original floating image and the reference image may be medical images of the same modality, such as any one of images obtained by Computed Radiography (CR), Digital Subtraction Angiography (DSA), Direct Digital Radiography (DDR), Computed Tomography (CT), Magnetic Resonance Imaging (MRI), Ultrasound imaging (US), Y-Scintigraphy (Y-Scintigraphy), Single Photon Emission Tomography (SPECT), Positron Emission Tomography (PET), etc. The original floating image and the reference image may also be medical images of different modalities, for example, the two images may be respectively anatomical imaging (such as CT, MRI, etc.) and functional imaging (such as PET, SPECT, etc.), wherein the resolution of the anatomical imaging is high, and anatomical morphological information of organs in a human body can be provided (for example, the CT image can clearly display a structural image of a bone, and MRI is suitable for soft tissue morphological imaging); functional imaging can provide functional metabolic information of organs and brains in a human body (for example, PET can reflect the metabolic conditions of organs in the body). The morphological information and the functional information obtained by different imaging technologies on the same anatomical structure of a human body are mutually different and complementary, and the imaging results can be combined and analyzed by registering images of different modalities, so that the accuracy of medical diagnosis is improved.
The embodiment provides an image registration method, when the body position directions of an original floating image and a reference image are not consistent, the image registration is not directly performed, but the original floating image is subjected to orientation correction by means of a direction correction image with the same body position direction as the reference image, and then the image registration is performed, so that the orientation consistency of the registered image can be ensured, the registration accuracy is improved, and the robustness of the image registration method is better.
In one embodiment, as shown in fig. 2, step S300 is to obtain a first spatial transformation matrix according to the original floating image and the direction-corrected image, and perform posture direction transformation on the original floating image according to the first spatial transformation matrix to obtain a first transformed image, including steps S310 to S330.
Step S310, acquiring image information of a direction correction image, wherein the image information at least comprises a body position direction of the direction correction image;
step S320, obtaining a first spatial transformation matrix according to the image information of the direction correction image and the body position direction of the original floating image;
and step S330, carrying out posture direction transformation on the original floating image according to the first space transformation matrix to obtain a first transformation image.
Specifically, the image information of the direction correction image can be obtained through the image information of the original floating image and the body position direction of the reference image, then a first spatial transformation matrix is obtained according to the image information of the direction correction image and the body position direction of the original floating image, and further body position direction transformation processing is carried out according to the first spatial transformation matrix.
In one embodiment, the image information of the orientation corrected image further includes: the orientation corrects at least one of an origin of the image, an image size, and a voxel size. Because the image registration process involves image transformation processing, to some extent, a certain degree of precision loss exists in the image registration process, so that in the registration process, calculation is performed by combining other image information such as an origin, an image size and a voxel size, the precision loss caused by image registration can be reduced as much as possible, and the accuracy of the registration result is ensured.
In one embodiment, the image registration processing of the original floating image and the reference image is performed to obtain a corresponding image registration result, including: and step 410, performing image registration processing on the original floating image and the reference image according to the reference image and the first transformation image to obtain a corresponding first image registration result.
As shown in fig. 3, step 410 performs image registration processing on the original floating image and the reference image according to the reference image and the first transformed image to obtain a corresponding first image registration result, which includes steps S412 to S414.
Step S412, a second spatial transformation matrix is obtained according to the reference image and the first transformation image;
and S414, carrying out image space transformation on the first transformed image according to the second space transformation matrix to obtain a first registration result of the original floating image and the reference image.
Specifically, the processor further performs image conversion processing on the first converted image after obtaining the first converted image. At this time, the body position direction of the first transformed image is consistent with that of the reference image, and the coordinate systems of the two images are different, so that after a second spatial transformation matrix representing the image transformation relation of the two images is obtained, the first transformed image is subjected to image spatial transformation according to the second spatial transformation matrix, that is, the first transformed image is transformed to the coordinate system of the reference image, and thus the image registration of the first transformed image and the reference image is completed. In addition, since the first transformed image is obtained by transforming the original floating image through the posture direction, the image registration of the original floating image and the reference image can also be considered to be completed, so that the image registration result of the first transformed image and the reference image can be used as the first registration result of the original floating image and the reference image, and the first registration result and the reference image are in the same coordinate system. In addition, the gray value information of the first registration result is derived from the original floating image.
In one embodiment, the image registration processing of the original floating image and the reference image is performed to obtain a corresponding image registration result, including: and step 420, performing image registration processing on the original floating image and the reference image according to the reference image, the first transformed image, the first spatial transformation matrix and the original floating image to obtain a corresponding first image registration result.
As shown in fig. 4, in step 420, the original floating image and the reference image are subjected to image registration processing according to the reference image, the first transformed image, the first spatial transformation matrix, and the original floating image, so as to obtain a corresponding first image registration result, which includes steps S422 to S426.
Step S422, a second spatial transformation matrix is obtained according to the reference image and the first transformation image;
step S424, obtaining a third spatial transformation matrix according to the first spatial transformation matrix and the second spatial transformation matrix;
and step S426, carrying out image space transformation on the original floating image according to the third space transformation matrix to obtain a first registration result of the original floating image and the reference image.
Specifically, the processor may perform image transformation processing on the original floating image after obtaining the first transformed image. At this time, a second spatial transformation matrix is obtained according to the reference image and the first transformation image, since the first spatial transformation matrix represents the image transformation relationship between the original floating image and the direction correction image, and the second spatial transformation matrix represents an image transformation relationship of the first transformed image to the reference image, a third spatial transformation matrix representing an image transformation relationship of the original floating image and the reference image may be obtained based on the first spatial transformation matrix and the second spatial transformation matrix, such that the original floating image is image-space transformed based on the third spatial transformation matrix, the original floating image is transformed to a coordinate system where the reference image is located, so that image registration of the original floating image and the reference image is completed, a first registration result of the original floating image and the reference image is obtained, and the first registration result and the reference image are in the same coordinate system. In addition, the gray value information of the first registration result is derived from the original floating image.
Optionally, when the third spatial transformation matrix is obtained according to the first spatial transformation matrix and the second spatial transformation matrix, the third spatial transformation matrix is a product of the first spatial transformation matrix and the second spatial transformation matrix. For example, with M1、M2、M3Respectively representing a first spatial transformation matrix, a second spatial transformation matrix and a third spatial transformation matrix, wherein the calculation formula of the third spatial transformation matrix is as follows: m3=M1×M2
In one embodiment, the image registration processing of the original floating image and the reference image is performed to obtain a corresponding image registration result, including: and 430, performing image registration processing on the original floating image and the reference image according to the reference image, the first transformed image, the first spatial transformation matrix and the reference image to obtain a corresponding second image registration result.
As shown in fig. 5, step 430 performs image registration processing on the original floating image and the reference image according to the reference image, the first transformed image, the first spatial transformation matrix, and the reference image, to obtain a corresponding second image registration result, which includes steps S432 to S438.
Step S432, a second spatial transformation matrix is obtained according to the reference image and the first transformation image;
step S434, obtaining a third spatial transformation matrix according to the first spatial transformation matrix and the second spatial transformation matrix;
step S436, obtaining a fourth spatial transformation matrix according to the third spatial transformation matrix, wherein the fourth spatial transformation matrix is an inverse matrix of the third spatial transformation matrix;
and step 438, performing image space transformation on the reference image according to the fourth space transformation matrix to obtain a second registration result of the original floating image and the reference image.
Specifically, the processor may perform image transformation processing on the reference image after obtaining the first transformed image. At this time, a second spatial transformation matrix is obtained according to the reference image and the first transformation image, since the first spatial transformation matrix represents the image transformation relationship between the original floating image and the direction correction image, and the second spatial transformation matrix represents the image transformation relationship from the first transformation image to the reference image, a third spatial transformation matrix representing the image transformation relationship between the original floating image and the reference image can be obtained according to the first spatial transformation matrix and the second spatial transformation matrix, and further, an inverse matrix of the third spatial transformation matrix, that is, a fourth spatial transformation matrix represents the image transformation relationship from the reference image to the original floating image, so that the reference image is subjected to image spatial transformation according to the fourth spatial transformation matrix, that is, the reference image is transformed to the coordinate system where the original floating image is located, thereby completing the image registration between the original floating image and the reference image, the image registration process may be understood as an "inverse registration process" to obtain a second registration result of the original floating image and the reference image, and the second registration result is in the same coordinate system as the original floating image. In addition, the gray value information of the second registration result is derived from the reference image.
In one embodiment, a specific application example of the image registration method of the present application is provided.
For the original floating image A, the corresponding image coordinate system and the world coordinate system have the mutual transformation relationship as follows:
for the reference image B, the corresponding image coordinate system and the world coordinate system have the mutual transformation relationship as follows:
the direction matrix of the original floating image A is as follows:
the direction matrix of the reference picture B is:
acquiring a direction correction image K consistent with the body position direction of the reference image B, wherein for the direction correction image K, the interconversion relationship between the corresponding image coordinate system and the world coordinate system is as follows:
due to the fact thatAnd assume thatA first spatial transformation Matrix M1 (expressed as Transform _ Matrix) of the original floating image a to the orientation transformed image K is obtained as:
the voxel values of the direction-transformed image K are:
under the previous assumption (unchanged image origin), the positions (f _ box _ coordinates) of the image coordinate system of the 8 corner points of the original floating image a are applied to the first spatial transformation matrix M1 to obtain the positions (t _ box _ coordinates) of the image coordinate system of the 8 corner points of the orientation transformed image K.
Comparing the positions (d _ box _ coordinates) of the image coordinate system of the 8 corner points of the direction correction image K with the minimum value and the maximum value of the x-axis y-axis z-axis respectively, and then the size of the direction correction image K is the absolute value of the difference between the minimum value and the maximum value. Wherein the world coordinate of the minimum value is the new origin position of the direction correction image KSince the origin is changed, the world coordinates of the orientation corrected image K are changed accordingly, this transformation is represented by the translation vector, and the first spatial transformation Matrix M1 (Transform _ Matrix) for posture correction is obtained by the following formula
M1=Transform_Matrix=Voxel_to_World_Matrixflo -1×Voxel_to_World_Matrixdst
Wherein,
specifically, for example, the direction matrix, voxel size, origin, and image size of the original floating image a are:
the direction matrix, voxel size, origin, and image size of the reference image B are respectively:
setting a direction matrix, a voxel size, an origin and an image size of the direction transformation image K as follows: orientationdst,Spacingdst,Origindst,Sizedst
The direction matrix of the direction correction image K is the same as that of the reference image B, and if the origin of the direction correction image K is the same as that of the original floating image A, the transformation relation between the world coordinate system and the image coordinate system in the direction correction image K is as follows:
World_Positiondst=Voxel_to_World_Matrxdst×Spacing_Matrixdst×Voxel_Positiondst
(1) solving the spatial transformation relation of posture correction under the condition of unchanged origin
The world coordinate system position is independent of which image coordinate system the patient is in, so, taking into account the voxel size, the spatial transformation of the original floating image a with the orientation-corrected image K is as follows,
for the above derivation, assuming that the origin of the direction correction image K is the same as the origin of the original floating image a, Transform _ Matrix is the spatial transformation relationship between the original floating image a and the direction correction image K, i.e. the first spatial transformation Matrix M1 in this application; voxel _ Transform _ Matrix takes into account the spatial transformation of Voxel size for the original floating image a and the orientation corrected image K.
In addition, if the elements in the rotation matrix are non-integer, when the original floating image a is spatially transformed onto the direction correction image K, the original floating image a needs to be interpolated, and the first transformed image obtained in this way loses the precision of the original floating image a, so that the rotation matrix needs to be rounded. The posture correction rotation matrix and the translation vector are as follows:
(2) solving the image basic information of the orientation corrected image K
Various image information of the orientation corrected image K can be obtained from the rotation matrix as follows:
the image direction is:
in order to ensure that the voxel sizes of the corresponding positions before and after the spatial transformation of the original floating image A and the orientation corrected image K are the same, i.e.
Voxe_Transform_Matrix=Transform_Matrix
The image voxel size is:
before calculating the image origin and the image size of the direction correction image K, it is necessary to calculate the correspondence relationship between the original floating image a and each Voxel position of the direction correction image K in the image coordinate system under the assumption that the origin is unchanged as follows, which is Voxel _ Transform _ Matrix, wherein:
respectively calculating 8 image corner points of the original floating image A in an image coordinate system according to the image size of the original floating image A to obtain an image corner point matrix Box _ Coords with the matrix size of the original floating image A being 4 multiplied by 8floAnd obtaining an image corner matrix of the direction correction image K under the corresponding image coordinate system under the condition that the original point is not changed by the image corner matrix of the original floating image A and the space transformation relation of the original floating image A and the direction correction image K under the image coordinate system:
wherein Box _ CoordsdstThe matrix size of (2) is 4 x 8, wherein the first 3 rows respectively represent the positions of 8 corner points on x, y and z image coordinate axes. Respectively comparing 8 values of each row of the matrix to obtain the minimum value Box _ Coords _ Min of the x, y and z axis positions of 8 angular points after the spatial transformation of an image coordinate system under the condition that the origin of the direction correction image K is consistent with the origin of the original floating image AdstAnd the maximum value Box _ Coords _ Maxdst
The image size is:
according to Box _ Coords _ Min at the same timedstUpdating the origin of the orientation corrected image, the new origin being Box _ Coords _ MindstThe corresponding world coordinate system positions, namely:
in the image registration process, the calculation is carried out by combining other image information such as an origin, the image size, the voxel size and the like, so that the precision loss caused by image registration can be reduced as much as possible, and the accuracy of the registration result is ensured.
(3) Solving spatial transformation matrix for postural correction
The spatial transformation relationship between the world coordinate system and the image coordinate system obtained by updating the image information of the direction corrected image K is updated to,
(4) solving process of third spatial transformation matrix from original image floating image A to reference image B
By means of Linear _ Transform _ Matrix representing the second spatial transformation Matrix M2 for Linear registration between the orientation corrected image K and the reference image B, the second spatial transformation Matrix M2 is calculated as:
after the second spatial transformation Matrix M2(Linear _ Transform _ Matrix) is obtained, the original floating image a is corrected in the body position direction according to the first spatial transformation Matrix M1 to obtain a first corrected image a ', and the first corrected image a' is subjected to image space transformation according to the second spatial transformation Matrix M2, so that the original floating image a can be transformed into the coordinate system of the reference image B, and the registration of the two images is completed.
In addition, when the combination _ Transform _ Matrix represents the third spatial Transform Matrix (joint spatial Transform Matrix) M3, the calculation formula for obtaining the third spatial Transform Matrix M3 from the first spatial Transform Matrix (posture correction Matrix) M1 and the second spatial Transform Matrix (linear registration Matrix) M2 is:
the third spatial transformation Matrix M3 (combination _ Transform _ Matrix) is a spatial transformation Matrix of the original floating image a and the reference image B, and after the third spatial transformation Matrix M3 is obtained, the original floating image a is subjected to image space transformation according to the third spatial transformation Matrix M3, that is, the original floating image a is transformed into a coordinate system where the reference image B is located, so that the registration of the two images is completed.
(5) Solving process of fourth space transformation matrix from reference image B to original floating image A
Expressing a fourth spatial transformation Matrix M4 (an Inverse Matrix of the three-spatial transformation Matrix M3) by Inverse _ transformation _ Matrix, and solving the Inverse Matrix of the third spatial transformation Matrix M3 to obtain a fourth spatial transformation Matrix M4 of the reference image B inverted to the original floating image A, wherein a calculation formula of the fourth spatial transformation Matrix M4 is specifically as follows:
after the fourth spatial transformation matrix M4 is obtained, the reference image B may also be subjected to image space transformation according to the fourth spatial transformation matrix M4, that is, the reference image B is transformed to the coordinate system of the original floating image a (i.e., inverse registration), so as to complete the registration of the two images.
In the embodiment, when the original floating image A and the reference image B are subjected to image registration, because the body position directions of the two images are different, the image registration is not directly performed, but the original floating image A is subjected to azimuth correction by using the direction correction image K with the same body position direction as the reference image B, and then the image registration is performed, so that the orientation consistency of the registered images can be ensured, the registration accuracy is improved, and the robustness of the image registration method is better.
It should be understood that, under reasonable circumstances, although the steps in the flowcharts referred to in the foregoing embodiments are shown in sequence as indicated by the arrows, the steps are not necessarily executed in sequence as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps in each flowchart may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the sub-steps or stages is not necessarily sequential, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
In one embodiment, as shown in fig. 6, there is provided an image registration apparatus including: a first image acquisition module 100, a second image acquisition module 200, a body position direction transformation module 300, and an image registration processing module 400.
The first image acquisition module 100 is configured to acquire an original floating image to be registered and a reference image;
the second image obtaining module 200 is configured to obtain a direction correction image when the body position directions of the original floating image and the reference image are not the same, where the body position directions of the direction correction image and the reference image are the same;
the body position direction transformation module 300 is configured to obtain a first spatial transformation matrix according to the original floating image and the direction correction image, and perform body position direction transformation on the original floating image according to the first spatial transformation matrix to obtain a first transformed image, where a body position direction of the first transformed image is consistent with a body position direction of the reference image;
the image registration processing module 400 is configured to perform image registration processing on the original floating image and the reference image to obtain a corresponding image registration result after obtaining a first transformed image by performing posture direction transformation on the original floating image.
For specific definition of the image registration apparatus, reference may be made to the above definition of the image registration method, which is not described herein again. The modules in the image registration apparatus can be implemented in whole or in part by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program: acquiring an original floating image to be registered and a reference image; when the body position directions of the original floating image and the reference image are not consistent, acquiring a direction correction image, wherein the body position directions of the direction correction image and the reference image are consistent; obtaining a first spatial transformation matrix according to the original floating image and the direction correction image, and carrying out body position direction transformation on the original floating image according to the first spatial transformation matrix to obtain a first transformation image, wherein the body position direction of the first transformation image is consistent with the body position direction of the reference image; after the body position direction transformation is carried out on the original floating image to obtain a first transformation image, image registration processing of the original floating image and the reference image is carried out to obtain a corresponding image registration result.
In one embodiment, the processor, when executing the computer program, further performs the steps of: acquiring image information of a direction correction image, wherein the image information at least comprises a body position direction of the direction correction image; obtaining a first spatial transformation matrix according to the image information of the direction correction image and the body position direction of the original floating image; and carrying out posture direction transformation on the original floating image according to the first space transformation matrix to obtain a first transformation image.
In one embodiment, the processor, when executing the computer program, further implements any of:
the first item: carrying out image registration processing on the original floating image and the reference image according to the reference image and the first transformation image to obtain a corresponding first image registration result;
the second term is: performing image registration processing on the original floating image and the reference image according to the reference image, the first transformation image, the first spatial transformation matrix and the original floating image to obtain a corresponding first image registration result;
the third item: and carrying out image registration processing on the original floating image and the reference image according to the reference image, the first transformation image, the first spatial transformation matrix and the reference image to obtain a corresponding second image registration result.
In one embodiment, the processor, when executing the computer program, further performs the steps of: obtaining a second spatial transformation matrix according to the reference image and the first transformation image; and performing image space transformation on the first transformed image according to the second space transformation matrix to obtain a first registration result of the original floating image and the reference image.
In one embodiment, the processor, when executing the computer program, further performs the steps of: obtaining a second spatial transformation matrix according to the reference image and the first transformation image; obtaining a third spatial transformation matrix according to the first spatial transformation matrix and the second spatial transformation matrix; and performing image space transformation on the original floating image according to the third space transformation matrix to obtain a first registration result of the original floating image and the reference image.
In one embodiment, the processor, when executing the computer program, further performs the steps of: obtaining a second spatial transformation matrix according to the reference image and the first transformation image; obtaining a third spatial transformation matrix according to the first spatial transformation matrix and the second spatial transformation matrix; obtaining a fourth spatial transformation matrix according to the third spatial transformation matrix, wherein the fourth spatial transformation matrix is an inverse matrix of the third spatial transformation matrix; and performing image space transformation on the reference image according to the fourth space transformation matrix to obtain a second registration result of the original floating image and the reference image.
FIG. 7 is a diagram illustrating an internal structure of a computer device in one embodiment. The computer device may specifically be a terminal (or server). As shown in fig. 7, the computer apparatus includes a processor, a memory, a network interface, an input device, and a display screen connected through a system bus. Wherein the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program that, when executed by the processor, causes the processor to implement a video bitrate control method and a video transcoding method. The internal memory may also store a computer program, which when executed by the processor, causes the processor to perform a video bitrate control method and a video transcoding method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the architecture shown in fig. 7 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of: acquiring an original floating image to be registered and a reference image; when the body position directions of the original floating image and the reference image are not consistent, acquiring a direction correction image, wherein the body position directions of the direction correction image and the reference image are consistent; obtaining a first spatial transformation matrix according to the original floating image and the direction correction image, and carrying out body position direction transformation on the original floating image according to the first spatial transformation matrix to obtain a first transformation image, wherein the body position direction of the first transformation image is consistent with the body position direction of the reference image; after the body position direction transformation is carried out on the original floating image to obtain a first transformation image, image registration processing of the original floating image and the reference image is carried out to obtain a corresponding image registration result.
In one embodiment, the computer program when executed by the processor further performs the steps of: acquiring image information of a direction correction image, wherein the image information at least comprises a body position direction of the direction correction image; obtaining a first spatial transformation matrix according to the image information of the direction correction image and the body position direction of the original floating image; and carrying out posture direction transformation on the original floating image according to the first space transformation matrix to obtain a first transformation image.
In one embodiment, the computer program when executed by the processor further implements any of:
the first item: carrying out image registration processing on the original floating image and the reference image according to the reference image and the first transformation image to obtain a corresponding first image registration result;
the second term is: performing image registration processing on the original floating image and the reference image according to the reference image, the first transformation image, the first spatial transformation matrix and the original floating image to obtain a corresponding first image registration result;
the third item: and carrying out image registration processing on the original floating image and the reference image according to the reference image, the first transformation image, the first spatial transformation matrix and the reference image to obtain a corresponding second image registration result.
In one embodiment, the computer program when executed by the processor further performs the steps of: obtaining a second spatial transformation matrix according to the reference image and the first transformation image; and performing image space transformation on the first transformed image according to the second space transformation matrix to obtain a first registration result of the original floating image and the reference image.
In one embodiment, the computer program when executed by the processor further performs the steps of: obtaining a second spatial transformation matrix according to the reference image and the first transformation image; obtaining a third spatial transformation matrix according to the first spatial transformation matrix and the second spatial transformation matrix; and performing image space transformation on the original floating image according to the third space transformation matrix to obtain a first registration result of the original floating image and the reference image.
In one embodiment, the computer program when executed by the processor further performs the steps of: obtaining a second spatial transformation matrix according to the reference image and the first transformation image; obtaining a third spatial transformation matrix according to the first spatial transformation matrix and the second spatial transformation matrix; obtaining a fourth spatial transformation matrix according to the third spatial transformation matrix, wherein the fourth spatial transformation matrix is an inverse matrix of the third spatial transformation matrix; and performing image space transformation on the reference image according to the fourth space transformation matrix to obtain a second registration result of the original floating image and the reference image.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware related to instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, the computer program can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims (10)

1. An image registration method, comprising:
acquiring an original floating image to be registered and a reference image;
when the body position directions of the original floating image and the reference image are not consistent, acquiring a direction correction image, wherein the body position directions of the direction correction image and the reference image are consistent;
obtaining a first spatial transformation matrix according to the original floating image and the direction correction image, and performing body position direction transformation on the original floating image according to the first spatial transformation matrix to obtain a first transformation image, wherein the body position direction of the first transformation image is consistent with the body position direction of the reference image;
and after the original floating image is subjected to body position direction transformation to obtain the first transformation image, carrying out image registration processing on the original floating image and the reference image to obtain a corresponding image registration result.
2. The method of claim 1, wherein obtaining a first spatial transformation matrix from the original floating image and the orientation-corrected image, and performing posture-orientation transformation on the original floating image according to the first spatial transformation matrix to obtain a first transformed image, comprises:
acquiring image information of the direction correction image, wherein the image information at least comprises a body position direction of the direction correction image;
obtaining the first spatial transformation matrix according to the image information of the direction correction image and the body position direction of the original floating image;
and carrying out posture direction transformation on the original floating image according to the first spatial transformation matrix to obtain the first transformation image.
3. The method of claim 2, wherein the image information of the orientation corrected image further comprises: the orientation corrects at least one of an origin of the image, an image size, and a voxel size.
4. The method according to claim 1, wherein performing image registration processing of the original floating image and the reference image results in corresponding image registration results, comprising any one of:
the first item: performing image registration processing on the original floating image and the reference image according to the reference image and the first transformation image to obtain a corresponding first image registration result;
the second term is: performing image registration processing on the original floating image and the reference image according to the reference image, the first transformed image, the first spatial transformation matrix and the original floating image to obtain a corresponding first image registration result;
the third item: and carrying out image registration processing on the original floating image and the reference image according to the reference image, the first transformation image, the first spatial transformation matrix and the reference image to obtain a corresponding second image registration result.
5. The method according to claim 4, wherein performing image registration processing on the original floating image and the reference image according to the reference image and the first transformed image to obtain a corresponding first image registration result comprises:
obtaining a second spatial transformation matrix according to the reference image and the first transformation image;
and performing image space transformation on the first transformed image according to the second space transformation matrix to obtain a first registration result of the original floating image and the reference image.
6. The method of claim 4, wherein performing image registration processing on the original floating image and the reference image according to the reference image, the first transformed image, the first spatial transformation matrix, and the original floating image to obtain a corresponding first image registration result comprises:
obtaining a second spatial transformation matrix according to the reference image and the first transformation image;
obtaining a third spatial transformation matrix according to the first spatial transformation matrix and the second spatial transformation matrix;
and performing image space transformation on the original floating image according to the third space transformation matrix to obtain a first registration result of the original floating image and the reference image.
7. The method of claim 4, wherein performing image registration processing on the original floating image and the reference image according to the reference image, the first transformed image, the first spatial transformation matrix, and the reference image to obtain a corresponding second image registration result comprises:
obtaining a second spatial transformation matrix according to the reference image and the first transformation image;
obtaining a third spatial transformation matrix according to the first spatial transformation matrix and the second spatial transformation matrix;
obtaining a fourth spatial transformation matrix according to the third spatial transformation matrix, wherein the fourth spatial transformation matrix is an inverse matrix of the third spatial transformation matrix;
and performing image space transformation on the reference image according to the fourth space transformation matrix to obtain a second registration result of the original floating image and the reference image.
8. The method according to claim 6 or 7, wherein the third spatial transform matrix is a product of the first spatial transform matrix and the second spatial transform matrix.
9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 8 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 8.
CN201910698059.5A 2019-07-31 2019-07-31 Image registration method, storage medium and computer device Active CN110473241B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201910698059.5A CN110473241B (en) 2019-07-31 2019-07-31 Image registration method, storage medium and computer device
CN202210284241.8A CN114820740A (en) 2019-07-31 2019-07-31 Image registration method, storage medium and computer device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910698059.5A CN110473241B (en) 2019-07-31 2019-07-31 Image registration method, storage medium and computer device

Related Child Applications (1)

Application Number Title Priority Date Filing Date
CN202210284241.8A Division CN114820740A (en) 2019-07-31 2019-07-31 Image registration method, storage medium and computer device

Publications (2)

Publication Number Publication Date
CN110473241A true CN110473241A (en) 2019-11-19
CN110473241B CN110473241B (en) 2022-03-01

Family

ID=68509279

Family Applications (2)

Application Number Title Priority Date Filing Date
CN202210284241.8A Pending CN114820740A (en) 2019-07-31 2019-07-31 Image registration method, storage medium and computer device
CN201910698059.5A Active CN110473241B (en) 2019-07-31 2019-07-31 Image registration method, storage medium and computer device

Family Applications Before (1)

Application Number Title Priority Date Filing Date
CN202210284241.8A Pending CN114820740A (en) 2019-07-31 2019-07-31 Image registration method, storage medium and computer device

Country Status (1)

Country Link
CN (2) CN114820740A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111932443A (en) * 2020-07-16 2020-11-13 江苏师范大学 Method for improving registration accuracy of ultrasound and magnetic resonance by combining multi-scale expression with contrast agent
CN112434572A (en) * 2020-11-09 2021-03-02 北京极豪科技有限公司 Fingerprint image calibration method and device, electronic equipment and storage medium

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103337065A (en) * 2013-05-22 2013-10-02 西安电子科技大学 Non-rigid registering method of mouse three-dimensional CT image
CN105389815A (en) * 2015-10-29 2016-03-09 武汉联影医疗科技有限公司 Breast image registration method and device
CN106846386A (en) * 2017-02-08 2017-06-13 南通大学 3D cranium method for registering images based on ROI and conformal geometric algebra property invariant
CN109276248A (en) * 2018-11-30 2019-01-29 上海联影医疗科技有限公司 Self-oscillate position method and Medical Imaging System for Medical Imaging System
JP2019109313A (en) * 2017-12-15 2019-07-04 株式会社シャルマン Method of setting reference front of three-dimensional facial image, method of selecting spectacles using the same, and method of generating customer records using the same

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103337065A (en) * 2013-05-22 2013-10-02 西安电子科技大学 Non-rigid registering method of mouse three-dimensional CT image
CN105389815A (en) * 2015-10-29 2016-03-09 武汉联影医疗科技有限公司 Breast image registration method and device
CN106846386A (en) * 2017-02-08 2017-06-13 南通大学 3D cranium method for registering images based on ROI and conformal geometric algebra property invariant
JP2019109313A (en) * 2017-12-15 2019-07-04 株式会社シャルマン Method of setting reference front of three-dimensional facial image, method of selecting spectacles using the same, and method of generating customer records using the same
CN109276248A (en) * 2018-11-30 2019-01-29 上海联影医疗科技有限公司 Self-oscillate position method and Medical Imaging System for Medical Imaging System

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
XIA LI ET AL.: "Automatic nonrigid registration of whole body CT mice images", 《MEDICAL PHYSICS》 *
刘树伟: "《功能神经影像学》", 31 August 2011, 山东科学技术出版社 *
南玲巧: "肺部非刚性医学图像配准算法的研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 *

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111932443A (en) * 2020-07-16 2020-11-13 江苏师范大学 Method for improving registration accuracy of ultrasound and magnetic resonance by combining multi-scale expression with contrast agent
CN111932443B (en) * 2020-07-16 2024-04-02 江苏师范大学 Method for improving registration accuracy of ultrasound and magnetic resonance by combining multiscale expression with contrast agent
CN112434572A (en) * 2020-11-09 2021-03-02 北京极豪科技有限公司 Fingerprint image calibration method and device, electronic equipment and storage medium
CN112434572B (en) * 2020-11-09 2022-05-06 北京极豪科技有限公司 Fingerprint image calibration method and device, electronic equipment and storage medium

Also Published As

Publication number Publication date
CN110473241B (en) 2022-03-01
CN114820740A (en) 2022-07-29

Similar Documents

Publication Publication Date Title
CN109993726B (en) Medical image detection method, device, equipment and storage medium
CN102667857B (en) Bone in X-ray photographs suppresses
JP6316671B2 (en) Medical image processing apparatus and medical image processing program
US20140323845A1 (en) Automated 3-d orthopedic assessments
CN110717961B (en) Multi-modal image reconstruction method and device, computer equipment and storage medium
US20080050043A1 (en) Methods and Systems for Registration of Images
US10796498B2 (en) Image processing apparatus, image processing method, and non-transitory computer-readable medium
JP2008546441A (en) Elastic image registration method based on a model for comparing first and second images
US20140294263A1 (en) Synchronized Navigation of Medical Images
CN110570483A (en) scanning method, scanning device, computer equipment and storage medium
CN111462018B (en) Image alignment method in image, computer equipment and storage medium
CN110473241B (en) Image registration method, storage medium and computer device
CN111223158B (en) Artifact correction method for heart coronary image and readable storage medium
CN115526929A (en) Image-based registration method and device
US20180025501A1 (en) Image processing apparatus, image processing method, and, non-transitory computer readable medium
CN111583099A (en) Image rectification method, computer device, and storage medium
US11903691B2 (en) Combined steering engine and landmarking engine for elbow auto align
CN113989110A (en) Lung image registration method and device, computer equipment and storage medium
CN112150485B (en) Image segmentation method, device, computer equipment and storage medium
CN111161330B (en) Non-rigid image registration method, device, system, electronic equipment and storage medium
CN111161369B (en) Image reconstruction storage method, device, computer equipment and storage medium
US12100075B2 (en) Image reconstruction by modeling image formation as one or more neural networks
CN117422724A (en) Training method of knee joint image segmentation model and knee joint image segmentation method
Rivest-Hénault et al. Inverse-consistent rigid registration of CT and MR for MR-based planning and adaptive prostate radiation therapy
Andronache Multi-modal non-rigid registration of volumetric medical images

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant