CN109934934B - Medical image display method and device based on augmented reality - Google Patents
Medical image display method and device based on augmented reality Download PDFInfo
- Publication number
- CN109934934B CN109934934B CN201910199816.4A CN201910199816A CN109934934B CN 109934934 B CN109934934 B CN 109934934B CN 201910199816 A CN201910199816 A CN 201910199816A CN 109934934 B CN109934934 B CN 109934934B
- Authority
- CN
- China
- Prior art keywords
- image
- medical
- matching
- virtual
- dimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000003190 augmentative effect Effects 0.000 title claims abstract description 60
- 238000000034 method Methods 0.000 title claims abstract description 43
- 238000002059 diagnostic imaging Methods 0.000 claims abstract description 14
- 238000007499 fusion processing Methods 0.000 claims abstract description 13
- 238000003745 diagnosis Methods 0.000 claims description 33
- 230000004927 fusion Effects 0.000 claims description 28
- 201000010099 disease Diseases 0.000 claims description 21
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 claims description 21
- 238000000605 extraction Methods 0.000 claims description 9
- 238000003672 processing method Methods 0.000 claims description 6
- 238000010276 construction Methods 0.000 claims description 3
- 238000009825 accumulation Methods 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 6
- 238000013178 mathematical model Methods 0.000 description 5
- 230000011218 segmentation Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 230000007547 defect Effects 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
Landscapes
- Measuring And Recording Apparatus For Diagnosis (AREA)
Abstract
The invention discloses a medical image display method and a device based on augmented reality, which are characterized in that the method comprises the following steps: acquiring an original medical image based on a medical imaging device; generating a virtual medical image based on the original medical image and personal information of the patient, and acquiring a matched medical matching image in a medical image database; performing three-dimensional fusion processing on the virtual medical image and the medical matching image to obtain a virtual three-dimensional medical characteristic image; and matching and fusing the original medical image and the virtual three-dimensional medical characteristic image, and obtaining and displaying an augmented reality medical image. In the embodiment of the invention, the display diversity and the friendliness of the medical image are improved.
Description
Technical Field
The invention relates to the technical field of medical display, in particular to a medical image display method and device based on augmented reality.
Background
With the development of technology, the continuous improvement of medical level and the continuous improvement of living standard of people, more and more people begin to pay attention to their own physical health condition, and the number of times of performing the examination such as X-ray, B-ultrasonic and CT is also continuously increased, and the medical images generated by the examination need to be displayed and printed.
However, in the prior art, the medical images of the examination results are displayed singly and then printed directly, so that the display contents are relatively single and unfriendly when displayed.
Disclosure of Invention
The invention aims to overcome the defects of the prior art, and provides a medical image display method and device based on augmented reality, which improves the display diversity and friendliness of medical images.
In order to solve the above technical problems, an embodiment of the present invention provides a medical image display method based on augmented reality, including:
acquiring an original medical image based on a medical imaging device;
generating a virtual medical image based on the original medical image and personal information of the patient, and acquiring a matched medical matching image in a medical image database;
performing three-dimensional fusion processing on the virtual medical image and the medical matching image to obtain a virtual three-dimensional medical characteristic image;
and matching and fusing the original medical image and the virtual three-dimensional medical characteristic image, and obtaining and displaying an augmented reality medical image.
Optionally, the generating a virtual medical image based on the original medical image and the personal information of the patient, and acquiring the matched medical matching image in the medical image database, includes:
acquiring a target image in the original medical image;
obtaining age information, sex information, and condition diagnosis information of a patient based on personal information of the patient;
constructing and generating a virtual medical image based on the target image, age information, sex information and disease diagnosis information;
and matching similar medical matching images in a medical image database based on the target image, the age information, the gender information and the illness state diagnosis information.
Optionally, the matching similar medical matching images in the medical image database based on the target image, the age information, the sex information and the disease diagnosis information includes:
matching a matched medical matching image set in the medical database based on the age information, the sex information and the disease diagnosis information;
and matching the target image with the medical matching image set to obtain similar medical matching images based on a picture similarity matching algorithm.
Optionally, the matching of the target image and the medical matching image set to similar medical matching images based on a picture similarity matching algorithm includes:
matching the image feature point group topological similarity of each medical matching image in the target image and the medical matching image set based on an image feature point group topological similarity algorithm;
matching the image feature point group direction similarity of each medical matching image in the target image and the medical matching image set based on an image feature point group direction similarity algorithm;
matching the image feature point group distance similarity of each medical matching image in the target image and the medical matching image set based on an image feature point group distance similarity algorithm;
and accumulating the topological similarity of the image feature point group, the directional similarity of the image feature point group and the distance similarity of the image feature point group of each medical matching image in the medical matching image set to obtain the medical matching image with the largest accumulated result.
Optionally, the performing three-dimensional fusion processing on the virtual medical image and the medical matching image to obtain a virtual three-dimensional medical feature image includes:
performing virtual three-dimensional modeling processing on the virtual medical image to obtain a three-dimensional virtual medical image;
performing virtual three-dimensional modeling processing on the medical matching image to obtain a three-dimensional medical matching image;
and carrying out three-dimensional matching fusion on the three-dimensional virtual medical image and the three-dimensional medical matching image to obtain a virtual three-dimensional medical characteristic image.
Optionally, the matching and fusing the original medical image and the virtual three-dimensional medical feature image to obtain and display an augmented reality medical image includes:
extracting feature points of the original medical image and extracting feature points of the virtual three-dimensional medical feature image;
the feature points of the original medical image and the feature points of the virtual three-dimensional medical feature image are utilized to carry out pairwise comparison, and feature points matched with each other are obtained;
constructing a feature correspondence between images based on the feature points matched with each other;
and acquiring and displaying the medical image of the augmented reality based on the characteristic corresponding relation between the images.
Optionally, the acquiring and displaying the medical image of augmented reality based on the feature correspondence between the images includes:
acquiring the size of a display area used for displaying the augmented reality on a display device;
and scaling the medical image of the augmented reality to be consistent with the size of the display area, and pushing the medical image to the display equipment for display.
In addition, the embodiment of the invention also provides a medical image display device based on augmented reality, which comprises:
an image acquisition module: for acquiring an original medical image based on the medical imaging device;
and an image generation and matching module: the medical image processing method comprises the steps of generating a virtual medical image based on the original medical image and personal information of a patient, and acquiring matched medical matching images in a medical image database;
and a fusion module: the method comprises the steps of performing three-dimensional fusion processing on the virtual medical image and the medical matching image to obtain a virtual three-dimensional medical characteristic image;
and (3) matching and fusing the display module: and the method is used for carrying out matching fusion on the original medical image and the virtual three-dimensional medical characteristic image, obtaining and displaying an augmented reality medical image.
Optionally, the fusion module includes:
a first three-dimensional modeling unit: the virtual medical image processing module is used for carrying out virtual three-dimensional modeling processing on the virtual medical image to obtain a three-dimensional virtual medical image;
a second three-dimensional modeling unit: the medical matching image processing method comprises the steps of performing virtual three-dimensional modeling processing on the medical matching image to obtain a three-dimensional medical matching image;
three-dimensional fusion unit: and the three-dimensional medical image processing module is used for carrying out three-dimensional matching fusion on the three-dimensional virtual medical image and the three-dimensional medical matching image to obtain a virtual three-dimensional medical characteristic image.
Optionally, the matching fusion display module includes:
feature point extraction unit: the method comprises the steps of extracting feature points of an original medical image and extracting feature points of a virtual three-dimensional medical feature image;
and a comparison unit: the method comprises the steps of comparing feature points of the original medical image with feature points of the virtual three-dimensional medical feature image in pairs to obtain feature points matched with each other;
the corresponding relation construction unit: the feature correspondence between the images is built based on the feature points matched with each other;
augmented reality acquisition and display unit: and acquiring and displaying the medical image of the augmented reality based on the characteristic corresponding relation between the images.
In the embodiment of the invention, the original medical image is acquired based on the medical imaging equipment; generating a virtual medical image based on the original medical image and personal information of the patient, and acquiring a matched medical matching image in a medical image database; performing three-dimensional fusion processing on the virtual medical image and the medical matching image to obtain a virtual three-dimensional medical characteristic image; matching and fusing the original medical image and the virtual three-dimensional medical characteristic image to obtain and display an augmented reality medical image; the display diversity and the friendliness of the medical image are improved.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings which are required in the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the description below are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a medical image display method based on augmented reality in an embodiment of the invention;
fig. 2 is a schematic structural composition diagram of an augmented reality-based medical image display apparatus in an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Examples
Referring to fig. 1, fig. 1 is a flowchart illustrating a medical image display method based on augmented reality according to an embodiment of the invention.
As shown in fig. 1, a medical image display method based on augmented reality, the medical image display method includes:
s11: acquiring an original medical image based on a medical imaging device;
in the implementation process of the invention, the medical imaging equipment comprises an X-ray machine, an endoscope, a CT machine, a B-ultrasonic machine and the like, and the obtained original medical images are subjected to duplicate removal and unclear removal, and the sharpest medical image in the medical images obtained by the medical imaging equipment is selected as the original medical image.
S12: generating a virtual medical image based on the original medical image and personal information of the patient, and acquiring a matched medical matching image in a medical image database;
in the implementation process of the invention, the generating the virtual medical image based on the original medical image and the personal information of the patient and the obtaining the matched medical matching image in the medical image database comprises the following steps: acquiring a target image in the original medical image; obtaining age information, sex information, and condition diagnosis information of a patient based on personal information of the patient; constructing and generating a virtual medical image based on the target image, age information, sex information and disease diagnosis information; and matching similar medical matching images in a medical image database based on the target image, the age information, the gender information and the illness state diagnosis information.
Further, the matching of similar medical matching images in the medical image database based on the target image, the age information, the sex information and the disease diagnosis information includes: matching a matched medical matching image set in the medical database based on the age information, the sex information and the disease diagnosis information; and matching the target image with the medical matching image set to obtain similar medical matching images based on a picture similarity matching algorithm.
Further, the matching of the target image and the medical matching image set to similar medical matching images based on a picture similarity matching algorithm includes: matching the image feature point group topological similarity of each medical matching image in the target image and the medical matching image set based on an image feature point group topological similarity algorithm; matching the image feature point group direction similarity of each medical matching image in the target image and the medical matching image set based on an image feature point group direction similarity algorithm; matching the image feature point group distance similarity of each medical matching image in the target image and the medical matching image set based on an image feature point group distance similarity algorithm; and accumulating the topological similarity of the image feature point group, the directional similarity of the image feature point group and the distance similarity of the image feature point group of each medical matching image in the medical matching image set to obtain the medical matching image with the largest accumulated result.
Specifically, performing target segmentation in an original medical image through an image target segmentation algorithm, and then obtaining a target image in the original medical image; obtaining personal information such as age information, sex information, and disease diagnosis information of the patient through information contained in the personal information of the patient; then constructing and producing a virtual medical image according to the age information, sex information and disease diagnosis information of the patient; according to the target image, matching similar medical matching images in a medical image database according to age information, sex information and disease diagnosis information of the patient; the target image is an image formed by a target which is required to be obtained by a doctor; the medical image database is a database storing various medical images.
Specifically, matching a medical image which accords with the age information, the sex information and the disease diagnosis information of a patient in the medical database to be used as a medical matching image set; and then matching the similarity of the target image and the medical matching image set by using a picture similarity matching algorithm to obtain a medical matching image.
Different from the judgment of the topological relation in the geographic space, the definition of the distance in the raster image has no uniform standard, and the Voronoi neighbors are used as the description of the topological relation of the point group, the number of the neighbors does not need to be manually fixed, the absolute relation of the distance between the feature points does not need to be considered, and the method has certain adaptability to the scale and density change; the calculation formula is as follows:
wherein,,wherein N is 1 Representing the total point number of the target image point group, B 1 Representing the total number of topological neighbors of the target image; />Wherein N is 2 Representing the total point number of each medical matching image point group in the medical matching image set, B 2 Representing a total number of topological neighbors for each medical matching image within the set of medical matching images; s is S 1 And representing the similarity of the directions of the image feature point groups.
The direction of the image feature point group can change to a certain extent along with the rotation of the image, so that when the similarity of the directions of the 2 image feature point groups is calculated, the measurement cannot be directly carried out by using the calculated direction of the standard deviation ellipse; comprehensively considering the direction of the characteristic point group and the characteristic main direction of the characteristic point group, and measuring the similarity of directions by calculating the included angles between the obtained 2 directions; the specific calculation formula is as follows:
wherein θ 1 The respective directions of the feature point groups representing the target image,representing the main direction of the gradient of the characteristic point group of the target image; θ 2 Characteristic point group representing each medical matching image in medical matching image set is directed in a respective direction, +.>Representing a principal direction of feature point cluster gradients for each medical matching image within the set of medical matching images; s is S 2 And representing the distance similarity of the image feature point groups.
The similarity of the distance between the characteristic point groups is used for describing the concentration degree of the point groups, and is measured by the ratio of the length of a standard deviation ellipse to the distance of a short axis, and the specific calculation formula is as follows:
wherein b 1 ，a 1 Respectively representing the long and short axes of the point group standard deviation ellipse of the target image; b 2 ，a 2 Respectively representing the long and short axes of the point group standard deviation ellipse of each medical matching image in the medical matching image set; s is S 3 And representing the distance similarity of the image feature point groups.
In the implementation process of the invention, the image feature point group topological similarity, the image feature point group direction similarity and the image feature point group distance similarity of each medical matching image in the medical matching image set can be accumulated, then an accumulation result is obtained, and the largest accumulation result is used as the medical matching image.
Respectively carrying out image feature matching by using the image feature point group topological similarity algorithm, the image feature point group direction similarity algorithm and the image feature point group distance similarity algorithm, then carrying out corresponding accumulation, and obtaining similar medical matching images matched with each other based on accumulation results; thus, the matching precision problem possibly existing when a single similarity algorithm is used for matching can be reduced, and the most similar medical matching images can not be reasonably and accurately matched; in the medical field, quite high accuracy is required, so that the condition of a patient can be rapidly and accurately diagnosed to the maximum extent.
S13: performing three-dimensional fusion processing on the virtual medical image and the medical matching image to obtain a virtual three-dimensional medical characteristic image;
in the implementation process of the invention, the three-dimensional fusion processing is performed on the virtual medical image and the medical matching image to obtain a virtual three-dimensional medical characteristic image, which comprises the following steps: performing virtual three-dimensional modeling processing on the virtual medical image to obtain a three-dimensional virtual medical image; performing virtual three-dimensional modeling processing on the medical matching image to obtain a three-dimensional medical matching image; performing three-dimensional matching fusion on the three-dimensional virtual medical image and the three-dimensional medical matching image to obtain a virtual three-dimensional medical characteristic image
Specifically, virtual three-dimensional modeling processing is carried out on the virtual medical image through a mathematical model, and the virtual medical image is reconstructed into a three-dimensional virtual medical image; then, performing virtual three-dimensional modeling processing on the medical matching image through the mathematical model again, and reconstructing the medical matching image into a three-dimensional medical matching image; and finally, carrying out three-dimensional matching fusion on the three-dimensional virtual medical image and the three-dimensional medical matching image to obtain a virtual three-dimensional medical characteristic image.
In the step, firstly, three-dimensional modeling is carried out through a mathematical model to obtain a three-dimensional virtual medical image and a three-dimensional medical matching image, and then three-dimensional matching fusion is carried out to obtain a virtual three-dimensional medical characteristic image; the feature points of the original image can be reserved to the greatest extent through three-dimensional matching fusion, so that the loss of the features is reduced during fusion, errors generated during fusion are reduced, and the like.
S14: and matching and fusing the original medical image and the virtual three-dimensional medical characteristic image, and obtaining and displaying an augmented reality medical image.
In the implementation process of the invention, the matching and fusing the original medical image and the virtual three-dimensional medical feature image to obtain and display the augmented reality medical image comprises the following steps: extracting feature points of the original medical image and extracting feature points of the virtual three-dimensional medical feature image; the feature points of the original medical image and the feature points of the virtual three-dimensional medical feature image are utilized to carry out pairwise comparison, and feature points matched with each other are obtained; constructing a feature correspondence between images based on the feature points matched with each other; and acquiring and displaying the medical image of the augmented reality based on the characteristic corresponding relation between the images.
Further, the obtaining and displaying the medical image of augmented reality based on the feature correspondence between the images includes: acquiring the size of a display area used for displaying the augmented reality on a display device; and scaling the medical image of the augmented reality to be consistent with the size of the display area, and pushing the medical image to the display equipment for display.
Specifically, feature point extraction is performed in the original image and the virtual three-dimensional medical feature image respectively, the feature point extraction modes can be multiple, in the embodiment of the invention, the feature point extraction is performed in a binarization mode, after the feature point is extracted, the feature points of the original image and the virtual three-dimensional medical feature image are used for performing pairwise comparison, and then the feature points matched with each other are obtained; then, constructing a feature corresponding relation between the images by utilizing the feature points matched with each other; obtaining an augmented reality medical image through the feature correspondence between the images; after obtaining the augmented reality medical image, it is first necessary to obtain the size (resolution, etc.) of the display device and the size of the position for displaying the augmented reality medical image, then zoom the augmented reality medical image to the size for the display area, and push the augmented reality medical image onto the display device for display.
In the embodiment of the invention, the original medical image is acquired based on the medical imaging equipment; generating a virtual medical image based on the original medical image and personal information of the patient, and acquiring a matched medical matching image in a medical image database; performing three-dimensional fusion processing on the virtual medical image and the medical matching image to obtain a virtual three-dimensional medical characteristic image; matching and fusing the original medical image and the virtual three-dimensional medical characteristic image to obtain and display an augmented reality medical image; the display diversity and the friendliness of the medical image are improved.
Examples
Referring to fig. 2, fig. 2 is a schematic structural diagram of a medical image display apparatus based on augmented reality according to an embodiment of the present invention.
As shown in fig. 2, a medical image display apparatus based on augmented reality, the medical image display apparatus comprising:
the image acquisition module 11: for acquiring an original medical image based on the medical imaging device;
in the implementation process of the invention, the medical imaging equipment comprises an X-ray machine, an endoscope, a CT machine, a B-ultrasonic machine and the like, and the obtained original medical images are subjected to duplicate removal and unclear removal, and the sharpest medical image in the medical images obtained by the medical imaging equipment is selected as the original medical image.
Image generation and matching module 12: the medical image processing method comprises the steps of generating a virtual medical image based on the original medical image and personal information of a patient, and acquiring matched medical matching images in a medical image database;
in the implementation process of the invention, the generating the virtual medical image based on the original medical image and the personal information of the patient and the obtaining the matched medical matching image in the medical image database comprises the following steps: acquiring a target image in the original medical image; obtaining age information, sex information, and condition diagnosis information of a patient based on personal information of the patient; constructing and generating a virtual medical image based on the target image, age information, sex information and disease diagnosis information; and matching similar medical matching images in a medical image database based on the target image, the age information, the gender information and the illness state diagnosis information.
Further, the matching of similar medical matching images in the medical image database based on the target image, the age information, the sex information and the disease diagnosis information includes: matching a matched medical matching image set in the medical database based on the age information, the sex information and the disease diagnosis information; and matching the target image with the medical matching image set to obtain similar medical matching images based on a picture similarity matching algorithm.
Further, the matching of the target image and the medical matching image set to similar medical matching images based on a picture similarity matching algorithm includes: matching the image feature point group topological similarity of each medical matching image in the target image and the medical matching image set based on an image feature point group topological similarity algorithm; matching the image feature point group direction similarity of each medical matching image in the target image and the medical matching image set based on an image feature point group direction similarity algorithm; matching the image feature point group distance similarity of each medical matching image in the target image and the medical matching image set based on an image feature point group distance similarity algorithm; and accumulating the topological similarity of the image feature point group, the directional similarity of the image feature point group and the distance similarity of the image feature point group of each medical matching image in the medical matching image set to obtain the medical matching image with the largest accumulated result.
Specifically, performing target segmentation in an original medical image through an image target segmentation algorithm, and then obtaining a target image in the original medical image; obtaining personal information such as age information, sex information, and disease diagnosis information of the patient through information contained in the personal information of the patient; then constructing and producing a virtual medical image according to the age information, sex information and disease diagnosis information of the patient; according to the target image, matching similar medical matching images in a medical image database according to age information, sex information and disease diagnosis information of the patient; the target image is an image formed by a target which is required to be obtained by a doctor; the medical image database is a database storing various medical images.
Specifically, matching a medical image which accords with the age information, the sex information and the disease diagnosis information of a patient in the medical database to be used as a medical matching image set; and then matching the similarity of the target image and the medical matching image set by using a picture similarity matching algorithm to obtain a medical matching image.
Different from the judgment of the topological relation in the geographic space, the definition of the distance in the raster image has no uniform standard, and the Voronoi neighbors are used as the description of the topological relation of the point group, the number of the neighbors does not need to be manually fixed, the absolute relation of the distance between the feature points does not need to be considered, and the method has certain adaptability to the scale and density change; the calculation formula is as follows:
wherein,,wherein N is 1 Representing the total point number of the target image point group, B 1 Representing the total number of topological neighbors of the target image; />Wherein N is 2 Representing the total point number of each medical matching image point group in the medical matching image set, B 2 Representing a total number of topological neighbors for each medical matching image within the set of medical matching images; s is S 1 And representing the similarity of the directions of the image feature point groups.
The direction of the image feature point group can change to a certain extent along with the rotation of the image, so that when the similarity of the directions of the 2 image feature point groups is calculated, the measurement cannot be directly carried out by using the calculated direction of the standard deviation ellipse; comprehensively considering the direction of the characteristic point group and the characteristic main direction of the characteristic point group, and measuring the similarity of directions by calculating the included angles between the obtained 2 directions; the specific calculation formula is as follows:
wherein θ 1 The respective directions of the feature point groups representing the target image,representing the main direction of the gradient of the characteristic point group of the target image; θ 2 Characteristic point group representing each medical matching image in medical matching image set is directed in a respective direction, +.>Representing a principal direction of feature point cluster gradients for each medical matching image within the set of medical matching images; s is S 2 And representing the distance similarity of the image feature point groups.
The similarity of the distance between the characteristic point groups is used for describing the concentration degree of the point groups, and is measured by the ratio of the length of a standard deviation ellipse to the distance of a short axis, and the specific calculation formula is as follows:
wherein b 1 ，a 1 Respectively representing the long and short axes of the point group standard deviation ellipse of the target image; b 2 ，a 2 Respectively representing the long and short axes of the point group standard deviation ellipse of each medical matching image in the medical matching image set; s is S 3 And representing the distance similarity of the image feature point groups.
In the implementation process of the invention, the image feature point group topological similarity, the image feature point group direction similarity and the image feature point group distance similarity of each medical matching image in the medical matching image set can be accumulated, then an accumulation result is obtained, and the largest accumulation result is used as the medical matching image.
Fusion module 13: the method comprises the steps of performing three-dimensional fusion processing on the virtual medical image and the medical matching image to obtain a virtual three-dimensional medical characteristic image;
in the implementation of the present invention, the fusion module 13 includes: a first three-dimensional modeling unit: the virtual medical image processing module is used for carrying out virtual three-dimensional modeling processing on the virtual medical image to obtain a three-dimensional virtual medical image; a second three-dimensional modeling unit: the medical matching image processing method comprises the steps of performing virtual three-dimensional modeling processing on the medical matching image to obtain a three-dimensional medical matching image; three-dimensional fusion unit: and the three-dimensional medical image processing module is used for carrying out three-dimensional matching fusion on the three-dimensional virtual medical image and the three-dimensional medical matching image to obtain a virtual three-dimensional medical characteristic image.
Specifically, virtual three-dimensional modeling processing is carried out on the virtual medical image through a mathematical model, and the virtual medical image is reconstructed into a three-dimensional virtual medical image; then, performing virtual three-dimensional modeling processing on the medical matching image through the mathematical model again, and reconstructing the medical matching image into a three-dimensional medical matching image; and finally, carrying out three-dimensional matching fusion on the three-dimensional virtual medical image and the three-dimensional medical matching image to obtain a virtual three-dimensional medical characteristic image.
Matching fusion display module 14: and the method is used for carrying out matching fusion on the original medical image and the virtual three-dimensional medical characteristic image, obtaining and displaying an augmented reality medical image.
In the implementation of the present invention, the matching fusion display module 14 includes: feature point extraction unit: the method comprises the steps of extracting feature points of an original medical image and extracting feature points of a virtual three-dimensional medical feature image; and a comparison unit: the method comprises the steps of comparing feature points of the original medical image with feature points of the virtual three-dimensional medical feature image in pairs to obtain feature points matched with each other; the corresponding relation construction unit: the feature correspondence between the images is built based on the feature points matched with each other; augmented reality acquisition and display unit: and acquiring and displaying the medical image of the augmented reality based on the characteristic corresponding relation between the images.
Further, the obtaining and displaying the medical image of augmented reality based on the feature correspondence between the images includes: acquiring the size of a display area used for displaying the augmented reality on a display device; and scaling the medical image of the augmented reality to be consistent with the size of the display area, and pushing the medical image to the display equipment for display.
Specifically, feature point extraction is performed in the original image and the virtual three-dimensional medical feature image respectively, the feature point extraction modes can be multiple, in the embodiment of the invention, the feature point extraction is performed in a binarization mode, after the feature point is extracted, the feature points of the original image and the virtual three-dimensional medical feature image are used for performing pairwise comparison, and then the feature points matched with each other are obtained; then, constructing a feature corresponding relation between the images by utilizing the feature points matched with each other; obtaining an augmented reality medical image through the feature correspondence between the images; after obtaining the augmented reality medical image, it is first necessary to obtain the size (resolution, etc.) of the display device and the size of the position for displaying the augmented reality medical image, then zoom the augmented reality medical image to the size for the display area, and push the augmented reality medical image onto the display device for display.
In the embodiment of the invention, the original medical image is acquired based on the medical imaging equipment; generating a virtual medical image based on the original medical image and personal information of the patient, and acquiring a matched medical matching image in a medical image database; performing three-dimensional fusion processing on the virtual medical image and the medical matching image to obtain a virtual three-dimensional medical characteristic image; matching and fusing the original medical image and the virtual three-dimensional medical characteristic image to obtain and display an augmented reality medical image; the display diversity and the friendliness of the medical image are improved.
Those of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be implemented by a program to instruct related hardware, the program may be stored in a computer readable storage medium, and the storage medium may include: read Only Memory (ROM), random access Memory (RAM, random Access Memory), magnetic or optical disk, and the like.
In addition, the foregoing details of the method and apparatus for displaying a medical image based on augmented reality according to the embodiments of the present invention are presented in the following description, and specific examples should be adopted to illustrate the principles and embodiments of the present invention, where the foregoing description of the embodiments is only for helping to understand the method and core ideas of the present invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.
Claims (7)
1. A medical image display method based on augmented reality, the medical image display method comprising:
acquiring an original medical image based on a medical imaging device;
generating a virtual medical image based on the original medical image and personal information of the patient, and acquiring a matched medical matching image in a medical image database;
performing three-dimensional fusion processing on the virtual medical image and the medical matching image to obtain a virtual three-dimensional medical characteristic image;
matching and fusing the original medical image and the virtual three-dimensional medical characteristic image to obtain and display an augmented reality medical image;
the generating a virtual medical image based on the original medical image and personal information of the patient, and acquiring a matched medical matching image in a medical image database comprises the following steps:
acquiring a target image in the original medical image;
obtaining age information, sex information, and condition diagnosis information of a patient based on personal information of the patient;
constructing and generating a virtual medical image based on the target image, age information, sex information and disease diagnosis information;
matching similar medical matching images in a medical image database based on the target image, the age information, the gender information and the illness state diagnosis information;
the matching of similar medical matching images in a medical image database based on the target image, age information, gender information and illness state diagnosis information comprises the following steps:
matching a matched medical matching image set in the medical image database based on the age information, the sex information and the disease diagnosis information;
the target image and the medical matching image set are matched to form similar medical matching images based on a picture similarity matching algorithm;
the target image and the medical matching image set are matched to form similar medical matching images based on a picture similarity matching algorithm, and the method comprises the following steps:
matching the image feature point group topological similarity of each medical matching image in the target image and the medical matching image set based on an image feature point group topological similarity algorithm;
matching the image feature point group direction similarity of each medical matching image in the target image and the medical matching image set based on an image feature point group direction similarity algorithm;
matching the image feature point group distance similarity of each medical matching image in the target image and the medical matching image set based on an image feature point group distance similarity algorithm;
and accumulating the topological similarity of the image feature point group, the directional similarity of the image feature point group and the distance similarity of the image feature point group of each medical matching image in the medical matching image set to obtain the medical matching image with the largest accumulated result.
2. The medical image display method according to claim 1, wherein the performing three-dimensional fusion processing on the virtual medical image and the medical matching image to obtain a virtual three-dimensional medical feature image includes:
performing virtual three-dimensional modeling processing on the virtual medical image to obtain a three-dimensional virtual medical image;
performing virtual three-dimensional modeling processing on the medical matching image to obtain a three-dimensional medical matching image;
and carrying out three-dimensional matching fusion on the three-dimensional virtual medical image and the three-dimensional medical matching image to obtain a virtual three-dimensional medical characteristic image.
3. The medical image display method according to claim 1, wherein the matching and fusing the original medical image and the virtual three-dimensional medical feature image, obtaining and displaying an augmented reality medical image, includes:
extracting feature points of the original medical image and extracting feature points of the virtual three-dimensional medical feature image;
the feature points of the original medical image and the feature points of the virtual three-dimensional medical feature image are utilized to carry out pairwise comparison, and feature points matched with each other are obtained;
constructing a feature correspondence between images based on the feature points matched with each other;
and acquiring and displaying the medical image of the augmented reality based on the characteristic corresponding relation between the images.
4. The medical image display method according to claim 3, wherein the acquiring and displaying the augmented reality medical image based on the feature correspondence between the images includes:
acquiring the size of a display area used for displaying the augmented reality on a display device;
and scaling the medical image of the augmented reality to be consistent with the size of the display area, and pushing the medical image to the display equipment for display.
5. An augmented reality-based medical image display apparatus, characterized in that the medical image display apparatus comprises:
an image acquisition module: for acquiring an original medical image based on the medical imaging device;
and an image generation and matching module: the medical image processing method comprises the steps of generating a virtual medical image based on the original medical image and personal information of a patient, and acquiring matched medical matching images in a medical image database;
and a fusion module: the method comprises the steps of performing three-dimensional fusion processing on the virtual medical image and the medical matching image to obtain a virtual three-dimensional medical characteristic image;
and (3) matching and fusing the display module: the method comprises the steps of carrying out matching fusion on the original medical image and the virtual three-dimensional medical characteristic image, obtaining and displaying an augmented reality medical image;
the generating a virtual medical image based on the original medical image and personal information of the patient, and acquiring a matched medical matching image in a medical image database comprises the following steps:
acquiring a target image in the original medical image;
obtaining age information, sex information, and condition diagnosis information of a patient based on personal information of the patient;
constructing and generating a virtual medical image based on the target image, age information, sex information and disease diagnosis information;
matching similar medical matching images in a medical image database based on the target image, the age information, the gender information and the illness state diagnosis information;
the matching of similar medical matching images in a medical image database based on the target image, age information, gender information and illness state diagnosis information comprises the following steps:
matching a matched medical matching image set in the medical image database based on the age information, the sex information and the disease diagnosis information;
the target image and the medical matching image set are matched to form similar medical matching images based on a picture similarity matching algorithm;
the target image and the medical matching image set are matched to form similar medical matching images based on a picture similarity matching algorithm, and the method comprises the following steps:
matching the image feature point group topological similarity of each medical matching image in the target image and the medical matching image set based on an image feature point group topological similarity algorithm;
matching the image feature point group direction similarity of each medical matching image in the target image and the medical matching image set based on an image feature point group direction similarity algorithm;
matching the image feature point group distance similarity of each medical matching image in the target image and the medical matching image set based on an image feature point group distance similarity algorithm;
and accumulating the topological similarity of the image feature point group, the directional similarity of the image feature point group and the distance similarity of the image feature point group of each medical matching image in the medical matching image set to obtain the medical matching image with the largest accumulated result.
6. The medical image display apparatus according to claim 5, wherein the fusion module includes:
a first three-dimensional modeling unit: the virtual medical image processing module is used for carrying out virtual three-dimensional modeling processing on the virtual medical image to obtain a three-dimensional virtual medical image;
a second three-dimensional modeling unit: the medical matching image processing method comprises the steps of performing virtual three-dimensional modeling processing on the medical matching image to obtain a three-dimensional medical matching image;
three-dimensional fusion unit: and the three-dimensional medical image processing module is used for carrying out three-dimensional matching fusion on the three-dimensional virtual medical image and the three-dimensional medical matching image to obtain a virtual three-dimensional medical characteristic image.
7. The medical image display apparatus of claim 5, wherein the matching fusion display module comprises:
feature point extraction unit: the method comprises the steps of extracting feature points of an original medical image and extracting feature points of a virtual three-dimensional medical feature image;
and a comparison unit: the method comprises the steps of comparing feature points of the original medical image with feature points of the virtual three-dimensional medical feature image in pairs to obtain feature points matched with each other;
the corresponding relation construction unit: the feature correspondence between the images is built based on the feature points matched with each other;
augmented reality acquisition and display unit: and acquiring and displaying the medical image of the augmented reality based on the characteristic corresponding relation between the images.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910199816.4A CN109934934B (en) | 2019-03-15 | 2019-03-15 | Medical image display method and device based on augmented reality |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910199816.4A CN109934934B (en) | 2019-03-15 | 2019-03-15 | Medical image display method and device based on augmented reality |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109934934A CN109934934A (en) | 2019-06-25 |
| CN109934934B true CN109934934B (en) | 2023-09-29 |
Family
ID=66987382
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910199816.4A Active CN109934934B (en) | 2019-03-15 | 2019-03-15 | Medical image display method and device based on augmented reality |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109934934B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH0824227A (en) * | 1994-07-19 | 1996-01-30 | Hitachi Medical Corp | Medical image diagnosing apparatus |
| JP2003265408A (en) * | 2002-03-19 | 2003-09-24 | Mitsubishi Electric Corp | Endoscope guide device and method |
| JP2005065728A (en) * | 2003-08-25 | 2005-03-17 | Fuji Photo Film Co Ltd | Similar image retrieval system |
| JP2012179336A (en) * | 2011-03-02 | 2012-09-20 | Stat Lab:Kk | Pathology image diagnosis support system |
| JP2014064606A (en) * | 2012-09-24 | 2014-04-17 | Hitachi Medical Corp | Image processing device and method for generating time-lapse difference image |
| KR20140054783A (en) * | 2012-10-29 | 2014-05-09 | 한국과학기술원 | 4d fusion display technique of motion compensated pet, and ct and/or mr |
| CN105078514A (en) * | 2014-04-22 | 2015-11-25 | 重庆海扶医疗科技股份有限公司 | Construction method and device of three-dimensional model, image monitoring method and device |
| CN106202379A (en) * | 2016-07-09 | 2016-12-07 | 兰州交通大学 | A kind of matching inquiry method based on spatial scene similarity |
| CN106846317A (en) * | 2017-02-27 | 2017-06-13 | 北京连心医疗科技有限公司 | A kind of feature based extracts the method for retrieving medicine image with Similarity matching |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004005364A (en) * | 2002-04-03 | 2004-01-08 | Fuji Photo Film Co Ltd | Similar image retrieval system |
-
2019
- 2019-03-15 CN CN201910199816.4A patent/CN109934934B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH0824227A (en) * | 1994-07-19 | 1996-01-30 | Hitachi Medical Corp | Medical image diagnosing apparatus |
| JP2003265408A (en) * | 2002-03-19 | 2003-09-24 | Mitsubishi Electric Corp | Endoscope guide device and method |
| JP2005065728A (en) * | 2003-08-25 | 2005-03-17 | Fuji Photo Film Co Ltd | Similar image retrieval system |
| JP2012179336A (en) * | 2011-03-02 | 2012-09-20 | Stat Lab:Kk | Pathology image diagnosis support system |
| JP2014064606A (en) * | 2012-09-24 | 2014-04-17 | Hitachi Medical Corp | Image processing device and method for generating time-lapse difference image |
| KR20140054783A (en) * | 2012-10-29 | 2014-05-09 | 한국과학기술원 | 4d fusion display technique of motion compensated pet, and ct and/or mr |
| CN105078514A (en) * | 2014-04-22 | 2015-11-25 | 重庆海扶医疗科技股份有限公司 | Construction method and device of three-dimensional model, image monitoring method and device |
| CN106202379A (en) * | 2016-07-09 | 2016-12-07 | 兰州交通大学 | A kind of matching inquiry method based on spatial scene similarity |
| CN106846317A (en) * | 2017-02-27 | 2017-06-13 | 北京连心医疗科技有限公司 | A kind of feature based extracts the method for retrieving medicine image with Similarity matching |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109934934A (en) | 2019-06-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2020215985A1 (en) | Medical image segmentation method and device, electronic device and storage medium | |
| CN109872364B (en) | Image area positioning method, device, storage medium and medical image processing equipment | |
| CN111553892B (en) | Lung nodule segmentation calculation method, device and system based on deep learning | |
| US9858716B2 (en) | Fast three-dimensional visualization of object volumes without image reconstruction by direct display of acquired sensor data | |
| CN110599528A (en) | Unsupervised three-dimensional medical image registration method and system based on neural network | |
| US20120189178A1 (en) | Method and apparatus for automatically generating optimal 2-dimensional medical image from 3-dimensional medical image | |
| CN110599421A (en) | Model training method, video fuzzy frame conversion method, device and storage medium | |
| CN111932492B (en) | Medical image processing method and device and computer readable storage medium | |
| CN111488872B (en) | Image detection method, image detection device, computer equipment and storage medium | |
| CN115965750B (en) | Vascular reconstruction method, vascular reconstruction device, vascular reconstruction computer device, and vascular reconstruction program | |
| CN111369564B (en) | Image processing method, model training method and model training device | |
| CN112070785A (en) | Medical image analysis method based on computer vision | |
| CN116758087B (en) | Lumbar vertebra CT bone window side recess gap detection method and device | |
| CN116958217B (en) | MRI and CT multi-mode 3D automatic registration method and device | |
| CN110634554A (en) | Spine image registration method | |
| Zhang et al. | 3D reconstruction for super-resolution CT images in the Internet of health things using deep learning | |
| CN115170401A (en) | Image completion method, device, equipment and storage medium | |
| CN111105476B (en) | Three-dimensional reconstruction method for CT image based on Marching Cubes | |
| CN114298997A (en) | Method and device for detecting forged picture and storage medium | |
| CN115439453A (en) | Vertebral body positioning method and device, electronic equipment and storage medium | |
| JP2009301254A (en) | Image processing device, method and program | |
| CN110738702A (en) | three-dimensional ultrasonic image processing method, device, equipment and storage medium | |
| CN113538372A (en) | Three-dimensional target detection method and device, computer equipment and storage medium | |
| CN109934934B (en) | Medical image display method and device based on augmented reality | |
| CN114266848A (en) | Medical image three-dimensional reconstruction method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address |
Address after: Room 1402, Building F, No. 98 Xiangxueba Road, Huangpu District, Guangzhou City, Guangdong Province, 510000 (for office purposes only) Patentee after: Guangzhou Xinran Technology Co.,Ltd. Address before: No. 321-323, Building G2, No. 39 Ruihe Road, Huangpu District, Guangzhou City, Guangdong Province, 510700 Patentee before: GUANGZHOU JIUSAN ZHIXIN TECHNOLOGY Co.,Ltd. |
|
| CP03 | Change of name, title or address |