CN111487320B - Three-dimensional ultrasonic imaging method and system based on three-dimensional optical imaging sensor - Google Patents

Three-dimensional ultrasonic imaging method and system based on three-dimensional optical imaging sensor Download PDF

Info

Publication number
CN111487320B
CN111487320B CN201910086302.8A CN201910086302A CN111487320B CN 111487320 B CN111487320 B CN 111487320B CN 201910086302 A CN201910086302 A CN 201910086302A CN 111487320 B CN111487320 B CN 111487320B
Authority
CN
China
Prior art keywords
dimensional
information
marker
optical imaging
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910086302.8A
Other languages
Chinese (zh)
Other versions
CN111487320A (en
Inventor
郑永平
孟强
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Telefield Medical Imaging Ltd
Original Assignee
Telefield Medical Imaging Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Telefield Medical Imaging Ltd filed Critical Telefield Medical Imaging Ltd
Priority to CN201910086302.8A priority Critical patent/CN111487320B/en
Priority to PCT/CN2020/073105 priority patent/WO2020156299A1/en
Publication of CN111487320A publication Critical patent/CN111487320A/en
Application granted granted Critical
Publication of CN111487320B publication Critical patent/CN111487320B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N29/00—Investigating or analysing materials by the use of ultrasonic, sonic or infrasonic waves; Visualisation of the interior of objects by transmitting ultrasonic or sonic waves through the object
    • G01N29/04—Analysing solids
    • G01N29/06—Visualisation of the interior, e.g. acoustic microscopy
    • G01N29/0654—Imaging
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B17/00—Measuring arrangements characterised by the use of infrasonic, sonic or ultrasonic vibrations
    • G01B17/06—Measuring arrangements characterised by the use of infrasonic, sonic or ultrasonic vibrations for measuring contours or curvatures
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N29/00—Investigating or analysing materials by the use of ultrasonic, sonic or infrasonic waves; Visualisation of the interior of objects by transmitting ultrasonic or sonic waves through the object
    • G01N29/04—Analysing solids
    • G01N29/06—Visualisation of the interior, e.g. acoustic microscopy
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N29/00—Investigating or analysing materials by the use of ultrasonic, sonic or infrasonic waves; Visualisation of the interior of objects by transmitting ultrasonic or sonic waves through the object
    • G01N29/22—Details, e.g. general constructional or apparatus details
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N29/00—Investigating or analysing materials by the use of ultrasonic, sonic or infrasonic waves; Visualisation of the interior of objects by transmitting ultrasonic or sonic waves through the object
    • G01N29/22—Details, e.g. general constructional or apparatus details
    • G01N29/26—Arrangements for orientation or scanning by relative movement of the head and the sensor
    • G01N29/265—Arrangements for orientation or scanning by relative movement of the head and the sensor by moving the sensor relative to a stationary material
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S15/00—Systems using the reflection or reradiation of acoustic waves, e.g. sonar systems
    • G01S15/02—Systems using the reflection or reradiation of acoustic waves, e.g. sonar systems using reflection of acoustic waves
    • G01S15/06—Systems determining the position data of a target
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S3/00—Direction-finders for determining the direction from which infrasonic, sonic, ultrasonic, or electromagnetic waves, or particle emission, not having a directional significance, are being received
    • G01S3/80—Direction-finders for determining the direction from which infrasonic, sonic, ultrasonic, or electromagnetic waves, or particle emission, not having a directional significance, are being received using ultrasonic, sonic or infrasonic waves
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
    • G01N2291/00—Indexing codes associated with group G01N29/00
    • G01N2291/02—Indexing codes associated with the analysed material
    • G01N2291/023—Solids

Abstract

A three-dimensional ultrasound imaging system based on a three-dimensional optical imaging sensor, comprising: the ultrasonic probe is used for carrying out ultrasonic scanning on the region of interest of the target; a two-dimensional ultrasound imaging device that generates a two-dimensional ultrasound image of a region of interest of the target based on the ultrasound scan; a three-dimensional optical imaging sensor, which acquires distance information between at least one marker and a camera of the three-dimensional optical imaging sensor and image information of the marker in a visible range of the three-dimensional optical imaging sensor; a spatial information processing module for acquiring three-dimensional spatial information of the ultrasonic probe based on the distance information and the image information; and the three-dimensional reconstruction module is used for reconstructing a three-dimensional ultrasonic image based on the three-dimensional space information and the two-dimensional ultrasonic image. The invention also relates to a three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor. The invention can reconstruct three-dimensional ultrasonic images flexibly, with low cost and small volume, and can effectively avoid interference.

Description

Three-dimensional ultrasonic imaging method and system based on three-dimensional optical imaging sensor
Technical Field
The invention relates to the field of three-dimensional ultrasonic imaging, in particular to a three-dimensional ultrasonic imaging method and system based on a three-dimensional optical imaging sensor.
Background
Free-hand three-dimensional imaging, namely, the ultrasonic probe is freely moved by hands to scan on a target object, and the position and direction information of the ultrasonic probe is captured by utilizing an optical three-dimensional space sensing technology. Currently, three-dimensional spatial sensing techniques include spatial references or signals and corresponding detectors. For example, an electromagnetic transmitter is used to transmit electromagnetic waves as a reference signal, and a detector determines the position and direction change of the probe from the change of the electromagnetic field strength. For another example, one or more visual markers mounted on the probe surface are used as references and one or more cameras surrounding the ultrasound probe are used to detect the position and orientation of the probe.
The three-dimensional space sensing technologies have own advantages and limitations. In the case of electromagnetic sensing technology, it is subject to interference from surrounding metal objects. Camera-based sensing systems are typically bulky and expensive.
Disclosure of Invention
The invention aims to solve the technical problems of the prior art and provides a three-dimensional ultrasonic imaging method and system based on a three-dimensional optical imaging sensor, which have strong anti-interference capability, low cost and small volume.
The technical scheme adopted for solving the technical problems is as follows: constructing a three-dimensional ultrasound imaging system based on a three-dimensional optical imaging sensor, comprising:
the ultrasonic probe is used for carrying out ultrasonic scanning on the region of interest of the target;
a two-dimensional ultrasound imaging device for generating a two-dimensional ultrasound image of a region of interest of the target based on the ultrasound scan;
a three-dimensional optical imaging sensor for acquiring distance information between at least one marker and a camera of the three-dimensional optical imaging sensor within a visible range of the three-dimensional optical imaging sensor and image information of the marker; the spatial information processing module is used for acquiring three-dimensional spatial information of the ultrasonic probe based on the distance information and the image information;
and the three-dimensional reconstruction module is used for reconstructing a three-dimensional ultrasonic image based on the three-dimensional space information and the two-dimensional ultrasonic image.
In the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor, the marker is at least one part of the ultrasonic probe.
In the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor, the marker comprises at least one visual marker arranged on the ultrasonic probe.
In the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor, the three-dimensional optical imaging sensor is arranged on the ultrasonic probe, and the marker comprises at least one visual marker arranged in the visual range of the three-dimensional optical imaging sensor.
In the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor, the spatial information processing module comprises:
a marker two-dimensional position recognition unit for acquiring two-dimensional position information of the marker based on the marker image information;
and the three-dimensional space information acquisition unit is used for identifying respective distances between at least three pixel points in the image information of the marker and the camera based on the distance information image and the marker image information to obtain the position and direction information of the marker, and obtaining the three-dimensional space information of the ultrasonic probe based on the position and direction information of the marker.
In the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor, a plurality of markers are arranged in the visible range of the three-dimensional optical imaging sensor; the three-dimensional optical imaging sensor is used for acquiring each group of distance information images between each marker and the camera and each group of marker image information; the two-dimensional position identification unit of the marker is used for acquiring each group of two-dimensional position information of each marker based on each group of image information of the marker; the three-dimensional space information acquisition unit is configured to identify respective distances between at least three pixel points in image information of each of the markers and the camera based on each of the sets of distance information images and each of the sets of marker image information to obtain position and direction information of each of the markers, and to obtain three-dimensional space information of the ultrasonic probe based on the position and direction information of each of the markers.
In the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor, the marker image information is RGB and/or infrared image information, and the marker two-dimensional position identification unit is used for acquiring the two-dimensional position information of the marker based on the color, the shape, the pattern mode or the light darkness of the marker in the RGB and/or infrared image information.
In the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor, the three-dimensional optical imaging sensor is further used for detecting target three-dimensional contour information of a region of interest of the target;
the spatial information processing module further includes:
and the correction unit is used for acquiring three-dimensional motion information of the target in the scanning process of the ultrasonic probe based on the three-dimensional contour information of the target and correcting three-dimensional space information of the ultrasonic probe based on the three-dimensional motion information.
In the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor, the three-dimensional ultrasonic imaging system further comprises:
and the calibration unit is used for converting the pixel points of each frame of the two-dimensional ultrasonic image into a three-dimensional space to calibrate the three-dimensional ultrasonic image.
In the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor, the ultrasonic probe is further provided with at least one angle sensor for acquiring three-dimensional direction information of the ultrasonic probe, and/or the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor comprises a plurality of three-dimensional optical imaging sensors.
In the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor, the system further comprises a display device for displaying the three-dimensional contour information of the target, the three-dimensional ultrasonic image and/or the ultrasonic probe image on the same three-dimensional space.
The invention solves the technical problem by adopting another technical scheme that a three-dimensional ultrasonic imaging method based on a three-dimensional optical imaging sensor is constructed, and the method comprises the following steps:
s1, an ultrasonic probe is used for carrying out ultrasonic scanning on a region of interest of a target, and a two-dimensional ultrasonic image of the region of interest of the target is generated based on the ultrasonic scanning;
s2, acquiring distance information between at least one marker and a camera of the three-dimensional optical imaging sensor in a visible range of the three-dimensional optical imaging sensor and image information of the marker;
S3, acquiring three-dimensional space information of the ultrasonic probe based on the distance information and the image information;
s4, reconstructing a three-dimensional ultrasonic image based on the three-dimensional space information and the two-dimensional ultrasonic image.
In the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor, the marker is at least one part of the ultrasonic probe.
In the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor, the marker comprises at least one visual marker arranged on the ultrasonic probe.
In the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor, the three-dimensional optical imaging sensor is arranged on the ultrasonic probe, and the marker comprises at least one visual marker arranged in the visual range of the three-dimensional optical imaging sensor.
In the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor of the present invention, the step S3 further includes:
s31, acquiring two-dimensional position information of the marker based on the marker image information;
s32, identifying respective distances between at least three pixel points in the image information of the marker and the camera based on the distance information image and the marker image information to obtain position and direction information of the marker, and obtaining three-dimensional space information of the ultrasonic probe based on the position and direction information of the marker.
In the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor, in the step S2, a plurality of markers are arranged in the visible range of the three-dimensional optical imaging sensor, and each group of distance information images and each group of marker image information between each marker and the camera are acquired; in the step S31, each set of two-dimensional position information of each of the markers is acquired based on each set of the marker image information; in the step S32, distances between at least three pixels in the image information of each of the markers and the camera are identified based on each of the sets of distance information images and each of the sets of marker image information to obtain position and direction information of each of the markers, and three-dimensional space information of the ultrasound probe is obtained based on the position and direction information of each of the markers.
In the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor, the marker image information is RGB and/or infrared image information, and in the step S31, the two-dimensional position information of the marker is obtained based on the color, shape, pattern or darkness of the marker in the RGB and/or infrared image information.
In the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor, the method further comprises the following steps: detecting target three-dimensional contour information of a region of interest of the target before ultrasonic scanning;
the step S3 further includes:
s33, acquiring three-dimensional motion information of the target in the scanning process of the ultrasonic probe based on the three-dimensional contour information of the target, and correcting three-dimensional space information of the ultrasonic probe based on the three-dimensional motion information.
In the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor of the present invention, the step S3 further includes:
s34, converting pixel points of each frame of the two-dimensional ultrasonic image into a three-dimensional space to calibrate the three-dimensional ultrasonic image.
The three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor further comprises the following steps of
And S5, displaying the three-dimensional outline information of the target, the three-dimensional ultrasonic image and/or the ultrasonic probe image on the same three-dimensional space.
In the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor of the present invention, the step S3 further includes:
s35, acquiring three-dimensional direction information of the ultrasonic probe by using at least one angle sensor arranged on the ultrasonic probe, and fusing the three-dimensional direction information with three-dimensional space information of the ultrasonic probe acquired by the three-dimensional optical imaging sensor so as to improve the measurement accuracy of the three-dimensional space information.
By implementing the three-dimensional ultrasonic imaging system and method based on the three-dimensional optical imaging sensor, the three-dimensional ultrasonic image can be reconstructed flexibly, with low cost and small volume by adopting the three-dimensional optical imaging sensor to acquire the three-dimensional space information, and the interference can be effectively avoided. Further, by adopting a mode of a marker or an angle sensor, the accuracy of the three-dimensional space information can be improved, and the quality of the three-dimensional ultrasonic image is further improved. Still further, by means of setting a plurality of markers, not only relevant position information but also movement direction information can be obtained, so that the detection of the three-dimensional space information is more effective and reliable, and the quality of the three-dimensional ultrasonic image is further improved. Still further, the obtained three-dimensional space information and/or the three-dimensional ultrasonic image can be corrected and/or calibrated, so that a more accurate and reliable three-dimensional ultrasonic image can be obtained.
Drawings
The invention will be further described with reference to the accompanying drawings and examples, in which:
FIG. 1 is a system schematic diagram of a first preferred embodiment of a three-dimensional ultrasound imaging system based on a three-dimensional optical imaging sensor of the present invention;
FIG. 2 is a system schematic diagram of a second preferred embodiment of a three-dimensional ultrasound imaging system based on a three-dimensional optical imaging sensor of the present invention;
FIGS. 3A-3C are schematic illustrations of different placement positions of the markers of the three-dimensional optical imaging sensor-based three-dimensional ultrasound imaging system shown in FIG. 2;
FIG. 4 is a system schematic diagram of a third preferred embodiment of a three-dimensional ultrasound imaging system based on a three-dimensional optical imaging sensor of the present invention;
FIG. 5 is a system diagram of a fourth preferred embodiment of a three-dimensional ultrasound imaging system based on a three-dimensional optical imaging sensor of the present invention;
FIGS. 6A-6C are schematic illustrations of exemplary ArUco identification codes used in the embodiment of FIG. 5;
FIG. 7 is a system diagram of a fifth preferred embodiment of a three-dimensional ultrasound imaging system based on a three-dimensional optical imaging sensor of the present invention;
FIG. 8 is a system diagram of a sixth preferred embodiment of a three-dimensional ultrasound imaging system based on a three-dimensional optical imaging sensor of the present invention;
FIG. 9 is a flow chart of a first preferred embodiment of a three-dimensional ultrasound imaging method based on a three-dimensional optical imaging sensor of the present invention;
fig. 10 is a flow chart of a second preferred embodiment of the three-dimensional ultrasound imaging method of the present invention based on a three-dimensional optical imaging sensor.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
The traditional camera can only generate two-dimensional images, and the three-dimensional optical imaging sensor can record two-dimensional images of objects by using a plurality of sets of optical sensing systems or applying various distance measuring methods in the camera, and can acquire distance information of different positions in the two-dimensional images from the camera, wherein the distance information can be absolute distance or equivalent distance. For example, microsoft's Kinect sensor and intel's Realsense sensor are both three-dimensional optical imaging sensors as described above. This distance information can be obtained in a number of ways, such as using two separate optical cameras with several distances to generate a stereoscopic image with depth of field. Or an infrared camera is additionally arranged to detect the infrared pattern projected by the optical imaging sensor on the surface of the object to acquire depth information. The latter is widely used in recent three-dimensional optical imaging sensors, such as the intel Realsense three-dimensional optical imaging sensor 03, which has an integrated infrared emitter and an imaging sensor integrated with a pair of RGB cameras. The three-dimensional optical imaging sensor can simultaneously provide a colored RGB image, an infrared image reflecting infrared intensity and a distance map of the object surface from the camera. All of the images described above may be provided in real time (frame rate higher than 25 frames per second).
Accordingly, one inventive concept of the present invention is to perform three-dimensional image reconstruction using three-dimensional spatial information of an ultrasound probe provided by a three-dimensional optical imaging sensor and two-dimensional ultrasound image information provided by a two-dimensional ultrasound imaging device. A further inventive concept of the present invention is to improve the accuracy of three-dimensional spatial information provided by a three-dimensional optical imaging sensor by adding a marker. A further inventive concept of the present invention is to use the ultrasound probe itself as a marker; or setting a visual marker on the ultrasonic probe; or the three-dimensional optical sensor is arranged on the ultrasonic probe, and the visual marker is arranged in the visual range of the three-dimensional optical sensor.
Fig. 1 is a system schematic diagram of a first preferred embodiment of a three-dimensional ultrasound imaging system based on a three-dimensional optical imaging sensor of the present invention. As shown in fig. 1, the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor of the present invention includes an ultrasonic probe 01, a two-dimensional ultrasonic imaging device 05, a three-dimensional optical imaging sensor 03, a spatial information processing module 06, a three-dimensional reconstruction module 07, and a display device 08. The ultrasonic probe 01 is arranged in the visible range 04 of the three-dimensional optical imaging sensor 03 and is used for carrying out ultrasonic scanning on the interested region of the target 02. The two-dimensional ultrasound imaging device 05 is communicatively connected to the ultrasound probe 01 for generating a two-dimensional ultrasound image of the region of interest of the target 02 based on the ultrasound scan. It is known to a person skilled in the art that the region of interest may be at least a part of the object 02, or the whole thereof, and that the ultrasound probe 01, the two-dimensional ultrasound imaging device 05 of the present invention may be constructed using any ultrasound probe, two-dimensional ultrasound imaging device known in the art.
The three-dimensional optical imaging sensor 03 may be communicatively connected to the ultrasonic probe 01 and the two-dimensional ultrasonic imaging device 05, so as to obtain distance information between the ultrasonic probe 01 and a camera of the three-dimensional optical imaging sensor 03 and image information of the ultrasonic probe 01. It will be appreciated by those skilled in the art that any three-dimensional optical imaging sensor in the art, particularly microsoft's Kinect sensor and intel's Realsense sensor, and similar devices developed in the future, may be employed. The image information may be RGB and/or infrared image information, or any other image information that may be used to obtain three-dimensional surface information and/or motion information of an object. Preferably, intel's Realsense three-dimensional optical imaging sensor 03 can be used, which carries an integrated infrared emitter and an imaging sensor integrated with a pair of RGB cameras. The three-dimensional optical imaging sensor can simultaneously provide a colored RGB image, an infrared image reflecting infrared intensity and a distance map of the object surface from the camera. All of the images described above may be provided in real time (frame rate higher than that of ultrasound imaging, such as 25 frames per second). During operation, the ultrasound probe 01 will be in real-time monitoring of the three-dimensional optical imaging sensor 03 and the three-dimensional surface information (i.e., contours) of the ultrasound probe 01 will be detected in real-time. From the motion of the obtained three-dimensional surface information, the motion of the ultrasonic probe 01 in the three-dimensional space can be known if it moves.
The spatial information processing module 06 is in communication connection with the three-dimensional optical imaging sensor 03, so as to acquire three-dimensional spatial information of the ultrasonic probe 01 based on the distance information and the image information. Preferably, respective distances between at least three pixel points in the image information of the ultrasonic probe and a camera in the three-dimensional optical imaging sensor can be obtained through distance information between the ultrasonic probe and each part of the ultrasonic probe, which is provided by the three-dimensional optical imaging sensor, and the distance between the whole ultrasonic probe and the camera can be obtained through calculating average distances between at least three pixel points in the image information and the camera. If the object or the region of interest of the object moves simultaneously with the ultrasound probe, three-dimensional spatial information of the object or the region of interest of the object and the movement in three-dimensional space can thus be known. In addition, the pointing direction (gradient, namely, space three-dimensional direction) of the ultrasonic probe can be calculated according to the obtained distances between at least three pixel points of the ultrasonic probe and the three-dimensional optical imaging sensor, and can be related to the direction of the ultrasonic probe. With knowledge of the three-dimensional spatial position of at least three pixels of a surface, the method of calculating the object surface orientation is standardized and will not be described in detail. Those skilled in the art will appreciate that the communication connection may be a wireless communication connection or a wired communication connection. The method of acquiring three-dimensional space information of the ultrasonic probe from the distance information and the image information may also employ any standardized method known in the art, and thus will not be described in detail herein. Of course, after the three-dimensional spatial information is acquired, those skilled in the art can further perform noise reduction on various signal processing methods, such as a moving average method on spatial position information and spatial angle information, and the like. In addition, when the spatial three-dimensional direction of the ultrasonic probe is acquired through the distance information, at least three pixels of the ultrasonic probe image, at most, the distance information of all pixels may be utilized.
The three-dimensional reconstruction module 07 is communicatively connected with the spatial information processing module 06 and the two-dimensional ultrasound imaging device 05, thereby reconstructing a three-dimensional ultrasound image based on the three-dimensional spatial information and the two-dimensional ultrasound image. Those skilled in the art will appreciate that the communication connection may be a wireless communication connection or a wired communication connection. Also, the reconstruction of the three-dimensional ultrasound image may be accomplished using any reconstruction method known in the art, and will not be described in detail herein.
The display device 08 and the three-dimensional reconstruction module 07 thereby display the three-dimensional ultrasound image. The display process may be real-time or non-real-time. Further, the three-dimensional optical imaging sensor 03 may also transmit the three-dimensional surface profile information of the object 02 to the display device 08 at the same time, so that the display device 08 may display the three-dimensional surface profile information of the object 02 and the three-dimensional ultrasound image in the same three-dimensional space, preferably in different colors. In a further preferred embodiment of the present invention, the image of the ultrasonic probe 01 may also be sent to the display device 08, and the display device 08 may display the three-dimensional surface profile information of the target 02, the three-dimensional ultrasonic image and the image of the ultrasonic probe 01 in the same three-dimensional space, preferably in different colors, or may display any of the three, for example, in a switching manner. In other simplified embodiments of the invention, the display device 08 may be omitted.
By implementing the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor, three-dimensional ultrasonic images can be reconstructed flexibly, with low cost and small volume by adopting the three-dimensional optical imaging sensor to acquire three-dimensional space information, and interference can be effectively avoided.
Fig. 2 is a system schematic diagram of a second preferred embodiment of a three-dimensional ultrasound imaging system based on a three-dimensional optical imaging sensor of the present invention. As shown in fig. 2, the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor of the present invention includes an ultrasonic probe 01, a two-dimensional ultrasonic imaging device 05, a three-dimensional optical imaging sensor 03, a spatial information processing module 06, a three-dimensional reconstruction module 07, and a display device 13. In the preferred embodiment shown in fig. 2, the ultrasound probe 01 is arranged within the field of view 04 of the three-dimensional optical imaging sensor 03 and is used for ultrasound scanning of a region of interest of the object 02. A visual marker 11 is provided at the front end portion of the ultrasonic probe 01, and the visual marker 11 is also located within the visual range 04. In other preferred embodiments of the present invention, as shown in fig. 3A-3C, the visual marker 11 may be located at the front end, side, front middle of the ultrasound probe 01. In a further preferred embodiment of the invention, the visual marker 11 may also be provided at other locations of the ultrasound probe 01, such as the bottom, the rear end, etc. Still alternatively, the visual marker 11 may be at least a part of the ultrasound probe 01, or the whole. Further, the visual marker 11 may be any shape, such as a circle, a triangle, and any regular or irregular shape, and may also be a code attached to any portion of the ultrasonic probe 01, such as various types of ArUco identification codes, and the like. The visual marker 11 may be any color (such that the visual marker 11 and the background may be distinguished by color), or a highly reflective infrared coating (such that the visual marker 11 and the background may be clearly distinguished in infrared imaging); or has itself a lighting characteristic such as using LED lamps or infrared lamps of different colors (which can be set to emit light when needed, which can be made more clear in RGB or infrared images due to its own brightness). The LED or infrared light may be further synchronized with the three-dimensional optical imaging sensor such that the LED or infrared light emits light only when the measurement by the three-dimensional optical imaging sensor is performed. In this way, the accuracy of the measurement of the visual marker 11 can be further increased. Still further, the visual marker 11 may be removable or retractable into the ultrasound probe when not in use, or may be fixedly disposed. In other preferred embodiments of the present invention, the three-dimensional optical imaging sensor 03 may be provided on the ultrasound probe 01 instead of the visual marker 11. For example, the three-dimensional optical imaging sensor 03 may be disposed at any position on the ultrasonic probe 01 as long as the visual marker 11 is still within the visible range 04. The advantage of this approach is that when we use a large visual marker 11, and/or place it in any one or more locations within the relevant space, some or a portion of the visual marker 11 can be detected, regardless of the adjustment angle and position. In other preferred embodiments of the present invention, a plurality of visual markers may be further provided, for example, visual markers of different shapes, types or colors are provided at different positions of the ultrasonic probe 01, so as to enhance the detection positioning effect.
In this embodiment, the two-dimensional ultrasound imaging device 05 is communicatively connected to the ultrasound probe 01 so as to generate a two-dimensional ultrasound image of the region of interest of the target 02 based on the ultrasound scan. The three-dimensional optical imaging sensor 03 may be communicatively connected to the ultrasonic probe 01 and the two-dimensional ultrasonic imaging device 05, and is configured to acquire a distance information image between the visual marker and the camera, and visual marker image information. The visual identifier image information may preferably be RGB and/or infrared image information.
In the present embodiment, the spatial information processing module 06 may preferably include a visual marker two-dimensional position recognition unit 61 and a three-dimensional spatial information acquisition unit 62. The visual marker two-dimensional position identifying unit 61 is configured to acquire two-dimensional position information of the visual marker based on the visual marker image information. Preferably, the visual marker two-dimensional position identifying unit 61 is configured to obtain the two-dimensional position information of the visual marker based on the color, shape, pattern of graphics, or darkness of the visual marker in the RGB and/or infrared image information. The three-dimensional space information obtaining unit 62 is configured to identify distances between at least three pixels in the image information of the visual marker and the camera based on the distance information image and the visual marker image information to obtain position and direction information of the visual marker, and obtain three-dimensional space information of the ultrasonic probe 01 based on the position and direction information of the visual marker.
As shown in fig. 2, when the three-dimensional optical imaging sensor 03 acquires visual marker image information of the circular visual marker 11 provided at the end of the ultrasonic probe 01, the visual marker two-dimensional position identifying unit 61 may acquire two-dimensional position information of at least three (but may be more or even all) pixels in the visual marker image information. The three-dimensional space information obtaining unit 62 may obtain three-dimensional space information (i.e., the distance between the camera and the left and right and up and down) of at least three pixels based on the two-dimensional position information of at least three pixels in the visual marker image information and the distance information image between the at least three pixels in the visual marker image information, so as to push out the position and direction information of the visual marker, and further obtain the three-dimensional space information of the ultrasonic probe 01.
In a further preferred embodiment of the invention, a triangular visual marker, preferably a triangular visual marker marked by a specific color or highly reflective infrared coating, may be employed. When the special designed visual marker is identified, the position and direction information of the visual marker can be deduced according to the three-dimensional space information (left, right, up, down and distance between the camera) contained in at least three pixels in the corresponding image, so that the three-dimensional space information of the ultrasonic probe can be obtained. For example, we can record the location of a visual marker with its center and use the pointing direction of the plane in which the visual marker lies in three-dimensional space to detect the direction of the visual marker. Once the position and direction of the visual marker can be detected, we can acquire the position and direction information of the image.
In a further preferred embodiment of the present invention, at least one angle sensor for acquiring three-dimensional direction information of the ultrasonic probe 01 is further provided on the ultrasonic probe 01. For example, to further improve the accuracy of the three-dimensional spatial direction detection of the ultrasonic probe, one or more angle sensors capable of detecting angles may be additionally installed in or on the surface of the ultrasonic probe 10: such as an accelerometer, gyroscope or magnetic sensor. These additional detected directional information can be compared to, combined with, the results of the three-dimensional optical imaging sensor to reduce interference from environmental factors. For example, one of the sensors is severely disturbed, and the other sensor can correct three-dimensional space information according to the detection result. Because these sensors employ disparate sensing technologies, an sporadic independent disturbance will not normally affect all sensors. For example, three-dimensional information obtained by a three-dimensional optical imaging sensor can be interfered by sudden strong light, and the operations of an accelerometer, a gyroscope and a magnetic sensor are not affected. In this way, at least one angle sensor arranged on the ultrasonic probe can be utilized to acquire three-dimensional direction information of the ultrasonic probe and the three-dimensional direction information is fused with three-dimensional space information of the ultrasonic probe acquired by the three-dimensional optical imaging sensor so as to improve the measurement accuracy of the three-dimensional space information
In a further preferred embodiment of the present invention, a plurality of three-dimensional optical imaging sensors 03 may be used to monitor the condition of the visual marker 11, each of which records the position and direction information of the ultrasonic probe, and the combination of the results of the plurality of sensors may help to improve the stability and reliability of the system. In addition, when the ultrasonic probe freely converts different directions, at least one three-dimensional optical imaging sensor 03 can timely acquire the position and direction information of the visual marker.
Of course, after the three-dimensional spatial information is acquired, those skilled in the art can further reduce noise for various signal processing methods, such as a moving average method, and the like. The three-dimensional reconstruction module 07 is communicatively connected with the spatial information processing module 06 and the two-dimensional ultrasound imaging device 05, thereby reconstructing a three-dimensional ultrasound image based on the three-dimensional spatial information and the two-dimensional ultrasound image. Those skilled in the art will appreciate that the communication connection may be a wireless communication connection or a wired communication connection. Also, the reconstruction of the three-dimensional ultrasound image may be accomplished using any reconstruction method known in the art, and will not be described in detail herein.
The display device 13 and the three-dimensional reconstruction module 07 thereby display the three-dimensional ultrasound image. The display process may be real-time or non-real-time. The display device 13 may further display the coordinates 12 of the visual marker 11.
In this embodiment, the above-mentioned visual marker and the three-dimensional optical imaging sensor are combined, and two-dimensional position information in the image and at least three pixel-to-camera distances provided by the three-dimensional optical imaging sensor can be acquired. Thus, for a region of interest of an object, we can learn where it is located in three-dimensional space by selecting the surface features (i.e., markers) or additional markers of the object. The three-dimensional space information of the ultrasonic probe can be calculated by the three-dimensional space information of the marker. First, the distance between at least three pixel points in the detected visual marker region and the camera in the three-dimensional optical imaging sensor can be obtained through the distance information provided by the three-dimensional optical imaging sensor and each part of the marker. The distance between the whole marker and the camera can be obtained by calculating at least three visual markers, or the average distance between all pixel points and the camera. The movement of the object in three dimensions can thus be known if the object moves simultaneously with the visual marker. In addition, the pointing direction (gradient) of the surface of the marker can be calculated according to the obtained distances between all the pixel points of the marker region and the sensor, and can be related to the direction of the object, namely the ultrasonic probe. In the case where the three-dimensional spatial positions of all points of a surface are known, there is a standardized way to calculate the object surface orientation and therefore will not be described in detail here. In order to make the detection of the distance of the visual marker from the three-dimensional optical imaging sensor and the direction of the visual marker itself more reliable, a number of enhancement algorithms can be employed: including using a median filter or the like to reduce noise, or wavelet analysis to process the distance data for all pixels on the marker. Further, after three-dimensional information (including three-dimensional position and direction) of the marker is obtained, various signal processing methods, such as moving average, can be used to reduce noise.
By implementing the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor, three-dimensional ultrasonic images can be reconstructed flexibly, with low cost and small volume by adopting the three-dimensional optical imaging sensor to acquire three-dimensional space information, and interference can be effectively avoided. Further, by adopting a visual marker or an angle sensor, the accuracy of the three-dimensional space information can be improved, and the quality of the three-dimensional ultrasonic image can be further improved.
Fig. 4 is a system schematic diagram of a third preferred embodiment of a three-dimensional ultrasound imaging system based on a three-dimensional optical imaging sensor of the present invention. In the embodiment shown in fig. 4, it differs from the embodiment shown in fig. 3 in that it is provided with a plurality of visual markers 11. Although only three circular visual markers are shown in fig. 4 disposed on one side of the ultrasound probe 11, one skilled in the art will appreciate that different numbers, shapes, or types of visual markers may be disposed at different locations of the ultrasound probe. These visual markers 11 may be coded in different colors (i.e. objects may be separated from the background by color) or marked with a highly reflective infrared coating (a bright mark that is clearly distinguishable from the background in the infrared). The spatial orientation of the ultrasound probe 11 can be detected using a plurality of visual markers. For example, we can record the position of a visual marker using its center and use the orientation of the plane in which the marker lies in three dimensions to detect the direction of the marker (the triangular marker shown in fig. 7). When positioning by using multiple markers, we can obtain multiple sets of three-dimensional spatial information (position and direction) at the same time, which helps to increase reliability of three-dimensional positioning, such as calculating three-dimensional spatial position and direction by using average value of multiple sets of information.
Thus, in the present embodiment, the three-dimensional optical imaging sensor 03 is configured to acquire each set of distance information images between each of the markers and the camera, and each set of marker image information; the two-dimensional position identification unit of the marker is used for acquiring each group of two-dimensional position information of each marker based on each group of image information of the marker; the three-dimensional space information acquisition unit is configured to identify distances between at least three pixel points in image information of each of the markers and the camera based on each of the sets of distance information images and each of the sets of marker image information to obtain position and direction information of each of the markers, and to obtain three-dimensional space information of the ultrasonic probe 01 based on the position and direction information of each of the markers.
Also, in this embodiment, the positions of the three-dimensional optical imaging sensor and the visual marker may be interchanged. In particular, the three-dimensional optical imaging sensor may be mounted at different locations of the ultrasound probe, and the marker may be placed within the visual range of the three-dimensional optical imaging sensor. The method has the advantages that large visual markers are adopted and are placed at a plurality of positions in space, and the three-dimensional optical imaging sensor can always detect some markers no matter how the positions and the angles are adjusted. Also, to further enhance the accuracy of detecting the distance of the markers, at least one, and preferably all, of the visual markers may be coated with an infrared reflective material to enhance the effect of the infrared pattern projected onto the surface of the markers. At least some or all of the visual indicia may also be made with a detachable feature so that it is only necessary to use (e.g., use magnetic material) when tracking in three dimensions. Likewise, in the method of fixing the three-dimensional optical imaging sensor to the ultrasound probe, the sensor may also be detachable. At least one or all of the visual markers may also have self-illuminating properties, such as using different colored LED or infrared lamps. The LED or infrared light may be further synchronized with the three-dimensional optical imaging sensor such that the LED or infrared light emits light only when the measurement by the three-dimensional optical imaging sensor is performed. Furthermore, the luminescence may have different time series characteristics, spatial transformation patterns to facilitate localization detection. The advantage of using self-illuminating markers is that it can be set that self-illuminating markers illuminate when needed or in dim conditions, self-illuminating makes it more clear in RGB or infrared images.
By implementing the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor, three-dimensional ultrasonic images can be reconstructed flexibly, with low cost and small volume by adopting the three-dimensional optical imaging sensor to acquire three-dimensional space information, and interference can be effectively avoided. Further, by adopting a mode of a marker or an angle sensor, the accuracy of the three-dimensional space information can be improved, and the quality of the three-dimensional ultrasonic image is further improved. Still further, by means of setting a plurality of markers, not only relevant position and direction information but also movement direction information can be obtained, so that the detection of the three-dimensional space information is more effective and reliable, and the quality of the three-dimensional ultrasonic image is further improved.
Fig. 5 is a system schematic diagram of a fourth preferred embodiment of a three-dimensional ultrasound imaging system based on a three-dimensional optical imaging sensor of the present invention. The embodiment of fig. 5 is similar to the embodiment of fig. 3, except that in the embodiment of fig. 5, an ArUco identification code is used as the visual identifier. In this embodiment, in order to make detection of the visual marker more efficient and reliable, an ArUco identification code is used. Typical ArUco identification codes are shown in FIGS. 6A-6C. ArUco identification codes have found considerable application in tracking objects in virtual reality. Generally, an RGB camera is used to capture an image containing an ArUco identification code, and depth and direction information of an object containing the identification code are determined according to the size and deformation of the ArUco identification code. Of course, in other preferred embodiments of the present invention, other ArUco identification codes, and even other codes, may be employed. The use of ArUco identification codes or other codes will make detection and tracking of the identifier easier because the location and orientation of the ArUco identification code in the image can be quickly identified by a number of algorithms. The obtained position of the identification code identifier can be further processed to obtain the distance between at least three pixels of the identification code identifier and the camera so as to further obtain the three-dimensional direction information of the identification code identifier. The direction information (relatively inaccurate) from detecting the ArUco identifier may also be incorporated into the calculation of the three-dimensional direction information of the identifier, such as to assist in removing noise that may be present in the distance measurement.
Fig. 7 is a system schematic diagram of a fifth preferred embodiment of a three-dimensional ultrasound imaging system based on a three-dimensional optical imaging sensor of the present invention. In the embodiment shown in fig. 7, three different types of visual markers are used, namely a circular visual marker 111, a triangular visual marker 121, and an arco identification code 131. And none of these markers is provided on the ultrasonic probe, but instead, the three-dimensional optical imaging sensor 103 is provided at the end of the ultrasonic probe 01, and the circular visual marker 111, the triangular visual marker 121, and the ArUco identification code 131 are placed at different positions in the visual range of the three-dimensional optical sensor, so that the three-dimensional optical imaging sensor can detect some markers regardless of the adjustment of position and angle. In implementations, the shape, size, and number of visual markers may be combined differently depending on the particular application. The position and the direction of the three-dimensional optical sensor in the three-dimensional space can be calculated through the distance and the image information of the related visual marker obtained by the three-dimensional optical sensor, so that the three-dimensional space information of the ultrasonic probe connected with the three-dimensional optical sensor can be obtained. Those skilled in the art will appreciate that the remainder of the three-dimensional ultrasound imaging system based on the three-dimensional optical imaging sensor of this embodiment may be configured with reference to any of the embodiments shown in fig. 1-6 and will not be discussed further herein.
Fig. 8 is a system schematic diagram of a sixth preferred embodiment of a three-dimensional ultrasound imaging system based on a three-dimensional optical imaging sensor of the present invention. In the embodiment shown in fig. 8, which differs from the embodiment shown in fig. 2 in that a triangular visual marker 21 covered with an infrared reflecting material is used and the three-dimensional optical imaging sensor 03 is further used for detecting three-dimensional profile information of the object 02 of the region of interest of the object 02. With a triangular visual marker 21 we can record the position of the marker with its centre and use the orientation of the plane in which the marker lies in three dimensions to detect the direction of the marker. Once the position and orientation of the marker can be detected, we can acquire the position and orientation of the image. And the spatial information processing module 06 further includes: a correction unit 63 and a calibration unit 64. The correction unit 63 is configured to obtain three-dimensional motion information of the target 02 during the scanning process of the ultrasonic probe 01 based on the three-dimensional contour information of the target 02, and correct three-dimensional space information of the ultrasonic probe 01 based on the three-dimensional motion information. The calibration unit 64 is used for converting the pixel points of each frame of the two-dimensional ultrasound image into a three-dimensional space to calibrate the three-dimensional ultrasound image.
In the present embodiment, the correction is made by using the three-dimensional space information of the probe detected by the correction unit 63. The motion of the object during scanning can cause errors in three-dimensional ultrasound imaging. This step may reduce or eliminate disturbances caused by the movement of the object. The obtained three-dimensional ultrasound image may be further calibrated using the calibration unit 64. The associated correction and calibration methods may be any correction and calibration methods known in the art and will not be described in detail herein.
By implementing the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor, three-dimensional ultrasonic images can be reconstructed flexibly, with low cost and small volume by adopting the three-dimensional optical imaging sensor to acquire three-dimensional space information, and interference can be effectively avoided. Further, by adopting a mode of a marker or an angle sensor, the accuracy of the three-dimensional space information can be improved, and the quality of the three-dimensional ultrasonic image is further improved. Still further, by means of setting a plurality of markers, not only relevant position and direction information but also movement direction information can be obtained, so that the detection of the three-dimensional space information is more effective and reliable, and the quality of the three-dimensional ultrasonic image is further improved. Still further, the obtained three-dimensional space information and/or the three-dimensional ultrasonic image can be corrected and/or calibrated, so that a more accurate and reliable three-dimensional ultrasonic image can be obtained.
Fig. 9 is a flow chart of a first preferred embodiment of a three-dimensional ultrasound imaging method based on a three-dimensional optical imaging sensor of the present invention. As shown in fig. 9, in step S1, an ultrasound probe is employed for ultrasound scanning a region of interest of a target and a two-dimensional ultrasound image of the region of interest of the target is generated based on the ultrasound scanning. Those skilled in the art will appreciate that the ultrasound probe may employ the ultrasound probe configuration of any of the embodiments of fig. 1-8. In step S2, distance information between at least one marker and a camera of the three-dimensional optical imaging sensor within a visible range of the three-dimensional optical imaging sensor and image information of the marker are acquired. Preferably, the three-dimensional optical imaging sensor is communicably connected to the ultrasonic probe and the two-dimensional ultrasonic imaging device, so as to be used for acquiring distance information between the ultrasonic probe and a camera of the three-dimensional optical imaging sensor and image information of the ultrasonic probe. It will be appreciated by those skilled in the art that this step may be implemented using the three-dimensional optical imaging sensor 03 and the two-dimensional ultrasound imaging device 05 of any of the embodiments of fig. 1-8. In a preferred embodiment of the invention, the marker is at least part of the ultrasound probe. In a further preferred embodiment of the invention, the marker comprises at least one visual marker provided on the ultrasound probe. In yet a further preferred embodiment of the present invention, the three-dimensional optical imaging sensor is disposed on the ultrasound probe, and the marker comprises at least one visual marker disposed within a visual range of the three-dimensional optical imaging sensor.
In step S3, three-dimensional spatial information of the ultrasound probe is acquired based on the distance information and the image information. Preferably, the distance between at least three pixel points in the image information of the ultrasonic probe and the camera in the three-dimensional optical imaging sensor can be obtained through the distance information between the ultrasonic probe and each part of the ultrasonic probe provided by the three-dimensional optical imaging sensor, and the distance between the whole ultrasonic probe and the camera can be obtained through calculating the average distance between at least three pixel points in the image information and the camera. The movement of the object or the region of interest of the object in three dimensions can thus be known if the object or the region of interest of the object moves simultaneously with the ultrasound probe. In addition, the pointing direction (gradient) of the ultrasonic probe can be calculated according to the obtained distances between all the pixel points of the ultrasonic probe and the three-dimensional optical imaging sensor, and can be related to the direction of the ultrasonic probe. With the three-dimensional spatial locations of all points of a surface known, the method of computing the object surface orientation is standardized and will not be described in detail. In step S4, a three-dimensional ultrasound image is reconstructed based on the three-dimensional spatial information and the two-dimensional ultrasound image. It is known to those skilled in the art that the reconstruction of three-dimensional ultrasound images can be accomplished using any reconstruction method known in the art, and will not be described in detail herein.
Those skilled in the art will appreciate that in other further preferred embodiments of the present invention, the three-dimensional optical imaging sensor-based three-dimensional ultrasound imaging system constructed in any of the embodiments of fig. 1-8 may be employed to implement the three-dimensional optical imaging sensor-based three-dimensional ultrasound imaging method described above.
By implementing the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor, three-dimensional ultrasonic images can be reconstructed flexibly, with low cost and small volume by adopting the three-dimensional optical imaging sensor to acquire three-dimensional space information, and interference can be effectively avoided.
Fig. 10 is a flow chart of a second preferred embodiment of the three-dimensional ultrasound imaging method of the present invention based on a three-dimensional optical imaging sensor. In step S1, at least one marker is arranged in the visible range of the three-dimensional optical imaging sensor. In the preferred embodiment, the marker is disposed on the ultrasound probe. In other preferred embodiments of the present invention, the three-dimensional optical imaging sensor is disposed on the ultrasound probe. The type, number, kind, and placement of the markers described above may be referred to in the three-dimensional ultrasound imaging configuration based on three-dimensional optical imaging sensors shown in fig. 1-8. In other embodiments of the invention, the three-dimensional optical imaging sensor may also be disposed on the ultrasound probe.
In step S2, an ultrasound probe is employed for ultrasound scanning of a region of interest of a target and a two-dimensional ultrasound image of the region of interest of the target is generated based on the ultrasound scanning. Those skilled in the art will appreciate that the ultrasound probe may employ the ultrasound probe configuration of any of the embodiments of fig. 1-8.
In step 3, a distance information image between the marker and the camera and marker image information are acquired. As previously mentioned, any three-dimensional optical imaging sensor in the art may be employed, particularly microsoft's Kinect sensor and intel's Realsense sensor. The image information may be RGB and/or infrared image information, or any other image information that may be used to obtain three-dimensional surface information and/or motion information of an object.
Two-dimensional position information of the marker is acquired based on the marker image information in step S4. In a further preferred embodiment of the invention, the two-dimensional position information of the marker may be acquired based on the color, shape, pattern of graphics or degree of darkness of the marker in the RGB and/or infrared image information. In step S5, distances between at least three pixels in the image information of the marker and the camera are identified based on the distance information image and the marker image information to obtain position and direction information of the marker, and three-dimensional space information of the ultrasound probe is obtained based on the position and direction information of the marker. Preferably, the distance between at least three pixel points in the image information of the marker and the camera in the three-dimensional optical imaging sensor can be obtained through the distance information provided by the three-dimensional optical imaging sensor and each part of the marker, and the distance between the whole marker and the camera can be obtained through calculating the average distance between at least three pixel points in the image information and the camera. The movement of the object or the region of interest of the object in three dimensions can thus be known if the object or the region of interest of the object moves simultaneously with the marker. In addition, the pointing direction (gradient) of the marker can be calculated according to the obtained distances between all the pixel points of the marker and the three-dimensional optical imaging sensor, and can be related to the direction of the marker. With the three-dimensional spatial locations of all points of a surface known, the method of computing the object surface orientation is standardized and will not be described in detail. When the specially designed marker is identified, the position and direction information of the marker can be deduced according to the three-dimensional space information (left, right, up, down and distance between the marker and the camera) contained in at least three pixels in the corresponding image, so that the three-dimensional space information of the ultrasonic probe can be obtained. It is known to those skilled in the art that the above steps S4-S5 may be implemented by the spatial information processing module 06 of any one of the embodiments of fig. 2-8.
In step S6, a three-dimensional ultrasound image is reconstructed based on the three-dimensional spatial information and the two-dimensional ultrasound image. It is known to those skilled in the art that the reconstruction of three-dimensional ultrasound images can be accomplished using any reconstruction method known in the art, and will not be described in detail herein. In step S7, the three-dimensional ultrasound image may be displayed using a display device.
In a further preferred embodiment of the invention, a plurality of markers may be provided within the visible range of the three-dimensional optical imaging sensor. However, an ultrasound probe is employed for ultrasound scanning a region of interest of a target and generating a two-dimensional ultrasound image of the region of interest of the target based on the ultrasound scanning. In a further embodiment, the target three-dimensional profile information of the region of interest of the target may also be detected prior to the ultrasound scan. Thus, in the subsequent step, acquiring each group of distance information images between each marker and the camera and each group of marker image information; acquiring each set of two-dimensional position information of each marker based on each set of marker image information; and identifying distances between at least three pixel points in the image information of each marker and the camera based on each group of the distance information images and each group of the marker image information to obtain position and direction information of each marker, and obtaining three-dimensional space information of the ultrasonic probe based on the position and direction information of each marker.
Further, in a preferred embodiment of the present invention, three-dimensional motion information of the target in the scanning process of the ultrasonic probe may be acquired based on the three-dimensional profile information of the target, and three-dimensional spatial information of the ultrasonic probe may be corrected based on the three-dimensional motion information. Movement of the target during scanning can cause errors in three-dimensional ultrasound imaging. This step may reduce or eliminate disturbances caused by the movement of the object.
In a further preferred embodiment of the invention, at least one angle sensor for acquiring three-dimensional direction information of the ultrasound probe is further provided on the ultrasound probe, so that a subsequent correction can be further performed based on the three-dimensional direction information. The three-dimensional direction information of the ultrasonic probe may be acquired by, for example, additional sensors (including accelerometers, gyroscopes and magnetometers) internal or external to the ultrasonic probe. For example, at least one angle sensor arranged on the ultrasonic probe can be used for acquiring three-dimensional direction information of the ultrasonic probe and fusing the three-dimensional direction information with three-dimensional space information of the ultrasonic probe acquired by the three-dimensional optical imaging sensor so as to improve measurement accuracy of the three-dimensional space information
In a further preferred embodiment of the present invention, the method further comprises converting pixels of each frame of the two-dimensional ultrasound image into three-dimensional space to calibrate the three-dimensional ultrasound image. The specific calibration method may employ any known calibration method in the art.
In a further preferred embodiment of the present invention, the target three-dimensional contour information and the three-dimensional ultrasound image may be displayed on the same three-dimensional space. For example, a fused display of surface profile information of the target with the three-dimensional ultrasound image may be combined. That is, the three-dimensional ultrasound image may be displayed in a different manner, such as in a different color, on the same three-dimensional space as the three-dimensional surface profile information image of the object. This step may also be displayed in real time during the ultrasound probe scanning process. In still further preferred embodiments of the present invention, the ultrasound probe and its position/orientation information may also be displayed so that the operator can see the position, orientation and motion of the ultrasound probe. This function may be combined with the three-dimensional surface contour shape image in the previous step. That is, the three-dimensional surface contour information image of the object is displayed in a certain color or gray scale in the display module, and the ultrasonic image is displayed in the same three-dimensional space in real time according to the corresponding three-dimensional space information. So that the operator can scan the object better.
By implementing the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor, three-dimensional ultrasonic images can be reconstructed flexibly, with low cost and small volume by adopting the three-dimensional optical imaging sensor to acquire three-dimensional space information, and interference can be effectively avoided. Further, by adopting a mode of a marker or an angle sensor, the accuracy of the three-dimensional space information can be improved, and the quality of the three-dimensional ultrasonic image is further improved. Still further, by means of setting a plurality of markers, not only relevant position and direction information but also movement direction information can be obtained, so that the detection of the three-dimensional space information is more effective and reliable, and the quality of the three-dimensional ultrasonic image is further improved. Still further, the obtained three-dimensional space information and/or the three-dimensional ultrasonic image can be corrected and/or calibrated, so that a more accurate and reliable three-dimensional ultrasonic image can be obtained.
Those skilled in the art will further appreciate that the three-dimensional ultrasound imaging system and method based on a three-dimensional optical imaging sensor of the present invention may be interrelated in that the functions and steps described in each may be combined with, or substituted for one another.
The invention has been described above also with the aid of functional modules illustrating certain important functions. The boundaries of these functional building blocks have been defined specifically herein for the convenience of description. When these important functions are properly implemented, varying the limits thereof is allowed. Similarly, flow chart modules are also specifically defined herein to illustrate certain important functions, and for a wide range of applications, the boundaries and sequence of flow chart modules may be otherwise defined so long as such important functions are still achieved. Variations in the boundaries and sequence of the above described functional blocks, flowchart functional blocks should still be considered within the scope of the claims.
The present invention can also be realized by a computer program product, which comprises all the features enabling the implementation of the methods described herein, and which when installed in a computer system is able to carry out these methods. The computer program in this document refers to: any expression, in any programming language, code or notation, of a set of instructions intended to cause a system having an information processing capability to perform a particular function either directly or after either or both of the following: a) Conversion to other languages, codes or symbols; b) Reproduced in a different format.
While the invention has been described with reference to specific embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the invention without departing from its scope. Therefore, it is intended that the invention not be limited to the particular embodiment disclosed, but that the invention will include all embodiments falling within the scope of the appended claims.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, and alternatives falling within the spirit and principles of the invention.

Claims (17)

1. A three-dimensional ultrasound imaging system based on a three-dimensional optical imaging sensor, comprising:
the ultrasonic probe is used for carrying out ultrasonic scanning on the region of interest of the target;
a two-dimensional ultrasound imaging device for generating a two-dimensional ultrasound image of a region of interest of the target based on the ultrasound scan;
a three-dimensional optical imaging sensor for acquiring a distance information image between at least one marker and a camera of the three-dimensional optical imaging sensor within a visible range of the three-dimensional optical imaging sensor and image information of the marker;
The spatial information processing module is used for acquiring three-dimensional spatial information of the ultrasonic probe based on the distance information image and the image information; the spatial information processing module includes:
a marker two-dimensional position recognition unit for acquiring two-dimensional position information of the marker based on the marker image information;
a three-dimensional space information acquisition unit for identifying respective distances between at least three pixel points in the image information of the marker and the camera based on the distance information image and the marker image information to obtain position and direction information of the marker, and obtaining three-dimensional space information of the ultrasonic probe based on the position and direction information of the marker;
the three-dimensional optical imaging sensor is further used for detecting target three-dimensional contour information of a region of interest of the target;
the spatial information processing module further includes:
a correction unit, configured to acquire three-dimensional motion information of the target in a scanning process of the ultrasonic probe based on the three-dimensional profile information of the target, and correct three-dimensional space information of the ultrasonic probe based on the three-dimensional motion information;
the three-dimensional reconstruction module is used for reconstructing a three-dimensional ultrasonic image based on the three-dimensional space information and the two-dimensional ultrasonic image;
The ultrasonic probe is further provided with at least one angle sensor for acquiring three-dimensional direction information of the ultrasonic probe, and the three-dimensional direction information is fused with the three-dimensional space information of the ultrasonic probe acquired by the three-dimensional optical imaging sensor so as to improve the measurement accuracy of the three-dimensional space information.
2. The three-dimensional ultrasound imaging system based on a three-dimensional optical imaging sensor of claim 1, wherein the marker is at least a portion of the ultrasound probe.
3. The three-dimensional ultrasound imaging system based on three-dimensional optical imaging sensors of claim 1, wherein the markers comprise at least one visual marker disposed on the ultrasound probe.
4. The three-dimensional ultrasound imaging system of claim 1, wherein the three-dimensional optical imaging sensor is disposed on the ultrasound probe, and wherein the marker comprises at least one visual marker disposed within a visual range of the three-dimensional optical imaging sensor.
5. The three-dimensional ultrasound imaging system based on a three-dimensional optical imaging sensor of claim 1, wherein a plurality of markers are disposed within a viewable range of the three-dimensional optical imaging sensor; the three-dimensional optical imaging sensor is used for acquiring each group of distance information images between each marker and the camera and each group of marker image information; the two-dimensional position identification unit of the marker is used for acquiring each group of two-dimensional position information of each marker based on each group of image information of the marker; the three-dimensional space information acquisition unit is configured to identify respective distances between at least three pixel points in image information of each of the markers and the camera based on each of the sets of distance information images and each of the sets of marker image information to obtain position and direction information of each of the markers, and to obtain three-dimensional space information of the ultrasonic probe based on the position and direction information of each of the markers.
6. The three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor according to claim 5, wherein the marker image information is RGB and/or infrared image information, and the marker two-dimensional position recognition unit is configured to acquire the two-dimensional position information of the marker based on the color, shape, pattern or darkness of the marker in the RGB and/or infrared image information.
7. The three-dimensional ultrasound imaging system based on the three-dimensional optical imaging sensor of claim 5, further comprising:
and the calibration unit is used for converting the pixel points of each frame of the two-dimensional ultrasonic image into a three-dimensional space to calibrate the three-dimensional ultrasonic image.
8. The three-dimensional optical imaging sensor-based three-dimensional ultrasound imaging system of claim 1, wherein the three-dimensional optical imaging sensor-based three-dimensional ultrasound imaging system comprises a plurality of three-dimensional optical imaging sensors.
9. The three-dimensional ultrasound imaging system based on three-dimensional optical imaging sensor of claim 1, further comprising a display device for displaying the target three-dimensional profile information, the three-dimensional ultrasound image and/or ultrasound probe image on the same three-dimensional space.
10. A three-dimensional ultrasound imaging method based on a three-dimensional optical imaging sensor, comprising:
s1, an ultrasonic probe is used for carrying out ultrasonic scanning on a region of interest of a target, and a two-dimensional ultrasonic image of the region of interest of the target is generated based on the ultrasonic scanning;
s2, acquiring a distance information image between at least one marker and a camera of the three-dimensional optical imaging sensor and image information of the marker in a visible range of the three-dimensional optical imaging sensor;
s3, acquiring three-dimensional space information of the ultrasonic probe based on the distance information image and the image information; the step S3 further includes:
s31, acquiring two-dimensional position information of the marker based on the marker image information;
s32, identifying respective distances between at least three pixel points in the image information of the marker and the camera based on the distance information image and the marker image information to obtain position and direction information of the marker, and obtaining three-dimensional space information of the ultrasonic probe based on the position and direction information of the marker; further comprises: detecting target three-dimensional contour information of a region of interest of the target before ultrasonic scanning;
The step S3 further includes:
s33, acquiring three-dimensional motion information of the target in the scanning process of the ultrasonic probe based on the three-dimensional contour information of the target, and correcting three-dimensional space information of the ultrasonic probe based on the three-dimensional motion information;
s35, acquiring three-dimensional direction information of the ultrasonic probe by using at least one angle sensor arranged on the ultrasonic probe, and fusing the three-dimensional direction information with three-dimensional space information of the ultrasonic probe acquired by the three-dimensional optical imaging sensor to improve the measurement accuracy of the three-dimensional space information;
s4, reconstructing a three-dimensional ultrasonic image based on the three-dimensional space information and the two-dimensional ultrasonic image.
11. The three-dimensional ultrasonic imaging method based on a three-dimensional optical imaging sensor of claim 10, wherein the marker is at least a portion of the ultrasonic probe.
12. The three-dimensional ultrasound imaging method based on three-dimensional optical imaging sensors of claim 10, wherein the markers comprise at least one visual marker disposed on the ultrasound probe.
13. The three-dimensional ultrasound imaging method based on a three-dimensional optical imaging sensor of claim 10, wherein the three-dimensional optical imaging sensor is disposed on the ultrasound probe, and the marker comprises at least one visual marker disposed within a visual range of the three-dimensional optical imaging sensor.
14. The three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor according to claim 10, wherein in the step S2, a plurality of markers are set in a visible range of the three-dimensional optical imaging sensor, and each set of distance information images and each set of marker image information between each of the markers and the camera are acquired; in the step S31, each set of two-dimensional position information of each of the markers is acquired based on each set of the marker image information; in the step S32, distances between at least three pixels in the image information of each of the markers and the camera are identified based on each of the sets of distance information images and each of the sets of marker image information to obtain position and direction information of each of the markers, and three-dimensional space information of the ultrasound probe is obtained based on the position and direction information of each of the markers.
15. The three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor according to claim 14, wherein the marker image information is RGB and/or infrared image information, and in the step S31, two-dimensional position information of the marker is acquired based on a color, a shape, a pattern, or a degree of darkness of the marker in the RGB and/or infrared image information.
16. The three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor according to claim 10, wherein the step S3 further comprises:
s34, converting pixel points of each frame of the two-dimensional ultrasonic image into a three-dimensional space to calibrate the three-dimensional ultrasonic image.
17. The three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor according to claim 10, further comprising
And S5, displaying the three-dimensional outline information of the target, the three-dimensional ultrasonic image and/or the ultrasonic probe image on the same three-dimensional space.
CN201910086302.8A 2019-01-29 2019-01-29 Three-dimensional ultrasonic imaging method and system based on three-dimensional optical imaging sensor Active CN111487320B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201910086302.8A CN111487320B (en) 2019-01-29 2019-01-29 Three-dimensional ultrasonic imaging method and system based on three-dimensional optical imaging sensor
PCT/CN2020/073105 WO2020156299A1 (en) 2019-01-29 2020-01-20 Three-dimensional ultrasonic imaging method and system based on three-dimensional optical imaging sensor

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910086302.8A CN111487320B (en) 2019-01-29 2019-01-29 Three-dimensional ultrasonic imaging method and system based on three-dimensional optical imaging sensor

Publications (2)

Publication Number Publication Date
CN111487320A CN111487320A (en) 2020-08-04
CN111487320B true CN111487320B (en) 2023-07-21

Family

ID=71793920

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910086302.8A Active CN111487320B (en) 2019-01-29 2019-01-29 Three-dimensional ultrasonic imaging method and system based on three-dimensional optical imaging sensor

Country Status (2)

Country Link
CN (1) CN111487320B (en)
WO (1) WO2020156299A1 (en)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112617902A (en) * 2020-12-31 2021-04-09 上海联影医疗科技股份有限公司 Three-dimensional imaging system and imaging method
CN115248446A (en) * 2021-04-28 2022-10-28 中慧医学成像有限公司 Three-dimensional ultrasonic imaging method and system based on laser radar
CN113576528A (en) * 2021-08-31 2021-11-02 深圳迈瑞动物医疗科技有限公司 Operation method of posture map information for ultrasound and ultrasound imaging system
CN114396904A (en) * 2021-11-29 2022-04-26 北京银河方圆科技有限公司 Positioning device and positioning system
CN114469169A (en) * 2021-12-30 2022-05-13 上海深至信息科技有限公司 Ultrasonic three-dimensional reconstruction method and system
CN114533111A (en) * 2022-01-12 2022-05-27 电子科技大学 Three-dimensional ultrasonic reconstruction system based on inertial navigation system

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9451928B2 (en) * 2006-09-13 2016-09-27 Elekta Ltd. Incorporating internal anatomy in clinical radiotherapy setups
CN101569541B (en) * 2008-04-29 2011-04-06 香港理工大学 Three-dimensional ultrasonic imaging system
US9113817B2 (en) * 2010-10-20 2015-08-25 Siemens Medical Solutions Usa, Inc. System for locating anatomical objects in ultrasound imaging
CN103829973A (en) * 2014-01-16 2014-06-04 华南理工大学 Ultrasonic probe scanning system and method for remote control
CN109223030B (en) * 2017-07-11 2022-02-18 中慧医学成像有限公司 Handheld three-dimensional ultrasonic imaging system and method
CN107582098B (en) * 2017-08-08 2019-12-06 南京大学 three-dimensional ultrasonic imaging method for two-dimensional ultrasonic image set reconstruction
CN107714082A (en) * 2017-09-04 2018-02-23 北京航空航天大学 A kind of ultrasonic probe caliberating device and method based on optical alignment
CN107944422B (en) * 2017-12-08 2020-05-12 业成科技（成都）有限公司 Three-dimensional camera device, three-dimensional camera method and face recognition method
CN107865671B (en) * 2017-12-12 2023-05-26 成都优途科技有限公司 Three-dimensional ultrasonic scanning system based on monocular vision positioning and control method
CN109171808A (en) * 2018-09-07 2019-01-11 东南大学 Three-dimension ultrasonic imaging system based on measuring three-dimensional profile

Also Published As

Publication number Publication date
WO2020156299A1 (en) 2020-08-06
CN111487320A (en) 2020-08-04

Similar Documents

Publication Publication Date Title
CN111487320B (en) Three-dimensional ultrasonic imaging method and system based on three-dimensional optical imaging sensor
CN101068344B (en) Object detection apparatus
CN106643699B (en) Space positioning device and positioning method in virtual reality system
JP6404954B2 (en) POSITIONING SYSTEM, HAND-HELD POSITIONING UNIT, EVALUATION UNIT, AND METHOD FOR POSITIONING
EP3262439B1 (en) Using intensity variations in a light pattern for depth mapping of objects in a volume
JP3951984B2 (en) Image projection method and image projection apparatus
JP5576937B2 (en) Vehicle periphery monitoring device
US20140375812A1 (en) Method for representing a vehicle's surrounding environment
JP2018518750A (en) Enhancement of depth map representation by reflection map representation
CN110009682A (en) A kind of object recognition and detection method based on monocular vision
WO2019135726A1 (en) Metal detector capable of visualizing the target shape
CN110895676B (en) dynamic object tracking
US20160171702A1 (en) Optical tracking
GB2602588A (en) Electronic device for automated user identification
CN111667526A (en) Method and apparatus for determining size and distance of multiple objects in an environment
CN111095357A (en) Positioning system
JPH06189906A (en) Visual axial direction measuring device
JP2005326405A (en) Method for measuring attitude and/or position of object by means of image processing method
CN115388891A (en) Space positioning method and system for large-view-field moving target
KR100913165B1 (en) Apparatus and method for detecting a localization of mobility
CN114724104A (en) Method, device, electronic equipment, system and medium for detecting visual recognition distance
KR100844640B1 (en) Method for object recognizing and distance measuring
CN113340405A (en) Bridge vibration mode measuring method, device and system
JP2007003462A (en) Sight line recognition device and sight line recognition method
JPWO2020175085A1 (en) Image processing device and image processing method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant