WO2020156299A1

WO2020156299A1 - Three-dimensional ultrasonic imaging method and system based on three-dimensional optical imaging sensor

Info

Publication number: WO2020156299A1
Application number: PCT/CN2020/073105
Authority: WO
Inventors: 郑永平; 孟强
Original assignee: 中慧医学成像有限公司
Priority date: 2019-01-29
Filing date: 2020-01-20
Publication date: 2020-08-06
Also published as: CN111487320B; CN111487320A

Abstract

A three-dimensional ultrasonic imaging system and method based on a three-dimensional optical imaging sensor. The system comprises: an ultrasonic probe (01) for performing ultrasonic scanning on a region of interest of a target; a two-dimensional ultrasonic imaging device (05) for generating a two-dimensional ultrasonic image of the region of interest of the target on the basis of the ultrasonic scanning; a three-dimensional optical imaging sensor (03) for obtaining distance information from at least one marker (11) within a visual range (04) of the three-dimensional optical imaging sensor (03) to a camera of the three-dimensional optical imaging sensor (03) and image information of the marker (11); a spatial information processing module (06) for obtaining three-dimensional spatial information of the ultrasonic probe (01) on the basis of the distance information and the image information; and a three-dimensional reconstruction module (07) for reconstructing a three-dimensional ultrasonic image on the basis of the three-dimensional spatial information and the two-dimensional ultrasonic image. The system and method can flexibly reconstruct a three-dimensional ultrasonic image with low costs and a small volume, and can effectively avoid interference.

Description

Three-dimensional ultrasonic imaging method and system based on three-dimensional optical imaging sensor

Technical field

The invention relates to the field of three-dimensional ultrasound imaging, and more specifically, to a three-dimensional ultrasound imaging method and system based on a three-dimensional optical imaging sensor.

Background technique

Free-hand three-dimensional imaging, that is, the human hand freely moves the ultrasound probe to scan on the target, and uses the optical three-dimensional space sensing technology to capture the position and direction information of the ultrasound probe. Currently commonly used three-dimensional space sensing technologies include spatial reference objects or signals and corresponding detectors. For example, an electromagnetic transmitter is used to emit electromagnetic waves as a reference signal, and the detector determines the position and direction of the probe according to the change of the electromagnetic wave field strength. For another example, one or more visual markers placed on the surface of the probe are used as reference objects, and one or more cameras surrounding the ultrasound probe are used to detect the position and direction of the probe.

technical problem

The above-mentioned three-dimensional space sensing technologies each have their own advantages and limitations. As far as electromagnetic sensing technology is concerned, it will be interfered by surrounding metal objects. However, camera-based sensing systems are usually bulky and expensive.

Technical solutions

The technical problem to be solved by the present invention is to provide a three-dimensional ultrasonic imaging method and system based on a three-dimensional optical imaging sensor with strong anti-interference ability, low cost and small volume in view of the above-mentioned defects of the prior art.

The technical solution adopted by the present invention to solve its technical problems is to construct a three-dimensional ultrasonic imaging system based on a three-dimensional optical imaging sensor, including:

Ultrasound probe, used for ultrasonic scanning of the target area of interest;

A two-dimensional ultrasound imaging device for generating a two-dimensional ultrasound image of the region of interest of the target based on the ultrasound scan;

A three-dimensional optical imaging sensor for acquiring distance information between at least one marker and the camera of the three-dimensional optical imaging sensor within the visible range of the three-dimensional optical imaging sensor and image information of the marker; spatial information processing A module for acquiring three-dimensional space information of the ultrasound probe based on the distance information and the image information;

The three-dimensional reconstruction module is used to reconstruct a three-dimensional ultrasound image based on the three-dimensional space information and the two-dimensional ultrasound image.

In the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor of the present invention, the marker is at least a part of the ultrasonic probe.

In the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor of the present invention, the marker includes at least one visual marker provided on the ultrasonic probe.

In the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor of the present invention, the three-dimensional optical imaging sensor is arranged on the ultrasonic probe, and the marker includes a three-dimensional optical imaging sensor arranged in the visible range of the three-dimensional optical imaging sensor. At least one visual identifier.

In the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor of the present invention, the spatial information processing module includes:

A marker two-dimensional position recognition unit, configured to obtain the two-dimensional position information of the marker based on the marker image information;

A three-dimensional information acquisition unit, configured to identify the respective distances between at least three pixels in the image information of the marker and the camera based on the distance information image and the marker image information to obtain the marker The three-dimensional space information of the ultrasound probe is obtained based on the position and direction information of the marker.

In the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor of the present invention, a plurality of markers are arranged in the visible range of the three-dimensional optical imaging sensor; the three-dimensional optical imaging sensor is used to acquire each of the markers Each group of distance information images and each group of marker image information between the camera and the camera; the marker two-dimensional position recognition unit is used to obtain each group of two markers based on each group of the marker image information Dimensional position information; the three-dimensional space information acquisition unit is used to identify the difference between at least three pixels in the image information of each of the markers and the camera based on each group of the distance information image and each group of the marker image information The position and direction information of each of the markers is obtained by the respective distances between the two, and the three-dimensional space information of the ultrasound probe is obtained based on the position and direction information of each of the markers.

In the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor of the present invention, the marker image information is RGB and/or infrared image information, and the marker two-dimensional position recognition unit is configured to be based on the RGB and/or infrared image information. Or the color, shape, pattern or lightness of the marker in the infrared image information is used to obtain the two-dimensional position information of the marker.

In the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor of the present invention, the three-dimensional optical imaging sensor is further used to detect the target three-dimensional contour information of the region of interest of the target;

The spatial information processing module further includes:

The correction unit is configured to obtain the three-dimensional movement information of the target during the scanning process of the ultrasound probe based on the three-dimensional contour information of the target, and correct the three-dimensional spatial information of the ultrasound probe based on the three-dimensional movement information.

In the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor of the present invention, it further includes:

The calibration unit is used to convert the pixel points of each frame of the two-dimensional ultrasound image into a three-dimensional space to calibrate the three-dimensional ultrasound image.

In the three-dimensional ultrasound imaging system based on the three-dimensional optical imaging sensor of the present invention, the ultrasound probe is further provided with at least one angle sensor for acquiring three-dimensional direction information of the ultrasound probe and/or the three-dimensional optical imaging sensor The three-dimensional ultrasound imaging system includes multiple three-dimensional optical imaging sensors.

The three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor of the present invention further includes a display device for displaying the target three-dimensional contour information, the three-dimensional ultrasound image and/or the ultrasound probe image on the same three-dimensional space.

Another technical solution adopted by the present invention to solve its technical problems is to construct a three-dimensional ultrasonic imaging method based on a three-dimensional optical imaging sensor, including:

S1. An ultrasound probe is used to perform an ultrasound scan of a region of interest of a target and generate a two-dimensional ultrasound image of the region of interest of the target based on the ultrasound scan;

S2. Obtain distance information between at least one marker and the camera of the three-dimensional optical imaging sensor within the visible range of the three-dimensional optical imaging sensor and image information of the marker;

S3: Acquire three-dimensional space information of the ultrasound probe based on the distance information and the image information;

S4. Reconstruct a three-dimensional ultrasound image based on the three-dimensional space information and the two-dimensional ultrasound image.

In the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor of the present invention, the marker is at least a part of the ultrasonic probe.

In the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor of the present invention, the marker includes at least one visual marker provided on the ultrasonic probe.

In the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor of the present invention, the three-dimensional optical imaging sensor is arranged on the ultrasonic probe, and the identifier includes a three-dimensional optical imaging sensor arranged in the visible range of the three-dimensional optical imaging sensor. At least one visual identifier.

In the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor of the present invention, the step S3 further includes:

S31: Acquire two-dimensional position information of the marker based on the image information of the marker;

S32: Recognizing the respective distances between at least three pixels in the image information of the marker and the camera based on the distance information image and the marker image information to obtain position and direction information of the marker; And obtain the three-dimensional space information of the ultrasound probe based on the position and direction information of the marker.

In the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor of the present invention, in the step S2, a plurality of markers are set within the visible range of the three-dimensional optical imaging sensor, and each of the markers is acquired Each group of distance information images and each group of marker image information between the camera and the camera; in the step S31, each group of two-dimensional position information of each of the markers is acquired based on each group of the marker image information; In the step S32, the distance between at least three pixels in the image information of each of the markers and the camera is identified based on each group of the distance information image and each group of the marker image information to obtain each The position and direction information of the markers, and the three-dimensional space information of the ultrasound probe is obtained based on the position and direction information of each marker.

In the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor of the present invention, the marker image information is RGB and/or infrared image information, and in step S31, based on the RGB and/or infrared image information The color, shape, pattern or lightness of the marker is used to obtain the two-dimensional position information of the marker.

In the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor of the present invention, it further includes: before the ultrasonic scanning, detecting the target three-dimensional contour information of the region of interest of the target;

The step S3 further includes:

S33: Acquire three-dimensional motion information of the target during the scanning process of the ultrasound probe based on the target three-dimensional contour information, and correct the three-dimensional space information of the ultrasound probe based on the three-dimensional motion information.

S34. Convert the pixel points of each frame of the two-dimensional ultrasound image into a three-dimensional space to calibrate the three-dimensional ultrasound image.

In the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor of the present invention, it further includes

S5. Display the target three-dimensional contour information, the three-dimensional ultrasound image and/or the ultrasound probe image on the same three-dimensional space.

S35. Use at least one angle sensor provided on the ultrasound probe to obtain the three-dimensional direction information of the ultrasound probe and combine the three-dimensional direction information with the three-dimensional space information of the ultrasound probe obtained by the three-dimensional optical imaging sensor Fusion to improve the measurement accuracy of the three-dimensional spatial information.

Beneficial effect

Implementing the three-dimensional ultrasonic imaging system and method based on the three-dimensional optical imaging sensor of the present invention, by using the three-dimensional optical imaging sensor to acquire three-dimensional spatial information, three-dimensional ultrasound images can be reconstructed in a flexible, low-cost and small-volume manner, and interference can be effectively avoided. Further, by adopting a marker or an angle sensor, the accuracy of the three-dimensional spatial information can be improved, thereby further improving the quality of the three-dimensional ultrasound image. Furthermore, by setting multiple identifiers, not only relevant position information can be obtained, but also movement direction information can be obtained, so that the detection of the three-dimensional spatial information is more effective and reliable, and the quality of the three-dimensional ultrasound image is further improved. Still further, the obtained three-dimensional spatial information and/or three-dimensional ultrasound image can also be corrected and/or calibrated, so as to obtain a more accurate and reliable three-dimensional ultrasound image.

Description of the drawings

The present invention will be further described below in conjunction with the accompanying drawings and embodiments. In the accompanying drawings:

1 is a system schematic diagram of the first preferred embodiment of a three-dimensional ultrasonic imaging system based on a three-dimensional optical imaging sensor of the present invention;

2 is a system schematic diagram of a second preferred embodiment of a three-dimensional ultrasonic imaging system based on a three-dimensional optical imaging sensor of the present invention;

3A-3C are schematic diagrams of different setting positions of the markers of the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor shown in FIG. 2;

4 is a system schematic diagram of a third preferred embodiment of a three-dimensional ultrasonic imaging system based on a three-dimensional optical imaging sensor of the present invention;

5 is a system schematic diagram of a fourth preferred embodiment of a three-dimensional ultrasonic imaging system based on a three-dimensional optical imaging sensor of the present invention;

6A-6C are schematic diagrams of typical ArUco identification codes used in the embodiment shown in FIG. 5;

7 is a system schematic diagram of a fifth preferred embodiment of a three-dimensional ultrasonic imaging system based on a three-dimensional optical imaging sensor of the present invention;

8 is a system schematic diagram of a sixth preferred embodiment of a three-dimensional ultrasonic imaging system based on a three-dimensional optical imaging sensor of the present invention;

9 is a flowchart of the first preferred embodiment of the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor of the present invention;

Fig. 10 is a flowchart of a second preferred embodiment of a three-dimensional ultrasonic imaging method based on a three-dimensional optical imaging sensor of the present invention.

Embodiments of the invention

In order to make the objectives, technical solutions, and advantages of the present invention clearer, the following further describes the present invention in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, but not to limit the present invention.

The traditional camera can only generate two-dimensional images, and because the three-dimensional optical imaging sensor uses multiple sets of optical sensing systems in the camera or uses various distance measurement methods, it can not only record two-dimensional images of objects, but also obtain two-dimensional images. The distance information of different positions from the camera can be an absolute distance or a considerable distance. For example, Microsoft's Kinect sensor and Intel's Realsense sensor belong to the aforementioned three-dimensional optical imaging sensors. Various methods can be used to obtain this distance information. For example, two independent optical cameras with several distances can be used to generate a stereo image with depth of field. Or add an infrared camera to detect the infrared pattern projected by the optical imaging sensor on the surface of the object to obtain depth information. The latter is widely used in three-dimensional optical imaging sensors in recent years, such as Intel’s Realsense three-dimensional optical imaging sensor 03, which comes with an integrated infrared transmitter and an imaging sensor that integrates a pair of RGB cameras. The three-dimensional optical imaging sensor can simultaneously provide color RGB images, infrared images reflecting infrared intensity, and distance maps from the surface of the object to the camera. All the above images can be provided in real time (frame rate higher than 25 frames per second).

Therefore, an inventive concept of the present invention is to use the three-dimensional spatial information of the ultrasound probe provided by the three-dimensional optical imaging sensor and the two-dimensional ultrasound image information provided by the two-dimensional ultrasound imaging device for three-dimensional image reconstruction. A further inventive concept of the present invention is to increase the accuracy of the three-dimensional spatial information provided by the three-dimensional optical imaging sensor by adding markers. A further inventive concept of the present invention is to use the ultrasound probe itself as a marker; or set a visual marker on the ultrasound probe; or set a three-dimensional optical sensor on the ultrasound probe and set a visual mark within the visible range of the three-dimensional optical sensor Things.

Fig. 1 is a system schematic diagram of a first preferred embodiment of a three-dimensional ultrasonic imaging system based on a three-dimensional optical imaging sensor of the present invention. As shown in Figure 1, the 3D ultrasonic imaging system based on the 3D optical imaging sensor of the present invention includes an ultrasonic probe 01, a 2D ultrasonic imaging device 05, a 3D optical imaging sensor 03, a spatial information processing module 06, a 3D reconstruction module 07, and a display device. 08. The ultrasonic probe 01 is set within the visible range 04 of the three-dimensional optical imaging sensor 03 and is used for ultrasonic scanning of the region of interest of the target 02. The two-dimensional ultrasound imaging device 05 is communicatively connected with the ultrasound probe 01 so as to generate a two-dimensional ultrasound image of the region of interest of the target 02 based on the ultrasound scan. Those skilled in the art know that the region of interest can be at least a part of the target 02 or the whole, and any ultrasound probe or two-dimensional ultrasound imaging device known in the art can be used to construct the ultrasound probe 01, Two-dimensional ultrasound imaging device 05.

The three-dimensional optical imaging sensor 03 may be communicatively connected with the ultrasound probe 01 and the two-dimensional ultrasound imaging device 05, so as to obtain distance information between the ultrasound probe 01 and the camera of the three-dimensional optical imaging sensor 03 And the image information of the ultrasound probe 01. Those skilled in the art know that any three-dimensional optical imaging sensor in the field can be used, especially the Kinect sensor of Microsoft and the Realsense sensor of Intel, and similar devices developed in the future. The image information can be RGB and/or infrared image information, and can also be any other image information that can be used to obtain three-dimensional surface information and/or motion information of an object. Preferably, Intel’s Realsense three-dimensional optical imaging sensor 03 can be used, which comes with an integrated infrared transmitter and an imaging sensor integrated with a pair of RGB cameras. The three-dimensional optical imaging sensor can simultaneously provide color RGB images, infrared images reflecting infrared intensity, and distance maps from the surface of the object to the camera. All the above images can be provided in real time (frame rate higher than that of ultrasound imaging, such as 25 frames per second). In the working process, the ultrasonic probe 01 will be under real-time monitoring of the three-dimensional optical imaging sensor 03, and the three-dimensional surface information (ie contour) of the ultrasonic probe 01 will be detected in real time. According to the movement of the obtained three-dimensional surface information, if the ultrasound probe 01 moves, its movement in the three-dimensional space can also be known.

The spatial information processing module 06 is in communication connection with the three-dimensional optical imaging sensor 03, so as to obtain the three-dimensional spatial information of the ultrasound probe 01 based on the distance information and the image information. Preferably, the respective distances between at least three pixels in the image information of the ultrasound probe and the camera in the three-dimensional optical imaging sensor can be obtained from the distance information provided by the three-dimensional optical imaging sensor to each part of the ultrasound probe. The distance to the camera can be obtained by calculating the average distance between at least three pixels in the image information and the camera. If the target or the region of interest of the target moves simultaneously with the ultrasound probe, the three-dimensional space information of the target or the region of interest of the target and the movement in the three-dimensional space can be known accordingly. In addition, according to the obtained distance between at least three pixels of the ultrasound probe and the three-dimensional optical imaging sensor, the orientation (inclination, that is, the spatial three-dimensional direction) of the ultrasound probe can be calculated, and it can be compared with the direction of the ultrasound probe. contact. When the three-dimensional space position of at least three pixels on a surface is known, there is a standardized method for calculating the surface orientation of an object, which will not be repeated here. Those skilled in the art know that the communication connection may be a wireless communication connection or a wired communication connection. The method for obtaining the three-dimensional space information of the ultrasound probe through the distance information and the image information can also adopt any standardized method known in the art, so it will not be repeated here. Of course, after acquiring the three-dimensional spatial information, those skilled in the art can further reduce noise by using various signal processing methods, such as using a moving average method for spatial position information and spatial angle information. In addition, when acquiring the spatial three-dimensional direction of the ultrasound probe through the distance information, the distance information of at least three pixels and at most all pixels of the ultrasound probe image can be used.

The three-dimensional reconstruction module 07 is in communication connection with the spatial information processing module 06 and the two-dimensional ultrasound imaging device 05, so as to reconstruct a three-dimensional ultrasound image based on the three-dimensional spatial information and the two-dimensional ultrasound image. Those skilled in the art know that the communication connection may be a wireless communication connection or a wired communication connection. Moreover, any reconstruction method known in the art can be used to achieve the reconstruction of the three-dimensional ultrasound image, which will not be repeated here.

The display device 08 and the three-dimensional reconstruction module 07 thus display the three-dimensional ultrasound image. The display process can be real-time or non-real-time. Further, the three-dimensional optical imaging sensor 03 can also simultaneously transmit the three-dimensional surface profile information of the target 02 to the display device 08, so that the display device 08 can be in the same three-dimensional space, preferably in different colors The three-dimensional surface contour information of the target 02 and the three-dimensional ultrasound image are displayed. In a further preferred embodiment of the present invention, the image of the ultrasound probe 01 can also be sent to the display device 08, and the display device 08 can be in the same three-dimensional space, preferably displaying the three-dimensional image of the target 02 in different colors. The surface profile information, the three-dimensional ultrasound image, and the image of the ultrasound probe 01 can of course also be displayed in any one of the three, for example, displayed in a switching manner. In other simplified embodiments of the present invention, the display device 08 may be omitted.

The three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor of the present invention can obtain three-dimensional spatial information by adopting the three-dimensional optical imaging sensor, which can reconstruct the three-dimensional ultrasound image in a flexible, low-cost and small-volume manner, and can effectively avoid interference.

Fig. 2 is a system schematic diagram of a second preferred embodiment of a three-dimensional ultrasonic imaging system based on a three-dimensional optical imaging sensor of the present invention. As shown in Figure 2, the 3D ultrasonic imaging system based on the 3D optical imaging sensor of the present invention includes an ultrasonic probe 01, a 2D ultrasonic imaging device 05, a 3D optical imaging sensor 03, a spatial information processing module 06, a 3D reconstruction module 07, and a display device. 13. In the preferred embodiment shown in FIG. 2, the ultrasonic probe 01 is set within the visible range 04 of the three-dimensional optical imaging sensor 03 and is used for ultrasonic scanning of the region of interest of the target 02. A visual marker 11 is provided at the front end of the ultrasonic probe 01, and the visual marker 11 is also located within the visible range 04. In other preferred embodiments of the present invention, as shown in FIGS. 3A-3C, the visual marker 11 may be located at the front end, the side, and the middle of the front end of the ultrasonic probe 01. In a further preferred embodiment of the present invention, the visual marker 11 may also be arranged at other positions of the ultrasonic probe 01, such as the bottom, the rear end, and so on. Alternatively, the visual identifier 11 may also be at least a part or all of the ultrasonic probe 01. Further, the visual identifier 11 can be any shape, such as a circle, a triangle, and any regular or irregular shape, and it can also be a code attached to any part of the ultrasound probe 01, such as various types. ArUco identification code and so on. The above-mentioned visual marker 11 can be in any color (so that the visual marker 11 can be distinguished from the background by color), or a highly reflective infrared coating (so that the visual marker 11 can be clearly distinguished from the background in infrared imaging) ); Or it has its own luminous characteristics, such as using different colors of LED lights or infrared lights (this can be set to emit light when needed, because its own light can make it clearer in RGB or infrared images). The LED lamp or infrared lamp can be further synchronized with the three-dimensional optical imaging sensor, so that the LED lamp or infrared lamp emits light only when the measurement of the three-dimensional optical imaging sensor is in progress. In this way, the accuracy of the measurement of the visual marker 11 can be further increased. Furthermore, the visual identifier 11 may be detachable or retractable into the ultrasound probe when not in use, or may be fixedly arranged. In other preferred embodiments of the present invention, instead of the visual marker 11, the three-dimensional optical imaging sensor 03 may be arranged on the ultrasonic probe 01. For example, the three-dimensional optical imaging sensor 03 can be arranged at any position on the ultrasonic probe 01, as long as the visual marker 11 is still within the visible range 04. The advantage of this method is that when we use large visual markers 11 and/or place them in any or multiple positions in the relevant space, no matter how the angle and position are adjusted, some or part of the visual can be detected. Marker 11. In other preferred embodiments of the present invention, multiple visual markers may also be set, for example, visual markers of different shapes, types or colors are set at different positions of the ultrasonic probe 01 to improve the detection and positioning effect.

In this embodiment, the two-dimensional ultrasound imaging device 05 is in communication connection with the ultrasound probe 01, so as to generate a two-dimensional ultrasound image of the region of interest of the target 02 based on the ultrasound scan. The three-dimensional optical imaging sensor 03 can be communicatively connected with the ultrasonic probe 01 and the two-dimensional ultrasonic imaging device 05, and is used to obtain the distance information image between the visual marker and the camera and the visual marker image information . The visual marker image information may preferably be RGB and/or infrared image information.

In this embodiment, the spatial information processing module 06 may preferably include a visual marker two-dimensional position recognition unit 61 and a three-dimensional spatial information acquisition unit 62. The visual marker two-dimensional position recognition unit 61 is configured to obtain the two-dimensional position information of the visual marker based on the visual marker image information. Preferably, the visual marker two-dimensional position recognition unit 61 is configured to obtain the visual marker based on the color, shape, pattern, or lightness of the visual marker in the RGB and/or infrared image information The two-dimensional position information of the object. The three-dimensional information acquisition unit 62 is configured to identify the distance between at least three pixels in the image information of the visual identifier and the camera based on the distance information image and the visual identifier image information to obtain the distance information. The position and direction information of the visual marker, and the three-dimensional space information of the ultrasonic probe 01 is obtained based on the position and direction information of the visual marker.

As shown in FIG. 2, when the three-dimensional optical imaging sensor 03 acquires the visual marker image information of the circular visual marker 11 disposed at the end of the ultrasonic probe 01, the visual marker two-dimensional position recognition unit 61 can acquire the visual marker The two-dimensional position information of at least three (or more, or even all) pixels in the object image information. The three-dimensional space information acquisition unit 62 may be based on the two-dimensional position information of at least three pixels in the visual marker image information and the distance information image between at least three pixels in the visual marker image information and the camera. Obtain the three-dimensional space information of at least three pixels (ie, the left, right, up, down, and the distance from the camera), so as to push out the position and direction information of the visual marker, and then obtain the three-dimensional space information of the ultrasound probe 01.

In a further preferred embodiment of the present invention, a triangular visual marker may be used, preferably a triangular visual marker marked by a specific color or a highly reflective infrared coating. When this specially designed visual marker is recognized, the position of the visual marker can be derived based on the three-dimensional information (left, right, up, down, and distance from the camera) contained in at least three pixels in the corresponding image And direction information, so that the three-dimensional spatial information of the ultrasound probe can be obtained. For example, we can use the center of the visual marker to record the position of the visual marker, and use the direction of the plane on which the visual marker is located in the three-dimensional space to detect the direction of the visual marker. Once the position and direction of the visual marker can be detected, we can obtain the position and direction information of the image.

In a further preferred embodiment of the present invention, at least one angle sensor for acquiring three-dimensional direction information of the ultrasonic probe 01 is further provided on the ultrasonic probe 01. For example, in order to further improve the accuracy of the three-dimensional spatial direction detection of the ultrasonic probe, we can additionally install one or more angle sensors capable of detecting angles on the inner or surface of the ultrasonic probe 10, such as an accelerometer, a gyroscope or a magnetic sensor. These additional detected direction information can be compared and combined with the results of the three-dimensional optical imaging sensor to reduce the interference of environmental factors. For example, if one of the sensors is severely disturbed, the other sensors can correct the three-dimensional information based on their own detection results. Because these sensors use completely different sensing technologies, under normal circumstances, an occasional independent disturbance will not affect all sensors. For example, the three-dimensional spatial information obtained by the three-dimensional optical imaging sensor can be disturbed by sudden strong light, while the operation of the accelerometer, gyroscope and magnetic sensor will not be affected. In this way, the at least one angle sensor provided on the ultrasonic probe can be used to obtain the three-dimensional direction information of the ultrasonic probe and the three-dimensional direction information can be compared with the three-dimensional space of the ultrasonic probe obtained by the three-dimensional optical imaging sensor. Information fusion to improve the measurement accuracy of the three-dimensional spatial information

In a further preferred embodiment of the present invention, multiple three-dimensional optical imaging sensors 03 can also be used to monitor the condition of the visual marker 11. Each sensor separately records the position and direction information of the ultrasonic probe, and combines multiple sensors. The result can help improve the stability and reliability of the system. In addition, when the ultrasonic probe is free to switch to different directions, at least one three-dimensional optical imaging sensor 03 can obtain the position and direction information of the visual marker in time.

Of course, after acquiring the three-dimensional spatial information, those skilled in the art can further reduce noise by various signal processing methods, such as moving average method and so on. The three-dimensional reconstruction module 07 is in communication connection with the spatial information processing module 06 and the two-dimensional ultrasound imaging device 05, so as to reconstruct a three-dimensional ultrasound image based on the three-dimensional spatial information and the two-dimensional ultrasound image. Those skilled in the art know that the communication connection may be a wireless communication connection or a wired communication connection. Moreover, any reconstruction method known in the art can be used to achieve the reconstruction of the three-dimensional ultrasound image, which will not be repeated here.

The display device 13 and the three-dimensional reconstruction module 07 thus display the three-dimensional ultrasound image. The display process can be real-time or non-real-time. The display device 13 may further display the coordinates 12 of the visual identifier 11.

In this embodiment, by combining the above-mentioned visual marker and the three-dimensional optical imaging sensor, the two-dimensional position information in the image and the distance between at least three pixels and the camera provided by the three-dimensional optical imaging sensor can be acquired. Therefore, for the region of interest of the target, we can know its position in the three-dimensional space through the selected surface features (namely markers) or additional markers of the target. The three-dimensional space information of the ultrasound probe can be calculated through the three-dimensional space information of the marker. First, the detected distance between at least three pixels in the visual marker area and the camera in the three-dimensional optical imaging sensor can be obtained from the distance information from each part of the marker provided by the three-dimensional optical imaging sensor. The distance between the entire marker and the camera can be obtained by calculating at least three visual markers, or the average distance between all pixels and the camera. If the object and the visual marker move at the same time, the movement of the object in the three-dimensional space can be known. In addition, the direction (inclination) of the surface of the marker can be calculated based on the obtained distance between all the pixels of the marker area and the sensor, and it can be related to the direction of the object, that is, the ultrasonic probe. In the case that the three-dimensional space positions of all points on a surface are known, there is a standardized method for calculating the surface orientation of an object, so it will not be repeated here. In order to make the detection of the distance between the visual marker and the three-dimensional optical imaging sensor and the direction of the visual marker more reliable, a variety of enhancement algorithms can be used: including the use of median filters to reduce noise, or wavelet analysis to process the marker The distance data of all pixels on the top. Furthermore, after obtaining the three-dimensional spatial information of the marker (including the three-dimensional position and direction), we can use a variety of signal processing methods to reduce noise, such as the moving average method.

The three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor of the present invention can obtain three-dimensional spatial information by adopting the three-dimensional optical imaging sensor, which can reconstruct the three-dimensional ultrasound image in a flexible, low-cost and small-volume manner, and can effectively avoid interference. Further, by adopting a visual identifier or an angle sensor, the accuracy of the three-dimensional spatial information can be improved, thereby further improving the quality of the three-dimensional ultrasound image.

4 is a system schematic diagram of a third preferred embodiment of a three-dimensional ultrasonic imaging system based on a three-dimensional optical imaging sensor of the present invention. In the embodiment shown in FIG. 4, it is different from the embodiment shown in FIG. 3 in that it is provided with a plurality of visual identifiers 11. Although only three circular visual markers arranged on one side of the ultrasonic probe 11 are shown in FIG. 4, those skilled in the art will know that different numbers, different shapes or types of visual markers can be set at different positions of the ultrasonic probe. Marker. These visual markers 11 can be coded in different colors (that is, the object can be separated from the background by color) or use highly reflective infrared coating markers (bright markers that are clearly distinguished from the background in infrared). The spatial direction of the ultrasonic probe 11 can be detected by applying multiple visual markers. For example, we can use the center of the visual marker to record the position of the marker, and use the direction of the plane on which the marker is located in the three-dimensional space to detect the direction of the marker (the triangular marker shown in Figure 7) . When multiple markers are used for positioning, we can obtain multiple sets of three-dimensional spatial information (position and direction) at the same time. This information helps to increase the reliability of three-dimensional positioning. For example, the average value of multiple sets of information is used to calculate the three-dimensional spatial position and direction.

Therefore, in this embodiment, the three-dimensional optical imaging sensor 03 is used to obtain each set of distance information images between each of the markers and the camera and each set of marker image information; the markers are two-dimensional The position recognition unit is used to obtain each set of two-dimensional position information of each of the markers based on each set of the marker image information; the three-dimensional space information acquisition unit is used to obtain each set of the distance information image and each set of the The marker image information identifies the distance between at least three pixels in the image information of each marker and the camera to obtain the position and direction information of each marker, and is based on the position and direction of each marker. Direction information to obtain the three-dimensional spatial information of the ultrasound probe 01.

Similarly, in this embodiment, the positions of the three-dimensional optical imaging sensor and the visual marker can be interchanged. Specifically, the three-dimensional optical imaging sensor can be installed at different positions of the ultrasound probe, and the marker can be placed within the visible range of the three-dimensional optical imaging sensor. The advantage of this method is that we use large visual markers and place them in multiple locations in space. No matter how the position and angle are adjusted, the three-dimensional optical imaging sensor can always detect some markers. Similarly, in order to further enhance the accuracy of detecting the distance of the marker, at least one preferably all of the visual markers can be covered with an infrared reflective material to improve the effect of the infrared pattern projected on the surface of the marker. At least part or all of the visual markers can also be made to have easy-to-detach features, so that they are only needed for three-dimensional tracking (for example, using magnetic materials). Similarly, in the method in which the three-dimensional optical imaging sensor is fixed on the ultrasonic probe, the sensor may also be detachable. At least one or all of the visual identifiers may also have self-luminous characteristics, such as using LED lights or infrared lights of different colors. The LED lamp or infrared lamp can be further synchronized with the three-dimensional optical imaging sensor, so that the LED lamp or infrared lamp emits light only when the measurement of the three-dimensional optical imaging sensor is in progress. In addition, the luminescence can have different time series characteristics and spatial transformation modes to facilitate positioning and detection. The advantage of using self-luminous markers is that the self-luminous markers can be set to emit light when they need to be used, or used in dim conditions, and their own light makes it clearer in the RGB or infrared image.

The three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor of the present invention can obtain three-dimensional spatial information by adopting the three-dimensional optical imaging sensor, which can reconstruct the three-dimensional ultrasound image in a flexible, low-cost and small-volume manner, and can effectively avoid interference. Further, by adopting a marker or an angle sensor, the accuracy of the three-dimensional spatial information can be improved, thereby further improving the quality of the three-dimensional ultrasound image. Furthermore, by setting multiple markers, not only relevant position and direction information can be obtained, but also movement direction information can be obtained, thereby making the detection of the three-dimensional spatial information more effective and reliable, and further improving the performance of three-dimensional ultrasound images. quality.

Fig. 5 is a system schematic diagram of a fourth preferred embodiment of a three-dimensional ultrasonic imaging system based on a three-dimensional optical imaging sensor of the present invention. The embodiment shown in FIG. 5 is similar to the embodiment shown in FIG. 3, with the difference that in the embodiment shown in FIG. 5, the ArUco identification code is used as the visual identifier. In this embodiment, in order to make the detection of visual markers more effective and reliable, use ArUco identification code. A typical ArUco identification code is shown in Figure 6A-6C. ArUco identification codes have a wide range of applications for tracking objects in virtual reality. Generally speaking, an RGB camera is used to take an image containing an ArUco identification code, and the depth and direction of the object containing the identification code are determined according to the size and deformation of the ArUco identification code. Of course, in other preferred embodiments of the present invention, other ArUco identification codes or even other codes can also be used. The use of ArUco identification codes or other codes will make the detection and tracking of markers easier, because many algorithms can quickly identify the position and direction of the ArUco identification code in the image. The position of the obtained identification code marker can be further obtained by obtaining the distance between at least three pixels and the camera to further obtain the three-dimensional direction information of the identification code marker. The direction information (relatively inaccurate) obtained by detecting the ArUco identification code can also be incorporated into the calculation of the three-dimensional direction information of the identification code marker, for example, to help eliminate noise that may exist in the distance measurement.

Fig. 7 is a system schematic diagram of a fifth preferred embodiment of a three-dimensional ultrasonic imaging system based on a three-dimensional optical imaging sensor of the present invention. In the embodiment shown in FIG. 7, three different types of visual markers are used, namely, a circular visual marker 111, a triangular visual marker 121, and an ArUco identification code 131. And these markers are not set on the ultrasonic probe, on the contrary, a three-dimensional optical imaging sensor 103 is set at the end of the ultrasonic probe 01, and the circular visual marker 111, the triangular visual marker 121 and the ArUco identification code 131 are placed on the Different positions in the visual range of the 3D optical sensor, so that no matter how the position and angle are adjusted, the 3D optical imaging sensor can always detect some markers. In a specific implementation, the shape, size, and number of visual markers can be combined in different ways according to specific applications. The distance and image information of the visual marker obtained by the three-dimensional optical sensor can calculate the position and direction of the three-dimensional optical sensor itself in the three-dimensional space, so as to obtain the three-dimensional space information of the ultrasound probe connected to it. Those skilled in the art know that the rest of the three-dimensional ultrasound imaging system based on the three-dimensional optical imaging sensor in this embodiment can be constructed with reference to any of the embodiments shown in FIGS. 1-6, and will not be repeated here.

Fig. 8 is a system schematic diagram of a sixth preferred embodiment of a three-dimensional ultrasonic imaging system based on a three-dimensional optical imaging sensor of the present invention. In the embodiment shown in FIG. 8, the difference from the embodiment shown in FIG. 2 is that a triangular visual marker 21 covered with an infrared reflective material is used, and the three-dimensional optical imaging sensor 03 is further used to detect the Target 02 three-dimensional contour information of the region of interest of target 02. Using the triangular visual marker 21, we can use the center of the marker to record the position of the marker, and use the direction of the plane on which the marker is located in the three-dimensional space to detect the direction of the marker. Once the location and direction of the marker can be detected, we can obtain the location and direction of the image. The spatial information processing module 06 further includes: a correction unit 63 and a calibration unit 64. The correction unit 63 is used to obtain the three-dimensional motion information of the target 02 during the scanning process of the ultrasound probe 01 based on the three-dimensional contour information of the target 02, and to correct the three-dimensional motion information of the ultrasound probe 01 based on the three-dimensional motion information. Spatial information. The calibration unit 64 is configured to convert the pixel points of each frame of the two-dimensional ultrasound image into a three-dimensional space to calibrate the three-dimensional ultrasound image.

In this embodiment, the correction is made by using the three-dimensional spatial information of the probe detected by the correction unit 63. The movement of the target object during the scanning process will cause errors in the three-dimensional ultrasound imaging. So this step can reduce or eliminate the interference caused by the movement of the target. The calibration unit 64 can further calibrate the obtained three-dimensional ultrasound image. The related correction and calibration methods can be any correction and calibration methods known in the art, which will not be repeated here.

The three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor of the present invention can obtain three-dimensional spatial information by adopting the three-dimensional optical imaging sensor, which can reconstruct the three-dimensional ultrasound image in a flexible, low-cost and small-volume manner, and can effectively avoid interference. Further, by adopting a marker or an angle sensor, the accuracy of the three-dimensional spatial information can be improved, thereby further improving the quality of the three-dimensional ultrasound image. Furthermore, by setting multiple markers, not only relevant position and direction information can be obtained, but also movement direction information can be obtained, thereby making the detection of the three-dimensional spatial information more effective and reliable, and further improving the performance of three-dimensional ultrasound images. quality. Still further, the obtained three-dimensional spatial information and/or three-dimensional ultrasound image can also be corrected and/or calibrated, so as to obtain a more accurate and reliable three-dimensional ultrasound image.

FIG. 9 is a flowchart of the first preferred embodiment of the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor of the present invention. As shown in FIG. 9, in step S1, an ultrasound probe is used to perform ultrasound scanning of the region of interest of the target and generate a two-dimensional ultrasound image of the region of interest of the target based on the ultrasound scan. Those skilled in the art know that the ultrasonic probe may adopt the ultrasonic probe structure in any one of the embodiments in FIGS. 1-8. In step S2, the distance information between at least one marker and the camera of the three-dimensional optical imaging sensor within the visible range of the three-dimensional optical imaging sensor and the image information of the marker are acquired. Preferably, the three-dimensional optical imaging sensor can be communicatively connected with the ultrasound probe and the two-dimensional ultrasound imaging device, so as to obtain the distance information between the ultrasound probe and the camera of the three-dimensional optical imaging sensor and the ultrasound Image information of the probe. Those skilled in the art know that the three-dimensional optical imaging sensor 03 and the two-dimensional ultrasonic imaging device 05 in any one of the embodiments in FIGS. 1-8 can be used to implement this step. In a preferred embodiment of the present invention, the marker is at least a part of the ultrasonic probe. In a further preferred embodiment of the present invention, the marker includes at least one visual marker provided on the ultrasonic probe. In a still further preferred embodiment of the present invention, the three-dimensional optical imaging sensor is arranged on the ultrasonic probe, and the identifier includes at least one visual identifier arranged within the visible range of the three-dimensional optical imaging sensor.

In step S3, the three-dimensional space information of the ultrasound probe is acquired based on the distance information and the image information. Preferably, the distance between at least three pixels in the image information of the ultrasound probe and the camera in the three-dimensional optical imaging sensor can be obtained through the distance information provided by the three-dimensional optical imaging sensor to each part of the ultrasound probe. The distance to the camera can be obtained by calculating the average distance between at least three pixels in the image information and the camera. If the target or the region of interest of the target moves simultaneously with the ultrasound probe, the movement of the target or the region of interest of the target in the three-dimensional space can be known accordingly. In addition, the orientation (tilt) of the ultrasound probe can be calculated based on the obtained distance between all the pixels of the ultrasound probe and the three-dimensional optical imaging sensor, and can be related to the direction of the ultrasound probe. In the case that the three-dimensional space position of all points on a surface is known, there is a standardized method for calculating the surface orientation of an object, which will not be repeated here. In step S4, a three-dimensional ultrasound image is reconstructed based on the three-dimensional space information and the two-dimensional ultrasound image. Those skilled in the art know that any reconstruction method known in the art can be used to achieve the reconstruction of the three-dimensional ultrasound image, which will not be repeated here.

Those skilled in the art know that, in other further preferred embodiments of the present invention, the three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor constructed in any of the embodiments in FIGS. 1-8 can be used to realize the above-mentioned three-dimensional optical imaging sensor-based Three-dimensional ultrasound imaging method.

Implementing the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor of the present invention, by using the three-dimensional optical imaging sensor to obtain three-dimensional spatial information, three-dimensional ultrasound images can be reconstructed in a flexible, low-cost and small-volume manner, and interference can be effectively avoided.

Fig. 10 is a flowchart of a second preferred embodiment of a three-dimensional ultrasonic imaging method based on a three-dimensional optical imaging sensor of the present invention. In step S1, at least one marker is set within the visible range of the three-dimensional optical imaging sensor. In this preferred embodiment, the marker is provided on the ultrasonic probe. In other preferred embodiments of the present invention, the three-dimensional optical imaging sensor is arranged on the ultrasonic probe. For the type, quantity, type, and location of the above-mentioned markers, refer to the three-dimensional ultrasonic imaging structure based on the three-dimensional optical imaging sensor shown in FIGS. 1-8. In other embodiments of the present invention, the three-dimensional optical imaging sensor may also be arranged on the ultrasonic probe.

In step S2, an ultrasound probe is used to perform ultrasound scanning on the region of interest of the target and generate a two-dimensional ultrasound image of the region of interest of the target based on the ultrasound scan. Those skilled in the art know that the ultrasonic probe may adopt the ultrasonic probe structure in any one of the embodiments in FIGS. 1-8.

In step 3, the distance information image between the marker and the camera and the marker image information are acquired. As mentioned earlier, any three-dimensional optical imaging sensor in the field can be used, especially the Kinect sensor of Microsoft and the Realsense sensor of Intel. The image information can be RGB and/or infrared image information, and can also be any other image information that can be used to obtain three-dimensional surface information and/or motion information of an object.

In step S4, the two-dimensional position information of the marker is obtained based on the image information of the marker. In a further preferred embodiment of the present invention, the two-dimensional position information of the marker can be acquired based on the color, shape, pattern or light and darkness of the marker in the RGB and/or infrared image information. In step S5, the distance between at least three pixels in the image information of the marker and the camera is recognized based on the distance information image and the marker image information to obtain the position and direction of the marker Information, and obtain the three-dimensional space information of the ultrasound probe based on the position and direction information of the marker. Preferably, the distance between at least three pixels in the image information of the marker and the camera in the three-dimensional optical imaging sensor can be obtained through the distance information from each part of the marker provided by the three-dimensional optical imaging sensor. The distance of the camera can be obtained by calculating the average distance between at least three pixels in the image information and the camera. If the target or the region of interest of the target and the marker move at the same time, the movement of the target or the region of interest of the target in the three-dimensional space can be known accordingly. In addition, based on the obtained distance between all the pixels of the marker and the three-dimensional optical imaging sensor, the direction (inclination) of the marker can be calculated, and it can be related to the direction of the marker. In the case that the three-dimensional space position of all points on a surface is known, there is a standardized method for calculating the surface orientation of an object, which will not be repeated here. When this kind of specially designed marker is recognized, the position and direction of this marker can be derived based on the three-dimensional information (left, right, up, down, and distance from the camera) contained in at least three pixels in the corresponding image Information, so that the three-dimensional spatial information of the ultrasound probe can be obtained. Those skilled in the art know that the above steps S4-S5 can be implemented by the spatial information processing module 06 of any one of the embodiments in FIGS. 2-8.

In step S6, a three-dimensional ultrasound image is reconstructed based on the three-dimensional space information and the two-dimensional ultrasound image. Those skilled in the art know that any reconstruction method known in the art can be used to achieve the reconstruction of the three-dimensional ultrasound image, which will not be repeated here. In step S7, a display device may be used to display the three-dimensional ultrasound image.

In a further preferred embodiment of the present invention, multiple markers may be set within the visible range of the three-dimensional optical imaging sensor. However, an ultrasound probe is used to perform ultrasound scanning of the region of interest of the target and generate a two-dimensional ultrasound image of the region of interest of the target based on the ultrasound scan. In a further embodiment, the target three-dimensional contour information of the region of interest of the target may also be detected before the ultrasound scan. In this way, in the subsequent steps, each group of distance information images between each of the markers and the camera and the image information of each group of markers are obtained; each group of the marker image information is obtained based on each group of the marker image information. Group of two-dimensional position information; based on each group of the distance information image and each group of the marker image information to identify the distance between at least three pixels in the image information of each marker and the camera to obtain each The position and direction information of the markers, and the three-dimensional space information of the ultrasound probe is obtained based on the position and direction information of each marker.

Further, in a preferred embodiment of the present invention, the three-dimensional motion information of the target during the scanning process of the ultrasound probe can be obtained based on the three-dimensional contour information of the target, and the ultrasound can be corrected based on the three-dimensional motion information. Three-dimensional spatial information of the probe. The movement of the target during scanning can cause errors in three-dimensional ultrasound imaging. So this step can reduce or eliminate the interference caused by the movement of the target.

In a further preferred embodiment of the present invention, at least one angle sensor for acquiring three-dimensional direction information of the ultrasonic probe is further provided on the ultrasonic probe, so that subsequent correction can be further performed based on the three-dimensional direction information. For example, additional sensors (including accelerometers, gyroscopes, and magnetic sensors) built into or external to the ultrasound probe can be used to obtain the three-dimensional direction information of the ultrasound probe. For example, at least one angle sensor provided on the ultrasonic probe may be used to obtain the three-dimensional direction information of the ultrasonic probe and the three-dimensional direction information may be combined with the three-dimensional space of the ultrasonic probe obtained by the three-dimensional optical imaging sensor. Information fusion to improve the measurement accuracy of the three-dimensional spatial information

In a further preferred embodiment of the present invention, the method further includes converting the pixel points of each frame of the two-dimensional ultrasound image into a three-dimensional space to calibrate the three-dimensional ultrasound image. The specific calibration method can use any known calibration method in the art.

In a further preferred embodiment of the present invention, the target three-dimensional contour information and the three-dimensional ultrasound image can be displayed on the same three-dimensional space. For example, the fusion display of the surface contour information of the target and the three-dimensional ultrasound image can be combined. That is, the three-dimensional ultrasound image and the three-dimensional surface contour information image of the target object can be displayed in different ways in the same three-dimensional space, such as using different colors. This step can also be displayed in real time during the scanning process of the ultrasound probe. In a further preferred embodiment of the present invention, the ultrasonic probe and its position/direction information can also be displayed, so that the operator can see the position, direction and movement of the ultrasonic probe. This function can be combined with the three-dimensional surface contour shape image in the previous step. That is, the three-dimensional surface contour information image of the target is displayed in the display module in a certain color or gray scale, and the ultrasound image is displayed in the same three-dimensional space in real time according to its corresponding three-dimensional space information. In this way, the operator can better scan the target.

Implementing the three-dimensional ultrasonic imaging method based on the three-dimensional optical imaging sensor of the present invention, by using the three-dimensional optical imaging sensor to obtain three-dimensional spatial information, three-dimensional ultrasound images can be reconstructed in a flexible, low-cost and small-volume manner, and interference can be effectively avoided. Further, by adopting a marker or an angle sensor, the accuracy of the three-dimensional spatial information can be improved, thereby further improving the quality of the three-dimensional ultrasound image. Furthermore, by setting multiple markers, not only relevant position and direction information can be obtained, but also movement direction information can be obtained, thereby making the detection of the three-dimensional spatial information more effective and reliable, and further improving the performance of three-dimensional ultrasound images. quality. Still further, the obtained three-dimensional spatial information and/or three-dimensional ultrasound image can also be corrected and/or calibrated, so as to obtain a more accurate and reliable three-dimensional ultrasound image.

Those skilled in the art further know that the three-dimensional ultrasonic imaging system and method based on the three-dimensional optical imaging sensor of the present invention can be mutually verified and illustrated, and the functions and steps described in each can be combined, combined or replaced with each other.

The present invention has also been described above with the help of functional modules that illustrate some important functions. For the convenience of description, the boundaries of these functional modules are specifically defined here. When these important functions are properly implemented, changing their boundaries is allowed. Similarly, flowchart modules are also specifically defined here to illustrate some important functions. For wide application, the boundaries and sequence of flowchart modules can be defined separately, as long as these important functions can still be achieved. Changes in the boundaries and sequence of the above-mentioned functional modules and flowchart functional modules shall still be regarded as within the protection scope of the claims.

The present invention can also be implemented by a computer program product. The program contains all the features that can implement the method of the present invention, and when it is installed in a computer system, the method of the present invention can be implemented. The computer program in this document refers to any expression of a set of instructions that can be written in any programming language, code, or symbol. The instruction set enables the system to have information processing capabilities to directly achieve specific functions, or to perform After one or two steps, a specific function is realized: a) conversion into other languages, codes or symbols; b) reproduction in a different format.

Although the present invention is described through specific embodiments, those skilled in the art should understand that various changes and equivalent substitutions can be made to the present invention without departing from the scope of the present invention. In addition, various modifications can be made to the present invention for specific situations or materials without departing from the scope of the present invention. Therefore, the present invention is not limited to the disclosed specific embodiments, but should include all implementations falling within the scope of the claims of the present invention.

The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modification, equivalent replacement and improvement made within the spirit and principle of the present invention shall be included in the protection of the present invention. Within range.

Claims

A three-dimensional ultrasonic imaging system based on a three-dimensional optical imaging sensor is characterized in that it comprises:

Ultrasound probe, used for ultrasonic scanning of the target area of interest;

A two-dimensional ultrasound imaging device for generating a two-dimensional ultrasound image of the region of interest of the target based on the ultrasound scan;

A three-dimensional optical imaging sensor for acquiring distance information between at least one marker and a camera of the three-dimensional optical imaging sensor within the visible range of the three-dimensional optical imaging sensor and image information of the marker;

A spatial information processing module, configured to obtain three-dimensional spatial information of the ultrasound probe based on the distance information and the image information;

The three-dimensional reconstruction module is used to reconstruct a three-dimensional ultrasound image based on the three-dimensional space information and the two-dimensional ultrasound image.
The three-dimensional ultrasound imaging system based on a three-dimensional optical imaging sensor according to claim 1, wherein the marker is at least a part of the ultrasound probe.
The three-dimensional ultrasonic imaging system based on a three-dimensional optical imaging sensor according to claim 1, wherein the marker includes at least one visual marker provided on the ultrasonic probe.
The three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor according to claim 1, wherein the three-dimensional optical imaging sensor is arranged on the ultrasonic probe, and the marker includes a camera arranged on the three-dimensional optical imaging sensor. At least one visual identifier within the visual range.
The three-dimensional ultrasonic imaging system based on a three-dimensional optical imaging sensor according to any one of claims 1-4, wherein:

The spatial information processing module includes:

A marker two-dimensional position recognition unit, configured to obtain the two-dimensional position information of the marker based on the marker image information;

A three-dimensional information acquisition unit, configured to identify the respective distances between at least three pixels in the image information of the marker and the camera based on the distance information image and the marker image information to obtain the marker The three-dimensional space information of the ultrasound probe is obtained based on the position and direction information of the marker.
The three-dimensional ultrasonic imaging system based on the three-dimensional optical imaging sensor according to claim 5, wherein a plurality of markers are set within the visible range of the three-dimensional optical imaging sensor; the three-dimensional optical imaging sensor is used to obtain each Each group of distance information images between the marker and the camera and each group of marker image information; the marker two-dimensional position recognition unit is used to obtain each of the markers based on each group of the marker image information Each set of two-dimensional position information; the three-dimensional space information acquisition unit is used to identify at least three pixels in the image information of each of the markers based on each set of the distance information image and each set of the marker image information The respective distances between the cameras obtain position and direction information of each of the markers, and the three-dimensional space information of the ultrasound probe is obtained based on the position and direction information of each of the markers.
The three-dimensional ultrasonic imaging system based on a three-dimensional optical imaging sensor according to claim 6, wherein the image information of the marker is RGB and/or infrared image information, and the two-dimensional position recognition unit of the marker is configured to The color, shape, pattern or light and darkness of the marker in the RGB and/or infrared image information is used to obtain the two-dimensional position information of the marker.
The three-dimensional ultrasonic imaging system based on a three-dimensional optical imaging sensor according to claim 5, wherein the three-dimensional optical imaging sensor is further used to detect target three-dimensional contour information of the region of interest of the target;

The spatial information processing module further includes:

The correction unit is configured to obtain the three-dimensional movement information of the target during the scanning process of the ultrasound probe based on the three-dimensional contour information of the target, and correct the three-dimensional spatial information of the ultrasound probe based on the three-dimensional movement information.
The three-dimensional ultrasonic imaging system based on a three-dimensional optical imaging sensor according to claim 5, further comprising:

The calibration unit is used to convert the pixel points of each frame of the two-dimensional ultrasound image into a three-dimensional space to calibrate the three-dimensional ultrasound image.
The three-dimensional ultrasound imaging system based on a three-dimensional optical imaging sensor according to claim 5, wherein the ultrasound probe is further provided with at least one angle sensor for acquiring three-dimensional direction information of the ultrasound probe and/or the The three-dimensional ultrasonic imaging system of the three-dimensional optical imaging sensor includes a plurality of three-dimensional optical imaging sensors.
The three-dimensional ultrasound imaging system based on a three-dimensional optical imaging sensor according to claim 8, further comprising a display device for displaying the three-dimensional contour information of the target, the three-dimensional ultrasound image and/or ultrasound in the same three-dimensional space. Probe image.
A three-dimensional ultrasonic imaging method based on a three-dimensional optical imaging sensor is characterized in that it comprises:

S1. An ultrasound probe is used to perform an ultrasound scan of a region of interest of a target and generate a two-dimensional ultrasound image of the region of interest of the target based on the ultrasound scan;

S2. Obtain distance information between at least one marker and the camera of the three-dimensional optical imaging sensor within the visible range of the three-dimensional optical imaging sensor and image information of the marker;

S3: Acquire three-dimensional space information of the ultrasound probe based on the distance information and the image information;

S4. Reconstruct a three-dimensional ultrasound image based on the three-dimensional space information and the two-dimensional ultrasound image.
The three-dimensional ultrasonic imaging method based on a three-dimensional optical imaging sensor according to claim 12, wherein the marker is at least a part of the ultrasonic probe.
The three-dimensional ultrasonic imaging method based on a three-dimensional optical imaging sensor according to claim 12, wherein the marker includes at least one visual marker provided on the ultrasonic probe.
The three-dimensional ultrasonic imaging method based on a three-dimensional optical imaging sensor according to claim 12, wherein the three-dimensional optical imaging sensor is arranged on the ultrasonic probe, and the marker includes a camera that is arranged on the three-dimensional optical imaging sensor. At least one visual identifier within the visual range.
The three-dimensional ultrasonic imaging method based on a three-dimensional optical imaging sensor according to any one of claims 12-15, wherein:

The step S3 further includes:

S31: Acquire two-dimensional position information of the marker based on the image information of the marker;

S32: Recognizing the respective distances between at least three pixels in the image information of the marker and the camera based on the distance information image and the marker image information to obtain position and direction information of the marker; And obtain the three-dimensional space information of the ultrasound probe based on the position and direction information of the marker.
The three-dimensional ultrasonic imaging method based on a three-dimensional optical imaging sensor according to claim 16, wherein in the step S2, a plurality of identifiers are set within the visible range of the three-dimensional optical imaging sensor, and each Each group of distance information images between the identifier and the camera and each group of identifier image information; in the step S31, the two groups of each of the identifiers are acquired based on each group of the identifier image information. Dimensional position information; in the step S32, the distance between at least three pixels in the image information of each of the markers and the camera is identified based on each group of the distance information image and each group of the marker image information The distance obtains the position and direction information of each of the markers, and the three-dimensional space information of the ultrasound probe is obtained based on the position and direction information of each of the markers.
The three-dimensional ultrasonic imaging method based on a three-dimensional optical imaging sensor according to claim 16, wherein the marker image information is RGB and/or infrared image information, and in step S31, based on the RGB and/or In the infrared image information, the color, shape, pattern or light and darkness of the marker is used to obtain the two-dimensional position information of the marker.
The three-dimensional ultrasonic imaging method based on a three-dimensional optical imaging sensor according to claim 16, further comprising: detecting the target three-dimensional contour information of the region of interest of the target before the ultrasonic scanning;

The step S3 further includes:

S33: Acquire three-dimensional motion information of the target during the scanning process of the ultrasound probe based on the target three-dimensional contour information, and correct the three-dimensional space information of the ultrasound probe based on the three-dimensional motion information.
The three-dimensional ultrasonic imaging method based on a three-dimensional optical imaging sensor according to claim 16, wherein the step S3 further comprises:

S34. Convert the pixel points of each frame of the two-dimensional ultrasound image into a three-dimensional space to calibrate the three-dimensional ultrasound image.
The three-dimensional ultrasonic imaging method based on a three-dimensional optical imaging sensor according to claim 19, further comprising:

S5. Display the target three-dimensional contour information, the three-dimensional ultrasound image and/or the ultrasound probe image on the same three-dimensional space.
The three-dimensional ultrasonic imaging method based on a three-dimensional optical imaging sensor according to claim 16, wherein the step S3 further comprises:

S35. Use at least one angle sensor provided on the ultrasound probe to obtain the three-dimensional direction information of the ultrasound probe and combine the three-dimensional direction information with the three-dimensional space information of the ultrasound probe obtained by the three-dimensional optical imaging sensor Fusion to improve the measurement accuracy of the three-dimensional spatial information.