CN115526919A - Image registration method and related device - Google Patents

Abstract

An image registration method and a related device provided by the embodiment of the application are provided, and the method comprises the following steps: acquiring an ultrasonic profile and a Computed Tomography (CT) profile of a target organ; the ultrasonic contour is obtained according to an ultrasonic image acquired by ultrasonic equipment, and the CT contour is obtained according to a CT image acquired by CT equipment; rotating the CT profile to obtain a rotated CT profile; an included angle between a direction vector of a reference direction in the space of the ultrasound contour and a direction vector of the reference direction in the space of the rotated CT contour is smaller than or equal to a first threshold value, the reference direction being common to the space of the ultrasound contour and the space of the CT contour; registration is performed based on the rotated CT profile and the ultrasound profile. The registration time can be shortened through the method and the device.

Description

Image registration method and related device

Technical Field

The embodiment of the application relates to the technical field of computers, in particular to an image registration method and a related device.

Background

With the continuous development of computer science technology, the image registration technology is more and more mature. Image registration techniques have wide application in the fields of computer vision, materials mechanics, and medical image processing.

For example, in a renal puncture surgery, it is necessary to register a renal ultrasound contour obtained from an ultrasound image and a renal CT contour obtained from a Computed Tomography (CT) image so that an operator such as a doctor can obtain intra-renal information during the surgery, thereby reducing surgical errors.

However, for example, the organ itself such as the kidney is a three-dimensional structure, the amount of three-dimensional image data corresponding to the three-dimensional structure is large, and the registration time for performing registration by the conventional image space feature is long.

Disclosure of Invention

The embodiment of the application provides an image registration method and a related device, and the registration time can be shortened through the application.

In a first aspect, an embodiment of the present application provides an image registration method, including:

acquiring an ultrasonic profile and a Computed Tomography (CT) profile of a target organ; the ultrasonic contour is obtained according to an ultrasonic image acquired by ultrasonic equipment, and the CT contour is obtained according to a CT image acquired by CT equipment;

rotating the CT outline to obtain a rotated CT outline; an angle between a direction vector of a reference direction in a space of the ultrasound contour and a direction vector of the reference direction in a space of the rotated CT contour is smaller than or equal to a first threshold value, the reference direction being common to the space of the ultrasound contour and the space of the CT contour;

and registering based on the rotated CT contour and the ultrasonic contour.

With reference to the first aspect, in a possible implementation manner, the rotating the CT contour to obtain a rotated CT contour includes:

determining a first direction vector of the reference direction in the space of the CT contour, and determining a second direction vector of the reference direction in the space of the ultrasound contour;

determining a rotation matrix according to an included angle between the first direction vector and the second direction vector and an included angle between the second direction vector and a coordinate axis;

and rotating the CT contour based on the rotation matrix to obtain the rotated CT contour.

With reference to the first aspect, in a possible implementation manner, the reference direction includes a first sub-reference direction, a second sub-reference direction, and a third sub-reference direction; the direction vectors of the first sub-reference direction, the second sub-reference direction, and the third sub-reference direction in the space of the ultrasound contour and the direction vector of the rotated CT contour are respectively less than or equal to the first threshold.

With reference to the first aspect, in a possible implementation manner, the first sub-reference direction is a direction pointing from a foot to a head of the implementation subject, the second sub-reference direction is a direction pointing vertically outward from a back of the implementation subject, and the third sub-reference direction is a direction pointing from a body of the implementation subject to a left hand.

With reference to the first aspect, in a possible implementation manner, the registering based on the rotated CT contour and the ultrasound contour includes:

and performing the registration step based on the rotated CT contour and the ultrasound contour when a distance between the center coordinate of the rotated CT contour and the center coordinate of the ultrasound contour is smaller than a second threshold.

With reference to the first aspect, in a possible implementation manner, the method further includes:

translating the rotated CT contour to a position where the distance between the rotated CT contour and the ultrasonic contour is less than or equal to the second threshold value to obtain a rotationally translated CT contour when the distance between the rotated CT contour and the ultrasonic contour is greater than the second threshold value;

and registering the CT contour after the rotation translation with the ultrasonic contour.

With reference to the first aspect, in one possible implementation manner, the similarity between the ultrasound contour and the CT contour is greater than or equal to a third threshold.

In a second aspect, an embodiment of the present application provides an image registration method, including:

acquiring an ultrasonic profile and a Computed Tomography (CT) profile of a target organ; the ultrasonic contour is obtained according to an ultrasonic image acquired by ultrasonic equipment, and the CT contour is obtained according to a CT image acquired by CT equipment;

rotating the ultrasonic profile to obtain a rotated ultrasonic profile; an angle between a direction vector of a reference direction in a space of the ultrasound contour and a direction vector of the reference direction in a space of the rotated CT contour is smaller than or equal to a first threshold value, the reference direction being common to the space of the ultrasound contour and the space of the CT contour;

and registering based on the rotated ultrasonic contour and the CT contour.

With reference to the second aspect, in a possible implementation manner, the rotating the ultrasonic profile to obtain a rotated ultrasonic profile includes:

determining a first direction vector of the reference direction in the space of the CT profile, and determining a second direction vector of the reference direction in the space of the ultrasound profile;

determining a rotation matrix according to an included angle between the first direction vector and the second direction vector and an included angle between the first direction vector and a coordinate axis;

and rotating the ultrasonic profile based on the rotation matrix to obtain the rotated ultrasonic profile.

With reference to the second aspect, in a possible implementation manner, the reference direction includes a first sub-reference direction, a second sub-reference direction, and a third sub-reference direction; the direction vectors of the first sub-reference direction, the second sub-reference direction, and the third sub-reference direction in the space of the ultrasound contour and the direction vector of the rotated ultrasound contour are respectively less than or equal to the first threshold.

With reference to the second aspect, in a possible implementation manner, the first sub-reference direction is a direction from a foot to a head of the implementation object, the second sub-reference direction is a direction from a back of the implementation object to the outside, and the third sub-reference direction is a direction from a body of the implementation object to a left hand.

With reference to the second aspect, in a possible implementation manner, the registering based on the rotated ultrasound contour and the CT contour includes:

and performing the registration step based on the rotated ultrasound contour and the CT contour when a distance between the center coordinates of the rotated ultrasound contour and the center coordinates of the CT contour is smaller than a second threshold.

With reference to the second aspect, in a possible implementation manner, the method further includes:

translating the rotated ultrasonic contour to a position where the distance between the rotated ultrasonic contour and the CT contour is less than or equal to the second threshold value when the distance between the rotated ultrasonic contour and the CT contour is greater than the second threshold value, thereby obtaining a rotationally translated ultrasonic contour;

and registering the ultrasonic contour after the rotation translation with the CT contour.

With reference to the second aspect, in a possible implementation manner, the similarity between the ultrasound contour and the CT contour is greater than or equal to a third threshold.

In a third aspect, an embodiment of the present application provides an image registration apparatus, including:

an acquisition unit for acquiring an ultrasound profile and a Computed Tomography (CT) profile of a target organ; the ultrasonic contour is obtained according to an ultrasonic image acquired by ultrasonic equipment, and the CT contour is obtained according to a CT image acquired by CT equipment;

a rotating unit, configured to rotate the CT profile to obtain a rotated CT profile; an angle between a direction vector of a reference direction in a space of the ultrasound contour and a direction vector of the reference direction in a space of the rotated CT contour is smaller than or equal to a first threshold value, the reference direction being common to the space of the ultrasound contour and the space of the CT contour;

and the registration unit is used for registering based on the rotated CT contour and the ultrasonic contour.

In a fourth aspect, an embodiment of the present application provides an image registration apparatus, including:

an acquisition unit for acquiring an ultrasound profile and a Computed Tomography (CT) profile of a target organ; the ultrasonic contour is obtained according to an ultrasonic image acquired by ultrasonic equipment, and the CT contour is obtained according to a CT image acquired by CT equipment;

the rotating unit is used for rotating the ultrasonic profile to obtain a rotated ultrasonic profile; an angle between a direction vector of a reference direction in a space of the ultrasound contour and a direction vector of the reference direction in a space of the rotated CT contour is smaller than or equal to a first threshold value, the reference direction being common to the space of the ultrasound contour and the space of the CT contour;

and the registration unit is used for registering based on the rotated ultrasonic contour and the CT contour.

In a fifth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, where the memory is used for storing a computer program, where the computer program includes program instructions, and the processor is configured to invoke the program instructions, so that the method in the first aspect or any one of the possible implementations of the first aspect is executed, or so that the method in the second aspect or any one of the possible implementations of the second aspect is executed.

In a sixth aspect, an embodiment of the present application provides a chip, including a logic circuit and an interface, where the logic circuit is coupled to the interface; the interface is used for inputting and/or outputting code instructions, and the logic circuit is used for executing the code instructions so as to cause the method in the first aspect or any possible implementation manner of the first aspect to be executed; or causing a method of the second aspect to be performed, or in any possible implementation of the second aspect.

In a seventh aspect, an embodiment of the present application discloses a computer program product, which includes program instructions that, when executed by a processor, cause the method in the first aspect or any possible implementation manner of the first aspect to be performed; or cause a method of the second aspect or any possible implementation of the second aspect to be performed.

In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, where a computer program is stored, and when the computer program runs on a processor, the computer program causes the method in the first aspect or any possible implementation manner of the first aspect to be performed; or cause a method of the second aspect or any possible implementation of the second aspect to be performed. Illustratively, the computer program product may be a software installation package.

Because the pose difference between the original CT outline and the ultrasonic outline is large, namely the included angle between the direction vector of the reference direction in the space of the CT outline and the direction vector of the reference direction in the space of the ultrasonic outline is larger than a first threshold value, the image registration method provided by the embodiment of the application rotates the obtained original CT outline to obtain the rotated CT outline, and the included angle between the direction vector of the reference direction in the space of the rotated CT outline and the direction vector of the reference direction in the space of the ultrasonic outline is smaller than or equal to the first threshold value, so that the pose between the rotated CT outline and the ultrasonic outline is closer, the difference between the two is reduced, the calculation efficiency is improved, the registration time is shortened, and the registration efficiency is improved.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments or the background art of the present application, the drawings used in the embodiments or the background art of the present application will be briefly described below.

Fig. 1 is a schematic view of a scene for acquiring an ultrasound image according to an embodiment of the present application;

FIG. 2 is a schematic diagram of a CT image provided by an embodiment of the present application;

FIG. 3 is a schematic view of another example of a scene for acquiring an ultrasound image according to an embodiment of the present application;

FIG. 4 is a schematic diagram of a relative position relationship between a CT-renal contour and an ultrasound-renal contour according to an embodiment of the present application;

FIG. 5 is a schematic diagram of an example of an orientation transformation of an ultrasound-renal contour provided by an embodiment of the present application;

FIG. 6 is a schematic diagram illustrating the result of the registration of a CT-renal contour with an ultrasound-renal contour according to an embodiment of the present application;

fig. 7 is a schematic flowchart of an image registration method provided in an embodiment of the present application;

fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

fig. 9 is a schematic structural diagram of another electronic device provided in an embodiment of the present application.

Detailed Description

The terminology used in the following embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in the specification of the present application and the appended claims, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the listed items. The terms "first," "second," and the like in the description and claims of the present application and in the above-described drawings are used for distinguishing between different objects and not for describing a particular order.

Currently, an operator such as a doctor can perform an operation by combining medical images during a percutaneous renal operation. Illustratively, in percutaneous nephrolithotripsy, the physician may view the needle position in real time in conjunction with an ultrasound (ultrasound) image of the kidney.

In the embodiment of the present application, the ultrasound image may be understood as an image acquired by an ultrasound apparatus. It is understood that the fluctuation that can cause the sound sensation in the auditory organ may be referred to as sound waves, and the sound waves that are not felt by the human sense organ may be referred to as ultrasonic waves. In the embodiment of the present application, the above-mentioned ultrasound apparatus may be understood as an apparatus that scans an object to be scanned with an ultrasonic beam, receives a reflection signal of the ultrasonic beam, and processes the reflection signal to obtain an image of an organ in the body of the object to be scanned.

For ease of understanding, referring to fig. 1, fig. 1 is a schematic view of a scene for acquiring an ultrasound image according to an embodiment of the present application. As shown in fig. 1, where a portion 101 can be understood as an object to be scanned, in the embodiment of the present application, the ultrasound apparatus may include an ultrasound probe, such as an ultrasound probe 102 in fig. 1, the ultrasound probe includes a transmitting ultrasound unit and a receiving ultrasound unit, and is operable to transmit ultrasound waves and receive ultrasound waves reflected by the object to be scanned, where the ultrasound waves transmitted by the ultrasound probe are reflected to the probe after attenuation of tissue of the object to be scanned. It will be appreciated that the ultrasound probe described above may also be referred to as a probe, a super probe, etc.

The ultrasound device may further comprise a data processing unit, such as data processing unit 104 in fig. 1. The data processing unit 104 may be connected to the ultrasound probe 102, and process the ultrasound waves reflected by the object to be scanned and received by the ultrasound probe 102 to obtain an ultrasound image. Optionally, the ultrasound apparatus may further include a display unit, such as the display unit 103 in fig. 1, and the display unit 103 may be configured to display the ultrasound image obtained by the data processing unit 104.

It is understood that the ultrasound image acquired by the ultrasound device is a two-dimensional image corresponding to a slice. Since organs such as the kidney or other tissues have a three-dimensional spatial stereo structure, an image acquirer who acquires an ultrasound image using an ultrasound probe may operate to relatively completely acquire ultrasound image information of the entire region by rotating or moving the ultrasound probe, as shown in fig. 1 by section 105.

The ultrasonic image has the advantages of strong real-time performance, low examination cost and the like, but the ultrasonic image has the characteristics of low spatial resolution, small visual field and the like, so that the amount of information provided for an operator in the operation process is too small, and the operation difficulty of the operator is increased. Thus, in some embodiments, CT image data, i.e., CT images, may be acquired preoperatively by performing a Computed Tomography (CT) scan of the subject; the CT image is then registered with the intraoperatively acquired ultrasound image. The CT image has the advantages of high resolution, capability of keeping more details and the like, so that an operator can obtain more information in the operation, and the operation difficulty is reduced. Illustratively, in percutaneous nephrolithotomy, the registration of an ultrasonic image and a CT image enables an operator to obtain more image information of the kidney, so that the puncture precision of the renal surgery is improved, and the probability of puncture errors is reduced.

In the embodiment of the present application, the implementation object may be understood as any object that is acquired data to perform registration of an ultrasound image and a CT image, and may be, for example, a human, such as a patient, a volunteer, or the like; or may be an animal.

In the embodiment of the present application, the organ capable of acquiring the CT image and the ultrasound image may be understood as a target organ. Illustratively, the target organ may be a kidney (including a left kidney and/or a right kidney), gallbladder, spleen, or the like. For example, after permission of the subject or a guardian of the subject, an operator such as a doctor can acquire an ultrasound image and a CT image of a kidney of the subject and perform correlation processing.

In the embodiment of the present application, the ultrasound contour of the target organ can be understood from the target organ segmented from the ultrasound image, and the CT contour of the target organ can be understood from the target organ segmented from the CT image. It will be appreciated that since the target organ itself is three-dimensional, the ultrasound contour and the CT contour may each be considered to comprise a plurality of images, for example as shown in fig. 2, and that registration of the ultrasound contour and the CT contour may be understood as registration between the ultrasound image and the CT image.

For the registration of the ultrasound image and the CT image, an Iterative Closest Point (ICP) algorithm may be used for implementation. Wherein the ICP algorithm can transform sets of points in different coordinate systems into a common coordinate system by minimizing registration errors. Illustratively, the ICP algorithm can be represented by equation (1):

P _t ＝R·P _s +T (1)

wherein, P _t Can be understood as a target point cloud, P _s The ICP algorithm process is to solve a rotation matrix R and a translation matrix T so as to obtain a point cloud P _s Through the transformation (rotation and translation) and the point cloud P _t And overlapping and continuously iterating to minimize the mean square error.

For example, the rotation matrix R and the translation matrix T between the source point cloud and the target point cloud may be represented by a transformation matrix H, specifically formula (2):

the matrix R can be understood as a rotation amount, the matrix T can be understood as a translation amount, the matrix V can be understood as a perspective transformation amount, and the matrix S can be understood as a scale factor.

Exemplarily, the process of image registration can be divided into two stages of coarse registration and fine registration, wherein the coarse registration stage can be understood as coarser registration performed under the condition that the transformation between the source point cloud and the target point cloud is almost unknown, and the purpose of the coarse registration is mainly to provide a better initial value of a transformation matrix for the fine registration; fine registration may be understood as further optimization resulting in a more accurate transformation matrix, i.e. the final registration matrix, given an initial transformation matrix.

In general, in the coarse registration stage, the initial adjustment of the image orientation and the image approach may be performed according to the spatial features of the image, such as the normal vector corresponding to the image data. However, since the target organ inside the implementation object is a three-dimensional structure, the amount of image data is large, and the calculation efficiency of coarse registration through the spatial features of the images is slow, resulting in a long image registration time.

Based on the above problems, embodiments of the present application provide an image registration method and a related apparatus, after an ultrasound image is used to obtain an ultrasound contour of a target organ and a CT image is used to obtain a CT contour of the target organ, the ultrasound contour to be registered and the CT contour are subjected to orientation transformation, so that the poses of the ultrasound contour and the CT contour are close to each other, and thus the difference between the ultrasound contour and the CT contour is reduced, the time required for image registration is reduced, and the registration efficiency is improved.

It is to be understood that the image registration method may be performed by an image registration apparatus, which may be any apparatus capable of implementing the method provided in the present application, and for example, the image registration apparatus may be an electronic device such as a notebook computer and a desktop computer. It should be understood that the method embodiments of the present application may also be implemented by means of a processor executing computer program code.

In order to visually understand the image registration method provided by the present application, the above implementation object is a patient, the above target organ is a kidney, and the above image registration method is introduced by simulating a scenario of a percutaneous renal surgery performed by the patient.

It is understood that the main target of percutaneous renal surgery is the kidney of a patient, and therefore, the acquired CT image and ultrasound image should include the kidney information, such as the kidney contour of the patient or the blood vessels around the kidney.

1. CT image acquisition preparation

The doctor judges whether the enhanced CT image needs to be acquired according to the specific condition of the patient. If the doctor judges that the enhanced CT image does not need to be acquired, the patient lies on a CT scanning bed and then pushes the CT scanning bed into a CT scanning window to acquire a common CT image. If the doctor judges that the patient needs to acquire the enhanced CT image, the contrast agent is injected into the vein of the patient and then the patient is pushed into a CT scanning window to acquire the enhanced CT image.

It is understood that the contrast agent can be visualized through the blood circulation system at the location where the blood vessel is rich in blood supply, the visualization of these locations can be visually understood as reinforcement, and the CT image obtained by CT scan after reinforcement can obtain more information about the kidney, such as kidney stones and occupancy, kidney cysts, etc. It will be appreciated that subsequent registration of the enhanced CT image to the ultrasound image may also allow the physician to obtain more renal information than a normal CT image.

2. CT data acquisition

And (3) pushing the patient into a CT scanning window, and acquiring CT data according to the judgment result in the step (1), wherein the CT data comprises a CT image. And after the scanning is finished, taking the CT image data of the kidney of the patient.

It will be appreciated that if an enhanced CT scan is performed on a patient, including a sweep, arterial, venous, and delayed phase scan, generally, the follow-up steps can be performed using the sweep data.

3. CT data processing

The CT image acquired in step 2 is input into an image registration device (which may also be referred to as a renal surgery navigation system) provided by the present application. In the renal surgery navigation system, a three-dimensional renal contour is segmented and reconstructed from the CT data acquired in the step 2 through an artificial intelligence algorithm, and the three-dimensional renal contour is simply called as a CT-renal contour for convenience of understanding and description.

Illustratively, the artificial intelligence algorithm may be a deep learning algorithm, a Stochastic Gradient Descent (SGD), a back propagation (backward), a forward propagation (forward), and the like. It is understood that the CT-renal contours described above may include overall renal information, such as the contour of the kidney, the internal substance of the kidney, etc.

4. Ultrasound image acquisition

In the art, let the patient lie prone on the operation table, use the direction of ultrasonic equipment along patient's back vertebra to scan the observation to combine optical tracking equipment to fix a position the real-time spatial position of the ultrasonic image that ultrasonic equipment gathered. It can be understood that the patient can be reminded that the respiratory frequency is kept as slow and stable as possible in the ultrasonic image acquisition process, and the spatial position fluctuation error caused by respiration is reduced.

For ease of understanding, please refer to fig. 3 for example, and fig. 3 is a schematic view of another scene for acquiring an ultrasound image according to an embodiment of the present application. As shown in fig. 3, the ultrasound probe 302 is connected to a support 303. The bracket 303 includes markers, for example, the number of the markers is greater than or equal to 3, fig. 3 exemplarily illustrates 4 markers as an example, and 5 or 6 markers may also be set according to actual situations, which is not limited in the present application.

It is understood that the connection between the ultrasonic probe 302 and the bracket 303 may be a fixed connection or a movable connection, for example, a connection through a threaded structure, which is not limited in the present application.

First, it can be understood that without the optical tracking device for localization, the ultrasound image acquired by the ultrasound device is only a two-dimensional image, without three-dimensional spatial information. After the optical tracking device is turned on, the three-dimensional spatial position information of the ultrasonic image can be obtained through the coordinates of the marker on the ultrasonic probe.

In the embodiment of the present application, the markers on the gantry 303 are within the tracking range of the optical tracking device 305. In practice, the optical tracking device 305 can be moved or the position of the ultrasound probe 302 can be moved to bring the markers on the support 303 within the tracking range of the optical tracking device 305, so that the optical tracking device 305 can track the positions of the markers.

It will be appreciated that the optical tracking device 305 may determine the three-dimensional spatial coordinates of the various markers on the stent 303 by a light-reflective coating on the marker surfaces. Then, the optical tracking apparatus 305 may obtain the transformation matrix H from the coordinates of the marker _us Other objects having a definite positional relationship with the marker can be passed through the transformation matrix H _us Spatial location information is determined.

In the embodiment of the present application, the transformation matrix H _us It may be understood as a conversion matrix between the reference coordinates of the reference object (e.g., the coordinates of the above-mentioned marker) and the origin coordinates acquired by the optical tracking apparatus. The transformation matrix H is a three-dimensional space coordinate system _us Including rotationA component and a translational component. Illustratively, the above conversion matrix may be represented by equation (3):

since the structure of the scaffold 303 is determined, it is possible to determine the structural relationship between the marker and the scaffold 303, and the transformation matrix H _us Real-time spatial position information within the scanning range below the ultrasound probe 302 is determined.

Alternatively, after each two-dimensional ultrasound image is acquired, the renal surgery navigation system may segment a renal contour from the ultrasound image and display the segmentation result, i.e., the segmented renal contour, on the display unit 304, where it is understood that the renal contour is a two-dimensional renal contour.

It can be understood that the kidney is a three-dimensional space structure, and the doctor can change the position of the ultrasonic probe and acquire a plurality of ultrasonic images at the kidney and the nearby area so as to avoid missing the kidney information.

5. Ultrasound image processing

In the renal surgery navigation system, a three-dimensional renal contour is segmented and reconstructed according to the plurality of ultrasound images acquired in the step 4 and the corresponding spatial position information through an artificial intelligence algorithm, and is simply referred to as an ultrasound-renal contour for convenience of understanding and description.

In some embodiments, the renal surgery navigation system may further determine a similarity between the ultrasound-renal contour and the CT-renal contour, and if the similarity between the two is greater than the threshold a, the three-dimensional reconstruction of the ultrasound image is considered successful, otherwise, the ultrasound image is re-acquired to perform the three-dimensional reconstruction to obtain a new ultrasound-renal contour until the similarity between the obtained ultrasound-renal contour and the CT-renal contour is greater than the threshold a.

It is understood that the threshold a may be set according to actual situations, and may be any value greater than 90%, for example, generally, the greater the similarity, the higher the efficiency of subsequent registration. Illustratively, in one possible implementation, the similarity between the ultrasound-renal contour and the CT-renal contour may be obtained based on a euclidean distance therebetween. In another possible implementation manner, the similarity may be a Dice Similarity Coefficient (DSC), and in general, the higher the Dice similarity coefficient is, the higher the accuracy of the registration is.

It will be appreciated that steps 1-5 above may be considered as a data preparation phase, while the subsequent steps 6-7 may be considered as an image registration phase.

6. Based on orientation transformation matrix R _ct-us Performing coarse registration

In the embodiment of the application, an orientation transformation matrix is determined and subjected to orientation transformation for the ultrasound image and the CT image to be registered in the image registration stage, wherein the orientation transformation matrix may include a rotation matrix and/or a translation matrix.

It is understood that the ultrasound image is acquired by an ultrasound device, the CT image is acquired by a CT device, and finally the two images need to be registered in a world coordinate system. The CT device automatically converts the acquired images into a world coordinate system, and the spatial position and the direction of the CT images converted into the world coordinate system are fixed. However, the ultrasound device acquires a two-dimensional ultrasound image, and the identification of the optical tracking device is required to convert the two-dimensional ultrasound image to a world coordinate system, and therefore, the three-dimensional spatial position of the converted ultrasound image is correlated with the position of the optical tracking device. However, in practice, the position of the optical tracking device is generally random, for example, according to the operating room environment or according to the custom arrangement of the user, which results in a large difference in the position (e.g., position, angle) between the ultrasound-renal contour and the CT-renal contour in the world coordinate system.

Referring to fig. 4, fig. 4 is a schematic diagram illustrating a relative position relationship between a CT-renal contour and an ultrasound-renal contour according to an embodiment of the present application.

The coordinate system as shown in fig. 4 can be understood as a world coordinate system. Illustratively, the CT-kidney contour may be located in quadrant 1 and quadrant 4 in the world coordinate system, as shown in fig. 4, the kidney contour 401 and the kidney contour 402 may be understood as a CT-kidney contour, the kidney contour 401 may be understood as a left kidney contour, and the kidney contour 402 may be understood as a right kidney contour. Similarly, the above-mentioned ultrasound-kidney contour may be located in quadrant 5 in the world coordinate system, and as shown in fig. 4, the kidney contour 403 and the kidney contour 404 may be understood as the ultrasound-kidney contour, and exemplarily, the kidney contour 403 may be understood as the left kidney contour and the kidney contour 404 may be understood as the right kidney contour.

It is to be understood that in the registration of the CT-kidney contour with the ultrasound-kidney contour, it is the left kidney in the CT-kidney contour that is registered with the left kidney in the ultrasound-kidney contour and the right kidney in the CT-kidney contour that is registered with the right kidney in the ultrasound-kidney contour. In addition, since the registration between the left kidneys is similar to the registration between the right kidneys, for economy, the registration between the left kidneys (i.e., the left kidney contour 401 and the left kidney contour 403) is described herein as an example.

As shown in fig. 4, the CT-kidney contour and the ultrasound-kidney contour have different positions (also understood as center positions) and different angles. The method and the device perform orientation transformation between the kidney outlines before registration, and enable the CT-kidney outline and the ultrasound-kidney outline to be as close as possible from relative positions and angular positions so as to reduce the difference between the CT-kidney outline and the ultrasound-kidney outline and achieve the purpose of improving the registration efficiency.

Referring to fig. 5 by way of example, fig. 5 is a schematic diagram illustrating an orientation transformation of an ultrasound-renal contour according to an embodiment of the present application. It will be appreciated that one of the kidney contours in fig. 5 is pattern-filled to facilitate distinguishing between two kidney contours, wherein the kidney contour 501 in fig. 5 can be understood as the CT-kidney contour 401 in fig. 4, and the kidney contour 502 in fig. 5 can be understood as the ultrasound-kidney contour 403 in fig. 4. As shown in fig. 5, the renal contours 501 and 502 are highly coincident in both position and angle.

In order to distinguish the rotation matrix and the translation matrix provided by the embodiment of the present application from those in the existing registration algorithm, such as the rotation matrix R and the translation matrix T, the rotation matrix provided by the embodiment of the present application is referred to as a rotation orientation matrix, and the translation matrix provided by the embodiment of the present application is referred to as a translation orientation matrix. Next, an implementation of determining the rotational orientation matrix and the translational orientation matrix will be described separately.

And 6.1, determining a rotation orientation matrix.

In the embodiment of the application, under the condition that the posture of the patient is not changed, the rotation orientation matrix between the ultrasound-renal contour and the CT-renal contour is determined by taking the direction determined on the basis of the patient as a reference.

6.1.1 determining the relationship between the patient and the coordinate system of the ultrasound device

As described in step 4 above, when the ultrasound apparatus is used to scan the kidney of the patient along the direction of the spine of the back of the patient, the X direction of the coordinate system of the ultrasound apparatus is the same as the direction in which the feet of the patient point to the head (abbreviated as head direction), and the Y direction of the coordinate system is perpendicular to and outward from the back of the patient (abbreviated as back vertical direction). Thus, the X direction of the obtained ultrasound image is the head direction, and the Y direction is the back vertical direction.

It is understood that although the two-dimensional ultrasound image acquired by the ultrasound apparatus does not have three-dimensional spatial information, i.e., Z-axis information, the Z-direction of the ultrasound image can be considered as a direction in which the body of the patient points to the left of the patient (referred to as a left direction for short) according to the existing right-hand rule that the X-axis, the Y-axis and the coordinate system must satisfy.

Since the spatial position of the ultrasound-renal contours is based on the transformation matrix H of the optical tracking device _us The coordinates of the original ultrasound image are obtained along with the transformation matrix H _us And (6) carrying out transformation. Illustratively, the original head direction is the X direction of the ultrasound-renal contours, via the transformation matrix H _us After transformation to the world coordinate system, the direction vector of the head direction

Is (A) ₁₁ ,A ₂₁ ,A ₃₁ ). The direction vector of the back vertical direction can be obtained by the same way

Is (A) ₁₂ ,A ₂₂ ,A ₃₂ ) Direction vector of left direction

Is (A) ₁₃ ,A ₂₃ ,A ₃₃ )。

6.1.2 determining the relationship between the patient and the coordinate system of the CT apparatus

It is understood that the CT image acquired by the CT apparatus includes a plurality of two-dimensional images, which can be understood as a group of two-dimensional image sequences with spatial information. Since the CT device itself automatically converts the coordinate system to the world coordinate system, the coordinate information of the CT image acquired by the T device is information in the world coordinate system.

Because the patient is located at a special position when the CT image is acquired, for example, the head of the patient enters the CT device first, the X direction of the CT image obtained by CT scanning the patient is the left direction, the Y direction of the CT image is the vertical direction of the back, and the Z direction of the CT image is the head direction.

Thereby, a direction vector of the head direction can be obtained

Is (0,0,1). The direction vector of the vertical direction of the back can be obtained by the same way

Is (0,1,0), direction vector of left direction

Is (1,0,0).

As can be understood from the above analysis, the head direction, the back vertical direction, and the left direction of the patient are themselves certain in the real world without the posture of the patient changing. However, since the coordinate systems adopted by the apparatus are different, after the ultrasound-renal contour and the CT-renal contour are placed under the same world coordinate system, the direction vectors corresponding to the head direction, the back vertical direction and the left direction of the patient are different, and thus the situation that the positions and angles of the renal contours are different as shown in fig. 4 appears.

6.1.3, calculating the included angle relation between the CT direction and the ultrasonic direction for correction

a. Correct head direction

Calculating head direction

And

angle of (theta) _z 。

By

Vector projected in X direction

Is (A) ₁₁ ,A ₂₁ 0), calculating

Angle theta with positive X-axis unit vector (1,0,0) _x 。

Thus, the head direction (i.e., Z direction) of the CT image is rotated counterclockwise by θ about the Y axis _z Then rotated by theta about the Z axis _x The head direction correction of the CT-kidney contour can be aligned with the head direction of the ultrasound-kidney contour. Wherein, it is understood that in A ₂₁ In the case of > 0, rotate counterclockwise about the direction of rotation, A ₂₁ And rotating clockwise around the rotating direction under the condition that the rotating speed is less than or equal to 0.

b. Correct the vertical direction and the left direction of the back

According to the above-mentioned included angle theta _z And the above-mentioned angle theta _x A rotation matrix R for correcting the head direction of the CT-kidney contour can be obtained _ct-h The above transformation matrix R _ct-h Can be represented by formula (4):

thus, through the transformation matrix R _ct-h Left direction of CT-renal contour after transformation

Is (B) ₁₁ ,B ₂₁ ,B ₃₁ ) Perpendicular to the back

Is (B) ₁₂ ,B ₂₂ ,B ₃₂ ) Direction of head

Is (B) ₁₃ ,B ₂₃ ,B ₃₃ )。

Calculating the Back vertical orientation of the CT-Kidney contour

(B ₁₂ ,B ₂₂ ,B ₃₂ ) Perpendicular to the back of the ultrasound-renal contour

(A ₁₂ ,A ₂₂ ,A ₃₂ ) Angle alpha of ₁ 。

Calculating the Back vertical orientation of the CT-Kidney contour

(B ₁₂ ,B ₂₂ ,B ₃₂ ) Left direction from the ultrasound-renal contour

Is (A) ₁₃ ,A ₂₃ ,A ₃₃ ) Included angle alpha ₂ Above angle of inclination α ₂ For judging CT-kidney contour rotation alpha ₁ The direction of rotation of (c). Illustratively, at α ₂ Under the condition of less than or equal to 90 degrees, the CT-kidney contour is wound in the direction

(B ₁₃ ,B ₂₃ ,B ₃₃ ) Rotate counterclockwise by alpha ₁ (ii) a At α ₂ Under the condition of more than 90 degrees, the CT-kidney contour is wound around the head direction

(B ₁₃ ,B ₂₃ ,B ₃₃ ) Clockwise rotation alpha ₁ 。

Since the rotation is based on a coordinate axis (e.g., rotation about the X, Y, or Z axis), the above-mentioned correction of the vertical direction and the left direction of the back can be performed before the correction of the head. That is, may be at α ₂ Under the condition of less than or equal to 90 degrees, the CT-kidney contour is wound in the direction

(0,0,1) counterclockwise rotation of alpha ₁ ；α ₂ Under the condition of more than 90 degrees, the CT-kidney contour is wound in the direction of head

(0,0,1) clockwise rotation α ₁ . Then, the calibration of the head direction in step a is performed, so as to complete the calibration of the head direction, the left direction and the back vertical direction.

Finally, according to the angle theta _z The above-mentioned angle theta _x The above-mentioned included angle alpha ₁ And an included angle alpha ₂ The final rotation orientation matrix R can be obtained _ct-us . The above-mentioned rotation orientation matrix R _ct-us Can be represented by formula (5):

it is to be understood that the angle relation may be calculated by taking any one of the head direction, the back vertical direction and the left direction as the first direction of the calculated angle, the correction in the head direction is only described as an example, and the left direction or the back vertical direction may be corrected first, in a manner similar to that described above.

6.2, determining a translation orientation matrix

Let the central coordinate of the CT-kidney contour be (C) _x ，C _y ，C _z ) The ultrasound-renal contour has a center coordinate of (U) _x ，U _y ，U _z ). Therefore, under the scene that the ultrasound-kidney contour is the source point cloud and the CT-kidney contour is the target point cloud, the CT-kidney contour is close to the ultrasound-kidney wheelThe translation parameters of the profile can be represented by equation (6):

wherein, the above-mentioned T _x Representing the amount of translation of the ultrasound-renal contour in the X-direction, T _y Represents the translation of the ultrasound-renal contours in the Y direction, T _z Indicating the amount of translation of the ultrasound-renal contour in the Z-direction. It will be appreciated that the ultrasound-renal contours may actually be translated according to the above-described T _x 、T _y And T _z Moves in the positive direction of the coordinate axis if the positive and negative values are regular, and moves in the negative direction of the coordinate axis if the negative value is negative.

In some embodiments, the renal contours may be rotationally transformed based on a rotational orientation matrix, i.e., equation (5) above; in another embodiment, the renal contours may be translationally transformed based on a translational orientation matrix, i.e., equation (6) above, in still other embodiments, the renal contours may be rotationally and translationally transformed based on the rotational orientation matrix and the translational orientation matrix, illustratively, the transformation matrix for rotation and translation may be the transformation matrix H of equation (7) _ct-us Represents:

and performing coarse registration on the CT-kidney contour and the ultrasound-kidney contour after the orientation transformation through the transformation of the rotation orientation matrix and/or the translation orientation matrix, wherein the CT-kidney contour and the ultrasound-kidney contour are closer in position, so that the efficiency of the coarse registration is improved. In general, the renal contours that undergo rotation and translation transformations are more efficient to match than renal contours that undergo rotation or translation alone.

For ease of understanding, please refer to fig. 6. Fig. 6 is a schematic diagram illustrating a registration result of a CT-kidney contour and an ultrasound-kidney contour according to an embodiment of the present application. The result as shown in fig. 6 can be understood as registration after rotational and translational transformation.

7. Image fine registration

In the embodiment of the present application, the renal surgery navigation system may use the ICP algorithm to realize the fine registration of the CT-renal contour and the ultrasound-renal contour. Illustratively, the fine registration may include the steps of:

(1) Finding the nearest corresponding point

In the step, an initial rotation matrix and an initial translation matrix are used, or a rotation matrix and a translation matrix obtained by last iteration are used for transforming an initial point cloud, a CT-kidney contour and an ultrasound-kidney contour are converted into temporary transformation point clouds, then, a source point cloud corresponding to the ultrasound-kidney contour and a target point cloud corresponding to the CT-kidney contour are compared, and the nearest neighbor point of each point in the source point cloud in the target point cloud is found out.

(2) Solving for optimal transformations

In this step, the optimal transformation is a closed-form solution, and may be calculated by means of matrix Singular Value Decomposition (SVD) to obtain an optimal translation matrix and an optimal rotation matrix.

(3) Iterative transformation of kidney images

It will be appreciated that each iteration results in the current optimal transformation parameters, i.e., the current optimal translation matrix T _k And a rotation matrix R _k (where subscript k is used to denote the kth iteration, k may be an integer greater than or equal to 1), and after each iteration is complete, the matrix T will be translated _k And a rotation matrix R _k The transformation acts on the current source point cloud.

And after the last iteration and transformation are finished, continuously iterating according to the step (1) of finding the nearest corresponding points and the step (2) of solving the optimal transformation until the iteration termination condition is met. In this embodiment of the application, the termination condition may be that a variation of the obtained translation matrix and a variation of the rotation matrix in two adjacent iterations are both smaller than or equal to a threshold, or may be that a loss value of the iteration is smaller than or equal to a threshold, or may be that the number of iterations reaches a preset number. The threshold and the preset times may be set according to actual conditions, which is not limited in the present application.

8. Iterative post-processing of kidney images

It can be understood that, in the above-mentioned fine registration stage, the ultrasound-kidney contour is a source point cloud, and the CT-kidney contour is a target point cloud, so that the registration matrix for registering the ultrasound-kidney contour to the CT-kidney contour is obtained through the above-mentioned step 7, and for the convenience of understanding, the registration matrix for registering the ultrasound-kidney contour to the CT-kidney contour in the fine registration stage is denoted as F _us-ct 。

It can be understood that the ultrasound image has lower precision than the CT image, that is, the image information of the CT image is more comprehensive and accurate, and the ultrasound-renal contour is used as the source point cloud and the CT-renal contour is used as the target point cloud for registration in the registration stage, so that a more accurate registration effect can be obtained. However, since the ultrasound image is an image obtained in the actual operation space, and the CT image is an image (which can be considered as a virtual space) in the CT image acquisition room space, the CT image needs to be applied to the actual operation scene during the operation, so the CT-kidney contour needs to be registered to the ultrasound-kidney contour finally. Therefore, it is desirable to obtain a registration matrix F for CT-renal contour registration to ultrasound-renal contour _ct-us 。

In some embodiments, the final registration matrix F may be obtained by multiplying the registration matrix obtained in step 7 (pre-matrix multiplication of post-iteration) _us-ct For the above registration matrix F _us-ct Obtaining a registration matrix F by inversion _ct-us And further realize the registration of the CT-kidney contour to the ultrasound-kidney contour.

It will be appreciated that if the above-mentioned registration matrix F is used _ct-us The registration effect is not ideal, and the orientation can be adjusted through manual registration on the basis of the current registration and the fine registration is carried out again until the satisfactory precision is achieved.

The image registration method provided by the embodiment of the present application is introduced in the above scenario in which the implementation object is a patient, the target organ is a kidney, and the patient is simulated to perform a percutaneous renal operation, and then, in combination with other scenarios to which the present application is applicable, the image registration method provided by the present application is integrally introduced.

Referring to fig. 7 by way of example, fig. 7 is a schematic flowchart of an image registration method according to an embodiment of the present application. As shown in fig. 7, the method includes:

701: acquiring an ultrasonic profile and a Computed Tomography (CT) profile of a target organ; the ultrasound contour is obtained from an ultrasound image acquired by an ultrasound device and the CT contour is obtained from a CT image acquired by a CT device.

It is understood that the target organ in the embodiment of the present application is an organ capable of acquiring an ultrasound image and a CT image, such as a kidney, a gallbladder, a spleen, and the like. The ultrasound device may be used to perform ultrasound scanning on a target organ to obtain a plurality of ultrasound images, and then the ultrasound contour of the target organ is segmented from the plurality of ultrasound images. The CT device can be used for carrying out CT scanning on a target organ to obtain a plurality of CT images, and then the CT outline of the target organ is segmented from the plurality of CT images.

In the case that the target organ is a kidney, the ultrasound contour may be understood as an ultrasound-kidney contour in the foregoing, and the CT contour may be understood as a CT-kidney contour in the foregoing, specifically, refer to the description of step 1 to step 5 in the foregoing. It is understood that, in the case where the target organ is an organ other than the kidney, the processing manner is similar to that of the kidney, and is not described herein again.

In some embodiments, the similarity between the ultrasound profile and the CT profile is greater than or equal to a third threshold.

It can be understood that, in the process of acquiring the ultrasound contour, it is necessary to determine the space of the ultrasound scanning space (for example, the scene shown in fig. 3) by means of the optical tracking device, further determine the position information of the multiple ultrasound images scanned by the ultrasound device, and then perform the segmentation reconstruction to obtain the ultrasound contour. In the above process, the position information of some ultrasound images may have a deviation, so that the obtained ultrasound profile is not accurate enough. And the CT image is an image acquired by a CT apparatus. When the CT image is acquired, the implementation object needs to be suffocated, and the like, the image information acquired by the CT equipment is comprehensive and accurate, and the acquired CT contour can be considered as the accurate CT contour of the target organ. Therefore, the registration efficiency can be improved by controlling the similarity between the ultrasound contour and the CT contour, that is, by registering the ultrasound contour with the CT contour, the similarity of which is greater than or equal to the third threshold.

In this embodiment, the similarity may be a Dice similarity coefficient or other reasonable index, and the third threshold may be set according to an actual situation, for example, may be any value greater than or equal to 80%. Optionally, in the description of this embodiment, reference may also be made to step 5, and the third threshold may also be understood as the threshold a.

702: rotating the CT profile to obtain a rotated CT profile; an angle between a direction vector of a reference direction in the space of the ultrasound contour and a direction vector of the reference direction in the space of the rotated CT contour is smaller than or equal to a first threshold, the reference direction being common to the space of the ultrasound contour and the space of the CT contour.

In the embodiment of the present application, the reference direction is common to the space of the ultrasound contour and the space of the CT contour, and it can be understood that the reference direction may determine a direction vector in the space of the ultrasound contour, and may also determine a direction vector in the space of the CT contour.

Illustratively, the above-mentioned reference direction may be determined according to an implementation object. It is understood that both the ultrasound image and the CT image are acquired based on the subject, and that the relative relationship between the target organ and the subject is determined even though the time, spatial information, etc. of acquiring the ultrasound image differs from the time of acquiring the CT image. For example, when a CT image is acquired, the relative relationship between the spatial coordinate information of the CT image and the orientation of the human body is determined; when the ultrasonic image is acquired, the relative relationship between the spatial coordinate information of the ultrasonic image and the position of the human body is also determined. Therefore, the reference direction may be a direction determined according to the implementation object, and may be, for example, the head direction, the back vertical direction, the left direction, a direction in which the head of the implementation object points to the feet (foot direction for short), a direction in which the body of the implementation object points to the right of the patient (right direction for short), or the like.

It is understood that before the CT contour is not rotated, i.e. the ultrasound contour and the CT contour obtained in step 701, an angle between a direction vector of the reference direction in the space of the ultrasound contour and a direction vector of the reference direction in the space of the CT contour is larger than the first threshold. In this embodiment, the CT contour is rotated to obtain a rotated CT contour, and an included angle between a direction vector of the reference direction in the space of the ultrasound contour and a direction vector of the reference direction in the space of the rotated CT contour is smaller than or equal to a first threshold, so that a difference between the rotated CT contour and the ultrasound contour is reduced compared with a difference between the CT contour before rotation, thereby saving registration time in a registration process and improving registration efficiency.

The image registration means may keep the ultrasound contour stationary and rotate the CT contour according to a relation between a direction vector of the reference direction in the space of the ultrasound contour (referred to as ultrasound-reference direction vector for short) and a direction vector of the reference direction in the space of the CT contour (referred to as CT-reference direction vector for short).

On the other hand, when the CT contour is rotated, the CT-reference direction vector may be the same as the ultrasound-reference direction vector (it may be understood that the included angle is 0), or the included angle between the CT-reference direction vector and the ultrasound-reference direction vector may be another value as long as the included angle is smaller than or equal to the first threshold. On the other hand, the reference direction may be one or more. For ease of understanding, the rotation process in this step may be referred to as rectification.

It is understood that the first threshold may be set according to practical situations, and may be, for example, any value less than 5 °, such as 2 ° or 3 °, and the like, which is not limited in this application. Generally, the smaller the above-mentioned angle, the higher the registration efficiency.

703: registration is performed based on the rotated CT profile and the ultrasound profile.

In this step, the image registration device registers the rotated CT contour and the ultrasound contour, wherein the registration may include coarse registration and fine registration, and the specific implementation may refer to the related descriptions of step 6 and step 7 in the foregoing.

Because the original CT contour and the ultrasound contour have a larger pose difference, that is, an included angle between a direction vector of the reference direction in the space of the CT contour and a direction vector of the reference direction in the space of the ultrasound contour is greater than a first threshold, the image registration method provided by the embodiment of the present application rotates the acquired original CT contour to obtain the rotated CT contour, and the included angle between the direction vector of the reference direction in the space of the rotated CT contour and the direction vector of the reference direction in the space of the ultrasound contour is less than or equal to the first threshold, so that the poses of the rotated CT contour and the ultrasound contour are closer, and the difference between the two poses is reduced, thereby improving the calculation efficiency, shortening the registration time and improving the registration efficiency.

In some embodiments, the image processing apparatus may directly rotate the CT contour by using the space vector of the reference direction in the two contour spaces, so that the above-mentioned included angle is 0. Illustratively, in one possible implementation, in the method shown in fig. 7, step 702 includes:

7021: determining a first direction vector of the reference direction in the space of the CT profile, and determining a second direction vector of the reference direction in the space of the ultrasound profile.

7022: and determining a rotation matrix according to an included angle between the first direction vector and the second direction vector and an included angle between the second direction vector and the coordinate axis.

7023: and rotating the CT outline based on the rotation matrix to obtain the rotated CT outline.

It can be understood that, with the method of the present embodiment, the angle between the direction vector of the reference direction in the space of the ultrasound contour and the direction vector of the reference direction in the space of the rotated CT contour is 0, and therefore, the direction vector of the reference direction in the space of the ultrasound contour and the direction vector of the reference direction in the space of the rotated CT contour can be considered to be the same. By rotating the CT contours in the above manner, the registration efficiency is higher than that of other manners.

It is understood that the reference direction is different in the direction vector of the space of the CT contour and the ultrasound contour, and the first direction vector and the second direction vector may be determined according to the relative positional relationship between the reference direction and the CT contour and the ultrasound contour. Then, an included angle between the first direction vector and the second direction vector and an included angle between the second direction vector and a coordinate axis can be obtained according to an included angle formula between the vectors; finally, after the rotation direction is determined according to the coordinate values in the second direction vector, a rotation matrix for rotating the first direction vector to the second direction vector can be determined, and the CT contour can be rotated based on the rotation matrix.

For example, in a case where the target organ is a kidney and the reference direction is a head direction, the first direction vector may be understood as the direction vector

The second direction vector can be understood as the direction vector

The angle between the first direction vector and the second direction vector may be understood as the angle θ _z The angle between the second direction vector and the coordinate axis can be understood as the angle θ _x The rotation matrix may be understood as the transformation matrix R _ct-h For example, reference may be made to the related description of step 6.1.3, which is not described herein again.

Alternatively, the reference direction may be multiple, and the image registration apparatus may achieve the effect of step 702 in each reference direction. Two references, referred to as a first sub-reference direction and a second sub-reference direction, may be exemplarily provided. In the above case, the image registration apparatus may first rotate the CT contour for a first time to obtain a first rotated CT contour, and ensure that an included angle between the CT-reference direction vector and the ultrasound-reference direction vector in the first sub-reference direction is smaller than or equal to the first threshold; and then, carrying out second rotation on the first rotation CT outline to ensure that the included angle between the CT-reference direction vector and the ultrasonic-reference direction vector is less than or equal to the first threshold value in the second sub-reference direction.

Because the CT contour and the ultrasonic contour comprise a plurality of images, the CT contour and the ultrasonic contour can be understood as three-dimensional, and the difference between the rotated CT contour and the ultrasonic contour can be further reduced through correction in three directions, so that the calculation efficiency is further improved, the registration time is shortened, and the registration efficiency is improved.

Thus, in some embodiments, the reference direction comprises a first sub-reference direction, a second sub-reference direction, and a third sub-reference direction; the direction vectors of the first sub-reference direction, the second sub-reference direction, and the third sub-reference direction in the space of the ultrasound contour and the direction vector of the rotated CT contour are respectively less than or equal to the first threshold.

In this embodiment, the reference directions may be 3 different directions, that is, the first sub-reference direction, the second sub-reference direction, and the third sub-reference direction. It is to be understood that any sub-reference direction may be understood as a reference direction, and therefore, any sub-reference direction may be determined according to the manner of determining the reference direction, which may be referred to in the foregoing description of step 702.

Illustratively, the image registration apparatus may correct the first sub-reference direction based on the above steps 7021 to 7024, and then further correct the second sub-reference direction and the third sub-reference direction based on an angle relationship between the second sub-reference direction and the third sub-reference direction.

In some embodiments, the first sub-reference direction is a head direction, the second sub-reference direction is a back vertical direction, and the third sub-reference direction is a left direction, and the three directions are perpendicular to each other, so that the CT contour can be effectively corrected. Based on the setting of the three directions, if the target organ is a kidney, the rotation direction can be calculated in step 6Matrix R _ct-us The related description of (1).

In addition to the above correction process, the translation may be further performed according to the central positions of the CT contour and the ultrasound contour, and in some embodiments, in the method shown in fig. 7, step 703 includes:

7031: and performing the registration based on the rotated CT contour and the ultrasound contour when a distance between the center coordinate of the rotated CT contour and the center coordinate of the ultrasound contour is smaller than a second threshold value.

In this embodiment, after the rotated CT contour is obtained, the distance between the center coordinate of the CT contour and the center coordinate of the ultrasound contour may be determined first, and registration may be performed when the distance is smaller than a second threshold. By the method, the difference between the CT outline and the ultrasonic outline can be within a controllable range, and the registration efficiency is higher compared with other registration methods.

It is understood that the second threshold may be set according to practical situations, for example, may be a value smaller than the minimum distance between the center of the contour of the target organ and the edge of the contour of the target organ, or may be an empirical value, which is not limited in this application.

7032: and translating the rotated CT contour to a position where the distance between the rotated CT contour and the ultrasonic contour is less than or equal to the second threshold value when the distance between the rotated CT contour and the ultrasonic contour is greater than the second threshold value, thereby obtaining the rotated CT contour.

7033: and registering the CT contour after the rotation translation and the ultrasonic contour.

In this embodiment, after the rotated CT contour is obtained, a distance between a center coordinate of the CT contour and a center coordinate of the ultrasound contour may be determined first, and the rotated CT contour may be translated when the distance is greater than the second threshold. During translation, the distance between the center coordinate of the rotationally translated CT contour and the center coordinate of the ultrasound contour may be smaller than or equal to the second threshold. The embodiment can further reduce the difference between the two contours to be registered, thereby improving the calculation efficiency, shortening the registration time and improving the registration efficiency.

Optionally, the rotated CT contour may be translated, so that the center coordinate of the rotated and translated CT contour coincides with the center coordinate of the ultrasound contour, which may specifically refer to the description of determining the shift orientation matrix in step 6.2, and is not described herein again.

The embodiment of the present application further provides an image registration method, including:

acquiring an ultrasonic profile and a Computed Tomography (CT) profile of a target organ; the ultrasonic contour is obtained according to an ultrasonic image acquired by ultrasonic equipment, and the CT contour is obtained according to a CT image acquired by CT equipment;

rotating the ultrasonic profile to obtain a rotated ultrasonic profile; an angle between a direction vector of a reference direction in a space of the ultrasound contour and a direction vector of the reference direction in a space of the rotated CT contour is smaller than or equal to a first threshold value, the reference direction being common to the space of the ultrasound contour and the space of the CT contour;

and registering based on the rotated ultrasonic contour and the CT contour.

It will be appreciated that in the method shown in figure 7, the original CT contours and the ultrasound contours are acquired and then rotated. In this embodiment, after the original CT contour and the original ultrasound contour are obtained, the ultrasound contour is rotated, and the specific implementation is similar to the calculation manner and the rotation principle in the method corresponding to fig. 7, which can be understood as changing the CT contour into the ultrasound contour of the action object, and will not be described herein again.

The method provided by the embodiment of the application is explained in detail above, and the device provided by the embodiment of the application is introduced below.

Referring to fig. 8, fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. The electronic device 80 is configured to execute the image registration method provided in the embodiments of the present application. The electronic device 80 may be a desktop computer, a tablet computer, a portable notebook, and the like, for example, and the embodiments of the present application are not limited thereto.

As shown in fig. 8, the electronic apparatus 80 includes an acquisition unit 801, a rotation unit 802, and a registration unit 803. Optionally, the electronic device 80 may further include a determination unit 804 and a translation unit 805.

In the first implementation, the description of each unit is as follows:

an acquiring unit 801, configured to acquire an ultrasound profile and a Computed Tomography (CT) profile of a target organ; the ultrasonic contour is obtained according to an ultrasonic image acquired by ultrasonic equipment, and the CT contour is obtained according to a CT image acquired by CT equipment;

a rotating unit 802, configured to rotate the CT profile to obtain a rotated CT profile; an angle between a direction vector of a reference direction in a space of the ultrasound contour and a direction vector of the reference direction in a space of the rotated CT contour is smaller than or equal to a first threshold value, the reference direction being common to the space of the ultrasound contour and the space of the CT contour;

a registration unit 803, configured to perform registration based on the rotated CT contour and the ultrasound contour.

In a possible implementation manner, the determining unit 804 is configured to determine a first direction vector of the reference direction in the space of the CT contour, and determine a second direction vector of the reference direction in the space of the ultrasound contour;

the determining unit 804 is further configured to determine a rotation matrix according to an included angle between the first direction vector and the second direction vector and an included angle between the second direction vector and a coordinate axis;

the rotation unit 802 is further configured to rotate the CT contour based on the rotation matrix to obtain the rotated CT contour.

In a possible implementation manner, the reference direction includes a first sub-reference direction, a second sub-reference direction, and a third sub-reference direction; the direction vectors of the first sub-reference direction, the second sub-reference direction, and the third sub-reference direction in the space of the ultrasound contour and the direction vector of the rotated CT contour are respectively less than or equal to the first threshold.

In a possible implementation manner, the first sub-reference direction is a direction pointing from a foot to a head of the implementation subject, the second sub-reference direction is a direction pointing perpendicularly outward from a back of the implementation subject, and the third sub-reference direction is a direction pointing from a body to a left hand of the implementation subject.

In a possible implementation manner, the registration unit 803 is specifically configured to perform the registration step based on the rotated CT contour and the ultrasound contour when a distance between a center coordinate of the rotated CT contour and a center coordinate of the ultrasound contour is smaller than a second threshold.

In a possible implementation manner, the translating unit 805 is configured to translate the rotated CT contour to a position where a distance between the rotated CT contour and the ultrasound contour is less than or equal to the second threshold value when the distance between the center coordinate of the rotated CT contour and the center coordinate of the ultrasound contour is greater than the second threshold value, so as to obtain a rotationally translated CT contour;

a registration unit 803, configured to register the rotationally translated CT contour and the ultrasound contour.

In one possible implementation, the similarity between the ultrasound profile and the CT profile is greater than or equal to a third threshold.

In the second implementation, the description of each unit is as follows:

an acquisition unit 801, configured to acquire an ultrasound profile and a Computed Tomography (CT) profile of a target organ; the ultrasonic contour is obtained according to an ultrasonic image acquired by ultrasonic equipment, and the CT contour is obtained according to a CT image acquired by CT equipment;

a rotating unit 802, configured to rotate the ultrasonic profile to obtain a rotated ultrasonic profile; an angle between a direction vector of a reference direction in a space of the ultrasound contour and a direction vector of the reference direction in a space of the rotated CT contour is smaller than or equal to a first threshold value, the reference direction being common to the space of the ultrasound contour and the space of the CT contour;

a registration unit 803, configured to perform registration based on the rotated ultrasound contour and the CT contour.

In a possible implementation manner, the determining unit 804 is configured to determine a first direction vector of the reference direction in the space of the CT contour, and determine a second direction vector of the reference direction in the space of the ultrasound contour;

a determining unit 804, further configured to determine a rotation matrix according to an included angle between the first direction vector and the second direction vector, and an included angle between the first direction vector and a coordinate axis;

the rotating unit 802 is further configured to rotate the ultrasonic profile based on the rotation matrix to obtain the rotated ultrasonic profile.

In a possible implementation manner, the reference direction includes a first sub-reference direction, a second sub-reference direction, and a third sub-reference direction; the direction vectors of the first sub-reference direction, the second sub-reference direction, and the third sub-reference direction in the space of the ultrasound contour and the direction vector of the rotated ultrasound contour are respectively less than or equal to the first threshold.

In a possible implementation manner, the first sub-reference direction is a direction pointing from a foot to a head of the implementation subject, the second sub-reference direction is a direction pointing perpendicularly outward from a back of the implementation subject, and the third sub-reference direction is a direction pointing from a body to a left hand of the implementation subject.

In a possible implementation manner, the registration unit 803 is specifically configured to perform the registration step based on the rotated ultrasound contour and the CT contour when a distance between the center coordinate of the rotated ultrasound contour and the center coordinate of the CT contour is smaller than a second threshold.

In a possible implementation manner, the translating unit 805 is configured to translate the rotated ultrasound contour to a position where a distance between the rotated ultrasound contour and the CT contour is less than or equal to the second threshold value when the distance between the center coordinate of the rotated ultrasound contour and the center coordinate of the CT contour is greater than the second threshold value, so as to obtain a rotationally translated ultrasound contour;

a registration unit 803, specifically configured to register the rotationally translated ultrasound contour and the CT contour.

In one possible implementation, the similarity between the ultrasound profile and the CT profile is greater than or equal to a third threshold.

Referring to fig. 9, fig. 9 is a schematic structural diagram of another electronic device according to an embodiment of the present disclosure. As shown in fig. 9, the electronic device 90 includes a memory 901 and a processor 902. Further optionally, a communication interface 903 and a bus 904 may be further included, where the memory 901, the processor 902 and the communication interface 903 are communicatively connected to each other through the bus 904.

The memory 901 is used to provide a storage space, and data such as an operating system and a computer program may be stored in the storage space. The memory 901 includes, but is not limited to, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or a portable read-only memory (CD-ROM).

The processor 902 is a module for performing arithmetic operations and logical operations, and may be one or a combination of plural kinds of processing modules such as a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a microprocessor unit (MPU), or the like.

The memory 901 stores a computer program, and the processor 902 calls the computer program stored in the memory 901 to execute the image registration method described above. For example, in the case that the electronic device 90 is the electronic device 80, the content acquired by the acquiring unit 801 may be implemented by the communication interface 903, and the steps performed by the rotating unit 802, the registering unit 803, the determining unit 804 and the translating unit 805 may be implemented by the processor 902.

The present application also provides a computer-readable storage medium having stored therein computer code which, when run on a computer, causes the method of the above-described embodiments to be performed.

The present application also provides a computer program product comprising computer code or a computer program which, when run on a computer, causes the method of the above embodiments to be performed.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the above claims.

Claims (10)

1. A method of image registration, the method comprising:

acquiring an ultrasonic profile and a Computed Tomography (CT) profile of a target organ; the ultrasonic contour is obtained according to an ultrasonic image acquired by ultrasonic equipment, and the CT contour is obtained according to a CT image acquired by CT equipment;

rotating the CT outline to obtain a rotated CT outline; an included angle between a direction vector of a reference direction in the space of the ultrasound contour and a direction vector of the reference direction in the space of the rotated CT contour is smaller than or equal to a first threshold, the reference direction being common to the space of the ultrasound contour and the space of the CT contour;

registering based on the rotated CT profile and the ultrasound profile.

2. The method of claim 1, wherein the rotating the CT profile to obtain a rotated CT profile comprises:

determining a first direction vector of the reference direction in the space of the CT contour and determining a second direction vector of the reference direction in the space of the ultrasound contour;

determining a rotation matrix according to an included angle between the first direction vector and the second direction vector and an included angle between the second direction vector and the coordinate axis;

and rotating the CT outline based on the rotation matrix to obtain the rotated CT outline.

3. The method of claim 1, wherein the reference direction comprises a first sub-reference direction, a second sub-reference direction, and a third sub-reference direction; the direction vectors of the first sub-reference direction, the second sub-reference direction and the third sub-reference direction in the space of the ultrasound contour and the rotated CT contour are respectively less than or equal to the first threshold.

4. The method of claim 3, wherein the first sub-reference direction is a direction pointing from a foot to a head of a subject, the second sub-reference direction is a direction pointing vertically outward from a back of the subject, and the third sub-reference direction is a direction pointing from a body of the subject to a left hand.

5. The method according to any one of claims 1-4, wherein said registering based on the rotated CT profile and the ultrasound profile comprises:

performing the registering step based on the rotated CT profile and the ultrasound profile in case a distance between center coordinates of the rotated CT profile and center coordinates of the ultrasound profile is smaller than a second threshold.

6. The method of claim 5, further comprising:

under the condition that the distance between the center coordinate of the rotated CT contour and the center coordinate of the ultrasonic contour is larger than the second threshold value, translating the rotated CT contour to a position where the distance between the rotated CT contour and the ultrasonic contour is smaller than or equal to the second threshold value to obtain a rotationally translated CT contour;

registering the rotationally translated CT profile and the ultrasound profile.

7. The method according to any one of claims 1-5, wherein a similarity between the ultrasound profile and the CT profile is greater than or equal to a third threshold.

8. A method of image registration, the method comprising:

acquiring an ultrasonic profile and a Computed Tomography (CT) profile of a target organ; the ultrasonic contour is obtained according to an ultrasonic image acquired by ultrasonic equipment, and the CT contour is obtained according to a CT image acquired by CT equipment;

rotating the ultrasonic profile to obtain a rotated ultrasonic profile; an included angle between a direction vector of a reference direction in the space of the ultrasound contour and a direction vector of the reference direction in the space of the rotated CT contour is smaller than or equal to a first threshold value, and the reference direction is common to the space of the ultrasound contour and the space of the CT contour;

registering based on the rotated ultrasound contour and the CT contour.

9. An electronic device comprising a processor and a memory for storing a computer program comprising program instructions, the processor being configured to invoke the program instructions such that the method of any of claims 1-8 is performed.

10. A computer-readable storage medium, characterized in that it stores a computer program comprising program instructions which, when executed by a processor, cause the method according to any one of claims 1-8 to be performed.

Images