CN112884819A

CN112884819A - Image registration and neural network training method, device and equipment

Info

Publication number: CN112884819A
Application number: CN201911200249.6A
Authority: CN
Inventors: 何滨; 徐琦; 钱坤; 高连胜
Original assignee: Hangzhou Santan Medical Technology Co Ltd
Current assignee: Hangzhou Santan Medical Technology Co Ltd
Priority date: 2019-11-29
Filing date: 2019-11-29
Publication date: 2021-06-01
Anticipated expiration: 2039-11-29
Also published as: CN112884819B

Abstract

The application provides an image registration and neural network training method, device and equipment, wherein the method comprises the following steps: determining a two-dimensional image to be registered; inputting a two-dimensional image to be registered into a neural network; extracting the features of the two-dimensional image to be registered through a neural network, and determining the coordinates of projection points in the two-dimensional image to be registered according to the extracted features; and calculating the coordinates of the projection point in the two-dimensional image to be registered and the coordinates of the feature point of the three-dimensional bounding box of the current radiation object in the three-dimensional image according to the PNP algorithm so as to determine the pose information of the camera. By the technical scheme, the registration information of the two-dimensional image to be registered can be determined on the basis of accurately obtaining the characteristic information in the two-dimensional image to be registered, and the accuracy of the determined registration information is improved.

Description

Image registration and neural network training method, device and equipment

Technical Field

The application relates to the technical field of networks, in particular to a method, a device and equipment for training image registration and a neural network.

Background

With the rapid development of image imaging technology, various imaging devices are in a wide range, and meanwhile, due to the introduction of new imaging devices and the continuous improvement of existing imaging devices, the image imaging technology is widely applied in more and more industries.

Based on different imaging modes of different types of imaging equipment, images obtained by the different types of imaging equipment have different characteristics and application values. Therefore, how to integrate the advantages of various imaging devices and realize the comprehensive utilization of different types of images has a great research significance, wherein how to fuse images obtained in the same scene or similar scenes based on different imaging devices is a key process for acquiring the comprehensive utilization value of different types of images.

Disclosure of Invention

In view of this, the present application provides a method, an apparatus, and a device for training image registration and a neural network, which determine registration information of a two-dimensional image to be registered on the basis of accurately obtaining feature information in the two-dimensional image to be registered, so as to improve accuracy of the determined registration information.

In order to achieve the above purpose, the present application provides the following technical solutions:

according to a first aspect of the present application, an image registration method is provided, the method comprising:

determining a two-dimensional image to be registered, wherein the two-dimensional image to be registered comprises an imaging pattern of a current radiation object;

inputting the two-dimensional image to be registered into a neural network, wherein the neural network is trained by adopting a two-dimensional image sample comprising projection point coordinate marking information in advance, and the projection point coordinate marking information is the coordinates of projection points of feature points of a three-dimensional boundary box of a sample radiation object in the three-dimensional image in the two-dimensional image sample;

extracting features of the two-dimensional image to be registered through the neural network, and determining coordinates of projection points in the two-dimensional image to be registered according to the extracted features, wherein the projection points in the two-dimensional image to be registered correspond to feature points of a three-dimensional boundary frame of the current radiation object in the three-dimensional image;

and calculating the coordinates of the projection point in the two-dimensional image to be registered and the coordinates of the feature point of the three-dimensional bounding box of the current radiation object in the three-dimensional image according to a PNP algorithm so as to determine the pose information of the camera.

Optionally, the coordinates of the feature points of the three-dimensional bounding box are pre-labeled, and the feature points of the three-dimensional bounding box of the current radiation object in the three-dimensional image include: eight vertex coordinates of a three-dimensional boundary frame of the current radiation object in the three-dimensional image and at least three points in a central point coordinate; the characteristic points of the three-dimensional bounding box of the sample radiation object in the three-dimensional image comprise: and the sample radiation object comprises eight vertex coordinates of a three-dimensional boundary box in the three-dimensional image and at least three points in a central point coordinate.

Optionally, the two-dimensional image sample includes:

a two-dimensional reconstruction image corresponding to the three-dimensional image of the sample radiation object after the pose change is obtained; or, based on an actual two-dimensional image of the sample irradiation object acquired by an imaging device.

Optionally, the pose change includes:

changing the angle value of the sample irradiation object by a first preset value and/or changing the displacement value of the irradiation object by a second preset value based on a preset direction in a spatial coordinate system of the imaging device.

Optionally, the two-dimensional image to be registered is obtained when the projection distance of the imaging device satisfies a preset value.

According to a second aspect of the present application, a neural network training method for image registration is provided, the method comprising:

inputting a two-dimensional image sample serving as a training sample into a neural network, wherein the training sample comprises projection point coordinate marking information corresponding to a two-dimensional image to be registered, and the projection point coordinate marking information is coordinates of projection points of feature points of a three-dimensional boundary frame of a sample radiation object in the three-dimensional image in the two-dimensional image sample;

performing feature extraction on an input two-dimensional image sample through the neural network, and determining predicted projection point coordinate information corresponding to the two-dimensional image sample according to the extracted features;

determining the difference between the predicted projection point coordinate information and the projection point coordinate marking information;

network model parameters of the neural network are adjusted based on the difference.

Optionally, the coordinates of the feature points of the three-dimensional bounding box are pre-labeled, and the feature points of the three-dimensional bounding box of the sample radiation object in the three-dimensional image include: and the sample radiation object comprises eight vertex coordinates of a three-dimensional boundary box in the three-dimensional image and at least three points in a central point coordinate.

Optionally, the two-dimensional image sample as the training sample includes:

Optionally, the pose change includes:

Optionally, the two-dimensional image sample is obtained when the projection distance of the imaging device satisfies a preset value.

According to a third aspect of the present application, there is provided an image registration apparatus, the apparatus comprising:

the device comprises a determining unit, a judging unit and a judging unit, wherein the determining unit is used for determining a two-dimensional image to be registered, and the two-dimensional image to be registered comprises an imaging pattern of a current radiation object;

the input unit is used for inputting the two-dimensional image to be registered into a neural network, wherein the neural network is trained by adopting a two-dimensional image sample comprising projection point coordinate marking information in advance, and the projection point coordinate marking information is the coordinates of projection points of characteristic points of a three-dimensional boundary frame of a sample radiation object in the three-dimensional image in the two-dimensional image sample;

the characteristic extraction unit is used for extracting the characteristics of the two-dimensional image to be registered through the neural network and determining the coordinates of projection points in the two-dimensional image to be registered according to the extracted characteristics, wherein the projection points in the two-dimensional image to be registered correspond to the characteristic points of a three-dimensional boundary frame of the current radiation object in the three-dimensional image;

and the calculating unit is used for calculating the coordinates of the projection point in the two-dimensional image to be registered and the coordinates of the feature point of the three-dimensional bounding box of the current radiation object in the three-dimensional image according to a PNP algorithm so as to determine the pose information of the camera.

Optionally, the two-dimensional image sample includes:

Optionally, the pose change includes:

According to a fourth aspect of the present application, a neural network training device for image registration is provided, the device comprising:

the system comprises an input unit, a neural network and a control unit, wherein the input unit at least inputs a two-dimensional image sample serving as a training sample into the neural network, the training sample comprises projection point coordinate marking information corresponding to a to-be-registered two-dimensional image, and the projection point coordinate marking information is the coordinates of projection points of feature points of a three-dimensional boundary box of a sample radiation object in the three-dimensional image in the two-dimensional image sample;

the characteristic extraction unit is used for extracting the characteristics of the input two-dimensional image sample through the neural network and determining the coordinate information of the predicted projection point corresponding to the two-dimensional image sample according to the extracted characteristics;

the determining unit is used for determining the difference between the predicted projection point coordinate information and the projection point coordinate marking information;

a parameter adjusting unit that adjusts network model parameters of the neural network based on the difference.

Optionally, the two-dimensional image sample as the training sample includes:

Optionally, the pose change includes:

Optionally, the determining unit and the parameter adjusting unit are specifically configured to:

determining a loss function corresponding to a regression model containing the network model parameters; and determining network model parameters when the loss function obtains the minimum value based on the loss function, the predicted projection point coordinate information and the projection point coordinate marking information.

Optionally, the loss function is

Wherein, N is 9,

in order to predict the proxel coordinate information,

and marking information for the coordinates of the projection points.

According to a fifth aspect of the present application, there is provided an electronic device comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to execute instructions to implement the method of the first aspect.

According to a sixth aspect of the present application, a computer-readable storage medium is proposed, on which computer instructions are stored, which instructions, when executed by a processor, carry out the steps of the method according to the first aspect described above.

According to a seventh aspect of the present application, there is provided an electronic device comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to execute instructions to implement the method of the second aspect.

According to an eighth aspect of the present application, a computer-readable storage medium is proposed, on which computer instructions are stored, which instructions, when executed by a processor, carry out the steps of the method according to the second aspect described above.

According to the technical scheme, the determined to-be-registered two-dimensional image containing the imaging pattern of the current radiation object is subjected to feature extraction through the neural network, and then the coordinates of the projection points in the to-be-registered two-dimensional image are determined according to the extracted features, so that the accuracy and robustness of determining the coordinates of the projection points formed in the to-be-registered two-dimensional image are improved, the accuracy of the determined pose data is correspondingly improved for determining the pose data of the camera according to the PNP algorithm, and the stability of tracking and determining the pose data of the camera is enhanced.

Drawings

Fig. 1 is a flowchart of an image initial registration method according to an exemplary embodiment of the present application;

FIG. 2 is a flow chart of a neural network training method for image registration in accordance with an exemplary embodiment of the present application;

FIG. 3 is a flow chart of another image registration method according to an exemplary embodiment of the present application;

FIG. 4 illustrates a schematic diagram of determining camera pose information according to one of the exemplary embodiments of the present application;

FIG. 5 is a flow chart of another neural network training method for image registration in an exemplary embodiment according to the present application;

FIG. 6 is a schematic block diagram of an electronic device in an exemplary embodiment in accordance with the subject application;

fig. 7 is a block diagram of an image registration apparatus according to an exemplary embodiment of the present application;

FIG. 8 is a schematic block diagram of another electronic device in an exemplary embodiment in accordance with the subject application;

fig. 9 is a block diagram of a neural network training device for image registration according to an exemplary embodiment of the present application.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.

With the rapid development of image forming technology, the introduction of new image forming apparatuses and the continued improvement of existing image forming apparatuses have led to the widespread use of image forming technology in more and more industries. The imaging modes based on different types of imaging equipment are different, and images obtained by the different types of imaging equipment have different characteristics and application values, such as high density resolution based on CT (computed tomography), so that the CT images can better display organs consisting of soft tissues, and certainly, the CT images are very clear for imaging bones; the nuclear magnetic resonance utilizes a strong magnetic field to vibrate the water in the body, and forms images by utilizing the vibration difference of the water in different tissues, so that the MRI images have a clearer imaging effect on soft tissues; the ultrasonic image has the advantages of strong real-time performance, no radiation and the like on the basis of ensuring high resolution and strong contrast of the soft tissue structure.

In view of the fact that the images obtained by different types of imaging devices have different application values, in the practical application process, the images obtained based on a plurality of imaging devices are used comprehensively, and there is an urgent need for the fusion of different types of image information, therefore, an image registration technology has come to be developed, and a plurality of images obtained based on different imaging angles and different imaging devices of the same object can be subjected to one (or a series of) spatial transformation through image registration, so that the plurality of images are spatially consistent with respect to the same corresponding point, and the registration technology is widely applied to aspects of disease diagnosis, surgical navigation, human brain atlas, various medical evaluations and the like, and typical image registration is as follows: the registration of CT images and PET images, the registration of CT images and MRI images, the registration of ultrasonic images and CT images, the registration of fMRI image sequences, the registration of different MR weighted images and the like.

In order to improve the accuracy of the registration information before the image, the application provides an image registration and neural network training method, device and equipment. The technical solution of the present application will be described in detail by specific examples.

Referring to fig. 1, fig. 1 is a flowchart illustrating an initial image registration method according to an exemplary embodiment of the present application, as shown in fig. 1, the method may include the following steps:

step 101, determining a two-dimensional image to be registered, wherein the two-dimensional image to be registered comprises an imaging pattern of a current radiation object.

In the present application, an imaging pattern of a current radiation object may be represented by a two-dimensional image to be registered, where the current radiation object may be the same radiation object as a sample radiation object serving as a training sample, or the current radiation object may also be a radiation object belonging to the same class as the sample radiation object serving as the training sample.

In an actual application process, the two-dimensional image to be registered may be obtained when a projection distance of the imaging device satisfies a preset value, where the preset value may be parameter information of the imaging device when the imaging device leaves a factory, or may be a distance between a camera of the imaging device and a current radiation object, and then adjust the projection distance of the imaging device according to a result of the real-time measurement, so that the projection distances of the imaging device when the two-dimensional image to be registered or each two-dimensional image sample is obtained are kept consistent.

In an embodiment, the determined two-dimensional image to be registered may be an actual two-dimensional image or a two-dimensional reconstructed image, where the actual two-dimensional image may be a two-dimensional image actually determined based on irradiation of the radiation object by the imaging device; the two-dimensional reconstructed images can be a plurality of two-dimensional images determined by performing pose transformation on the radiation object, so that a three-dimensional image based on pose data after transformation is determined through digital simulation, a plurality of groups of two-dimensional reconstructed images are determined through the pose transformation of the radiation object and added into a training sample of the neural network, the regularization of the sample used for training the neural network model is improved, and the problem of over-fitting or under-fitting of the trained neural network caused by a specific sample is solved.

And 102, inputting the two-dimensional image to be registered into a neural network, wherein the neural network is trained by adopting a two-dimensional image sample comprising projection point coordinate marking information in advance, and the projection point coordinate marking information is the coordinates of the projection points of the characteristic points of the three-dimensional bounding box of the sample radiation object in the three-dimensional image in the two-dimensional image sample.

In one embodiment, the coordinates of the feature points of the three-dimensional bounding box may be pre-labeled, specifically, at least three points of eight vertex coordinates and one center point coordinate of the three-dimensional bounding box of the sample radiation object in the three-dimensional image.

In another embodiment, a two-dimensional image sample treasury serving as a training sample reconstructs a two-dimensional reconstruction image corresponding to a three-dimensional image of a sample radiation object after pose change; or based on actual two-dimensional images of the sample irradiation object acquired by the imaging device.

Further, the angle value of the sample irradiation object is changed by a first preset value, the displacement value of the irradiation object is changed by a second preset value, or the angle value of the sample irradiation object is changed by the first preset value and the displacement value of the irradiation object is changed by the second preset value along a preset direction based on a preset direction in a spatial coordinate system of the imaging apparatus.

103, extracting features of the two-dimensional image to be registered through the neural network, and determining coordinates of projection points in the two-dimensional image to be registered according to the extracted features, wherein the projection points in the two-dimensional image to be registered correspond to feature points of a three-dimensional boundary frame of the current radiation object in the three-dimensional image.

And 104, calculating the coordinates of the projection point in the two-dimensional image to be registered and the coordinates of the feature point of the three-dimensional bounding box of the current radiation object in the three-dimensional image according to a PNP algorithm to determine the pose information of the camera.

The three-dimensional coordinates of the N characteristic points in the three-dimensional frame of the known current radiation object in the three-dimensional image can be understood as coordinates in a world coordinate system, coordinates of a projection point of the current radiation object in an imaging pattern of the two-dimensional image to be registered predicted based on the neural network are coordinates in a camera coordinate system, and then the coordinates of the projection point in the two-dimensional image to be registered and the coordinates of the characteristic points of the three-dimensional boundary frame of the current radiation object in the three-dimensional image can be calculated according to a PNP algorithm to obtain pose information of the camera. Further, the feature points of the three-dimensional bounding box of the current radiation object may be pre-labeled, specifically, at least three points in eight vertex coordinates and one center point coordinate of the three-dimensional bounding box of the current radiation object in the three-dimensional image.

By the embodiment, the determined to-be-registered two-dimensional image containing the imaging pattern of the current radiation object is subjected to feature extraction through the neural network, and then the coordinates of the projection points in the to-be-registered two-dimensional image are determined according to the extracted features, so that the accuracy and robustness of determining the coordinates of the projection points formed in the to-be-registered two-dimensional image are improved, the accuracy of the determined pose data is correspondingly improved for determining the pose data of the camera according to the PNP algorithm, and the stability of tracking and determining the pose data of the camera is enhanced.

Fig. 2 is a flowchart of a neural network training method for image registration according to an exemplary embodiment of the present application, and as shown in fig. 2, the method may include the following steps:

step 201, inputting at least a two-dimensional image sample as a training sample into a neural network, wherein the training sample comprises projection point coordinate marking information corresponding to a two-dimensional image to be registered, and the projection point coordinate marking information is coordinates of projection points of feature points of a three-dimensional bounding box of a sample radiation object in the three-dimensional image in the two-dimensional image sample.

In one embodiment, the feature point coordinates of the three-dimensional bounding box may be pre-labeled. Further, the characteristic points of the three-dimensional bounding box of the sample radiation object in the three-dimensional image may be at least three points in eight vertex coordinates and one center point coordinate of the three-dimensional bounding box of the sample radiation object in the three-dimensional image.

In another embodiment, the two-dimensional image sample for constructing the sample data may be a two-dimensional image obtained by performing pose transformation on the radiation exclusive object through digital simulation, the two-dimensional image sample may be based on a two-dimensional image of a sample radiation object acquired by an imaging device, and specifically, the two-dimensional image sample may be obtained when a projection distance of the imaging device satisfies a preset value.

In the process of obtaining the sample radiation object, the position and posture of the sample radiation object can be transformed, corresponding two-dimensional images are further determined according to the radiation object after the position and posture transformation, a plurality of groups of two-dimensional reconstruction images are determined through the position and posture transformation of the radiation object, the training samples of the neural network are added, the regularization of the samples used for training the neural network model is improved, and the problem that the trained neural network is over-fitted or under-fitted due to specific samples is solved.

Further, the pose of the sample radiation object is transformed into a rigid body transformation, and the specific pose transformation mode may include changing the angle value of the radiation object by a first preset value, changing the displacement value of the radiation object by a second preset value along a preset direction, or changing the displacement value of the radiation object by the second preset value along the preset direction and changing the rotation angle value by the first preset value along the preset direction based on a preset direction in a spatial coordinate system of the imaging device.

Step 202, performing feature extraction on the input two-dimensional image sample through the neural network, and determining predicted projection point coordinate information corresponding to the two-dimensional image sample according to the extracted features.

Step 203, determining the difference between the predicted projection point coordinate information and the projection point coordinate marking information.

Step 204, network model parameters of the neural network are adjusted based on the difference.

In an embodiment, the constructed neural network regression model may be composed of several convolution layers, pooling layers, full-link layers, and the like, so that the neural network extracts image features based on the several convolution layers, pooling layers, and the like. In the practical application process, the number of input layers, hidden layers and input layers in the neural network regression model may be set, such as a perceptron neural network including an input layer with 8 neurons, a hidden layer with 12 neurons and an output layer with 6 neurons may be constructed, but the application does not limit the specific number of input layers, hidden layers and output layers, and it is understood that the neural network regression model including an input layer with 10 neurons, a hidden layer with 14 neurons and an output layer with 6 neurons is also applicable to the technical solution of the application.

And extracting features of the contour information in the two-dimensional image which is input by the neural network and used as a training sample, and outputting predicted registration information corresponding to the contour information in the two-dimensional image according to the extracted features. Specifically, feature extraction may be performed on the contour information of the input two-dimensional image through a combination of a plurality of convolutional layers and a plurality of pooling layers, that is, a plurality of fully-connected layers in the neural network.

The prediction registration information corresponding to the contour information of the pattern in the input two-dimensional image can be obtained after calculation of the neural network, the difference between the prediction registration information and the pre-labeled actual registration information can be determined based on the pre-labeled registration information in the two-dimensional image sample, the loss value of the loss function can be determined according to the difference between the pre-labeled actual registration information and the prediction registration information in the two-dimensional image sample, and then the determined loss value is reversely transmitted back to the neural network, so that the network parameters of the neural network are adjusted according to the received loss value, and the adjusted network parameters such as the weight of a full-connection layer in the neural network, the value of a convolution kernel and the like.

The loss function used to determine the loss value may be predetermined, and the selected loss function may be

Wherein, N is 9,

in order to predict the proxel coordinate information,

and marking information for the coordinates of the projection points. The neural network can reduce the loss value by adjusting the network parameters in the network structure after receiving the loss value, and determines that the training of the network parameters of the neural network is finished under the condition that a preset training finishing condition is reached, wherein the preset training finishing condition can be that the loss value obtained based on the determined loss function reaches a minimum value, or the network parameter optimization iteration number reaches a preset threshold value.

In order to explain the technical scheme of the present application in detail, the following two embodiments are used to explain the technical scheme of the present application in detail:

fig. 3 is a flowchart of another image registration method according to an exemplary embodiment of the present application, and as shown in fig. 3, the method may include the following steps:

step 301, determining a two-dimensional image to be registered including an imaging pattern of a current irradiation object.

An imaging pattern in the two-dimensional image to be registered, which characterizes a current radiation object, wherein the current radiation object may be the same radiation object as a sample radiation object as a training sample, such as the current radiation object and a left lower leg of a same patient a as the sample radiation object; similarly, the current radiation object may also be a radiation object belonging to the same class as the sample radiation object serving as the training sample, such as the chest cavity of both the current radiation object and the sample radiation object, and accordingly, the two-dimensional image to be registered may be an X-ray image including the chest cavity of the person, and in this application scenario, an imaging pattern presented by the X-ray image is a chest cavity pattern of the person. It is to be understood that the current irradiation target and the sample irradiation target may be any targets, and the present application does not limit the specific content of the current irradiation target and the sample irradiation target or the specific representation form of the two-dimensional image to be registered.

In an embodiment, the two-dimensional image to be registered is obtained when a projection distance of the imaging device satisfies a preset value, the projection distance of the imaging device may be parameter information of the imaging device when the imaging device leaves a factory, and the parameter information reflects an optimal imaging distance of a current radiation object compared with the imaging device when the two-dimensional image is obtained; or before the two-dimensional image to be registered is obtained, measuring the distance between the camera of the imaging device and the current irradiation object in real time, comparing the measured real-time information with the imaging distance when the two-dimensional image sample used for training the neural network is determined, and adjusting the distance between the camera of the imaging device for real-time measurement and the current irradiation object according to the comparison result so as to ensure that the two-dimensional image to be registered is obtained based on the projection distance between the camera of the imaging device and the sample irradiation object when the two-dimensional image sample is determined.

Step 302, determining a neural network trained in advance by a two-dimensional image sample containing the coordinate marking information of the projection point.

The two-dimensional image sample used for training the neural network can be an actual two-dimensional image or a two-dimensional reconstruction image, and the actual two-dimensional image is a two-dimensional image actually determined based on irradiation of the imaging equipment on the radiation object; the two-dimensional reconstructed images can be a plurality of two-dimensional images determined by performing pose transformation on the radiation object, so that a three-dimensional image based on pose data after transformation is determined through digital simulation, a plurality of groups of two-dimensional reconstructed images are determined through pose transformation of the radiation object and added into a training sample of the neural network, regularization of the sample used for training the neural network model is improved, and the problem of overfitting or under-fitting of the trained neural network caused by a specific sample is solved.

Specifically, in the process of using the actual two-dimensional image as the two-dimensional reconstructed image of the training sample, a two-dimensional reconstructed image matched with the actual two-dimensional image can be screened out from the two-dimensional reconstructed image set based on a similarity algorithm, and the pose data of the screened two-dimensional reconstructed image is determined as the pose data information of the three-dimensional image of the radiation object represented by the pattern in the actual two-dimensional image; in another embodiment, the pose data of the three-dimensional image of the radiation object may be determined based on the actual two-dimensional image of the sample radiation object actually acquired by the imaging device when the actual two-dimensional image is acquired.

The pose transformation process of the radiation object can be as follows: based on a preset direction in a space coordinate system of the imaging device, the angle value of the irradiation object is changed by a first preset value, the displacement value of the irradiation object is changed by a second preset value along the preset direction, or the displacement value of the irradiation object is changed by the second preset value along the preset direction and the rotation angle value is changed by the first preset value along the preset direction.

In the practical application process, the pose data of the radiation object can be displacement values or rotation angle values along a single direction of coordinate axes x, y and z. Using t in this application^x、t^y、t^zRespectively representing the offset, theta, in three directions relative to the x, y, z coordinate axes^x、θ^y、θ^zRespectively representing the rotation angle values relative to the x, y and z coordinate axes, the t and the t can be set^x、t^y、t^zAnd theta^x、θ^y、θ^zAnd carrying out rigid body transformation on the radiation object according to the related change rule, and uniquely marking the three-dimensional image of the radiation object after different rigid body transformations and the two-dimensional image corresponding to the three-dimensional image according to the displacement value and the rotation angle value after the rigid body transformation, namely taking the pose data of the radiation object corresponding to the three-dimensional image after the rigid body transformation of the radiation object as actual registration marking information. The specific transformation process of the irradiation target is explained in detail in the embodiment corresponding to fig. 4, and is not described in this embodiment.

The training sample used for training the neural network comprises a two-dimensional image sample containing projection point coordinate marking information, wherein the projection point coordinate marking information is the coordinates of the projection points of the characteristic points of the three-dimensional boundary box of the sample radiation object in the three-dimensional image in the two-dimensional image sample.

Step 303, inputting the determined two-dimensional image to be registered into the trained neural network, so as to determine the coordinates of the projection point in the two-dimensional image to be registered based on the features extracted from the two-dimensional image to be registered by the neural network.

Inputting the two-dimensional image to be registered of the current radiation object into the trained neural network, performing feature extraction on the input intersection point information through the pre-trained neural network, and further determining the coordinates of the projection points of the feature points of the three-dimensional bounding box of the current radiation object in the three-dimensional image in the two-dimensional image to be registered according to the extracted features.

And step 304, determining the three-dimensional coordinates of the characteristic points of the three-dimensional bounding box of the current irradiation object in the three-dimensional image.

In the application, the coordinates of the feature points of the three-dimensional bounding box of the radiation object in the three-dimensional image may be pre-labeled, that is, the coordinates of the feature points of the three-dimensional bounding box of the radiation object in the three-dimensional image are read and labeled in real time when the three-dimensional image of the radiation object is acquired.

The determined characteristic points of the three-dimensional bounding box of the radiation object in the three-dimensional image can be eight vertex coordinates and one central point coordinate of the three-dimensional bounding box of the radiation object in the three-dimensional image; or the determined characteristic points can also be at least three points in eight vertex coordinates and a central point coordinate of a three-dimensional bounding box of the radiation object in the three-dimensional image.

And 305, calculating the coordinates of the projection point in the two-dimensional image to be registered and the coordinates of the feature point of the three-dimensional bounding box of the current radiation object in the three-dimensional image according to a PNP algorithm to determine the pose information of the camera when the two-dimensional image is acquired.

The three-dimensional coordinates of the N characteristic points in the three-dimensional frame of the known current radiation object in the three-dimensional image can be understood as coordinates in a world coordinate system, coordinates of a projection point of the current radiation object in an imaging pattern of the two-dimensional image to be registered predicted based on the neural network are coordinates in a camera coordinate system, and then the coordinates of the projection point in the two-dimensional image to be registered and the coordinates of the characteristic points of the three-dimensional boundary frame of the current radiation object in the three-dimensional image can be calculated according to a PNP algorithm to obtain pose information of the camera.

In practical application, the obtained pose information of the camera can be simplified and expressed by an external parameter matrix [ R | t ] of the camera, wherein R is a rotation matrix, t is a translation matrix, and the rotation matrix R can be determined based on a Rodrigue formula, namely, the rotation in a three-dimensional space can be expressed by a combination form of a rotation angle and a vector. Accordingly, if the coordinate corresponding to the world coordinate system is X and the coordinate corresponding to the camera coordinate system is X ', the obtained pose information of the camera satisfies X' [ [ R | t ] ] X, as shown in fig. 4, fig. 4 is a schematic diagram for determining the pose information of the camera according to one exemplary embodiment of the present application, assuming that the three-dimensional coordinates of the point a, the point B, and the point C in the diagram are determined based on the world coordinate system, the point a, the point B, and the point C in the two-dimensional image m are determined based on the camera coordinate, and the point a is a projected point of the point a in the two-dimensional image m, the point B is a projected point of the point B in the two-dimensional image m, and the point C is a projected point of the point C in the two-dimensional image m, that is, the triangle ABC in the two-dimensional image m of the radiation object, the point a, the point B, and the triangle ABC in the two-dimensional image m are formed by the P3P algorithm, And (4) calculating the three-dimensional coordinates of the point B and the point C and the two-dimensional coordinates of the point a, the point B and the point C in the two-dimensional image m to determine the pose information of the camera P, namely the external parameter matrix [ R | t ] of the camera P.

In the embodiment, the coordinates of the projection points formed by the feature points of the three-dimensional frame of the current radiation object in the two-dimensional image to be registered are determined through the neural network trained according to the two-dimensional image sample containing the projection point coordinate marking information, so that the accuracy and robustness of determining the coordinates of the projection points formed in the two-dimensional image to be registered are improved, the accuracy of determining the pose data of the camera according to the PNP algorithm is correspondingly improved, and the stability of tracking and determining the pose data of the camera is enhanced.

Fig. 5 is a flowchart of another neural network training method for image registration according to an exemplary embodiment of the present application, and as shown in fig. 5, the method may include the following steps:

step 501, constructing sample data of a two-dimensional image sample containing the coordinate marking information of the projection point.

The method can be used for constructing sample data containing a two-dimensional image sample according to a two-dimensional reconstructed image or an actual two-dimensional image, wherein the two-dimensional reconstructed image is a two-dimensional image obtained by performing digital simulation on a radiation object after pose transformation, the actual two-dimensional image is a two-dimensional image of the sample radiation object obtained based on imaging equipment, and specifically, the two-dimensional image sample can be obtained under the condition that the projection distance of the imaging equipment meets a preset numerical value.

Specifically, in the process of using the actual two-dimensional image as the two-dimensional reconstructed image of the training sample, the two-dimensional reconstructed image matched with the actual two-dimensional image can be screened out from the two-dimensional reconstructed image set based on the similarity algorithm, and the pose data of the screened two-dimensional reconstructed image is determined as the pose data information of the three-dimensional image of the radiation object represented by the pattern in the actual two-dimensional image. When the two-dimensional image in the training sample is the actual two-dimensional image, the pose data of the three-dimensional image of the radiation object can be determined when the actual two-dimensional image is obtained.

Further, the pose of the sample radiation object is transformed into a rigid body transformation, and a specific pose transformation mode may include changing an angle value of the radiation object by a first preset value, changing a displacement value of the radiation object by a second preset value along a preset direction or changing a displacement value of the radiation object by a second preset value along a preset direction and changing a rotation angle value by a first preset value along a preset direction based on a preset direction in a spatial coordinate system of the imaging device, such as sampling the radiation object once every 3mm of translation along the preset direction, sampling once every 2 ° of rotation of the angle value along the preset direction, or sampling once after translating the angle value by 3mm along the preset direction and rotating the angle value by 2 °, which is easy to understand that the present application does not limit an offset direction and an offset amount, a rotation direction and a rotation angle value adopted in practical application.

In the practical application process, the pose data of the radiation object can be displacement values or rotation angle values along a single direction of coordinate axes x, y and z. Using t in this application^x、t^y、t^zRespectively representing the offset, theta, in three directions relative to the x, y, z coordinate axes^x、θ^y、θ^zIndicating the rotation angle values relative to the x, y, z coordinate axes, respectively, and in particular, the values can be used

And the offset of the x axis, the offset of the y axis, the offset of the z axis, the value of a rotation angle relative to the x axis, the value of a rotation angle relative to the y axis and the value of a rotation angle relative to the z axis after the change according to a preset rule are shown.

Accordingly, can be set by

The rigid body transformation of the irradiation object is realized, for example, the irradiation object is rotated only along the x-axis

In the case of (a), it can be obtained that the three-dimensional coordinates of a certain point on the irradiation target change from the original (x, y, z) to (x ', y', z '), and the following mapping relationship is satisfied between (x, y, z) and (x', y ', z'):

therein

To achieve a homogeneous transformation matrix of the irradiation object from the original coordinates (x, y, z) to (x ', y ', z ').

Similarly, if the irradiation target is rotated only along the y-axis

therein

To realize homogeneous transformation matrix of the irradiation object from the original three-dimensional coordinate (x, y, z) to (x ', y ', z ').

Similarly, if the irradiation target is rotated only along the z-axis

therein

Similarly, if the irradiation target is translated along the x, y, z axes, such as translating the irradiation target along the x axis

Translation along the y-axis

Translation along z-axis

The three-dimensional coordinate of a certain point on the irradiation object will be changed from the original (x, y, z) to (x ', y', z '), and the following mapping relationship is satisfied between (x, y, z) and (x', y ', z'), namely:

therein

By combining different rotation orders based on the rotation of the x-axis, the y-axis and the z-axis, different rotation situations in 6 can be obtained, the irradiation object is converted by sequentially rotating and translating according to the x-axis, the y-axis and the z-axis, the three-dimensional coordinate of a certain point on the irradiation object can be obtained by changing the original (x, y, z) into (x ', y', z '), and the following mapping relation is satisfied between (x, y, z) and (x', y ', z'):

therein

In order to realize the homogeneous transformation matrix of the irradiation object changed from the original three-dimensional coordinates (x, y, z) to (x ', y ', z '), in order to make the representation have uniformity, the variation rules possibly involved in the preset rules are uniformly distributed by the homogeneous transformation matrix under the coordinate system of the imaging equipment

Indicating, where R is a rotational transformation matrix of 3 x 3, T is a translation vector of 3 x 1, and 0 is a zero vector of 1 x 3.

For the three-dimensional image of the radiation object changed according to a plurality of preset rules, a two-dimensional reconstruction image matched with the three-dimensional image and the pose data of the three-dimensional image can be determined, and then a sample data set of a plurality of two-dimensional reconstruction images marked with the pose data can be determined.

And carrying out data splitting on the constructed sample set to obtain a training set, a testing set and a verification set corresponding to the constructed sample set.

In one embodiment, after the sample set is constructed, the amount of data for constructing the training set, the verification set, and the test set may be randomly selected from the created sample data set according to a certain ratio, such as: 80% of samples can be randomly selected from the established sample data set as a training set, 10% of samples can be selected as a verification set, and the rest 10% of samples can be selected as a test set.

In another embodiment, the data set may be divided into n parts, where n is usually 5 or 10 in practical applications, and then one part of the n parts is taken as the test set and the other (n-1) parts are taken as the training sets of the training models at a time without repeating the above steps.

In practical application, a sample set as a training set is mainly used for training parameters in a neural network; after the neural network is trained on the basis of the training set, the performance of each model can be compared and judged through the verification set, and the training optimization of the hyper-parameters which cannot be optimized on the basis of the training set can be realized by means of the verification set; and obtaining the evaluation index of the neural network model based on the sample of the test set.

Step 502, a neural network model for feature extraction of a two-dimensional image sample is constructed.

Furthermore, the number of training rounds and the batch size of the neural network can be set, the batch size of the constructed neural network model is not easy to be too small in order to ensure the accuracy of the training result, and the batch size of the constructed neural network model is not easy to be too large in order to ensure the training efficiency during the neural network training, so that the situation that the running speed of the equipment is low or even down due to too large batch size is avoided. As an exemplary setting, the number of training rounds may be 20000, and the number of batches of the neural network model may be set to 10, and it should be understood that the application also does not limit the training books and the number of batches for constructing the neural network.

And 503, performing feature extraction on the two-dimensional image sample serving as the training sample through the constructed neural network model.

In step 504, the neural network outputs the predicted projection point coordinate information corresponding to the two-dimensional image sample determined based on the extracted features.

And 505, adjusting network model parameters of the neural network based on the difference between the predicted projection point coordinate information and the projection point coordinate marking information.

Wherein, N is 9,

in order to predict the proxel coordinate information,

After training of the neural network training model is completed, optimization of model selection and adjustment of hyper-parameters can be performed through the verification set, and further, the fitting regression effect of the model can be evaluated by means of the test set. In the practical application process, aiming at one of the data set splitting modes, namely splitting the data set used as the training set and the test set into n parts, then repeatedly taking one part of the data set as the test set and taking the other (n-1) parts as the training set of the training model, correspondingly determining the k-time evaluation results, and further taking the performance average value of the k-time evaluation as the final evaluation result.

FIG. 6 is a schematic block diagram of an electronic device in an exemplary embodiment in accordance with the present application. Referring to fig. 6, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, but may also include hardware required for other services. The processor reads a corresponding computer program from the nonvolatile memory into the memory and then runs the computer program to form the image registration device on a logic level. Of course, besides the software implementation, the present application does not exclude other implementations, such as logic devices or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, and may also be hardware or logic devices.

Referring to fig. 7, fig. 7 is a block diagram of an image registration apparatus according to an exemplary embodiment of the present application, and in a software implementation, the image registration apparatus may include:

a determining unit 701, configured to determine a two-dimensional image to be registered, where the two-dimensional image to be registered includes an imaging pattern of a current radiation object;

the input unit 702 is configured to input the two-dimensional image to be registered into a neural network, where the neural network is trained in advance by using a two-dimensional image sample including projection point coordinate marking information, and the projection point coordinate marking information is coordinates of projection points of feature points of a three-dimensional bounding box of a sample radiation object in the three-dimensional image in the two-dimensional image sample;

the feature extraction unit 703 is configured to perform feature extraction on the two-dimensional image to be registered through a neural network, and determine coordinates of a projection point in the two-dimensional image to be registered according to the extracted features, where the projection point in the two-dimensional image to be registered corresponds to a feature point of a three-dimensional bounding box of the current radiation object in the three-dimensional image;

and the calculating unit 704 calculates the coordinates of the projection point in the two-dimensional image to be registered and the coordinates of the feature point of the three-dimensional bounding box of the current radiation object in the three-dimensional image according to the PNP algorithm so as to determine the pose information of the camera.

Optionally, the feature point coordinates of the three-dimensional bounding box are pre-labeled, and the feature points of the three-dimensional bounding box of the current radiation object in the three-dimensional image include:

eight vertex coordinates of a three-dimensional boundary frame of the current radiation object in the three-dimensional image and at least three points in a central point coordinate; the characteristic points of the three-dimensional bounding box of the sample radiation object in the three-dimensional image comprise: and the sample radiation object comprises eight vertex coordinates of a three-dimensional boundary box in the three-dimensional image and at least three points in a central point coordinate.

Optionally, the two-dimensional image sample includes:

a two-dimensional reconstruction image corresponding to the three-dimensional image of the sample radiation object after the pose change is obtained; or,

based on the actual two-dimensional image of the sample irradiation object acquired by the imaging device.

Optionally, the pose change includes:

FIG. 8 is a schematic block diagram of another electronic device in an exemplary embodiment in accordance with the present application. Referring to fig. 8, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, but may also include hardware required for other services. The processor reads a corresponding computer program from the nonvolatile memory into the memory and then runs the computer program to form the neural network training device for image registration on a logic level. Of course, besides the software implementation, the present application does not exclude other implementations, such as logic devices or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, and may also be hardware or logic devices.

Referring to fig. 9, fig. 9 is a block diagram of a neural network training device for image registration according to an exemplary embodiment of the present application, and in a software implementation, the neural network training device for image registration may include:

an input unit 901, which inputs at least a two-dimensional image sample as a training sample into a neural network, wherein the training sample includes projection point coordinate labeling information corresponding to a to-be-registered two-dimensional image, and the projection point coordinate labeling information is coordinates of projection points of feature points of a three-dimensional bounding box of a sample radiation object in the three-dimensional image in the two-dimensional image sample;

a feature extraction unit 902, configured to perform feature extraction on an input two-dimensional image sample through the neural network, and determine predicted projection point coordinate information corresponding to the two-dimensional image sample according to the extracted features;

a determining unit 903, which determines a difference between the predicted projection point coordinate information and the projection point coordinate marking information;

a parameter adjusting unit 904, which adjusts network model parameters of the neural network based on the difference.

Optionally, the two-dimensional image sample as the training sample includes:

Optionally, the pose change includes:

Optionally, the determining a difference between the predicted projection point coordinate information and the projection point coordinate labeling information, and adjusting a network model parameter of the neural network based on the difference includes:

determining a loss function corresponding to a regression model containing the network model parameters;

and determining network model parameters when the loss function obtains the minimum value based on the loss function, the predicted projection point coordinate information and the projection point coordinate marking information.

Optionally, the loss function is

Wherein, N is 9,

in order to predict the proxel coordinate information,

and marking information for the coordinates of the projection points.

The device corresponds to the method, and more details are not repeated.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the application. One of ordinary skill in the art can understand and implement it without inventive effort.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. In other instances, features described in connection with one embodiment may be implemented as discrete components or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. An image registration method, comprising:

2. The method as claimed in claim 1, wherein the coordinates of the feature points of the three-dimensional bounding box are pre-labeled, and the feature points of the three-dimensional bounding box of the current radiation object in the three-dimensional image comprise: eight vertex coordinates of a three-dimensional boundary frame of the current radiation object in the three-dimensional image and at least three points in a central point coordinate; the characteristic points of the three-dimensional bounding box of the sample radiation object in the three-dimensional image comprise: and the sample radiation object comprises eight vertex coordinates of a three-dimensional boundary box in the three-dimensional image and at least three points in a central point coordinate.

3. The method of claim 1, wherein the two-dimensional image sample comprises:

4. The method of claim 3, wherein the pose change comprises:

5. The method according to claim 1, wherein the two-dimensional image to be registered is obtained when a projection distance of an imaging device satisfies a preset value.

6. A neural network training method for image registration, the method comprising:

7. The method as claimed in claim 6, wherein the coordinates of the feature points of the three-dimensional bounding box are pre-labeled, and the feature points of the three-dimensional bounding box of the sample radiation object in the three-dimensional image comprise: and the sample radiation object comprises eight vertex coordinates of a three-dimensional boundary box in the three-dimensional image and at least three points in a central point coordinate.

8. The method of claim 6, wherein the two-dimensional image sample as the training sample comprises:

9. The method of claim 8, wherein the pose change comprises:

10. The method of claim 6, wherein the two-dimensional image sample is obtained when a projection distance of an imaging device satisfies a predetermined value.

11. The method of claim 6, wherein determining a difference between the predicted proxel coordinate information and the proxel coordinate annotation information, and adjusting network model parameters of the neural network based on the difference comprises:

12. The method of claim 10, wherein the loss function is

Wherein, N is 9,

in order to predict the proxel coordinate information,

and marking information for the coordinates of the projection points.

13. An image registration apparatus, comprising:

14. A neural network training apparatus for image registration, the apparatus comprising:

15. An electronic device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured with executable instructions to implement the method of any one of claims 1-5.

16. A computer-readable storage medium having stored thereon computer instructions, which when executed by a processor, perform the steps of the method according to any one of claims 1-5.

17. An electronic device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured with executable instructions to implement the method of any one of claims 6-12.

18. A computer-readable storage medium having stored thereon computer instructions, which, when executed by a processor, carry out the steps of the method according to any one of claims 6-12.