CN116823905A

CN116823905A - Image registration method, electronic device, and computer-readable storage medium

Info

Publication number: CN116823905A
Application number: CN202310767446.6A
Authority: CN
Inventors: 李孜; 田琳; 莫志榮; 白晓宇; 王普阳; 吕乐; 闫轲; 金达开
Original assignee: Alibaba Damo Institute Hangzhou Technology Co Ltd
Current assignee: Alibaba Damo Institute Hangzhou Technology Co Ltd
Priority date: 2023-06-26
Filing date: 2023-06-26
Publication date: 2023-09-29
Anticipated expiration: 2043-06-26
Also published as: CN116823905B

Abstract

The application discloses an image registration method, electronic equipment and a computer readable storage medium. Wherein the method comprises the following steps: acquiring at least two images, wherein any one image comprises shooting results of an object to be analyzed under different conditions; and respectively extracting features of at least two images to obtain image features of any one image, wherein the image features at least comprise: global features and local features; determining a target displacement vector field of the object to be analyzed based on the extracted image features; and carrying out image registration on at least two images based on the target displacement vector field of the object to be analyzed, and obtaining an image registration result. The application solves the technical problems of lower accuracy and limited application scene of the image registration method in the related technology.

Description

Image registration method, electronic device, and computer-readable storage medium

Technical Field

The present application relates to the field of image registration, and in particular, to an image registration method, an electronic device, and a computer-readable storage medium.

Background

Current image registration methods generally use feature descriptors that provide invariance information of modality and contrast, but because of limited information represented by the feature descriptors, challenges are presented in environments with large deformations or complex anatomical differences (e.g., inter-patient abdomen), and thus, the accuracy of processing registration tasks by existing image registration methods is low, which is difficult to apply to scenes that process complex registration tasks.

In view of the above problems, no effective solution has been proposed at present.

Disclosure of Invention

The embodiment of the application provides an image registration method, electronic equipment and a computer readable storage medium, which are used for at least solving the technical problems of lower accuracy and limited application scene of the image registration method in the related technology.

According to an aspect of an embodiment of the present application, there is provided an image registration method including: acquiring at least two images, wherein any one image comprises shooting results of an object to be analyzed under different conditions; and respectively extracting features of at least two images to obtain image features of any one image, wherein the image features at least comprise: global features and local features; determining a target displacement vector field of the object to be analyzed based on the extracted image features; and carrying out image registration on at least two images based on the target displacement vector field of the object to be analyzed, and obtaining an image registration result.

According to an aspect of the embodiment of the present application, there is also provided an image registration method, including: acquiring at least two medical images, wherein any one medical image comprises scanning results of a target part of the same biological object under different conditions by a computer tomography; respectively extracting features of at least two medical images to obtain image features of any image, wherein the image features comprise: global features and local features; determining a target displacement vector field of the target part based on the extracted image features; and carrying out image registration on at least two medical images based on the target displacement vector field of the target part to obtain an image registration result.

According to an aspect of the embodiment of the present application, there is also provided an image registration method, including: responding to an input instruction acted on an operation interface, and displaying at least two images on the operation interface, wherein any one image comprises shooting results of an object to be analyzed under different conditions; and responding to an image registration instruction acted on an operation interface, and displaying an image registration result on the operation interface, wherein the image registration result is obtained by carrying out image registration on at least two images based on a target displacement vector field of an object to be analyzed, the target displacement vector field is determined based on extracted image features, and the extracted image features comprise: the global features and the local features, and the extracted image features are obtained by respectively extracting features of at least two images.

According to an aspect of the embodiment of the present application, there is also provided an image registration method, including: displaying at least two images on a presentation picture of a Virtual Reality (VR) device or an Augmented Reality (AR) device, wherein any one image comprises shooting results of an object to be analyzed under different conditions; and respectively extracting features of at least two images to obtain image features of any one image, wherein the image features at least comprise: global features and local features; determining a target displacement vector field of the object to be analyzed based on the extracted image features; performing image registration on at least two images based on a target displacement vector field of an object to be analyzed to obtain an image registration result; and driving the VR device or the AR device to render and display the image registration result.

According to an aspect of the embodiment of the present application, there is also provided an image registration method, including: acquiring at least two images by calling a first interface, wherein the first interface comprises a first parameter, the parameter value of the first parameter is at least two images, and any image contains shooting results of an object to be analyzed under different conditions; and respectively extracting features of at least two images to obtain image features of any one image, wherein the image features at least comprise: global features and local features; determining a target displacement vector field of the object to be analyzed based on the extracted image features; performing image registration on at least two images based on a target displacement vector field of an object to be analyzed to obtain an image registration result; and outputting an image registration result by calling a second interface, wherein the second interface comprises a second parameter, and the parameter value of the second parameter is the image registration result.

According to an aspect of the embodiment of the present application, there is also provided an electronic device including: a memory storing an executable program; and a processor for running a program, wherein the program when run performs the method of any one of the above embodiments.

According to an aspect of the embodiments of the present application, there is also provided a computer-readable storage medium including a stored executable program, wherein the executable program when run controls a device in which the computer-readable storage medium is located to perform the method of any one of the above embodiments.

In the embodiment of the application, at least two images are acquired, wherein any one image contains shooting results of an object to be analyzed under different conditions; and respectively extracting features of at least two images to obtain image features of any one image, wherein the image features at least comprise: global features and local features; determining a target displacement vector field of the object to be analyzed based on the extracted image features; and carrying out image registration on at least two images based on a target displacement vector field of the object to be analyzed to obtain an image registration result, thereby achieving the aim of improving the accuracy of image registration. It is easy to note that feature extraction can be performed on at least two images respectively to obtain image features including global features and local features, a receptive field of the images can be enlarged according to the global features and the local features so as to obtain a more accurate target displacement vector field, and at least two images can be registered according to the target displacement vector field so as to obtain an image registration result with higher accuracy, and the method can be applied to scenes for processing complex registration tasks, and further solves the technical problems that an image registration method in the related art is lower in accuracy and limited in application scene.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application, as claimed.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:

fig. 1 is a schematic diagram of a hardware environment of a virtual reality device of an image registration method according to an embodiment of the application;

FIG. 2 is a block diagram of a computing environment for an image registration method according to an embodiment of the application;

fig. 3 is a flowchart of an image registration method according to embodiment 1 of the present application;

FIG. 4a is a block diagram of an image registration process according to an embodiment of the present application;

FIG. 4b is a block diagram of a convex optimization process in accordance with an embodiment of the application;

fig. 5 is a flowchart of an image configuration method according to embodiment 2 of the present application;

fig. 6 is a flowchart of an image configuration method according to embodiment 3 of the present application;

fig. 7 is a flowchart of an image registration method according to embodiment 4 of the present application;

Fig. 8 is a flowchart of an image registration method according to embodiment 5 of the present application;

fig. 9 is a schematic diagram of an image registration apparatus according to embodiment 6 of the present application;

fig. 10 is a schematic view of an image registration apparatus according to embodiment 7 of the present application;

fig. 11 is a schematic view of an image registration apparatus according to embodiment 8 of the present application;

fig. 12 is a schematic view of an image registration apparatus according to embodiment 9 of the present application;

fig. 13 is a schematic view of an image registration apparatus according to embodiment 10 of the present application;

fig. 14 is a block diagram of a computer terminal according to an embodiment of the present application.

Detailed Description

In order that those skilled in the art will better understand the present application, a technical solution in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present application without making any inventive effort, shall fall within the scope of the present application.

It should be noted that the terms "first," "second," and the like in the description and the claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

First, partial terms or terminology appearing in the course of describing embodiments of the application are applicable to the following explanation:

registration (registration): refers to the matching of geographic coordinates of different image patterns obtained by different imaging means in the same region. Image registration is a typical problem and technical difficulty in the field of image processing research, and aims to compare or fuse images acquired under different conditions for the same object, for example, the images may come from different acquisition devices, be taken from different times, different shooting angles, etc., and sometimes the problem of image registration for different objects is also required. In particular, for two images in a set of image data sets, one image is mapped to the other image by finding a spatial transformation, so that points corresponding to the same position in space in the two images are in one-to-one correspondence,

cost volume (cost volume or correlation volume): is a common intermediate representation in optical flow estimation or binocular matching, and is used to represent the similarity of pixel levels between feature images so as to find the correspondence.

Convex optimization: or convex optimization, convex minimization, is a sub-field of mathematical optimization, studying the problem of convex function minimization defined in convex sets, convex optimization is in some sense simpler than the mathematical optimization problem of the general case, e.g. in convex optimization the local optimum is the global optimum.

Currently, deformable image registration is a basic medical image analysis task, and is conventionally regarded as a problem of continuous update of dense displacement field space between image pairs, and has a problem of low efficiency of an iterative process of image registration. In order to solve the problem, a method for predicting a displacement field based on a learning depth network is proposed to iterate so as to improve iteration efficiency, but because different types of registration tasks need training, the image registration method is difficult to adapt to various application scenes.

Furthermore, collecting sufficient training data takes a lot of time, both iterative updating and learning-based methods rely on similarity measures calculated in terms of intensity, which are difficult to apply on anatomical correspondence, and some studies are processed using feature descriptors that provide modality and contrast invariant information, but generally only represent local information without global semantic information, thus image registration methods are challenging in environments with large deformations or complex anatomical differences (e.g. patient's inter-abdominal).

Registration is also expressed as a discrete update problem, and can be handled with a set of dense discrete displacements as a cost quantity, the main challenge of this type of approach is the huge scale of search space, since millions of voxels exist in a typical three-dimensional electronic computed tomography (Computed Tomography, abbreviated CT), and each voxel in a moving scan can be reasonably paired with thousands of points in other scans, resulting in a high computational burden. In order to obtain a fast registration by discrete updating, the search space may be pruned by constructing cost amounts in the neighborhood of different voxels, however, the range of deformation magnitudes that this approach can address is limited by the neighborhood window size, relying on accurate pre-alignment, and implementation is difficult.

A widely used deformable registration method with better performance may include: deeds, convexAdam, wherein Deeds and convexaadam use feature descriptors that provide modality and contrast invariant information, but still represent local information without global semantic information, and thus are also difficult to apply in environments with large deformations or complex anatomical differences (e.g., patient-to-patient abdomen); furthermore ConvexAdam relies on precise pre-alignment, which is difficult to implement.

Processing complex registration tasks relies on unique features with robustness to inter-subject variation, organ deformation, contrast agent injection, pathological changes and the like, and global/context information which is beneficial to improving the accuracy of complex deformation registration; in the process of calculating cost loss in the neighborhood, the method is limited by the size of the neighborhood, and in order to solve the problem, the application proposes an image processing process from a small-resolution image to a large-resolution image through a pyramid mode, and different levels have a smaller search range instead of a large search range under one resolution, which is beneficial to improving the efficiency of low calculation burden, quick running time and the like, expanding the receptive field and improving the registration accuracy.

Example 1

According to an embodiment of the present application, there is provided an image registration method, it being noted that the steps shown in the flowcharts of the figures may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that herein.

Fig. 1 is a schematic diagram of a hardware environment of a virtual reality device of an image registration method according to an embodiment of the application. As shown in fig. 1, the virtual reality device 104 is connected to the terminal 106, the terminal 106 is connected to the server 102 via a network, and the virtual reality device 104 is not limited to: the terminal 104 is not limited to a PC, a mobile phone, a tablet computer, etc., and the server 102 may be a server corresponding to a media file operator, and the network includes, but is not limited to: a wide area network, a metropolitan area network, or a local area network.

Optionally, the virtual reality device 104 of this embodiment includes: memory, processor, and transmission means. The memory is used to store an application program that can be used to perform: acquiring at least two images, wherein any one image comprises shooting results of an object to be analyzed under different conditions; and respectively extracting features of at least two images to obtain image features of any one image, wherein the image features at least comprise: global features and local features; determining a target displacement vector field of the object to be analyzed based on the extracted image features; and carrying out image registration on at least two images based on a target displacement vector field of the object to be analyzed to obtain an image registration result, thereby solving the technical problems of lower accuracy and limited application scene of the image registration method in the related technology.

The terminal of this embodiment may be configured to display the image registration result on a presentation screen of a Virtual Reality (VR) device or an augmented Reality (Augmented Reality, AR) device, and send the image registration result to the Virtual Reality device 104, where the Virtual Reality device 104 displays at the target delivery location after receiving the image registration result.

Optionally, the HMD (Head Mount Display, head mounted display) head display and the eye tracking module of the virtual reality device 104 of this embodiment have the same functions as those of the above embodiment, that is, a screen in the HMD head display is used for displaying a real-time picture, and the eye tracking module in the HMD is used for acquiring a real-time motion track of an eyeball of a user. The terminal of the embodiment obtains the position information and the motion information of the user in the real three-dimensional space through the tracking system, and calculates the three-dimensional coordinates of the head of the user in the virtual three-dimensional space and the visual field orientation of the user in the virtual three-dimensional space.

The hardware architecture block diagram shown in fig. 1 may be used not only as an exemplary block diagram for an AR/VR device (or mobile device) as described above, but also as an exemplary block diagram for a server as described above, and in an alternative embodiment, fig. 2 shows in block diagram form one embodiment of a computing node in a computing environment 201 using an AR/VR device (or mobile device) as described above in fig. 1. Fig. 2 is a block diagram of a computing environment for an image registration method according to an embodiment of the present application, as shown in fig. 2, the computing environment 201 includes a plurality of computing nodes (e.g., servers) running on a distributed network (shown as 210-1, 210-2, …). Different computing nodes contain local processing and memory resources and end user 202 may run applications or store data remotely in computing environment 201. The application may be provided as a plurality of services 220-1, 220-2, 220-3, and 220-4 in computing environment 201, representing services "A", "D", "E", and "H", respectively.

End user 202 may provide and access services through a web browser or other software application on a client, in some embodiments, provisioning and/or requests of end user 202 may be provided to portal gateway 230. Ingress gateway 230 may include a corresponding agent to handle provisioning and/or request for services (one or more services provided in computing environment 201).

Services are provided or deployed in accordance with various virtualization techniques supported by the computing environment 201. In some embodiments, services may be provided according to Virtual Machine (VM) based virtualization, container based virtualization, and/or the like. Virtual machine-based virtualization may be the emulation of a real computer by initializing a virtual machine, executing programs and applications without directly touching any real hardware resources. While the virtual machine virtualizes the machine, according to container-based virtualization, a container may be started to virtualize the entire Operating System (OS) so that multiple workloads may run on a single Operating System instance.

In one embodiment based on container virtualization, several containers of a service may be assembled into one Pod (e.g., kubernetes Pod). For example, as shown in FIG. 2, the service 220-2 may be equipped with one or more Pods 240-1, 240-2, …,240-N (collectively referred to as Pods). The Pod may include an agent 245 and one or more containers 242-1, 242-2, …,242-M (collectively referred to as containers). One or more containers in the Pod handle requests related to one or more corresponding functions of the service, and the agent 245 generally controls network functions related to the service, such as routing, load balancing, etc. Other services may also be equipped with similar Pod.

In operation, executing a user request from end user 202 may require invoking one or more services in computing environment 201, and executing one or more functions of one service may require invoking one or more functions of another service. As shown in FIG. 2, service "A"220-1 receives a user request of end user 202 from ingress gateway 230, service "A"220-1 may invoke service "D"220-2, and service "D"220-2 may request service "E"220-3 to perform one or more functions.

The computing environment may be a cloud computing environment, and the allocation of resources is managed by a cloud service provider, allowing the development of functions without considering the implementation, adjustment or expansion of the server. The computing environment allows developers to execute code that responds to events without building or maintaining a complex infrastructure. Instead of expanding a single hardware device to handle the potential load, the service may be partitioned to a set of functions that can be automatically scaled independently.

In the above-described operating environment, the present application provides an image registration method as shown in fig. 3. It should be noted that, the image registration method of this embodiment may be performed by the mobile terminal of the embodiment shown in fig. 1. Fig. 3 is a flowchart of an image registration method according to embodiment 1 of the present application. As shown in fig. 3, the method may include the steps of:

Step S302, at least two images are acquired.

Any one image contains shooting results of an object to be analyzed under different conditions.

The at least two images may be CT belonging to the medical field, or may be images of other fields, and the type of the images is not limited in any way.

The object to be analyzed contained in the above-mentioned image may be a portion of the two images that needs to be compared. In the medical field, the at least two images may be CT images of the abdomen of a patient, and the object to be analyzed may be a site of an abnormality in the abdomen, for example, a lesion of the abdomen of the patient. It should be noted that, the object to be analyzed is not limited at all, and the object to be analyzed may be determined according to the analysis requirement on the image, which is only illustrated here as an example.

The different conditions may be different time periods, different environments, different types of photographing devices, and different photographing angles, and the different conditions may be set according to actual scenes.

In an alternative embodiment, the object to be analyzed or the area where the object to be analyzed is located may be photographed in different time periods, so as to obtain at least two images; the method can also be used for shooting the object to be analyzed or the region where the object to be analyzed is located by using different shooting modes to obtain at least two images, and can also be used for shooting the object to be analyzed or the region where the object to be analyzed is located under other different conditions to obtain at least two images.

In the embodiment of the present application, the image registration is performed by taking two images as an example, but the method is not limited thereto, and the processing manner of acquiring other numbers of images is similar and will not be described herein.

Step S304, respectively extracting the characteristics of at least two images to obtain the image characteristics of any image.

Wherein the image features include at least: global features and local features.

The image feature of any one of the images may be an image feature of each of the at least two images.

In an alternative embodiment, the image features of each image may be obtained by feature extraction of at least two images by a feature extractor, which may be a Self-supervising anatomical embedding feature extractor (Self-supervised Autoencoder with Mosaicked Training, SAM for short), but is not limited thereto, and may be other types of feature extractors.

It should be noted that SAM is generally used to extract features of an image, and the algorithm uses a self-supervised learning method, and trains an anatomical embedding model in a self-learning manner, where the model can learn global semantic features and local semantic features of different points on the image, so that image features of any image can be obtained, and the image features at least include global features and local features.

Step S306, determining a target displacement vector field of the object to be analyzed based on the extracted image features.

The target displacement vector field may represent a spatial variation of the object to be analyzed in at least two images, and the displacement vector field estimation may be performed on the position of the object to be analyzed, where the vector in the target displacement vector field is used to represent the displacement direction and the size of the object to be analyzed in the images.

The above-mentioned object displacement vector field is geometrically and mechanically displaced as a vector whose length is the shortest distance from the initial position to the final position of the point undergoing motion, which quantifies the distance and direction of the net or total movement of the point trajectory along a straight line from the initial position to the final position. The displacement vector field may be described as a relative position, i.e. as the final position of a point relative to its initial position, may be defined as the difference between the final position and the initial position.

In an alternative embodiment, a cost volume may be constructed according to the extracted image features, and a target displacement vector field may be determined according to the cost volume, where the cost volume is a common intermediate representation in optical flow estimation or binocular matching, and may be used to represent similarity at pixel level between features, so as to find a correspondence between pixels, and the target displacement vector field of the object to be analyzed may be determined according to the correspondence.

The cost volume described above may be, but is not limited to, a 6D cost volume, where the 6D cost volume stores data matching costs for associating a pixel with its corresponding pixel on another image.

It should be noted that the cost volume (cost volume or correlation volume) is a common intermediate representation in optical flow estimation or binocular matching, and is used to represent the pixel-level similarity between feature images, so as to find the correspondence between pixels according to the similarity. Typically calculated is a dense cost volume (dense cost volume), whose complexity is the square of the number of pixels (H W) in the optical flow problem; in the 3D medical image registration problem, its complexity is the square of the number of pixels (hxw x D x H x W x D), where the size of the 3D image is hx W x D.

Constructing a 6D cost volume based on SAM features is a method of constructing a 6D cost volume by features, which uses features of two images to construct a 6D cost volume, which is used to measure the similarity of two 3D objects. By extracting features in the image from the SAM, a corresponding 6D cost volume can be obtained.

Alternatively, the convex optimization processing may be performed based on the cost volume to obtain the target displacement vector field. The convex optimization processing refers to a class of optimization problems for solving the objective function as a convex function.

Step S308, performing image registration on at least two images based on a target displacement vector field of the object to be analyzed, and obtaining an image registration result.

The image registration refers to matching geographic coordinates of different images acquired by different imaging means in the same region, and optionally, the image registration refers to mapping one image into another image by searching for a spatial transformation, so that points in the two images corresponding to the same position in space are in one-to-one correspondence.

In an alternative embodiment, any one of the at least two images can be mapped into the other image according to the target displacement vector field of the object to be analyzed, so that the change condition of the object to be analyzed can be observed in one image, thereby facilitating the analysis of the object to be analyzed by a user and improving the analysis efficiency.

In the medical field, the object to be analyzed may be an abdominal region of a patient, a target displacement vector field of the abdominal region may be determined according to at least two CTs of the abdominal region of the patient, and the two CTs may be registered according to the target displacement vector field, thereby obtaining an image registration result, and facilitating a doctor to analyze the problem of the abdominal region of the patient according to the image registration result.

Through the steps, at least two images are obtained, wherein any one image contains shooting results of an object to be analyzed under different conditions; and respectively extracting features of at least two images to obtain image features of any one image, wherein the image features at least comprise: global features and local features; determining a target displacement vector field of the object to be analyzed based on the extracted image features; and carrying out image registration on at least two images based on a target displacement vector field of the object to be analyzed to obtain an image registration result, thereby achieving the aim of improving the accuracy of image registration. It is easy to note that feature extraction can be performed on at least two images respectively to obtain image features including global features and local features, a receptive field of the images can be enlarged according to the global features and the local features so as to obtain a more accurate target displacement vector field, and at least two images can be registered according to the target displacement vector field so as to obtain an image registration result with higher accuracy, and the method can be applied to scenes for processing complex registration tasks, and further solves the technical problems that an image registration method in the related art is lower in accuracy and limited in application scene.

In the above embodiment of the present application, feature extraction is performed on at least two images respectively to obtain image features of any one image, including: deforming a first image based on a displacement vector field corresponding to the first image in at least two images to obtain a deformed image; and respectively extracting features of the deformed image and a second image in the at least two images to obtain image features of any one image, wherein the second image is used for representing images except the first image in the at least two images.

The first image and the second image may be images of the same area acquired at different time points, the first image may be an image acquired first, and the second image may be an image acquired later; the second image may be a first acquired image, and the first image may be a second acquired image, which is not limited herein.

The first image may be any one of at least two images, and the second image may be any other image than the at least two images. Alternatively, in the case where the at least two images include only two images, the first image may be a source image of the two images and the second image may be a target image.

The displacement vector field described above may also be referred to as a registration field (up-sampled field).

The displacement vector field corresponding to the first image may be determined according to the scale information of the first image, where the scale information of the first image may be divided into multiple levels, and the higher the level is, the larger the resolution of the image is, the lower the level is, the smaller the resolution of the image is, so if the scale of the first image is minimum, that is, the level is the lowest, the displacement vector field may be a preset vector field, and if the scale of the first image is not the minimum scale, that is, the lowest level, the displacement vector field may be a target displacement vector field determined by the image of the previous level.

In an alternative embodiment, the foregoing may be described in terms of three levels, where the first level is the level with the smallest image scale, the second level is the level with the next largest image scale, and the third level is the level with the largest image scale, where when the first image is in the first level, it is indicated that the resolution of the first image is the lowest, and at this time, the first displacement vector field between the deformed image and the second image of the first level may be directly acquired; when the first image is in the second level, the first image can be deformed based on the displacement vector field obtained after up-sampling the first displacement vector field to obtain a deformed image, and a second displacement vector field between the deformed image and a second image of the second level is obtained; when the first image is at the third level, the resolution of the first image is maximum, and at this time, the first image may be deformed based on the displacement vector field obtained by upsampling the second displacement vector field to obtain a deformed image, and a third displacement vector field between the deformed image and the second image at the maximum level may be obtained.

In another optional embodiment, feature extraction may be performed on the deformed image and the second image obtained at any one level, so as to obtain an image feature of any one image, and feature extraction may be performed on the deformed image and the second image obtained at the three levels, so as to obtain an image feature of any one image.

In the above embodiment of the present application, feature extraction is performed on a deformed image and a second image in at least two images, respectively, to obtain image features of any one image, including: extracting features of the deformed image and the second image to obtain global features and local features; and superposing the global features and the local features to obtain image features.

In an alternative embodiment, the global feature and the local feature in the deformed image and the second image may be captured simultaneously by the feature extractor, and the global feature and the local feature may be further overlapped, so as to obtain the image feature of any image.

In the above embodiment of the present application, determining a target displacement vector field of an object to be analyzed based on the extracted image features includes: obtaining a dot product of the extracted image features to obtain a target cost volume; and performing convex optimization processing on the target cost volume to obtain a target displacement vector field.

The target cost volume is used for finding the corresponding relation between the image features according to the similarity of the image features.

In an alternative embodiment, a dot product of the extracted image features may be obtained to measure the similarity between the first image and the second image, and a target cost volume may be constructed according to the similarity between the two images, and convex optimization processing (convex optimization) may be performed on the target cost volume to obtain the target displacement vector field. Wherein the target cost volume may be a 6D cost volume.

In the above embodiment of the present application, performing convex optimization processing on a target cost volume to obtain a target displacement vector field includes: performing parameter minimization treatment on the target cost volume to obtain an initial displacement vector field; and carrying out average pooling operation on the initial displacement vector field to obtain a target displacement vector field.

In an alternative embodiment, the parameter minimization process may be performed on the target cost volume, that is, the parameter minimization process is performed on the target cost volume by using Argmin (), so as to obtain a variable corresponding to the target cost volume when the output value of Argmin () is minimum, determine an initial displacement vector field according to the variable, and perform the average pooling operation on the initial displacement vector field, so that the initial displacement vector field may be divided into a plurality of areas with the same size, and then average the feature values in different areas, so as to obtain the target displacement vector field.

In the above embodiment of the present application, feature extraction is performed on at least two images respectively to obtain image features of any one image, including: respectively carrying out downsampling on at least two images for multiple times to obtain a plurality of downsampled images of any one image, wherein the resolutions of different downsampled images are different; deforming the downsampled image of the first image based on a sampling displacement vector field corresponding to the downsampled image of the first image to obtain a sampling deformed image; respectively extracting features of the downsampled images of the sampled deformed image and the second image to obtain downsampled image features of the downsampled image of any image; and summarizing the plurality of downsampled image features to obtain the image features of any one image.

In an alternative embodiment, at least two images may be downsampled for multiple times respectively, so as to obtain representations of at least two images on different scales, that is, multiple downsampled images of any one image may be obtained, resolutions of different downsampled images may be different, the downsampled images of the first image may be deformed according to a sampling displacement vector field corresponding to the downsampled images of the first image, a sampling deformed image may be obtained, sampling deformed images of different resolutions may be obtained, feature extraction may be performed on the downsampled images of the sampling deformed image and the downsampled images of the second image, downsampled image features of the downsampled image of any one image may be obtained, and the downsampled image features of the multiple different resolutions may be summarized, so as to obtain an image feature pyramid of any one image.

In another alternative embodiment, the sampled deformation image with the largest dimension can be obtained by processing the sampled image with the largest dimension, the sampled deformation image and the sampled image with the second image can be respectively subjected to feature extraction to obtain the sampled image features of the sampled image with any one image, the sampled displacement vector field required in the processing process of the next dimension can be constructed according to the sampled image features, alternatively, the cost volume can be constructed according to the sampled image features, convex optimization processing can be performed to obtain the sampled displacement vector field required in the processing process of the next dimension, and the sampled displacement vector fields corresponding to the images with all dimensions can be obtained in a circulating manner, so that a plurality of sampled deformation images can be sequentially obtained according to a plurality of sampled displacement vector fields, a plurality of sampled image features can be sequentially obtained according to the sampled deformation images, and the sampled image features can be summarized to obtain the image features of any one image.

The first image is deformed in a rough-to-fine mode under the condition of different scales to obtain a sampling deformed image, and the characteristics of the sampling deformed image and the downsampled image of the second image are extracted respectively to obtain downsampled image characteristics.

In the above embodiment of the present application, determining a target displacement vector field of an object to be analyzed based on the extracted image features includes: acquiring a dot product of the downsampled image features of a downsampled image of any one image to obtain a sampling cost volume; performing convex optimization processing based on the sampling cost volume to obtain a sampling displacement vector field; and superposing the plurality of sampling displacement vector fields to obtain a target displacement vector field.

In an alternative embodiment, dot products of downsampled image features of any one image may be obtained to measure similarity between downsampled images of the first image and downsampled images of the second image, a sampling cost volume may be constructed according to the similarity between the two downsampled images, and convex optimization processing may be performed based on the sampling cost volume to obtain a sampling displacement vector field.

The above operation can be performed on downsampled images of different scales in a cyclic manner, and it should be noted that the sampled displacement vector field obtained in the current scale can be deformed according to the sampled displacement vector field obtained in the previous scale, so as to improve accuracy of deformation, and after obtaining a plurality of sampled displacement vector fields of different scales, the plurality of sampled displacement vector fields can be superimposed to obtain the target displacement vector field.

In the above embodiment of the present application, the method further includes: under the condition that the downsampled image of the first image is a preset downsampled image, determining a sampling displacement vector field corresponding to the downsampled image of the first image as a preset displacement vector field, wherein the preset downsampled image is used for representing the downsampled image of the maximum scale of the first image; and under the condition that the downsampled image of the first image is not the preset downsampled image, upsampling the sampling displacement vector field corresponding to the downsampled image of the next scale of the first image to obtain the sampling displacement vector field corresponding to the downsampled image of the first image.

The preset downsampled image may be a downsampled image under the maximum scale, and since the resolution of the downsampled image under the maximum scale is low, the number of features to be processed is small, so that the global features of the downsampled image can be efficiently extracted, the downsampled image can be sequentially processed from the large resolution to the small resolution, and the feature extraction efficiency can be further improved under the condition that the global features and the local features can be ensured to be extracted.

In an alternative embodiment, when the downsampled image of the first image is the downsampled image of the maximum scale, the deformation is performed for the first time, so that the displacement vector field is not obtained yet, and at this time, the preset displacement vector field may be determined to be the sampling displacement vector field corresponding to the downsampled image of the first image; after the downsampled image features are obtained according to the downsampled image of the maximum scale, the dot product of the downsampled image features can be obtained to obtain a sampling cost volume, convex optimization processing can be performed based on the sampling cost volume to obtain a sampling displacement vector field required by the next downsampled image of the next scale when the downsampled image of the next scale is deformed, and based on the sampling displacement vector field, the downsampled images of different scales can be sequentially processed to obtain the sampling displacement vector fields obtained under different scales.

The preferred embodiment of the present application will be described in detail below taking a scene in which two images are registered as an example. Fig. 4a IS a block diagram of an image registration process according to an embodiment of the present application, where, as shown in fig. 4a, an image to be subjected to image registration includes a first image IS and a second image IT, and the first image and the second image may be respectively downsampled to obtain an image Pyramid (Pyramid Source) of the first image and an image Pyramid (Pyramid Target) of the second image, where an uppermost image of the two sampling pyramids IS a downsampled image with a largest scale, and a lowermost image of the sampling pyramids IS a downsampled image with a smallest scale. The downsampled image processing procedure for each scale is similar, and here, the downsampled image processing procedure with the smallest scale is described as an example: when the feature processing is performed on the downsampled image, an Up-sampling (Up-Sampled Field) can be performed on the registration Field obtained in the previous scale, that is, the Sampled displacement vector Field performs a deforming operation (Warping) on the downsampled image corresponding to the first image to obtain a Sampled deformed image, feature extraction can be performed on the downsampled images of the Sampled deformed image and the second image respectively to obtain downsampled image features, a convex optimization operation (Convex Optimization) can be performed on the downsampled image features to obtain a Sampled displacement vector Field, that is, a registration Field of the current scale is obtained, and finally, a plurality of Sampled displacement vector fields can be superimposed to obtain a target displacement vector Field, and the first image and the second image can be subjected to image registration according to the target displacement vector Field to obtain an image registration result. It should be noted that, for the processing flow of the downsampled image with the smallest scale, the downsampled image corresponding to the first image may be subjected to a deforming operation (Warping) according to the preset registration field, so as to obtain a sampled deformed image.

Fig. 4b is a block diagram of a convex optimization process according to an embodiment of the present application, as shown in fig. 4b, local features and global features of a first image may be superimposed to obtain image features of the first image and a second image, dot Product operation may be performed on the image features to obtain a target cost volume, a value of a minimum function argument of the target cost volume may be determined by using a preset function [ Argmin () ], to obtain an initial displacement vector field, that is, a first registration field u shown in the figure, and average pooling operation (Mean Conv) may be performed on the initial displacement vector field to obtain a target displacement vector field, that is, a second registration field ψ shown in the figure.

The image registration method of the application provides a self-supervision anatomical embedding network (SAMConvex), which can be used for rapidly carrying out a discrete optimization method for image registration, and comprises a decoupled convex optimization program, wherein a displacement vector field based on a self-supervision anatomical embedding feature extractor is obtained through the convex optimization program, the feature extractor can capture global features and local features at the same time, alternatively, the SAMConvex can extract features with different scales, and a 6D cost volume can be constructed based on the features, and the displacement vector field can be iteratively updated through a processing process from a large scale to a small scale.

It is noted that different levels of 6D cost volumes are calculated by using different resolution inputs, and the cost amount for each level is calculated by using different resolution inputs.

It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region, and provide corresponding operation entries for the user to select authorization or rejection.

It should be noted that, for simplicity of description, the foregoing method embodiments are all described as a series of acts, but it should be understood by those skilled in the art that the present application is not limited by the order of acts described, as some steps may be performed in other orders or concurrently in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily required for the present application.

From the description of the above embodiments, it will be clear to a person skilled in the art that the method according to the above embodiments may be implemented by means of software plus a necessary general hardware platform, but that it may also be implemented by means of hardware. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method of the various embodiments of the present application.

Example 2

There is also provided in accordance with an embodiment of the present application an image registration method, it being noted that the steps shown in the flowchart of the figures may be performed in a computer system, such as a set of computer executable instructions, and that although a logical order is shown in the flowchart, in some cases the steps shown or described may be performed in an order other than that shown.

Fig. 5 is a flowchart of an image configuration method according to embodiment 2 of the present application, as shown in fig. 5, the method including the steps of:

Step S502, at least two medical images are acquired.

Wherein, any medical image contains the scanning result of the same biological object target part under different conditions by the computer tomography.

The medical image includes but is not limited to CT image, skin image of patient, but is not limited thereto, and the type of medical image may be set according to actual situation.

Step S504, respectively extracting features of at least two medical images to obtain image features of any image.

Wherein the image features include: global features and local features.

Step S506, determining a target displacement vector field of the target portion based on the extracted image features.

And step S508, performing image registration on at least two medical images based on the target displacement vector field of the target part to obtain an image registration result.

Through the steps, at least two medical images are obtained, wherein any medical image contains scanning results of a target part of the same biological object under different conditions by a computer tomography technology; respectively extracting features of at least two medical images to obtain image features of any image, wherein the image features comprise: global features and local features; determining a target displacement vector field of the target part based on the extracted image features; and carrying out image registration on at least two medical images based on a target displacement vector field of the target part to obtain an image registration result, thereby achieving the aim of improving the accuracy of image registration. It is easy to note that feature extraction can be performed on at least two images respectively to obtain image features including global features and local features, a receptive field of the images can be enlarged according to the global features and the local features so as to obtain a more accurate target displacement vector field, and at least two images can be registered according to the target displacement vector field so as to obtain an image registration result with higher accuracy, and the method can be applied to scenes for processing complex registration tasks, and further solves the technical problems that an image registration method in the related art is lower in accuracy and limited in application scene.

It should be noted that, the preferred embodiment of the present application in the above examples is the same as the embodiment provided in example 1, the application scenario and the implementation process, but is not limited to the embodiment provided in example 1.

Example 3

Fig. 6 is a flowchart of an image configuration method according to embodiment 3 of the present application, as shown in fig. 6, the method including the steps of:

step S602, in response to an input instruction acting on the operation interface, displaying at least two images on the operation interface.

The operation interface may be a display interface capable of displaying at least two images, and the at least two images may be displayed by performing related touch operations on the display interface.

In step S604, in response to the image registration instruction acting on the operation interface, the image registration result is displayed on the operation interface.

The image registration result is obtained by performing image registration on at least two images based on a target displacement vector field of an object to be analyzed, the target displacement vector field is determined based on extracted image features, and the extracted image features comprise: the global features and the local features, and the extracted image features are obtained by respectively extracting features of at least two images.

The image registration instruction may be an instruction generated by touching a related control of the operation interface when at least two images need to be registered, and an image registration result may be displayed on the operation interface according to the image registration instruction.

Through the steps, at least two images are displayed on the operation interface in response to an input instruction acted on the operation interface, wherein any one image contains shooting results of an object to be analyzed under different conditions; and responding to an image registration instruction acted on an operation interface, and displaying an image registration result on the operation interface, wherein the image registration result is obtained by carrying out image registration on at least two images based on a target displacement vector field of an object to be analyzed, the target displacement vector field is determined based on extracted image features, and the extracted image features comprise: the global features and the local features are obtained by extracting features of at least two images respectively, so that the purpose of improving the image registration accuracy is achieved. It is easy to note that feature extraction can be performed on at least two images respectively to obtain image features including global features and local features, a receptive field of the images can be enlarged according to the global features and the local features so as to obtain a more accurate target displacement vector field, and at least two images can be registered according to the target displacement vector field so as to obtain an image registration result with higher accuracy, and the method can be applied to scenes for processing complex registration tasks, and further solves the technical problems that an image registration method in the related art is lower in accuracy and limited in application scene.

Example 4

There is also provided, in accordance with an embodiment of the present application, a method of image registration applicable in a virtual reality scene of a virtual reality VR device, an augmented reality AR device, etc., it being noted that the steps illustrated in the flowchart of the drawings may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order other than that illustrated herein.

Fig. 7 is a flowchart of an image registration method according to embodiment 4 of the present application. As shown in fig. 7, the method may include the steps of:

step S702, at least two images are presented on a presentation screen of a virtual reality VR device or an augmented reality AR device.

Step S704, extracting features of at least two images respectively to obtain image features of any one image.

Step S706, determining a target displacement vector field of the object to be analyzed based on the extracted image features.

Step S708, performing image registration on at least two images based on the target displacement vector field of the object to be analyzed, and obtaining an image registration result.

Step S710, driving the VR device or the AR device to render and display the image registration result.

Through the steps, at least two images are displayed on a display picture of the virtual reality VR device or the augmented reality AR device, wherein any one image contains shooting results of an object to be analyzed under different conditions; and respectively extracting features of at least two images to obtain image features of any one image, wherein the image features at least comprise: global features and local features; determining a target displacement vector field of the object to be analyzed based on the extracted image features; performing image registration on at least two images based on a target displacement vector field of an object to be analyzed to obtain an image registration result; and driving VR equipment or AR equipment to render and display an image registration result, so that the aim of improving the image registration accuracy is fulfilled. It is easy to note that feature extraction can be performed on at least two images respectively to obtain image features including global features and local features, a receptive field of the images can be enlarged according to the global features and the local features so as to obtain a more accurate target displacement vector field, and at least two images can be registered according to the target displacement vector field so as to obtain an image registration result with higher accuracy, and the method can be applied to scenes for processing complex registration tasks, and further solves the technical problems that an image registration method in the related art is lower in accuracy and limited in application scene.

Alternatively, in the present embodiment, the image registration method described above may be applied to a hardware environment constituted by a server, a virtual reality device. The image registration result is shown on a presentation screen of the virtual reality VR device or the augmented reality AR device, and the server may be a server corresponding to a media file operator, where the network includes but is not limited to: the virtual reality device is not limited to a wide area network, a metropolitan area network, or a local area network: virtual reality helmets, virtual reality glasses, virtual reality all-in-one machines, and the like.

Optionally, the virtual reality device comprises: memory, processor, and transmission means. The memory is used to store an application program that can be used to perform: displaying at least two images on a presentation picture of a Virtual Reality (VR) device or an Augmented Reality (AR) device, wherein any one image comprises shooting results of an object to be analyzed under different conditions; and respectively extracting features of at least two images to obtain image features of any one image, wherein the image features at least comprise: global features and local features; determining a target displacement vector field of the object to be analyzed based on the extracted image features; performing image registration on at least two images based on a target displacement vector field of an object to be analyzed to obtain an image registration result; and driving the VR device or the AR device to render and display the image registration result.

It should be noted that, the image registration method applied to the VR device or the AR device in this embodiment may include the method of the embodiment shown in fig. 7, so as to achieve the purpose of driving the VR device or the AR device to display the image registration result.

Alternatively, the processor of this embodiment may call the application program stored in the memory through the transmission device to perform the above steps. The transmission device can receive the media file sent by the server through the network and can also be used for data transmission between the processor and the memory.

Example 5

There is also provided in accordance with an embodiment of the present application a method of object tracking, it being noted that the steps shown in the flowcharts of the figures may be performed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order other than that shown or described herein.

Fig. 8 is a flowchart of an image registration method according to embodiment 5 of the present application, as shown in fig. 8, the method including the steps of:

step S802, at least two images are acquired by calling a first interface.

The first interface comprises a first parameter, the parameter value of the first parameter is at least two images, and any image contains shooting results of an object to be analyzed under different conditions.

The first interface in the above steps may be an interface for performing data interaction between the cloud server and the client, where the client may transmit at least two images into an interface function as a first parameter of the interface function, so as to achieve the purpose of uploading the at least two images to the cloud server.

Step S804, extracting features of at least two images respectively to obtain image features of any one image.

Step S806, determining a target displacement vector field of the object to be analyzed based on the extracted image features.

Step S808, performing image registration on at least two images based on the target displacement vector field of the object to be analyzed, and obtaining an image registration result.

Step S810, outputting an image registration result by calling a second interface.

The second interface comprises a second parameter, and the parameter value of the second parameter is an image registration result.

The second interface may be an interface for performing data interaction between the cloud server and the client, and the cloud server may transmit the image registration result to the interface function as a second parameter of the interface function, so as to achieve the purpose of transmitting the image registration result to the client.

Through the steps, at least two images are obtained by calling a first interface, wherein the first interface comprises a first parameter, the parameter value of the first parameter is at least two images, and any image contains shooting results of an object to be analyzed under different conditions; and respectively extracting features of at least two images to obtain image features of any one image, wherein the image features at least comprise: global features and local features; determining a target displacement vector field of the object to be analyzed based on the extracted image features; performing image registration on at least two images based on a target displacement vector field of an object to be analyzed to obtain an image registration result; and the second interface is called to output an image registration result, wherein the second interface comprises a second parameter, and the parameter value of the second parameter is the image registration result, so that the aim of improving the image registration accuracy is fulfilled. It is easy to note that feature extraction can be performed on at least two images respectively to obtain image features including global features and local features, a receptive field of the images can be enlarged according to the global features and the local features so as to obtain a more accurate target displacement vector field, and at least two images can be registered according to the target displacement vector field so as to obtain an image registration result with higher accuracy, and the method can be applied to scenes for processing complex registration tasks, and further solves the technical problems that an image registration method in the related art is lower in accuracy and limited in application scene.

Example 6

According to an embodiment of the present application, there is further provided an image registration apparatus for implementing the above image configuration method, and fig. 9 is a schematic diagram of an image registration apparatus according to embodiment 6 of the present application, as shown in fig. 9, and the apparatus 900 includes: an acquisition module 902, an extraction module 904, a determination module 906, a registration module 908.

The acquisition module is used for acquiring at least two images, wherein any one image comprises shooting results of an object to be analyzed under different conditions; the extraction module is used for respectively extracting features of at least two images to obtain image features of any one image, wherein the image features at least comprise: global features and local features; the determining module is used for determining a target displacement vector field of the object to be analyzed based on the extracted image features; the registration module is used for carrying out image registration on at least two images based on a target displacement vector field of the object to be analyzed, and an image registration result is obtained.

Here, it should be noted that the above-mentioned obtaining module 902, extracting module 904, determining module 906, and registering module 908 correspond to steps S302 to S308 in embodiment 1, and the four modules are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to those disclosed in embodiment 1 above. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (for example, the memory 104) and processed by one or more processors (for example, the processors 102a,102b, … …,102 n), and the above-mentioned modules may also be executed as a part of the apparatus in the computer terminal 10 provided in embodiment 1.

In the above embodiment of the present application, the extraction module is further configured to deform a first image of at least two images based on a displacement vector field corresponding to the first image, so as to obtain a deformed image; and respectively extracting features of the deformed image and a second image in the at least two images to obtain image features of any one image, wherein the second image is used for representing images except the first image in the at least two images.

In the above embodiment of the present application, the extraction module is further configured to perform feature extraction on the deformed image and the second image to obtain global features and local features; and superposing the global features and the local features to obtain image features.

In the above embodiment of the present application, the determining module is further configured to obtain a dot product of the extracted image feature, so as to obtain a target cost volume; and performing convex optimization processing on the target cost volume to obtain a target displacement vector field.

In the above embodiment of the present application, the determining module is further configured to perform a parameter minimization process on the target cost volume to obtain an initial displacement vector field; and carrying out average pooling operation on the initial displacement vector field to obtain a target displacement vector field.

In the above embodiment of the present application, the extraction module is further configured to perform downsampling on at least two images for multiple times, to obtain multiple downsampled images of any one image, where resolutions of different downsampled images are different; deforming the downsampled image of the first image based on a sampling displacement vector field corresponding to the downsampled image of the first image to obtain a sampling deformed image; respectively extracting features of the downsampled images of the sampled deformed image and the second image to obtain downsampled image features of the downsampled image of any image; and summarizing the plurality of downsampled image features to obtain the image features of any one image.

In the above embodiment of the present application, the determining module is further configured to obtain a dot product of a downsampled image feature of a downsampled image of any one image, to obtain a sampled cost volume; performing convex optimization processing based on the sampling cost volume to obtain a sampling displacement vector field; and superposing the plurality of sampling displacement vector fields to obtain a target displacement vector field.

In the above embodiment of the present application, the apparatus is further configured to determine, when the downsampled image of the first image is a preset downsampled image, that a sampled displacement vector field corresponding to the downsampled image of the first image is a preset displacement vector field, where the preset downsampled image is used to represent a downsampled image of a maximum scale of the first image; and under the condition that the downsampled image of the first image is not the preset downsampled image, upsampling the sampling displacement vector field corresponding to the downsampled image of the next scale of the first image to obtain the sampling displacement vector field corresponding to the downsampled image of the first image.

Example 7

There is further provided, according to an embodiment of the present application, an object tracking device for implementing the above object tracking method, and fig. 10 is a schematic diagram of an image registration device according to embodiment 7 of the present application, and as shown in fig. 10, the device 1000 includes: an acquisition module 1002, an extraction module 1004, a determination module 1006, a registration module 1008.

The acquisition module is used for acquiring at least two medical images, wherein any medical image comprises scanning results of a target part of the same biological object under different conditions by a computer tomography technology; the extraction module is used for respectively extracting features of at least two medical images to obtain image features of any image, wherein the image features comprise: global features and local features; the determining module is used for determining a target displacement vector field of the target part based on the extracted image features; the registration module is used for carrying out image registration on at least two medical images based on a target displacement vector field of the target part to obtain an image registration result.

Here, the above-mentioned obtaining module 1002, extracting module 1004, determining module 1006, and registering module 1008 correspond to steps S502 to S508 in embodiment 2, and the four modules are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to those disclosed in embodiment 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (for example, the memory 104) and processed by one or more processors (for example, the processors 102a,102b, … …,102 n), and the above-mentioned modules may also be executed as a part of the apparatus in the computer terminal 10 provided in embodiment 1.

Example 8

There is further provided, according to an embodiment of the present application, an object tracking device for implementing the above object tracking method, and fig. 11 is a schematic diagram of an image registration device according to embodiment 8 of the present application, and as shown in fig. 11, the device 1100 includes: a first display module 1102, a second display module 1104.

The first display module is used for responding to an input instruction acted on the operation interface, and displaying at least two images on the operation interface, wherein any one image comprises shooting results of an object to be analyzed under different conditions; the second display module is used for responding to an image registration instruction acting on the operation interface, and displaying an image registration result on the operation interface, wherein the image registration result is obtained by carrying out image registration on at least two images based on a target displacement vector field of an object to be analyzed, the target displacement vector field is determined based on extracted image features, and the extracted image features comprise: the global features and the local features, and the extracted image features are obtained by respectively extracting features of at least two images.

Here, it should be noted that the first display module 1102 and the second display module 1104 correspond to steps S602 to S604 in embodiment 3, and the two modules are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to those disclosed in embodiment 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (for example, the memory 104) and processed by one or more processors (for example, the processors 102a,102b, … …,102 n), and the above-mentioned modules may also be executed as a part of the apparatus in the computer terminal 10 provided in embodiment 1.

Example 9

There is further provided, according to an embodiment of the present application, an object tracking device for implementing the above object tracking method, and fig. 12 is a schematic diagram of an image registration device according to embodiment 9 of the present application, and as shown in fig. 12, the device 1200 includes: a presentation module 1202, an extraction module 1204, a determination module 1206, a registration module 1208, a drive module 1210.

The display module is used for displaying at least two images on a display picture of the virtual reality VR device or the augmented reality AR device, wherein any one image comprises shooting results of an object to be analyzed under different conditions; the extraction module is used for respectively extracting features of at least two images to obtain image features of any one image, wherein the image features at least comprise: global features and local features; the determining module is used for determining a target displacement vector field of the object to be analyzed based on the extracted image features; the registration module is used for carrying out image registration on at least two images based on a target displacement vector field of an object to be analyzed, so as to obtain an image registration result; the driving module is used for driving the VR equipment or the AR equipment to render and display the image registration result.

It should be noted that, the above-mentioned display module 1202, extraction module 1204, determination module 1206, registration module 1208 and driving module 1210 correspond to steps S702 to S710 in embodiment 4, and the five modules are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to those disclosed in embodiment 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (for example, the memory 104) and processed by one or more processors (for example, the processors 102a,102b, … …,102 n), and the above-mentioned modules may also be executed as a part of the apparatus in the computer terminal 10 provided in embodiment 1.

Example 10

There is further provided, according to an embodiment of the present application, an object tracking device for implementing the above object tracking method, and fig. 13 is a schematic diagram of an image registration device according to embodiment 10 of the present application, as shown in fig. 13, and the device 1300 includes: the acquisition module 1302, the extraction module 1304, the determination module 1306, the registration module 1308, the invocation module 1310.

The acquisition module is used for acquiring at least two images by calling a first interface, wherein the first interface comprises a first parameter, the parameter value of the first parameter is at least two images, and any image contains shooting results of an object to be analyzed under different conditions; the extraction module is used for respectively extracting features of at least two images to obtain image features of any one image, wherein the image features at least comprise: global features and local features; the determining module is used for determining a target displacement vector field of the object to be analyzed based on the extracted image features; the registration module is used for carrying out image registration on at least two images based on a target displacement vector field of an object to be analyzed, so as to obtain an image registration result; the calling module is used for outputting an image registration result by calling a second interface, wherein the second interface comprises a second parameter, and the parameter value of the second parameter is the image registration result.

It should be noted that, the above-mentioned obtaining module 1302, extracting module 1304, determining module 1306, registering module 1308, and calling module 1310 correspond to steps S802 to S810 in embodiment 5, and the five modules are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to those disclosed in embodiment 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory (for example, the memory 104) and processed by one or more processors (for example, the processors 102a,102b, … …,102 n), and the above-mentioned modules may also be executed as a part of the apparatus in the computer terminal 10 provided in embodiment 1.

Example 11

Embodiments of the present application may provide an electronic device, which may be an AR/VR device, which may be any one of a group of AR/VR devices. Alternatively, in this embodiment, the AR/VR device may be replaced by a terminal device such as a mobile terminal.

Alternatively, in this embodiment, the AR/VR device may be located in at least one network device among a plurality of network devices of the computer network.

In this embodiment, the AR/VR device may execute the program code for the following steps in the image registration method: acquiring at least two images, wherein any one image comprises shooting results of an object to be analyzed under different conditions; and respectively extracting features of at least two images to obtain image features of any one image, wherein the image features at least comprise: global features and local features; determining a target displacement vector field of the object to be analyzed based on the extracted image features; and carrying out image registration on at least two images based on the target displacement vector field of the object to be analyzed, and obtaining an image registration result.

Alternatively, fig. 14 is a block diagram of a computer terminal according to an embodiment of the present application. As shown in fig. 14, the computer terminal a may include: one or more (only one is shown) processors 102, memory 104, memory controller, and peripheral interfaces, where the peripheral interfaces are connected to the radio frequency module, audio module, and display.

The memory may be used to store software programs and modules, such as program instructions/modules corresponding to the image registration method and apparatus in the embodiments of the present application, and the processor executes the software programs and modules stored in the memory, thereby performing various functional applications and data processing, that is, implementing the image registration method described above. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory may further include memory remotely located with respect to the processor, which may be connected to terminal a through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The processor may call the information and the application program stored in the memory through the transmission device to perform the following steps: acquiring at least two images, wherein any one image comprises shooting results of an object to be analyzed under different conditions; and respectively extracting features of at least two images to obtain image features of any one image, wherein the image features at least comprise: global features and local features; determining a target displacement vector field of the object to be analyzed based on the extracted image features; and carrying out image registration on at least two images based on the target displacement vector field of the object to be analyzed, and obtaining an image registration result.

Optionally, the above processor may further execute program code for: deforming a first image based on a displacement vector field corresponding to the first image in at least two images to obtain a deformed image; and respectively extracting features of the deformed image and a second image in the at least two images to obtain image features of any one image, wherein the second image is used for representing images except the first image in the at least two images.

Optionally, the above processor may further execute program code for: extracting features of the deformed image and the second image to obtain global features and local features; and superposing the global features and the local features to obtain image features.

Optionally, the above processor may further execute program code for: obtaining a dot product of the extracted image features to obtain a target cost volume; and performing convex optimization processing on the target cost volume to obtain a target displacement vector field.

Optionally, the above processor may further execute program code for: performing parameter minimization treatment on the target cost volume to obtain an initial displacement vector field; and carrying out average pooling operation on the initial displacement vector field to obtain a target displacement vector field.

Optionally, the above processor may further execute program code for: respectively carrying out downsampling on at least two images for multiple times to obtain a plurality of downsampled images of any one image, wherein the resolutions of different downsampled images are different; deforming the downsampled image of the first image based on a sampling displacement vector field corresponding to the downsampled image of the first image to obtain a sampling deformed image; respectively extracting features of the downsampled images of the sampled deformed image and the second image to obtain downsampled image features of the downsampled image of any image; and summarizing the plurality of downsampled image features to obtain the image features of any one image.

Optionally, the above processor may further execute program code for: acquiring a dot product of the downsampled image features of a downsampled image of any one image to obtain a sampling cost volume; performing convex optimization processing based on the sampling cost volume to obtain a sampling displacement vector field; and superposing the plurality of sampling displacement vector fields to obtain a target displacement vector field.

Optionally, the above processor may further execute program code for: under the condition that the downsampled image of the first image is a preset downsampled image, determining a sampling displacement vector field corresponding to the downsampled image of the first image as a preset displacement vector field, wherein the preset downsampled image is used for representing the downsampled image of the maximum scale of the first image; and under the condition that the downsampled image of the first image is not the preset downsampled image, upsampling the sampling displacement vector field corresponding to the downsampled image of the next scale of the first image to obtain the sampling displacement vector field corresponding to the downsampled image of the first image.

The processor may call the information and the application program stored in the memory through the transmission device to perform the following steps: acquiring at least two medical images, wherein any one medical image comprises scanning results of a target part of the same biological object under different conditions by a computer tomography; respectively extracting features of at least two medical images to obtain image features of any image, wherein the image features comprise: global features and local features; determining a target displacement vector field of the target part based on the extracted image features; and carrying out image registration on at least two medical images based on the target displacement vector field of the target part to obtain an image registration result.

The processor may call the information and the application program stored in the memory through the transmission device to perform the following steps: responding to an input instruction acted on an operation interface, and displaying at least two images on the operation interface, wherein any one image comprises shooting results of an object to be analyzed under different conditions; and responding to an image registration instruction acted on an operation interface, and displaying an image registration result on the operation interface, wherein the image registration result is obtained by carrying out image registration on at least two images based on a target displacement vector field of an object to be analyzed, the target displacement vector field is determined based on extracted image features, and the extracted image features comprise: the global features and the local features, and the extracted image features are obtained by respectively extracting features of at least two images.

The processor may call the information and the application program stored in the memory through the transmission device to perform the following steps: displaying at least two images on a presentation picture of a Virtual Reality (VR) device or an Augmented Reality (AR) device, wherein any one image comprises shooting results of an object to be analyzed under different conditions; and respectively extracting features of at least two images to obtain image features of any one image, wherein the image features at least comprise: global features and local features; determining a target displacement vector field of the object to be analyzed based on the extracted image features; performing image registration on at least two images based on a target displacement vector field of an object to be analyzed to obtain an image registration result; and driving the VR device or the AR device to render and display the image registration result.

The processor may call the information and the application program stored in the memory through the transmission device to perform the following steps: acquiring at least two images by calling a first interface, wherein the first interface comprises a first parameter, the parameter value of the first parameter is at least two images, and any image contains shooting results of an object to be analyzed under different conditions; and respectively extracting features of at least two images to obtain image features of any one image, wherein the image features at least comprise: global features and local features; determining a target displacement vector field of the object to be analyzed based on the extracted image features; performing image registration on at least two images based on a target displacement vector field of an object to be analyzed to obtain an image registration result; and outputting an image registration result by calling a second interface, wherein the second interface comprises a second parameter, and the parameter value of the second parameter is the image registration result.

By adopting the embodiment of the application, at least two images are acquired, wherein any one image contains shooting results of an object to be analyzed under different conditions; and respectively extracting features of at least two images to obtain image features of any one image, wherein the image features at least comprise: global features and local features; determining a target displacement vector field of the object to be analyzed based on the extracted image features; and carrying out image registration on at least two images based on a target displacement vector field of the object to be analyzed to obtain an image registration result, thereby achieving the aim of improving the accuracy of image registration. It is easy to notice that feature extraction can be performed on at least two images respectively to obtain image features including global features and local features, the receptive field of the images can be enlarged according to the global features and the local features so as to obtain a more accurate target displacement vector field, and at least two images can be registered according to the target displacement vector field so as to obtain an image registration result with higher accuracy, so that the technical problem of lower accuracy of processing registration tasks in related technologies is solved.

It will be appreciated by those skilled in the art that the structure shown in fig. 14 is only illustrative, and the computer terminal may be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a mobile internet device (MobileInternetDevices, MID), a PAD, etc. Fig. 14 does not limit the structure of the electronic device. For example, the computer terminal a may also include more or fewer components (such as a network interface, a display device, etc.) than shown in fig. 14, or have a different configuration than shown in fig. 14.

Those of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be implemented by a program for instructing a terminal device to execute in association with hardware, the program may be stored in a computer readable storage medium, and the storage medium may include: flash disk, read-Only Memory (ROM), random-access Memory (Random Access Memory, RAM), magnetic or optical disk, and the like.

Example 6

Embodiments of the present application also provide a computer-readable storage medium. Alternatively, in this embodiment, the above-described computer-readable storage medium may be used to store the program code executed by the image registration method provided in embodiment 1 described above.

Alternatively, in this embodiment, the above-mentioned computer readable storage medium may be located in any one of the AR/VR device terminals in the AR/VR device network or in any one of the mobile terminals in the mobile terminal group.

Optionally, in the present embodiment, the computer readable storage medium is configured to store program code for performing the steps of: acquiring at least two images, wherein any one image comprises shooting results of an object to be analyzed under different conditions; and respectively extracting features of at least two images to obtain image features of any one image, wherein the image features at least comprise: global features and local features; determining a target displacement vector field of the object to be analyzed based on the extracted image features; and carrying out image registration on at least two images based on the target displacement vector field of the object to be analyzed, and obtaining an image registration result.

Optionally, the above-mentioned storage medium is further configured to store program code for performing the steps of: deforming a first image based on a displacement vector field corresponding to the first image in at least two images to obtain a deformed image; and respectively extracting features of the deformed image and a second image in the at least two images to obtain image features of any one image, wherein the second image is used for representing images except the first image in the at least two images.

Optionally, the above-mentioned storage medium is further configured to store program code for performing the steps of: extracting features of the deformed image and the second image to obtain global features and local features; and superposing the global features and the local features to obtain image features.

Optionally, the above-mentioned storage medium is further configured to store program code for performing the steps of: obtaining a dot product of the extracted image features to obtain a target cost volume; and performing convex optimization processing on the target cost volume to obtain a target displacement vector field.

Optionally, the above-mentioned storage medium is further configured to store program code for performing the steps of: performing parameter minimization treatment on the target cost volume to obtain an initial displacement vector field; and carrying out average pooling operation on the initial displacement vector field to obtain a target displacement vector field.

Optionally, the above-mentioned storage medium is further configured to store program code for performing the steps of: respectively carrying out downsampling on at least two images for multiple times to obtain a plurality of downsampled images of any one image, wherein the resolutions of different downsampled images are different; deforming the downsampled image of the first image based on a sampling displacement vector field corresponding to the downsampled image of the first image to obtain a sampling deformed image; respectively extracting features of the downsampled images of the sampled deformed image and the second image to obtain downsampled image features of the downsampled image of any image; and summarizing the plurality of downsampled image features to obtain the image features of any one image.

Optionally, the above-mentioned storage medium is further configured to store program code for performing the steps of: acquiring a dot product of the downsampled image features of a downsampled image of any one image to obtain a sampling cost volume; performing convex optimization processing based on the sampling cost volume to obtain a sampling displacement vector field; and superposing the plurality of sampling displacement vector fields to obtain a target displacement vector field.

Optionally, the above-mentioned storage medium is further configured to store program code for performing the steps of: under the condition that the downsampled image of the first image is a preset downsampled image, determining a sampling displacement vector field corresponding to the downsampled image of the first image as a preset displacement vector field, wherein the preset downsampled image is used for representing the downsampled image of the maximum scale of the first image; and under the condition that the downsampled image of the first image is not the preset downsampled image, upsampling the sampling displacement vector field corresponding to the downsampled image of the next scale of the first image to obtain the sampling displacement vector field corresponding to the downsampled image of the first image.

Optionally, in the present embodiment, the computer readable storage medium is configured to store program code for performing the steps of: acquiring at least two medical images, wherein any one medical image comprises scanning results of a target part of the same biological object under different conditions by a computer tomography; respectively extracting features of at least two medical images to obtain image features of any image, wherein the image features comprise: global features and local features; determining a target displacement vector field of the target part based on the extracted image features; and carrying out image registration on at least two medical images based on the target displacement vector field of the target part to obtain an image registration result.

Optionally, in the present embodiment, the computer readable storage medium is configured to store program code for performing the steps of: responding to an input instruction acted on an operation interface, and displaying at least two images on the operation interface, wherein any one image comprises shooting results of an object to be analyzed under different conditions; and responding to an image registration instruction acted on an operation interface, and displaying an image registration result on the operation interface, wherein the image registration result is obtained by carrying out image registration on at least two images based on a target displacement vector field of an object to be analyzed, the target displacement vector field is determined based on extracted image features, and the extracted image features comprise: the global features and the local features, and the extracted image features are obtained by respectively extracting features of at least two images.

Optionally, in the present embodiment, the computer readable storage medium is configured to store program code for performing the steps of: displaying at least two images on a presentation picture of a Virtual Reality (VR) device or an Augmented Reality (AR) device, wherein any one image comprises shooting results of an object to be analyzed under different conditions; and respectively extracting features of at least two images to obtain image features of any one image, wherein the image features at least comprise: global features and local features; determining a target displacement vector field of the object to be analyzed based on the extracted image features; performing image registration on at least two images based on a target displacement vector field of an object to be analyzed to obtain an image registration result; and driving the VR device or the AR device to render and display the image registration result.

Optionally, in the present embodiment, the computer readable storage medium is configured to store program code for performing the steps of: acquiring at least two images by calling a first interface, wherein the first interface comprises a first parameter, the parameter value of the first parameter is at least two images, and any image contains shooting results of an object to be analyzed under different conditions; and respectively extracting features of at least two images to obtain image features of any one image, wherein the image features at least comprise: global features and local features; determining a target displacement vector field of the object to be analyzed based on the extracted image features; performing image registration on at least two images based on a target displacement vector field of an object to be analyzed to obtain an image registration result; and outputting an image registration result by calling a second interface, wherein the second interface comprises a second parameter, and the parameter value of the second parameter is the image registration result.

The foregoing embodiment numbers of the present application are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.

In the foregoing embodiments of the present application, the descriptions of the embodiments are emphasized, and for a portion of this disclosure that is not described in detail in this embodiment, reference is made to the related descriptions of other embodiments.

In the several embodiments provided in the present application, it should be understood that the disclosed technology may be implemented in other manners. The above-described embodiments of the apparatus are merely exemplary, and the division of the units, such as the division of the units, is merely a logical function division, and may be implemented in another manner, for example, multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some interfaces, units or modules, or may be in electrical or other forms.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing program codes.

The foregoing is merely a preferred embodiment of the present application and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present application, which are intended to be comprehended within the scope of the present application.

Claims

1. A method of image registration, comprising:

acquiring at least two images, wherein any one image comprises shooting results of an object to be analyzed under different conditions;

and respectively extracting the characteristics of the at least two images to obtain the image characteristics of any one image, wherein the image characteristics at least comprise: global features and local features;

determining a target displacement vector field of the object to be analyzed based on the extracted image features;

and carrying out image registration on the at least two images based on the target displacement vector field of the object to be analyzed, and obtaining an image registration result.

2. The method according to claim 1, wherein the feature extraction is performed on the at least two images to obtain the image feature of the arbitrary image, respectively, including:

deforming a first image of the at least two images based on a displacement vector field corresponding to the first image to obtain a deformed image;

And respectively extracting features of the deformed image and a second image in the at least two images to obtain image features of any one image, wherein the second image is used for representing images except the first image in the at least two images.

3. The method according to claim 2, wherein the feature extraction is performed on the deformed image and a second image of the at least two images, respectively, to obtain image features of the arbitrary image, including:

extracting features of the deformed image and the second image to obtain the global features and the local features;

and superposing the global feature and the local feature to obtain the image feature.

4. The method according to claim 1, wherein determining the target displacement vector field of the object to be analyzed based on the extracted image features comprises:

obtaining a dot product of the extracted image features to obtain a target cost volume;

and performing convex optimization processing on the target cost volume to obtain the target displacement vector field.

5. The method of claim 4, wherein performing convex optimization on the target cost volume to obtain a target displacement vector field comprises:

Performing parameter minimization treatment on the target cost volume to obtain an initial displacement vector field;

and carrying out average pooling operation on the initial displacement vector field to obtain the target displacement vector field.

6. The method according to any one of claims 1 to 5, wherein the feature extraction is performed on the at least two images to obtain the image features of the any one image, respectively, including:

respectively carrying out downsampling on the at least two images for a plurality of times to obtain a plurality of downsampled images of any one image, wherein the resolutions of different downsampled images are different;

deforming a downsampled image of a first image based on a sampling displacement vector field corresponding to the downsampled image of the first image to obtain a sampling deformed image;

respectively extracting features of the downsampled images of the sampled deformed image and the second image to obtain downsampled image features of the downsampled image of any image;

and summarizing the plurality of downsampled image features to obtain the image features of any one image.

7. The method of claim 6, wherein determining the target displacement vector field of the object to be analyzed based on the extracted image features comprises:

Acquiring a dot product of the downsampled image features of the downsampled image of the arbitrary image to obtain a sampling cost volume;

performing convex optimization processing on the sampling cost volume to obtain a sampling displacement vector field;

and superposing a plurality of sampling displacement vector fields to obtain the target displacement vector field.

8. The method of claim 6, wherein the method further comprises:

determining a sampling displacement vector field corresponding to a downsampled image of the first image as a preset displacement vector field under the condition that the downsampled image of the first image is a preset downsampled image, wherein the preset downsampled image is used for representing the downsampled image of the first image with the largest scale;

and under the condition that the downsampled image of the first image is not the preset downsampled image, upsampling the sampled displacement vector field corresponding to the downsampled image of the next scale of the first image to obtain the sampled displacement vector field corresponding to the downsampled image of the first image.

9. A method of image registration, comprising:

acquiring at least two medical images, wherein any one medical image comprises scanning results of a target part of the same biological object under different conditions by a computer tomography;

And respectively extracting the characteristics of the at least two medical images to obtain the image characteristics of any one image, wherein the image characteristics comprise: global features and local features;

determining a target displacement vector field of the target part based on the extracted image features;

and carrying out image registration on the at least two medical images based on the target displacement vector field of the target part to obtain an image registration result.

10. A method of image registration, comprising:

responding to an input instruction acted on an operation interface, and displaying at least two images on the operation interface, wherein any one image comprises shooting results of an object to be analyzed under different conditions;

and in response to an image registration instruction acting on the operation interface, displaying an image registration result on the operation interface, wherein the image registration result is obtained by performing image registration on the at least two images based on a target displacement vector field of the object to be analyzed, the target displacement vector field is determined based on extracted image features, and the extracted image features comprise: global features and local features, wherein the extracted image features are obtained by respectively extracting features of the at least two images.

11. A method of image registration, comprising:

displaying at least two images on a presentation picture of a Virtual Reality (VR) device or an Augmented Reality (AR) device, wherein any one image comprises shooting results of an object to be analyzed under different conditions;

performing image registration on the at least two images based on the target displacement vector field of the object to be analyzed to obtain an image registration result;

and driving the VR equipment or the AR equipment to render and display the image registration result.

12. A method of image registration, comprising:

acquiring at least two images by calling a first interface, wherein the first interface comprises a first parameter, the parameter value of the first parameter is the at least two images, and any one image comprises shooting results of an object to be analyzed under different conditions;

and outputting the image registration result by calling a second interface, wherein the second interface comprises a second parameter, and the parameter value of the second parameter is the image registration result.

13. An electronic device, comprising:

a memory storing an executable program;

a processor for executing the program, wherein the program when run performs the method of any of claims 1 to 12.

14. A computer readable storage medium, characterized in that the computer readable storage medium comprises a stored executable program, wherein the executable program when run controls a device in which the computer readable storage medium is located to perform the method of any one of claims 1 to 12.