WO2024239287A1 - 图像配准的数据处理方法、装置及电子设备 - Google Patents

图像配准的数据处理方法、装置及电子设备 Download PDF

Info

Publication number
WO2024239287A1
WO2024239287A1 PCT/CN2023/096114 CN2023096114W WO2024239287A1 WO 2024239287 A1 WO2024239287 A1 WO 2024239287A1 CN 2023096114 W CN2023096114 W CN 2023096114W WO 2024239287 A1 WO2024239287 A1 WO 2024239287A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
registration
expression
sample
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/096114
Other languages
English (en)
French (fr)
Inventor
夏天祎
刘超
李美
张勇
黎宇翔
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
BGI Shenzhen Co Ltd
Original Assignee
BGI Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by BGI Shenzhen Co Ltd filed Critical BGI Shenzhen Co Ltd
Priority to CN202380083330.9A priority Critical patent/CN120457453A/zh
Priority to PCT/CN2023/096114 priority patent/WO2024239287A1/zh
Publication of WO2024239287A1 publication Critical patent/WO2024239287A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis

Definitions

  • the present application relates to the field of data processing technology, and in particular to a data processing method, device and electronic equipment for image registration.
  • Image registration refers to aligning one or more images with each other in terms of position and shape. It is an important task in image-related fields and has a wide range of applications in biology, medicine, and other engineering fields. Among them, the elastic registration of multimodal images has always been a difficult problem in image registration. Here, the elastic registration of expression maps based on tissue photography falls into this category.
  • the existing multimodal elastic registration scheme uses unsupervised deep learning methods for pairwise registration. This method requires the registered images to belong to the same modality, or two modalities with small differences, so as to achieve the loss calculation of the output results in each training round and the corresponding gradient calculation, and thus start to optimize the model parameters.
  • the present application provides a data processing method, device and electronic device for image registration, the main purpose of which is to improve the technical problems that the existing multimodal elastic registration scheme is difficult to implement or directly implemented will result in poor accuracy of image registration.
  • the present application provides a data processing method for image registration, comprising:
  • the machine learning model is obtained by pre-training the model according to the registration data of the sample image, and the registration data includes: the sample image and the image after deformation correction of the sample image;
  • the present application provides a data processing device for image registration, comprising:
  • An acquisition module configured to acquire a first image and a second image to be registered
  • a processing module configured to perform registration according to the first image and the second image
  • the acquisition module is further configured to input the registered first image and the second image into a machine learning model for calculation to obtain deformation field information, wherein the machine learning model is obtained by pre-training the model according to the registration data of the sample image, and the registration data includes: the sample image and the image after deformation correction of the sample image;
  • the determination module is configured to determine corresponding coordinate information after image registration based on the deformation field information.
  • the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the data processing method for image registration described in the first aspect is implemented.
  • the present application provides an electronic device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor implements the data processing method for image registration described in the first aspect when executing the computer program.
  • the present application provides a data processing method, device and electronic device for image registration.
  • the present application provides a supervised machine learning method, which pre-trains the model according to the registration data of the sample image to obtain a machine learning model, and the registration data includes: the sample image and the image after deformation correction of the sample image.
  • the first image and the second image to be registered are first registered; then the registered first image and the second image are input into the machine learning model for calculation to obtain deformation field information, and then the corresponding coordinate information after image registration is determined based on the deformation field information.
  • FIG1 is a schematic diagram showing a flow chart of a data processing method for image registration provided in an embodiment of the present application
  • FIG2 is a schematic flow chart of another data processing method for image registration provided in an embodiment of the present application.
  • FIG3 shows an example diagram of an expression level sample diagram provided in an embodiment of the present application
  • FIG4 shows an example diagram of histogram equalization processing of expression quantity sample graphs provided in an embodiment of the present application
  • FIG5 shows an example diagram of histogram equalization processing of a tissue photograph sample image provided by an embodiment of the present application
  • FIG6 is a schematic diagram showing a storage format of a transformation relationship of rigid registration provided in an embodiment of the present application.
  • FIG. 7 is a schematic diagram showing a storage format of a transformation relationship of elastic registration provided by an embodiment of the present application.
  • FIG8 is a schematic diagram showing a sample graph of expression quantity provided in an embodiment of the present application after rigid transformation and elastic transformation;
  • FIG9 is a schematic diagram showing the effect of elastic deformation image synthesis provided by an embodiment of the present application.
  • FIG10 is a schematic diagram showing an example of a model structure provided in an embodiment of the present application.
  • FIG11 is a schematic diagram showing the effect of optimizing the model structure provided in an embodiment of the present application.
  • FIG12 is a schematic diagram showing an overall example process provided by an embodiment of the present application.
  • FIG. 13 shows a schematic diagram of the structure of a data processing device for image registration provided in an embodiment of the present application.
  • this embodiment provides a data processing method for image registration, as shown in FIG1, the method includes:
  • Step 101 Acquire a first image and a second image to be registered.
  • the registration problem can be divided into rigid and elastic registration problems according to the transformation mode required for alignment; according to the modal difference of the input image, it can be divided into single-modal and multi-modal registration problems.
  • the first image and the second image to be registered can be images of the same modality or different modalities.
  • Step 102 perform registration based on the first image and the second image.
  • rigid registration may be performed based on the first image and the second image.
  • step 102 may specifically include: performing histogram equalization processing on the first image and the second image respectively; and registering the first image and the second image after the histogram equalization processing.
  • Histogram equalization is to enhance the contrast of the image, mainly by changing the histogram distribution of the image to a nearly uniform distribution, so that the difference between the pixel values of the image is strengthened, showing a contrast enhancement effect, so as to better perform image registration later.
  • the first image and the second image may not correspond in position and size. Therefore, in order to ensure the computational efficiency and accuracy of the subsequent machine learning model, this embodiment needs to register the first image and the second image after histogram equalization.
  • the first image is used as a reference image
  • the second image is registered, such as flipping, translating, and rotating.
  • the registration here may include modification of position and/or size, but does not include twisting or changing of shape.
  • Step 103 input the registered first image and the second image into a machine learning model for calculation to obtain deformation field information.
  • the machine learning model may be obtained by pre-training the model according to the registration data of the sample image, and the registration data of the sample image may include: the sample image and the image after deformation correction of the sample image.
  • the deformation field information may include elastic transformation (deformation) information of image registration.
  • Step 104 Determine the corresponding coordinate information after image registration based on the deformation field information.
  • the corresponding coordinate information after image registration obtained in step 104 can be used to obtain image information after deformation field correction.
  • this embodiment provides a supervised machine learning method, which pre-trains the model according to the registration data of the sample image to obtain a machine learning model.
  • the first image and the second image to be registered are firstly subjected to histogram equalization processing to enhance the contrast of the image and facilitate model prediction; then the first image and the second image after the histogram equalization processing are registered; then the registered first image and the second image are input into the machine learning model for calculation to obtain deformation field information, and then the corresponding coordinate information after image registration is determined according to the deformation field information.
  • the elastic registration problem of multimodal images can be solved, and in addition, the elastic registration problem of homomodal images can also be solved.
  • the technical solution of this embodiment can be used for elastic registration of the same modality and multi-modality, such as elastic registration of medical images, elastic registration of biological experiment images, etc., which can improve the accuracy of image registration.
  • this embodiment provides a specific method as shown in FIG. 2, which includes:
  • Step 201 Pre-train to obtain a machine learning model.
  • the machine learning model may be obtained by pre-training the model according to the registration data of the sample image, and the registration data includes: the sample image and the image after deformation correction of the sample image.
  • the sample image may specifically include: a tissue photograph sample image and an expression quantity sample image based on the tissue photograph sample image, and the image after deformation correction of the sample image is an image after deformation correction of the expression quantity sample image.
  • the training process of the machine learning model includes: first, performing histogram equalization processing on the tissue photography sample image and the expression sample image respectively; then, using the tissue photography sample image after the histogram equalization processing as a reference image, performing rigid alignment and elastic alignment on the expression sample image after the histogram equalization processing, to obtain a data file containing the alignment relationship and the aligned expression sample image; then, using the tissue photography sample image after the histogram equalization processing and the expression sample image as feature data, and using the data file and the aligned expression sample image as label data, to train a machine learning model.
  • an expression sample graph is generated based on the abundance statistics of the expression sample data.
  • the expression sample data is a table that records the spatiotemporal transcriptome expression sample data, each row of which represents the situation of a certain gene captured by a sequencing site; each column represents the attributes such as the gene name, site coordinates, and capture amount in the captured gene situation. As shown in Table 1 below, this is an example of the first 8 rows of an expression data.
  • the expression sample graph is obtained by abundance statistics of the expression sample data.
  • the statistical process includes: first, according to the coordinates of the expression sample data, the smallest square range covered by the entire tissue is divided into several small, adjacent square ranges, each small square range is also called a "Bin set"; then, the total capture of the captured genes represented by the above expression data is counted in each Binset range; finally, an image is drawn based on the gene statistics of these Binsets, and the pixel position of the image corresponds to the relative position of each Binset in the smallest square area covered by the above tissue (for example, the position of the pixel corresponding to the Binset in the first row and first column on the graph is also the first row and first column), and the pixel value is the total capture of the gene in the Binset.
  • An example of the expression sample graph is shown in Figure 3.
  • histogram equalization processing is performed.
  • histogram equalization is to enhance the contrast of the image, which is convenient for manual registration and model training. It mainly changes the histogram distribution of the image into an approximately uniform distribution, so that the difference between the pixel values of each image is enhanced, showing a contrast enhancement effect.
  • the color information of the tissue photography sample map does not play a decisive role in the registration of the tissue photography sample map and the expression sample map, it is converted into a grayscale map for the purpose of saving storage and memory space. Therefore, it is correspondingly optional to perform histogram equalization processing on the tissue photography sample map, which may specifically include: after processing the tissue photography sample map into a grayscale map, performing histogram equalization processing.
  • the expression level sample graph is histogram equalized.
  • the real tissue photograph sample graph is converted into a grayscale graph and then histogram equalized.
  • the ImageJ TrakEM2 plug-in After obtaining the tissue photography sample image and expression sample image after histogram equalization, for the images used for model training, if there is a mismatch in position and size between the two images, use the ImageJ TrakEM2 plug-in to first use the tissue photography sample image as the reference image for rigid registration. If there are other methods, such as automated algorithms, other methods can also be used for rigid registration; then perform elastic registration on them. Output a data file containing the registration relationship (i.e., the TrakEM2 project file) and the expression sample image after registration.
  • the TrakEM2 project file stores the transformation relationship recorded by the plug-in when performing manual rigid and elastic registration in this software plug-in.
  • the storage form of the transformation relationship of the rigid registration is a rigid transformation matrix, which describes the flipping, rotation angle and horizontal and vertical translation of the registered image, as shown in Figure 6.
  • the storage form of the transformation relationship of the elastic registration is the coordinates of the manually specified control points before and after the transformation. Based on these coordinates, the deformation field of the entire image can be calculated.
  • This coordinate example is shown in Figure 7.
  • the expression sample image after registration refers to the expression sample image that is subjected to rigid transformation (flipping, translation and rotation) and elastic transformation (deformation) with the Blockface image as the reference image, and the final image is generated. As shown in Figure 8, the results of the expression sample image before registration, after rigid transformation, and after elastic transformation are shown.
  • the parameter fitting of the multi-layer neural network is realized based on the data drive of thousands of pairs of registered images and reference images; however, for the elastic registration task of the expression map based on the tissue photography map, it is very difficult for R&D personnel to obtain large-scale data due to the limitations of its experimental cost and sequencing time. Therefore, there is a problem of high cost. For this reason, this embodiment can automatically synthesize elastic deformation images based on the collected experimental data, thereby reducing the cost of algorithm development.
  • the above-mentioned histogram equalization processing of the expression sample map includes: first, generating several pairs of control points according to the expression sample map; calculating the sample deformation field using the control points, and applying the sample deformation field to the expression sample map to generate an elastically deformed synthesized expression sample map; and then performing histogram equalization processing on the elastically deformed synthesized expression sample map.
  • the computer is first used to randomly generate several pairs of control points, including the original positions of the control points and the new positions they are moved to; at the same time, the control points manually anchored in the manual registration are added (the new positions of these control points are no longer randomly generated, but are consistent with the original positions).
  • the deformation field is calculated and applied to the expression sample image before registration to form a randomly deformed expression sample image.
  • the original image of the expression graph and the composite image of the elastic deformation are shown.
  • the left one is the original image
  • the right one is the composite image.
  • the composite image further adds elastic deformation to the original image and retains its rigid properties, such as position and size.
  • This composite method is used to amplify the training data by multiple times (10 times) for training the machine learning model.
  • this embodiment optimizes the smoothness of the output registration field and improves the accuracy of the registration result.
  • the smoothing layer module is a sliding convolution kernel of a preset size.
  • the above-mentioned tissue photo sample image and expression amount sample image after histogram equalization are used as feature data
  • the data file and the expression amount sample image after registration are used as label data to train the machine learning model, which may specifically include: creating a smoothing layer module, which can be a sliding convolution kernel of a preset size (such as a kernel size of 15*15); then adding the smoothing layer module to the preset position of the preset image registration model (such as the VoxelMorph model) after the activation function layer; then based on the feature data and label data, the preset image registration model is trained, and the preset image registration model that meets the training standard is used as the machine learning model.
  • a smoothing layer module which can be a sliding convolution kernel of a preset size (such as a kernel size of 15*15)
  • adding the smoothing layer module to the preset position of the preset image registration model (such as the VoxelMorph model) after the activation function layer
  • a BlurConv module smoothing layer module
  • its essence is to apply a two-dimensional sliding convolution kernel with a kernel size of 15*15 (other sizes can also be used according to actual needs, such as any other kernel size greater than 3*3, etc.), and the kernel calculates the median of the elements at the covered position when it slides to different positions of the input feature map (average kernel, Gaussian kernel, maximum kernel, etc. can also be used according to actual needs).
  • Add the BlurConv module to the last activation function layer and the second to last activation function layer of the VoxelMorph model preset image registration model.
  • the model structure diagram is shown in Figure 10.
  • the tissue photos and expression maps are used as input in pairs, where the tissue photos are all real data, some expression maps are real data, and some are deformed images randomly synthesized based on real data; the expression map and data file corresponding to the manually corrected deformation of the input data are used as labels; the loss function combines the difference between the label image and the predicted image, and the gradient of the x and y directions of the deformation field.
  • the loss function of the training set continues to decrease, and the loss function of the validation set reaches convergence and slowly rises after a certain number of steps; the training ends when the loss function of the validation set begins to rise, and the model is saved.
  • the input of the model is a tissue photograph sample map (alignment reference map) and an expression sample map
  • the intermediate output (marked with the ‘Local field’ position) is the deformation field predicted by the model.
  • the model structure adjacent to the input part is based on the transformation of the UNet and SpatialTransformer models. The specific changes are shown in the shaded arrows in FIG10 : After the information passes through the decoder and finally becomes a feature map of the same size as the input image, after each convolution, it undergoes the smoothing process of the BlurConv layer.
  • the models used in the prior art mainly learn the semantic information of images at a low-resolution level, which can be understood as the basic units used for registration tasks in expression maps and tissue photographs (such as the line shape of brain white matter); and learn how to register images in each part at a high-resolution level.
  • the low-resolution level extracts the characteristic shape of the image, it does not connect the neighboring areas in Euclidean space. Therefore, adding a smoothing layer to the model can fuse the information of each 15*15 neighboring area, thus creating a connection between neighboring areas. This allows areas without obvious features (areas with low brightness) to be affected by neighboring areas with obvious features, thereby making the registration field output by the model smoother.
  • the feature map has been restored to the same size as the input image, so it belongs to the above-mentioned high-resolution level learning.
  • the convolutional layer continuously fuses the features of different channels, and finally outputs the results of only two channels (representing the registration fields in the horizontal and vertical directions respectively).
  • Adding the smoothing layer not only to the output layer of these four layers, but also to the position farther away from the output layer can make the encoder and decoder at the lower level be affected by the neighboring region information fusion operation during training, optimize the degree of fitting, and finally show the accuracy improvement.
  • Step 202 When image registration is required, a tissue photograph and an expression quantity map based on the tissue photograph are obtained.
  • the tissue photograph and the expression level map based on the tissue photograph may be two images to be registered.
  • the process of acquiring the expression level map includes: generating the expression level map according to the abundance statistics of the expression level data.
  • the expression map is generated according to the abundance statistics of the expression data, including: dividing the minimum square range covered by the target tissue into a number of adjacent square areas according to the coordinates of the expression data; counting the total captured amount of captured genes in each square area; and drawing an expression map based on the total captured amount of genes counted correspondingly in each square area, wherein the pixel position in the expression map corresponds to the relative position of each square area in the minimum square range covered by the target tissue, and the pixel value of the pixel position is the total captured amount of genes counted correspondingly in each square area.
  • the smallest square range covered by the entire tissue is divided into several small, adjacent square ranges, and each small square range is also called a "Bin set"; then, the total captured amount of genes represented by the above expression data is counted in each Binset; finally, an image is drawn based on the gene statistics of these Binsets, and the pixel position of the image corresponds to the relative position of each Binset in the smallest square area covered by the above tissue (for example, the position of the pixel corresponding to the Binset in the first row and first column on the figure is also the first row and first column), and the pixel value is the total gene captured amount of the Binset.
  • Step 203 Perform histogram equalization processing on the tissue photography image and the expression level image respectively.
  • the tissue photograph is subjected to histogram equalization processing, which may specifically include: processing the tissue photograph into a grayscale image and then performing histogram equalization processing.
  • Step 204 using the tissue photograph after the histogram equalization processing as a reference image, registering the expression quantity map after the histogram equalization processing.
  • Step 205 input the expression quantity image after registration and the tissue photograph image after histogram equalization processing into a preset machine learning model for calculation to obtain deformation field information corresponding to the expression quantity image.
  • Step 206 perform linear interpolation calculation on the deformation field information corresponding to the expression amount graph to obtain updated deformation field information.
  • Step 207 bring the horizontal and vertical coordinates in the spatiotemporal group expression data corresponding to the expression graph into the updated deformation field information, and calculate the coordinate information of the expression graph after registration.
  • a First, generate an expression map based on the abundance statistics of the expression data.
  • b Process the tissue photograph into a grayscale image, then perform histogram equalization, and then perform histogram equalization on the expression map; use the tissue photograph as a reference and rigidly align the expression map.
  • c Input the rigidly aligned image pair into the machine learning model (obtained in step 201), and perform linear interpolation on the intermediate result output by the model: the deformation field, to obtain a more precise deformation prediction result.
  • d Substitute the horizontal and vertical coordinates in the spatiotemporal group expression data corresponding to the expression map into the deformation field output by c, and calculate its new coordinates after alignment.
  • Step 208 using the coordinate information of the expression quantity map after registration, obtain the expression quantity map after deformation correction.
  • Figure 12 it can be an example diagram of the overall technical process of the embodiment of Figure 2.
  • the left half of Figure 12 shows the preparation of training data and the training of the deep learning model
  • the right half shows the preprocessing of data in the actual problem when using the model for inference, the use of the model, and the post-processing of the output results, and finally the expression quantity map after deformation field correction can be output.
  • the method of this embodiment is based on a supervised machine learning method, and labels are manually provided to the model, which improves the feasibility of multimodal elastic registration.
  • the method of this embodiment automatically synthesizes elastic deformation images based on collected experimental data, reducing the cost of algorithm development.
  • the method of this embodiment optimizes the smoothness of the output registration field and improves the accuracy of the registration result.
  • this embodiment provides a data processing device for image registration.
  • the device includes: an acquisition module 31 , a processing module 32 , and a determination module 33 .
  • An acquisition module 31 is configured to acquire a first image and a second image to be registered
  • a processing module 32 configured to perform registration according to the first image and the second image
  • the acquisition module 31 is further configured to input the first image and the second image after registration into a machine learning model for calculation to obtain deformation field information, wherein the machine learning model is obtained by pre-training the model according to the registration data of the sample image, and the registration data includes: the sample image and the image after deformation correction of the sample image;
  • the determination module 33 is configured to determine corresponding coordinate information after image registration based on the deformation field information.
  • the first image is a tissue photograph
  • the second image is an expression quantity map based on the tissue photograph
  • the sample image includes: a tissue photograph sample image and an expression quantity sample map based on the tissue photograph sample image, and the image after deformation correction of the sample image is the image after deformation correction of the expression quantity sample image.
  • the processing module 32 is further configured to perform histogram equalization processing on the tissue photograph sample image and the expression sample image respectively; using the tissue photograph sample image after the histogram equalization processing as a reference image, aligning and elastically aligning the expression sample image after the histogram equalization processing to obtain a data file containing a registration relationship and an expression sample image after the alignment; using the tissue photograph sample image after the histogram equalization processing and the expression sample image as feature data, and using the data file and the expression sample image after the alignment as label data, to train and obtain the machine learning model.
  • the processing module 32 is further configured to generate several pairs of control points based on the expression sample graph; use the control points to calculate the sample deformation field, and apply the sample deformation field to the expression sample graph to generate an elastic deformation synthesized expression sample graph; and perform histogram equalization processing on the elastic deformation synthesized expression sample graph.
  • a smoothing layer module is provided after the activation function layer at a preset position of the neural network in the machine learning model, and the smoothing layer module is a sliding convolution kernel of a preset size.
  • the processing module 32 is further configured to process the tissue photograph sample image into a grayscale image and then perform a histogram equalization process.
  • the processing module 32 is further configured to process the tissue photograph into a grayscale image and then perform a histogram equalization process.
  • the deformation field information is the deformation field information corresponding to the expression quantity graph
  • the determination module 33 is specifically configured to perform linear interpolation calculation on the deformation field information corresponding to the expression map to obtain updated deformation field information; the horizontal and vertical coordinates in the spatiotemporal group expression data corresponding to the expression map are substituted into the updated deformation field information to calculate the coordinate information of the expression map after registration.
  • the acquisition module 31 is further configured to obtain the expression quantity map after deformation correction using the coordinate information after determining the corresponding coordinate information after image registration based on the deformation field information.
  • the processing module 32 is specifically configured to perform histogram equalization processing on the first image and the second image respectively; and perform registration based on the first image and the second image after the histogram equalization processing.
  • the processing module 32 is further configured to use the tissue photograph after the histogram equalization process as a reference image to register the expression quantity map after the histogram equalization process;
  • the processing module 32 is further specifically configured to input the expression quantity image after registration and the tissue photograph image after histogram equalization processing into a preset machine learning model for calculation.
  • the acquisition module 31 is specifically configured to generate the expression level graph according to the abundance statistics of the expression level data.
  • the acquisition module 31 is further configured to divide the minimum square range covered by the target tissue into a number of adjacent square areas according to the coordinates of the expression data; count the total captured amount of the captured genes in each square area; and draw the expression map based on the total captured amount of the genes counted in each square area, wherein the pixel position in the expression map corresponds to the relative position of each square area in the minimum square range covered by the target tissue, and the pixel value of the pixel position is the total captured amount of the genes counted in each square area.
  • this embodiment further provides a computer-readable storage medium on which a computer program is stored.
  • the computer program is executed by a processor, the above method as shown in FIG. 1 and FIG. 2 is implemented.
  • the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods of various implementation scenarios of the present application.
  • a non-volatile storage medium which can be a CD-ROM, USB flash drive, mobile hard disk, etc.
  • a computer device which can be a personal computer, server, or network device, etc.
  • the embodiment of the present application also provides an electronic device, which can be specifically a personal computer, a laptop computer, etc., and the device includes a storage medium and a processor; the storage medium is used to store computer programs; the processor is used to execute the computer program to implement the above-mentioned method as shown in Figures 1 and 2.
  • the above-mentioned physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc.
  • the user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc.
  • the network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.
  • the storage medium may also include an operating system and a network communication module.
  • the operating system is a program that manages the hardware and software resources of the above-mentioned physical device, and supports the operation of the information processing program and other software and/or programs.
  • the network communication module is used to realize the communication between the components inside the storage medium, and the communication with other hardware and software in the information processing physical device.
  • the present application can be implemented by means of software plus the necessary general hardware platform, or by hardware.
  • the method of this embodiment is based on a supervised machine learning method, and labels are manually provided to the model, thereby improving the feasibility of multimodal elastic registration.
  • the method of this embodiment automatically synthesizes elastic deformation images based on the collected experimental data, thereby reducing the cost of algorithm development.
  • the method of this embodiment optimizes the smoothness of the output registration field and improves the accuracy of the registration results.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

本申请涉及一种图像配准的数据处理方法、装置及电子设备，涉及数据处理技术领域，其中方法包括：首先获取待配准的第一图像和第二图像；再根据第一图像和第二图像进行配准；然后将配准后的第一图像和第二图像，输入到机器学习模型中进行计算，获得形变场信息，其中，机器学习模型是根据样本图像的配准数据预先进行模型训练得到的，配准数据包括：样本图像和样本图像形变校正后的图像；依据形变场信息，确定图像配准后相应的坐标信息。通过应用本申请的技术方案，解决了多模态图像的弹性配准问题，容易实施，并且可提高图像配准的准确性。

Description

图像配准的数据处理方法、装置及电子设备 技术领域
本申请涉及数据处理技术领域，具体涉及一种图像配准的数据处理方法、装置及电子设备。
背景技术
图像配准指将一张以上的图像彼此进行位置和形状上的对齐；它是图像相关领域中的一个重要任务，在生物、医疗和其他工程领域均有广泛的应用。其中，多模态图像的弹性配准问题一直是图像配准中的难点问题。这里基于组织拍照图的表达量图的弹性配准就属于此范畴。
目前，现有的多模态弹性配准方案是使用无监督的深度学习方法进行两两配准。这种方式需要配准图像属于同模态，或差异较小的两个模态，以此实现每个训练轮次中的输出结果的损失计算，和对应的梯度计算，并由此开始优化模型的参数。
然而，组织拍照图和表达量图的本身模态差异巨大，难以找到一个合适的损失函数来直接衡量模型给出的结果对应的梯度。因此，目前现有技术存在难以实施、或者直接实施会导致图像配准的准确性较差的技术问题。
发明内容
有鉴于此，本申请提供了一种图像配准的数据处理方法、装置及电子设备，主要目的在于改善目前现有的多模态弹性配准方案存在难以实施、或者直接实施会导致图像配准的准确性较差的技术问题。
第一方面，本申请提供了一种图像配准的数据处理方法，包括：
获取待配准的第一图像和第二图像；
根据所述第一图像和所述第二图像进行配准；
将配准后的所述第一图像和所述第二图像，输入到机器学习模型中进行计算，获得形变场信息，其中，所述机器学习模型是根据样本图像的配准数据预先进行模型训练得到的，所述配准数据包括：所述样本图像和所述样本图像形变校正后的图像；
依据所述形变场信息，确定图像配准后相应的坐标信息。
第二方面，本申请提供了一种图像配准的数据处理装置，包括：
获取模块，被配置为获取待配准的第一图像和第二图像；
处理模块，被配置为根据所述第一图像和所述第二图像进行配准；
所述获取模块，还被配置为将配准后的所述第一图像和所述第二图像，输入到机器学习模型中进行计算，获得形变场信息，其中，所述机器学习模型是根据样本图像的配准数据预先进行模型训练得到的，所述配准数据包括：所述样本图像和所述样本图像形变校正后的图像；
确定模块，被配置为依据所述形变场信息，确定图像配准后相应的坐标信息。
第三方面，本申请提供了一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现第一方面所述的图像配准的数据处理方法。
第四方面，本申请提供了一种电子设备，包括存储介质、处理器及存储在存储介质上并可在处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现第一方面所述的图像配准的数据处理方法。
借由上述技术方案，本申请提供的一种图像配准的数据处理方法、装置及电子设备，与目前现有的多模态弹性配准方案相比，本申请提供一种有监督的机器学习方法，根据样本图像的配准数据预先进行模型训练得到机器学习模型，该配准数据包括：样本图像和样本图像形变校正后的图像。在使用时，首先对待配准的第一图像和第二图像进行配准；然后将配准后的第一图像和第二图像，输入到机器学习模型中进行计算，获得形变场信息，进而依据形变场信息，确定图像配准后相应的坐标信息。通过应用本申请的技术方案，解决了多模态图像的弹性配准问题，容易实施，并且可提高图像配准的准确性。
上述说明仅是本申请技术方案的概述，为了能够更清楚了解本申请的技术手段，而可依照说明书的内容予以实施，并且为了让本申请的上述和其它目的、特征和优点能够更明显易懂，以下特举本申请的具体实施方式。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本申请的实施例，并与说明书一起用于解释本申请的原理。
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，对于本领域普通技术人员而言，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1示出了本申请实施例提供的一种图像配准的数据处理方法的流程示意图；
图2示出了本申请实施例提供的另一种图像配准的数据处理方法的流程示意图；
图3示出了本申请实施例提供的表达量样本图的示例图；
图4示出了本申请实施例提供的表达量样本图直方图均衡化处理的示例图；
图5示出了本申请实施例提供的组织拍照样本图直方图均衡化处理的示例图；
图6示出了本申请实施例提供的刚性配准的变换关系的存储形式的示意图；
图7示出了本申请实施例提供的弹性配准的变换关系的存储形式的示意图；
图8示出了本申请实施例提供的表达量样本图经过刚性变换和弹性变换的示意图；
图9示出了本申请实施例提供的弹性形变图像合成的效果示意图；
图10示出了本申请实施例提供的模型结构示例的示意图；
图11示出了本申请实施例提供的模型结构优化后的效果示意图；
图12示出了本申请实施例提供的总体示例流程的示意图；
图13示出了本申请实施例提供的一种图像配准的数据处理装置的结构示意图。
具体实施方式
为了能够更清楚地理解本申请的上述目的、特征和优点，下面将对本申请的方案进行进一步描述。需要说明的是，在不冲突的情况下，本申请的实施例及实施例中的特征可以相互组合。
为了改善目前现有的多模态弹性配准方案存在难以实施、或者直接实施会导致图像配准的准确性较差的技术问题。本实施例提供了一种图像配准的数据处理方法，如图1所示，该方法包括：
步骤101、获取待配准的第一图像和第二图像。
在实际应用中对于图像配准，按对齐所需的变换方式，配准问题可分为刚性和弹性等配准问题；按照输入图像的模态差异，它可分为单模态和多模态的配准问题。而在本实施例中，待配准的第一图像和第二图像可为同模态或者不同模态的图像。
步骤102、根据第一图像和第二图像进行配准。
具体可根据第一图像和第二图像进行刚性配准等。
在一些示例中，步骤102具体可包括：对第一图像和第二图像分别进行直方图均衡化处理；根据直方图均衡化处理后的第一图像和第二图像进行配准。
直方图均衡化是为了增强图像的对比度，主要是将图像的直方图分布变成近似均匀分布，使图像各个像素值之间的差异加强，呈现出对比度增强的效果。以便后续更好地进行图像配准。
在实际应用中，第一图像和第二图像这两张图像很可能存在位置和大小的不对应情况，因此为了保证后续机器学习模型的计算效率和计算准确性，本实施例需要根据直方图均衡化处理后的第一图像和第二图像进行配准。例如，以第一图像为参考图像，对第二图像进行配准，如翻转、平移和旋转等。可选的，这里的配准可以包括位置和/或大小的修改，不包括形状的扭动、改变。
步骤103、将配准后的第一图像和第二图像，输入到机器学习模型中进行计算，获得形变场信息。
其中，机器学习模型可以是根据样本图像的配准数据预先进行模型训练得到的，样本图像的配准数据可包括：样本图像和样本图像形变校正后的图像。形变场信息可包含图像配准的弹性变换(变形)信息。
步骤104、依据形变场信息，确定图像配准后相应的坐标信息。
本实施例在对第一图像和第二图像进行图像配准的过程中，可利用步骤104得到的图像配准后相应的坐标信息，获得经过形变场校正之后的图像信息。
目前，针对多模态弹性配准问题，尤其是模态差异较大的弹性配准问题，一直是配准中的难点问题。传统的特征点提取算法难以找到多模态图像上的对应特征点，且难以找全所有特征点，因此在多模态弹性配准问题上难以发挥作用。而本实施例与目前现有的多模态弹性配准方案(使用无监督的深度学习方案)相比，提供了一种有监督的机器学习方法，根据样本图像的配准数据预先进行模型训练得到机器学习模型。在使用时，首先对待配准的第一图像和第二图像分别进行直方图均衡化处理，以增强图像的对比度，方便模型预测；再根据直方图均衡化处理后的第一图像和第二图像进行配准；然后将配准后的第一图像和第二图像，输入到机器学习模型中进行计算，获得形变场信息，进而依据形变场信息，确定图像配准后相应的坐标信息。通过应用本实施例的技术方案，可解决多模态图像的弹性配准问题，并且除此之外还可解决同模态图像的弹性配准问题。总体来说，针对同模态和多模态的弹性配准，均可使用到本实施例的技术方案，如医疗影像的弹性配准，生物实验图像的弹性配准等，可提高图像配准的准确性。
进一步的，作为上述实施例的细化和扩展，为了完整说明本实施例方法的具体实现过程，以图1方法中的第一图像是组织拍照图、第二图像是基于组织拍照图的表达量图作为示例(如以猴脑的表达量数据和组织拍照图进行配准的案例为例)，本实施例提供了如图2所示的具体方法，该方法包括：
步骤201、预先训练得到机器学习模型。
机器学习模型可以是根据样本图像的配准数据预先进行模型训练得到的，该配准数据包括：样本图像和样本图像形变校正后的图像。该样本图像具体可包括：组织拍照样本图和基于组织拍照样本图的表达量样本图，样本图像形变校正后的图像为表达量样本图形变校正后的图像。
可选的，机器学习模型的训练过程，包括：首先对组织拍照样本图和表达量样本图分别进行直方图均衡化处理；再以直方图均衡化处理后的组织拍照样本图为参考图像，对直方图均衡化处理后的表达量样本图进行刚性配准和弹性配准，得到包含配准关系的数据文件和配准后的表达量样本图；然后将直方图均衡化处理后的组织拍照样本图和表达量样本图作为特征数据，并将该数据文件和配准后的表达量样本图作为标签数据，训练得到机器学习模型。
在本实施例中，为了能够准确训练得到有效的机器学习模型，需要进行数据准备，即准备模型训练所需的数据。首先根据表达量样本数据的丰度统计生成表达量样本图。表达量样本数据为记录时空转录组表达量样本数据的表格，其每一行表示的是，一个测序位点捕获的某一种基因的情况；每一列表示的是，捕获基因情况中的基因名称、位点坐标、捕获量等属性。如下表1所示，为一份表达量数据的前8行实例。
表1

表达量样本图是由表达量样本数据的丰度统计得到的，统计过程包括：首先根据表达量样本数据的坐标，把整个组织覆盖的最小方形范围划分为若干个小的，互相紧邻的正方形范围，每个小的正方形范围又被称为“Bin set”；然后统计上述表达量数据表示的，在每个Binset范围内统计被捕获的基因的总捕获量；最后根据这些Binset的基因统计结果绘制一张图像，该图像的像素位置对应各个Binset在上述组织覆盖的最小方形区域中的相对位置(例如第一行第一列的Binset对应的像素在图上的位置也是第一行第一列)，像素值是该Binset的基因总捕获量。表达量样本图示例如图3所示。
在得到组织拍照样本图和表达量样本图之后，进行直方图均衡化处理。其中，直方图均衡化是为了增强图像的对比度，方便人工配准和模型训练。它主要是将图像的直方图分布变成近似均匀分布，使图像各个像素值之间的差异加强，呈现出对比度增强的效果。而由于组织拍照样本图的颜色信息并不对组织拍照样本图和表达量样本图的配准起到决定性作用，因此出于节约存储和内存空间的考虑，将其转化为灰度图。因此相应可选的，对组织拍照样本图进行直方图均衡化处理，具体可包括：将组织拍照样本图处理成灰度图后，再进行直方图均衡化处理。
例如，如图4所示，对表达量样本图进行直方图均衡化。如图5所示，对真实的组织拍照样本图转换为灰度图后进行直方图均衡化。
在得到直方图均衡化处理后的组织拍照样本图和表达量样本图之后，对于用于模型训练的图像，如果两张图像存在位置和大小的不对应，则使用ImageJ的TrakEM2插件，先以组织拍照样本图为参考图像进行刚性配准，如果有其他方法，如自动化的算法，也可以采用其他方法进行刚性配准；然后对他们进行弹性的配准。输出包含配准关系的数据文件(即TrakEM2的项目文件)，以及配准后的表达量样本图。TrakEM2的项目文件中，存储着在此软件插件中，进行手动刚性与弹性配准时，插件记录下的变换关系。
其中的刚性配准的变换关系的存储形式为刚性变换矩阵，它描述了被配准的图像的翻转情况、旋转角度与横纵的平移量，如图6所示。其中的弹性配准的变换关系的存储形式为人为指定的控制点在变换前后的坐标，基于这些坐标，可以计算得到整张图像的形变场，此坐标示例如图7所示。配准后的表达量样本图指的是表达量样本图以Blockface图为参考图像，进行刚性变换(翻转、平移和旋转)，以及弹性变换(变形)后，最终生成的图像。如图8所示，展示的是表达量样本图在配准之前，经过刚性变换，和弹性变换后的结果。
在模型训练过程中，基于上千对被配准图像和参考图像的数据驱动，以此实现多层神经网络的参数拟合；而基于组织拍照图的表达量图弹性配准任务，出于其实验成本和测序时间的限制，研发人员获得大规模的数据十分困难。因此，具有成本高昂的问题。为此本实施例基于收集的实验数据，可自动化地进行弹性形变图像的合成，降低了算法开发的成本。相应可选的，上述对表达量样本图进行直方图均衡化处理，包括：首先根据表达量样本图生成若干对控制点；利用控制点计算出样本形变场，并将样本形变场应用在表达量样本图上，生成弹性形变合成的表达量样本图；然后对弹性形变合成的表达量样本图进行直方图均衡化处理。
例如，对于用于模型训练的图像，首先使计算机随机生成若干对控制点，具体信息包括控制点的原始位置和移动到的新位置；同时加入手动配准中人工锚定的控制点(这些控制点的新位置不再随机生成，而是和原始位置保持一致)。使用Moving Least Squares Deformation理论，计算出形变场，并将此形变场应用在配准前的表达量样本图像上，形成随机变形的表达量样本图。
如图9所示，分别展示出了表达量图的原始图像、和弹性形变的合成图像。其中左侧的是原始图像，右侧的是合成图像。可以看到，合成图像在原始图像的基础上，进一步增加了弹性的形变，并且保留了其刚性的特性：如位置和大小。使用该合成方法扩增了多倍(10倍)的训练数据，用于机器学习模型的训练。
目前的技术对应模型中，没有任何机制约束所给出配准场的平滑程度，这样不符合组织拍照图和表达量图之间形变较为平滑的常识。在图像配准过程中，会存在准确性差的技术问题。为此本实施例优化了输出配准场的平滑程度，提高了配准结果的准确性，本实施例所采用的机器学习模型中神经网络的预设位置的激活函数层之后具有平滑层模块，该平滑层模块为预设尺寸的滑动卷积核。相应可选的，上述将直方图均衡化处理后的组织拍照样本图和表达量样本图作为特征数据，并将数据文件和配准后的表达量样本图作为标签数据，训练得到机器学习模型，具体可包括：创建平滑层模块，该平滑层模块可为预设尺寸(如核的大小为15*15)的滑动卷积核；再将平滑层模块添加到预设图像配准模型(如VoxelMorph模型)的预设位置的激活函数层之后；然后基于特征数据和标签数据，训练预设图像配准模型，并将训练达标的预设图像配准模型作为机器学习模型。
例如，搭建BlurConv模块(平滑层模块)，其本质是应用一个二维的滑动卷积核，核的大小为15*15(也可根据实际需求使用其他尺寸，如也可以使用任意其他的超过3*3的核大小等)，核所计算的是其滑动到输入的特征图的不同位置时所覆盖位置上元素的中位数(也可根据实际需求选择使用平均数核，高斯核，最大值核等)。将BlurConv模块加入到VoxelMorph模型(预设图像配准模型)的最后一个激活函数层和倒数第二个激活函数层之后。模型结构图如图10所示。
将组织拍照图和表达量图两两一对作为输入，其中组织拍照图均为真实数据，表达量图有些是真实数据，有些是基于真实数据随机合成的形变图像；将对应于输入数据的人工校正好形变的表达量图和数据文件作为标签；损失函数结合标签图像和预测图像的差异、和形变场的x、y方向的梯度。训练开始，训练集的损失函数不断下降，验证集的损失函数在一定步数后达到收敛和缓慢上升；在验证集损失函数开始上升时结束训练，保存模型。
如图10所示，展现了本发明所采用的机器学习模型的结构。该模型的输入为组织拍照样本图(配准参考图)和表达量样本图，中间输出(标注‘Local field’位置)的是模型预测的形变场。该模型在紧邻输入部分的是基于UNet和SpatialTransformer模型改造的模型结构。具体的改变部分如图10加阴影的箭头所示：在信息经过解码器，最终成为与输入图像相同大小的特征图之后，在各次卷积之后，又分别经历BlurConv层的平滑处理。
为了说明本机器学习模型的结构优化设计效果，分别在不同样例上测试了不做优化和做了优化后的模型效果，如图11所示。可以看到，本实施例优化过的模型输出的配准场更加平滑，更加符合类似于手动配准得到的配准场的数据分布。为了揭示本实施例中的平滑层提高模型输出准确性的原理，在此公开相关的研究过程：
分别测试了(1)使用现有技术方案中的模型(2)不改变现有模型，只在其后面增加图像的平滑处理(3)在最后一层激活层后加入平滑层(BlurConv)，再训练模型(4)在倒数第二层和最后一层激活层后同时使用平滑层，再训练模型这四种方案的效果，并使用手动得到的配准场，计算模型在组织区域推导的配准场的准确性。在其中4个测试数据上，均方误差(Mean Squared Error，MSE)和平均绝对误差(Mean Absolute Error，MAE)。可以看到，虽然在模型输出数据上直接应用平滑处理，可以优化模型输出的准确性，但是在模型中添加平滑层，再训练模型，使得输出的结果更加准确。其中在最后和倒数第二个激活函数层之后同时加入平滑层，会比仅在最后一个激活函数层后加入平滑层，输出的准确性进一步提高。
以上的研究结果解释了创新点能显著提高准确性的原理：
A、现有技术中所用到的模型，主要是在低分辨率层面上学习图像的语义信息，在本任务中可以理解为表达量图和组织拍照图中用于配准任务的基本单元(大脑白质的线条形状等)；并在高分辨率层面上学习图片在每个局部如何进行配准。在这里，低分辨层面虽然提取了图像的特征形状，但是没有从欧式空间上把近邻的区域联系在一起。因此，在模型中加入平滑层，可以对每个15*15的近邻区域做信息的融合，也就创立了近邻区域的联系。这使得没有明显特征的区域(亮度低的区域)也可以受到近邻的有明显特征的区域的影响，从而使模型输出的配准场更加平滑。
B、在模型的最后四层处，特征图已经恢复到输入图像相同的大小，因此属于上述的高分辨率层面学习。在这四层中，卷积层不断地融合不同通道的特征，最终输出仅有两个通道的结果(分别表示横纵方向的配准场)。将平滑层不仅加入到这4层中的输出层，也加入到更远离输出层的位置，能使得更底层位置的编码器和解码器在训练中受到近邻区域信息融合操作的影响，优化拟合的程度，最终显示为准确性提升。
步骤202、在需要进行图像配准时，获取组织拍照图和基于组织拍照图的表达量图。
组织拍照图和基于组织拍照图的表达量图可为待配准的两个图像。可选的，表达量图的获取过程，包括：根据表达量数据的丰度统计，生成表达量图。
示例性的，根据表达量数据的丰度统计，生成所述表达量图，包括：根据表达量数据的坐标，将目标组织覆盖的最小方形范围划分为若干个互相紧邻的正方形区域；在每个正方形区域内统计被捕获的基因的总捕获量；基于每个正方形区域内对应统计到的基因的总捕获量，绘制得到表达量图，其中，表达量图中的像素位置对应各个正方形区域在目标组织覆盖的最小方形范围中的相对位置，像素位置的像素值为各个正方形区域内对应统计到的基因的总捕获量。
例如，首先根据表达量数据的坐标，把整个组织覆盖的最小方形范围划分为若干个小的，互相紧邻的正方形范围，每个小的正方形范围又被称为“Bin set”；然后统计上述表达量数据表示的，在每个Binset范围内统计被捕获的基因的总捕获量；最后根据这些Binset的基因统计结果绘制一张图像，该图像的像素位置对应各个Binset在上述组织覆盖的最小方形区域中的相对位置(例如第一行第一列的Binset对应的像素在图上的位置也是第一行第一列)，像素值是该Binset的基因总捕获量。
步骤203、对组织拍照图和表达量图分别进行直方图均衡化处理。
为了节约存储和内存空间，并且与机器学习模型的训练过程对应，对组织拍照图进行直方图均衡化处理，具体可包括：将组织拍照图处理成灰度图后，再进行直方图均衡化处理。
步骤204、以直方图均衡化处理后的组织拍照图为参考图像，对直方图均衡化处理后的表达量图进行配准。
步骤205、将配准后的所述表达量图像和直方图均衡化处理后的组织拍照图像，输入到预设机器学习模型中进行计算，获得表达量图对应的形变场信息。
步骤206、将表达量图对应的形变场信息进行线性插值计算，得到更新后的形变场信息。
步骤207、将表达量图对应的时空组表达量数据中的横纵坐标，带入到更新后的形变场信息中，计算得到表达量图配准后的坐标信息。
例如，a、首先根据表达量数据的丰度统计生成表达量图。b、将组织拍照图处理成灰度图，然后进行直方图均衡化，然后将表达量图进行直方图均衡化；将表达量图以组织拍照图为参照，刚性配准好。c、将刚性配准好的图像对输入到机器学习模型(步骤201中得到的)中，将模型输出的中间结果：形变场，进行线性插值，得到更加精细的形变预测结果。d、将表达量图对应的时空组表达量数据中的横纵坐标，带入到c输出的形变场中，计算得到其配准后的新坐标。
步骤208、利用表达量图配准后的坐标信息，获取经过形变校正后的表达量图。
例如，如图12所示，可为图2实施例的整体技术流程示例图。该图12的左半部分展示了训练数据的准备和深度学习模型的训练，右半部分展示了使用模型做推断时，实际问题中数据的预处理，和对模型的使用，以及输出结果的后处理，最后可输出经过形变场校正之后的表达量图。
与目前现有技术相比，本实施例方法基于有监督的机器学习方法，人工给模型提供标签，提高了多模态弹性配准的可行性。本实施例方法基于收集的实验数据，自动化地进行弹性形变图像的合成，降低了算法开发的成本。本实施例方法优化了输出配准场的平滑程度，提高了配准结果的准确性。
进一步的，作为图1和图2所示方法的具体实现，本实施例提供了一种图像配准的数据处理装置，如图13所示，该装置包括：获取模块31、处理模块32、确定模块33。
获取模块31，被配置为获取待配准的第一图像和第二图像；
处理模块32，被配置为根据所述第一图像和所述第二图像进行配准；
所述获取模块31，还被配置为将配准后的所述第一图像和所述第二图像，输入到机器学习模型中进行计算，获得形变场信息，其中，所述机器学习模型是根据样本图像的配准数据预先进行模型训练得到的，所述配准数据包括：所述样本图像和所述样本图像形变校正后的图像；
确定模块33，被配置为依据所述形变场信息，确定图像配准后相应的坐标信息。
在具体的应用场景中，可选的，所述第一图像为组织拍照图，所述第二图像为基于所述组织拍照图的表达量图；所述样本图像包括：组织拍照样本图和基于所述组织拍照样本图的表达量样本图，所述样本图像形变校正后的图像为所述表达量样本图形变校正后的图像。
在具体的应用场景中，处理模块32，还被配置为对所述组织拍照样本图和所述表达量样本图分别进行直方图均衡化处理；以直方图均衡化处理后的所述组织拍照样本图为参考图像，对直方图均衡化处理后的所述表达量样本图进行配准和弹性配准，得到包含配准关系的数据文件和配准后的表达量样本图；将直方图均衡化处理后的所述组织拍照样本图和所述表达量样本图作为特征数据，并将所述数据文件和配准后的表达量样本图作为标签数据，训练得到所述机器学习模型。
在具体的应用场景中，处理模块32，具体还被配置为根据所述表达量样本图生成若干对控制点；利用所述控制点计算出样本形变场，并将所述样本形变场应用在所述表达量样本图上，生成弹性形变合成的表达量样本图；对所述弹性形变合成的表达量样本图进行直方图均衡化处理。
在具体的应用场景中，所述机器学习模型中神经网络的预设位置的激活函数层之后具有平滑层模块，所述平滑层模块为预设尺寸的滑动卷积核。
在具体的应用场景中，处理模块32，具体还被配置为将所述组织拍照样本图处理成灰度图后，再进行直方图均衡化处理。
处理模块32，具体还被配置为将所述组织拍照图处理成灰度图后，再进行直方图均衡化处理。
在具体的应用场景中，可选的，所述形变场信息为所述表达量图对应的形变场信息；
相应的，确定模块33，具体被配置为将所述表达量图对应的形变场信息进行线性插值计算，得到更新后的形变场信息；将所述表达量图对应的时空组表达量数据中的横纵坐标，带入到所述更新后的形变场信息中，计算得到所述表达量图配准后的坐标信息。
在具体的应用场景中，获取模块31，还被配置为在所述依据所述形变场信息，确定图像配准后相应的坐标信息之后，利用所述坐标信息，获取经过形变校正后的表达量图。
在具体的应用场景中，处理模块32，具体被配置为对所述第一图像和所述第二图像分别进行直方图均衡化处理；根据直方图均衡化处理后的所述第一图像和所述第二图像进行配准。
在具体的应用场景中，处理模块32，具体还被配置为以直方图均衡化处理后的所述组织拍照图为参考图像，对直方图均衡化处理后的所述表达量图进行配准；
处理模块32，具体还被配置为将配准后的所述表达量图像和直方图均衡化处理后的所述组织拍照图像，输入到预设机器学习模型中进行计算。
在具体的应用场景中，获取模块31，具体被配置为根据表达量数据的丰度统计，生成所述表达量图。
在具体的应用场景中，获取模块31，具体还被配置为根据表达量数据的坐标，将目标组织覆盖的最小方形范围划分为若干个互相紧邻的正方形区域；在每个正方形区域内统计被捕获的基因的总捕获量；基于每个正方形区域内对应统计到的基因的总捕获量，绘制得到所述表达量图，其中，所述表达量图中的像素位置对应各个正方形区域在所述目标组织覆盖的最小方形范围中的相对位置，所述像素位置的像素值为各个正方形区域内对应统计到的基因的总捕获量。
需要说明的是，本实施例提供的一种图像配准的数据处理装置所涉及各功能单元的其它相应描述，可以参考图1和图2中的对应描述，在此不再赘述。
基于上述如图1和图2所示方法，相应的，本实施例还提供了一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现上述如图1和图2所示的方法。
基于这样的理解，本申请的技术方案可以以软件产品的形式体现出来，该软件产品可以存储在一个非易失性存储介质(可以是CD-ROM，U盘，移动硬盘等)中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施场景的方法。
基于上述如图1和图2所示的方法，以及图13所示的虚拟装置实施例，为了实现上述目的，本申请实施例还提供了一种电子设备，具体可为个人计算机、笔记本电脑等，该设备包括存储介质和处理器；存储介质，用于存储计算机程序；处理器，用于执行计算机程序以实现上述如图1和图2所示的方法。
可选的，上述实体设备还可以包括用户接口、网络接口、摄像头、射频(Radio Frequency，RF)电路，传感器、音频电路、WI-FI模块等等。用户接口可以包括显示屏(Display)、输入单元比如键盘(Keyboard)等，可选用户接口还可以包括USB接口、读卡器接口等。网络接口可选的可以包括标准的有线接口、无线接口(如WI-FI接口)等。
本领域技术人员可以理解，本实施例提供的上述实体设备结构并不构成对该实体设备的限定，可以包括更多或更少的部件，或者组合某些部件，或者不同的部件布置。
存储介质中还可以包括操作系统、网络通信模块。操作系统是管理上述实体设备硬件和软件资源的程序，支持信息处理程序以及其它软件和/或程序的运行。网络通信模块用于实现存储介质内部各组件之间的通信，以及与信息处理实体设备中其它硬件和软件之间通信。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到本申请可以借助软件加必要的通用硬件平台的方式来实现，也可以通过硬件实现。通过应用本实施例的方案，与目前现有技术相比，本实施例方法基于有监督的机器学习方法，人工给模型提供标签，提高了多模态弹性配准的可行性。本实施例方法基于收集的实验数据，自动化地进行弹性形变图像的合成，降低了算法开发的成本。本实施例方法优化了输出配准场的平滑程度，提高了配准结果的准确性。
需要说明的是，在本文中，诸如“第一”和“第二”等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。
以上所述仅是本申请的具体实施方式，使本领域技术人员能够理解或实现本申请。对这些实施例的多种修改对本领域的技术人员来说将是显而易见的，本文中所定义的一般原理可以在不脱离本申请的精神或范围的情况下，在其它实施例中实现。因此，本申请将不会被限制于本文所述的这些实施例，而是要符合与本文所申请的原理和新颖特点相一致的最宽的范围。

Claims (15)

  1. 一种图像配准的数据处理方法，其特征在于，包括：
    获取待配准的第一图像和第二图像；
    根据所述第一图像和所述第二图像进行配准；
    将配准后的所述第一图像和所述第二图像，输入到机器学习模型中进行计算，获得形变场信息，其中，所述机器学习模型是根据样本图像的配准数据预先进行模型训练得到的，所述配准数据包括：所述样本图像和所述样本图像形变校正后的图像；
    依据所述形变场信息，确定图像配准后相应的坐标信息。
  2. 根据权利要求1所述的方法，其特征在于，所述第一图像为组织拍照图，所述第二图像为基于所述组织拍照图的表达量图；所述样本图像包括：组织拍照样本图和基于所述组织拍照样本图的表达量样本图，所述样本图像形变校正后的图像为所述表达量样本图形变校正后的图像。
  3. 根据权利要求2所述的方法，其特征在于，所述机器学习模型的训练过程，包括：
    对所述组织拍照样本图和所述表达量样本图分别进行直方图均衡化处理；
    以直方图均衡化处理后的所述组织拍照样本图为参考图像，对直方图均衡化处理后的所述表达量样本图进行刚性配准和弹性配准，得到包含配准关系的数据文件和配准后的表达量样本图；
    将直方图均衡化处理后的所述组织拍照样本图和所述表达量样本图作为特征数据，并将所述数据文件和配准后的表达量样本图作为标签数据，训练得到所述机器学习模型。
  4. 根据权利要求3所述的方法，其特征在于，对所述表达量样本图进行直方图均衡化处理，包括：
    根据所述表达量样本图生成若干对控制点；
    利用所述控制点计算出样本形变场，并将所述样本形变场应用在所述表达量样本图上，生成弹性形变合成的表达量样本图；
    对所述弹性形变合成的表达量样本图进行直方图均衡化处理。
  5. 根据权利要求3所述的方法，其特征在于，所述机器学习模型中神经网络的预设位置的激活函数层之后具有平滑层模块，所述平滑层模块为预设尺寸的滑动卷积核。
  6. 根据权利要求3所述的方法，其特征在于，对所述组织拍照样本图进行直方图均衡化处理，包括：
    将所述组织拍照样本图处理成灰度图后，再进行直方图均衡化处理。
    对所述第一图像进行直方图均衡化处理，包括：
    将所述组织拍照图处理成灰度图后，再进行直方图均衡化处理。
  7. 根据权利要求2所述的方法，其特征在于，所述形变场信息为所述表达量图对应的形变场信息；
    所述依据所述形变场信息，确定图像配准后相应的坐标信息，包括：
    将所述表达量图对应的形变场信息进行线性插值计算，得到更新后的形变场信息；
    将所述表达量图对应的时空组表达量数据中的横纵坐标，带入到所述更新后的形变场信息中，计算得到所述表达量图配准后的坐标信息。
  8. 根据权利要求7所述的方法，其特征在于，在所述依据所述形变场信息，确定图像配准后相应的坐标信息之后，所述方法还包括：
    利用所述坐标信息，获取经过形变校正后的表达量图。
  9. 根据权利要求2所述的方法，其特征在于，所述根据所述第一图像和所述第二图像进行配准，包括：
    对所述第一图像和所述第二图像分别进行直方图均衡化处理；
    根据直方图均衡化处理后的所述第一图像和所述第二图像进行配准。
  10. 根据权利要求9所述的方法，其特征在于，所述根据直方图均衡化处理后的所述第一图像和所述第二图像进行配准，包括：
    以直方图均衡化处理后的所述组织拍照图为参考图像，对直方图均衡化处理后的所述表达量图进行配准；
    所述将配准后的所述第一图像和所述第二图像，输入到机器学习模型中进行计算，包括：
    将配准后的所述表达量图像和直方图均衡化处理后的所述组织拍照图像，输入到预设机器学习模型中进行计算。
  11. 根据权利要求2至10中任一项所述的方法，其特征在于，所述表达量图的获取过程，包括：
    根据表达量数据的丰度统计，生成所述表达量图。
  12. 根据权利要求11所述的方法，其特征在于，所述根据表达量数据的丰度统计，生成所述表达量图，包括：
    根据表达量数据的坐标，将目标组织覆盖的最小方形范围划分为若干个互相紧邻的正方形区域；
    在每个正方形区域内统计被捕获的基因的总捕获量；
    基于每个正方形区域内对应统计到的基因的总捕获量，绘制得到所述表达量图，其中，所述表达量图中的像素位置对应各个正方形区域在所述目标组织覆盖的最小方形范围中的相对位置，所述像素位置的像素值为各个正方形区域内对应统计到的基因的总捕获量。
  13. 一种图像配准的数据处理装置，其特征在于，包括：
    获取模块，被配置为获取待配准的第一图像和第二图像；
    处理模块，被配置为根据所述第一图像和所述第二图像进行配准；
    所述获取模块，还被配置为将配准后的所述第一图像和所述第二图像，输入到机器学习模型中进行计算，获得形变场信息，其中，所述机器学习模型是根据样本图像的配准数据预先进行模型训练得到的，所述配准数据包括：所述样本图像和所述样本图像形变校正后的图像；
    确定模块，被配置为依据所述形变场信息，确定图像配准后相应的坐标信息。
  14. 一种计算机可读存储介质，其上存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现权利要求1至12中任一项所述的方法。
  15. 一种电子设备，包括存储介质、处理器及存储在存储介质上并可在处理器上运行的计算机程序，其特征在于，所述处理器执行所述计算机程序时实现权利要求1至12中任一项所述的方法。
PCT/CN2023/096114 2023-05-24 2023-05-24 图像配准的数据处理方法、装置及电子设备 Ceased WO2024239287A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202380083330.9A CN120457453A (zh) 2023-05-24 2023-05-24 图像配准的数据处理方法、装置及电子设备
PCT/CN2023/096114 WO2024239287A1 (zh) 2023-05-24 2023-05-24 图像配准的数据处理方法、装置及电子设备

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2023/096114 WO2024239287A1 (zh) 2023-05-24 2023-05-24 图像配准的数据处理方法、装置及电子设备

Publications (1)

Publication Number Publication Date
WO2024239287A1 true WO2024239287A1 (zh) 2024-11-28

Family

ID=93588796

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/096114 Ceased WO2024239287A1 (zh) 2023-05-24 2023-05-24 图像配准的数据处理方法、装置及电子设备

Country Status (2)

Country Link
CN (1) CN120457453A (zh)
WO (1) WO2024239287A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN120374647A (zh) * 2025-04-11 2025-07-25 北京大学第三医院（北京大学第三临床医学院） 一种卵巢癌病灶分割模型的训练方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210209775A1 (en) * 2018-12-19 2021-07-08 Shanghai Sensetime Intelligent Technology Co., Ltd. Image Processing Method and Apparatus, and Computer Readable Storage Medium
CN113989338A (zh) * 2021-09-06 2022-01-28 北京东软医疗设备有限公司 图像配准方法及装置、存储介质及计算机设备
CN115830016A (zh) * 2023-02-09 2023-03-21 真健康（北京）医疗科技有限公司 医学图像配准模型训练方法及设备
CN115861394A (zh) * 2023-02-28 2023-03-28 福建自贸试验区厦门片区Manteia数据科技有限公司 医学图像的处理方法和装置、存储介质及电子设备

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210209775A1 (en) * 2018-12-19 2021-07-08 Shanghai Sensetime Intelligent Technology Co., Ltd. Image Processing Method and Apparatus, and Computer Readable Storage Medium
CN113989338A (zh) * 2021-09-06 2022-01-28 北京东软医疗设备有限公司 图像配准方法及装置、存储介质及计算机设备
CN115830016A (zh) * 2023-02-09 2023-03-21 真健康（北京）医疗科技有限公司 医学图像配准模型训练方法及设备
CN115861394A (zh) * 2023-02-28 2023-03-28 福建自贸试验区厦门片区Manteia数据科技有限公司 医学图像的处理方法和装置、存储介质及电子设备

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN120374647A (zh) * 2025-04-11 2025-07-25 北京大学第三医院（北京大学第三临床医学院） 一种卵巢癌病灶分割模型的训练方法

Also Published As

Publication number Publication date
CN120457453A (zh) 2025-08-08

Similar Documents

Publication Publication Date Title
CN108335303B (zh) 一种应用于手掌x光片的多尺度手掌骨骼分割方法
CN113129425A (zh) 一种人脸图像三维重建方法、存储介质及终端设备
CN110555835B (zh) 一种脑片图像区域划分方法及装置
CN106384383A (zh) 一种基于fast和freak特征匹配算法的rgb‑d和slam场景重建方法
CN113177592B (zh) 一种图像分割方法、装置、计算机设备及存储介质
CN114120013B (zh) 一种红外与rgb跨模态特征点匹配方法
CN117522817A (zh) 一种基于人工智能算法的医学图像处理方法及系统
CN104616247B (zh) 一种用于基于超像素sift航拍地图拼接的方法
CN111091567A (zh) 医学图像配准方法、医疗设备及存储介质
CN104123554B (zh) 基于mmtd的sift图像特征提取方法
WO2021233017A1 (zh) 图像处理方法、装置、设备及计算机可读存储介质
WO2024055446A1 (zh) 图像分割方法、装置、设备及可读存储介质
CN110363802A (zh) 基于自动分割和骨盆对齐的前列腺图像配准系统及方法
CN112102929A (zh) 医学图像标注方法、装置、存储介质及电子设备
CN112085017B (zh) 基于显著性检测和Grabcut算法的茶叶嫩芽图像分割方法
CN111414922A (zh) 特征提取方法、图像处理方法、模型训练方法及装置
CN109961435B (zh) 脑图像获取方法、装置、设备及存储介质
CN119648913A (zh) 一种街道场景的重建方法、电子设备及存储介质
CN114445462A (zh) 基于自适应卷积的跨模态视觉跟踪方法及装置
CN112464924A (zh) 一种构建训练集的方法及装置
WO2024239287A1 (zh) 图像配准的数据处理方法、装置及电子设备
CN109978897A (zh) 一种多尺度生成对抗网络的异源遥感图像配准方法及装置
CN105913423B (zh) 一种基于超像素的确定性模型拟合方法
Qu et al. An algorithm of image mosaic based on binary tree and eliminating distortion error
CN113658106B (zh) 一种基于腹部增强ct的肝脏病灶自动诊断系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23937961

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202380083330.9

Country of ref document: CN

WWP Wipo information: published in national office

Ref document number: 202380083330.9

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE