WO2021134506A1 - 宽景拼接方法、装置及存储介质 - Google Patents

宽景拼接方法、装置及存储介质 Download PDF

Info

Publication number
WO2021134506A1
WO2021134506A1 PCT/CN2019/130603 CN2019130603W WO2021134506A1 WO 2021134506 A1 WO2021134506 A1 WO 2021134506A1 CN 2019130603 W CN2019130603 W CN 2019130603W WO 2021134506 A1 WO2021134506 A1 WO 2021134506A1
Authority
WO
WIPO (PCT)
Prior art keywords
neural network
data
transformation matrix
convolutional neural
motion data
Prior art date
Application number
PCT/CN2019/130603
Other languages
English (en)
French (fr)
Inventor
殷晨
赵明昌
莫若理
Original Assignee
无锡祥生医疗科技股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 无锡祥生医疗科技股份有限公司 filed Critical 无锡祥生医疗科技股份有限公司
Priority to PCT/CN2019/130603 priority Critical patent/WO2021134506A1/zh
Priority to EP19958642.1A priority patent/EP3901896A4/en
Priority to US17/420,762 priority patent/US11983844B2/en
Publication of WO2021134506A1 publication Critical patent/WO2021134506A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/42—Details of probe positioning or probe attachment to the patient
    • A61B8/4245—Details of probe positioning or probe attachment to the patient involving determining the position of the probe, e.g. with respect to an external reference frame or to the patient
    • A61B8/4254—Details of probe positioning or probe attachment to the patient involving determining the position of the probe, e.g. with respect to an external reference frame or to the patient using sensors mounted on the probe
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B8/00—Diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/52—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves
    • A61B8/5215—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of medical diagnostic data
    • A61B8/5238—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of medical diagnostic data for combining image data of patient, e.g. merging several images from different acquisition modes into one image
    • A61B8/5246—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of medical diagnostic data for combining image data of patient, e.g. merging several images from different acquisition modes into one image combining images from the same or different imaging techniques, e.g. color Doppler and B-mode
    • A61B8/5253—Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves involving processing of medical diagnostic data for combining image data of patient, e.g. merging several images from different acquisition modes into one image combining images from the same or different imaging techniques, e.g. color Doppler and B-mode combining overlapping images, e.g. spatial compounding
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/10—Selection of transformation methods according to the characteristics of the input images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/14—Transformations for image registration, e.g. adjusting or mapping for alignment of images
    • G06T3/147—Transformations for image registration, e.g. adjusting or mapping for alignment of images using affine transformations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B34/00—Computer-aided surgery; Manipulators or robots specially adapted for use in surgery
    • A61B34/20—Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
    • A61B2034/2046—Tracking techniques
    • A61B2034/2048—Tracking techniques using an accelerometer or inertia sensor
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B2576/00—Medical imaging apparatus involving image processing or analysis
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/06—Devices, other than using radiation, for detecting or locating foreign bodies ; determining position of probes within or on the body of the patient
    • A61B5/065—Determining position of the probe employing exclusively positioning means located on or in the probe, e.g. using position sensors arranged on the probe
    • A61B5/067—Determining position of the probe employing exclusively positioning means located on or in the probe, e.g. using position sensors arranged on the probe using accelerometers or gyroscopes

Definitions

  • This application relates to the technical field of image splicing, in particular to a wide-view splicing method, device and storage medium.
  • Image stitching technology is a technology that stitches several images with overlapping parts into a seamless wide-view or high-resolution image.
  • image acquisition due to factors such as different time and different angles, it is impossible to see a complete overall image of the area of interest in an image.
  • the divergence range of the ultrasonic sound waves emitted by the traditional ultrasonic probe is limited, and the size of the probe is also fixed, so it can only generate ultrasound images within a specified range, but cannot generate an overall image of the area of interest.
  • doctors can only form an overall image of the area in the brain based on memory and their own experience in order to observe the situation of adjacent tissues. This affects the speed and speed of medical diagnosis to a certain extent. accuracy.
  • the accuracy rate of the panoramic images obtained by the above-mentioned panoramic splicing method is low, and when an electromagnetic positioning system is provided in the ultrasonic probe, the above-mentioned method for realizing the panoramic splicing has a high system cost and an expensive structure.
  • this application provides a transformation matrix acquisition, panoramic image splicing, neural network training method, device, equipment and storage medium.
  • a method for acquiring a transformation matrix includes the following steps: acquiring motion data detected by a sensor, wherein the sensor is set on a probe for acquiring images, and the motion data is used to represent all The motion trend of the probe in the process of acquiring images; input the motion data into a neural network obtained in advance, and use the neural network to calculate the matrix parameters; use the matrix parameters to calculate the transformation matrix, the transformation matrix It is used to stitch the images collected by the probe to obtain a wide-view image.
  • the neural network includes: a convolutional neural network, a recurrent neural network, and a fully connected network; wherein, the motion data is input into a neural network obtained in advance, and the neural network is used to calculate the transformation.
  • the parameters of the matrix include: performing convolution calculations on the motion data through the convolutional neural network to obtain the data characteristics of the motion data as the output of the convolutional neural network; The data characteristics output by the convolutional neural network are recursively calculated to obtain the recursive calculation result as the output of the recurrent neural network; the recursive calculation result of the recurrent neural network output by the fully connected network is regressively calculated to obtain the Matrix parameters.
  • the convolutional neural network includes a first convolutional neural network and multiple second convolutional neural networks in one-to-one correspondence with the multiple sensors, wherein the first convolutional neural network
  • the input of the convolutional neural network is connected to the outputs of the plurality of second convolutional neural networks.
  • the senor includes an accelerometer and a gyroscope.
  • the performing convolution calculation on the motion data through the convolutional neural network to obtain the data characteristics of the motion data includes: pairing with the second convolutional neural network through the second convolutional neural network Convolution processing is performed on the motion data detected by the sensor corresponding to the convolutional neural network; the output of multiple second convolutional neural networks is fused through the first convolutional neural network and convolution processing is performed to obtain the data feature.
  • the fusing and convolution processing on the outputs of a plurality of the second convolutional neural networks through the first convolutional neural network to obtain the data feature includes: combining each of the first convolutional neural networks The data output by the second convolutional neural network is tiled into one-dimensional data; all the one-dimensional data corresponding to the second convolutional neural network are superimposed together, and the deep convolution calculation is performed through the first convolutional neural network to obtain the Describe the characteristics of the data.
  • the acquiring motion data detected by the sensor includes: acquiring detection data of the duration to be measured detected by each sensor; and dividing each detection data into multiple segments at equal intervals according to the dimension of the duration to be measured Data; Fourier transform is performed on multiple pieces of data corresponding to each sensor to obtain the motion data.
  • the second aspect of the present application provides a panoramic image stitching method, including the following steps: using a probe to detect a plurality of consecutive images of a target area; using the transformation matrix acquisition method described in the first aspect to acquire phases in the plurality of images A transformation matrix between adjacent images; splicing the multiple images based on the acquired transformation matrix to obtain a panoramic image.
  • a neural network training method which includes the following steps: obtaining training sample data, the sample data including: motion data detected by a sensor and matrix parameters corresponding to the motion data, the sensor Is set on a probe for acquiring images, the motion data is used to indicate the motion trend of the probe in the process of acquiring images, and the matrix parameters are parameters in a transformation matrix used to splice panoramic images; using the The training sample data trains the pre-established neural network model to obtain the neural network used to obtain the transformation matrix.
  • acquiring training sample data includes: acquiring body membrane images collected by the probe; determining the transformation matrix of two adjacent body membrane images by using target coordinates set on adjacent body membrane images; The matrix parameter of the transformation matrix is obtained by two multiplication calculation; the movement data detected by the sensor is obtained, and the matrix parameter and the movement data are used as the training sample data.
  • a transformation matrix acquisition device which includes: a motion data acquisition module for acquiring motion data detected by a sensor, wherein the sensor is arranged on a probe for acquiring images, and the motion data The data is used to represent the movement trend of the probe in the process of image acquisition; the parameter calculation module is used to input the movement data into the neural network obtained in advance, and the matrix parameters are calculated by the neural network; the matrix calculation module , Used to calculate a transformation matrix by using the matrix parameters, and the transformation matrix is used to splice the images collected by the probe to obtain a panoramic image.
  • a panoramic image splicing device including: a detection module for detecting a plurality of consecutive images of a target area with a probe; the transformation matrix obtaining device of the fourth aspect is used for obtaining the A transformation matrix between adjacent images in the multiple images; a splicing module for splicing the multiple images based on the acquired transformation matrix to obtain a panoramic image.
  • a neural network training device including: a sample acquisition module for acquiring training sample data, the sample data including: motion data detected by a sensor and matrix parameters corresponding to the motion data ,
  • the sensor is arranged on a probe used to collect images, the movement data is used to indicate the movement trend of the probe in the process of collecting images, and the matrix parameter is a parameter in a transformation matrix used to splice a panoramic image
  • the training module is used to train the neural network model established in advance by using the training sample data to obtain the neural network used to obtain the transformation matrix.
  • a computer device including a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor implements any of the above methods when the computer program is executed A step of.
  • a computer-readable storage medium on which a computer program is stored, characterized in that: the computer program is executed by a processor to implement the steps of any one of the above methods.
  • the motion data of the image collected by the probe is obtained, and the neural network obtained by pre-training is used to calculate and analyze the motion data to obtain the movement change of the probe, and then calculate the transformation matrix of the image, that is, indirectly calculate
  • the transformation matrix can be calculated and image splicing can be performed without using the characteristics of the image itself. It is not affected by factors such as image brightness and characteristics, which improves the accuracy of transformation matrix calculation and further improves the image splicing effect.
  • the motion data is obtained through the sensor in the probe without adding an electromagnetic positioning system, which achieves the effect of reducing the system cost while improving the accuracy of wide-view splicing.
  • FIG. 1 is a flowchart of a specific example of a method for obtaining a transformation matrix in Embodiment 1 of this application;
  • FIG. 2 is a schematic diagram of a specific example of a neural network architecture in an embodiment of the application
  • FIG. 3 is a schematic diagram of a specific example of a convolutional neural network in an embodiment of the application
  • FIG. 4 is a schematic diagram of a specific example of a recurrent neural network in an embodiment of the application.
  • FIG. 5 is a functional block diagram of a specific example of the transformation matrix obtaining device in Embodiment 1 of the application; FIG. 5
  • FIG. 6 is a flowchart of a specific example of the panoramic image stitching method in Embodiment 2 of the application;
  • FIG. 7 is a schematic diagram of a specific example of image stitching in an embodiment of the application.
  • FIG. 8 is a functional block diagram of a specific example of a panoramic image splicing device in Embodiment 2 of the application; FIG.
  • FIG. 9 is a flowchart of a specific example of the neural network training method in Embodiment 3 of this application.
  • FIG. 10 is a schematic diagram of a specific example of a body membrane image in Embodiment 3 of the application.
  • FIG. 11 is a functional block diagram of a specific example of the neural network training device in Embodiment 3 of the application.
  • FIG. 12 is a schematic diagram of the hardware structure of a computer device according to an embodiment of the application.
  • the embodiment of the application discloses a method for acquiring a transformation matrix, which is mainly used to acquire a transformation matrix for image splicing. Specifically, it is mainly applicable to an image splicing technology collected by a probe provided with a sensor, as shown in FIG. 1, The method includes the following steps:
  • Step S101 Acquire motion data detected by a sensor, where the sensor is set on a probe used to collect an image, and the motion data is used to indicate a movement trend of the probe in the process of acquiring an image.
  • the sensor can be an accelerometer and a gyroscope, where the accelerometer is used to detect the acceleration of the probe in the process of moving the image acquisition, and the gyroscope is used to detect the angular change of the probe in three directions during the process of moving the image acquisition.
  • the acceleration sensor reflects the movement changes in the three directions of x, y, and z, and the gyroscope can calculate the change of the angle. These quantities can reflect the relative movement trend of the probe to a certain extent, and the position and angle of the probe movement can be measured. Quantification, which can calculate the law of change between the images scanned by the probe.
  • the probe in the embodiment of the present application may refer to an image acquisition device provided with a sensor for collecting motion data, including but not limited to an ultrasonic probe, and its specific form and structure are not limited.
  • Step S102 Input the motion data into a neural network obtained in advance, and use the neural network to calculate matrix parameters.
  • step S103 a transformation matrix is calculated by using the matrix parameters, and the transformation matrix is used to splice the images collected by the probe to obtain a panoramic image.
  • the neural network is a neural network obtained in advance by using motion data and corresponding matrix parameters as training samples. After training (the training process of the neural network described in the embodiments of this application will be introduced later), the neural network has the ability to recognize the relationship between motion data and matrix parameters. Therefore, when the motion data collected by the sensor is obtained After that, the neural network can be used to calculate and determine the corresponding matrix parameters, so that the transformation matrix can be calculated by using the combination of matrix parameters.
  • the transformation matrix to be obtained is as follows:
  • the matrix parameters involved include: a, b, c, d, e, f.
  • the neural network After inputting the motion data collected by the sensor into the neural network for learning and training, the neural network can be used to calculate the above-mentioned parameters a, b, c, d, e, f to obtain the transformation matrix, which is used for splicing to obtain a panoramic view image.
  • the motion data of the image collected by the probe is obtained, and the neural network obtained by pre-training is used to calculate and analyze the motion data to obtain the movement change of the probe, and then calculate the transformation matrix of the image, that is, indirectly calculate
  • the transformation matrix can be calculated and image splicing can be performed without using the characteristics of the image itself. It is not affected by factors such as image brightness and characteristics, which improves the accuracy of transformation matrix calculation and further improves the image splicing effect.
  • the motion data is acquired through the sensor in the probe, and the transformation matrix is accurately calculated without adding an electromagnetic positioning system, which achieves the effect of reducing the system cost while improving the accuracy of wide scene stitching.
  • the embodiment of the present application designs a relatively unified and integrated neural network structure, and inputs the data collected by the sensor into the neural network to calculate the transformation matrix M of the image at the current moment.
  • the neural networks described in the embodiments of this application include: Convolutional Neural Networks, Recurrent Neural Networks, and Fully Connected Networks, that is, the neural network is divided into three parts. The first part is Convolutional Neural Networks (CNN) ), the second part is a recursive neural network (recursive neural network, referred to as RNN), and the third part is a fully connected network (that is, a regression network) to calculate the final output result-the transformation matrix M.
  • CNN Convolutional Neural Networks
  • RNN recursive neural network
  • RNN fully connected network
  • the output of the convolutional neural network CNN is used as the input of the recurrent neural network N4, the output of the recurrent neural network N4 is used as the input of the fully connected network N5, and the fully connected network N5 calculates the final transformation matrix M.
  • the convolutional neural network includes a first convolutional neural network N3 and a plurality of second convolutional neural networks (N1 and N2) corresponding to a plurality of the sensors, wherein the first convolutional neural network The input is connected to the output of a plurality of said second convolutional neural networks.
  • the sensor may also include other sensors that can detect the movement of the probe, such as a speed sensor, etc., where the second convolutional neural network corresponds to the number of sensors one-to-one, and the first convolutional neural network can be used for After fusion processing of the data output by multiple second convolutional neural networks, deep learning and feature recognition are performed.
  • the first convolutional neural network and the second convolutional neural network described in the embodiments of the present application can also be called convolutional layers.
  • a neural network of multiple levels is set up to train, learn and calculate transformation matrices, especially when convolutional neural networks are used. The way the network corresponds to the sensor can enable the neural network to learn more accurate feature information, thereby improving the accuracy of the transformation matrix calculation.
  • the acquiring motion data detected by the sensor includes: acquiring detection data of the duration to be measured detected by each sensor; dividing each detection data into multiple pieces of data at equal intervals according to the dimension of the duration to be measured; The multiple pieces of data corresponding to each sensor are Fourier transformed to obtain the motion data.
  • the number of sensor types is K
  • K 2
  • the data generated by the two sensors is X
  • the dimension of is D*F
  • the total amount of data is D*F*n
  • F 2*f
  • f is the main f frequencies of the current data
  • 2 means the coefficient of the main frequency is required
  • the coefficient of the main frequency includes positive
  • the collected data is preprocessed according to sensor type and time, and then grouped into the corresponding convolutional neural network, where k1 represents the first sensor and k2 represents the second sensor.
  • the data detected by the sensor is processed by segmentation and Fourier transform to meet the requirements of neural network calculation and recognition, which can identify and calculate sensors of any data type, which improves the neural network.
  • the applicability of the neural network is improved, and the accuracy of neural network calculation and recognition is improved.
  • the inputting the motion data into the neural network obtained by pre-training, and using the neural network to calculate the parameters of the transformation matrix includes:
  • S1021 Perform convolution calculation on the motion data through the convolutional neural network to obtain data features of the motion data, which are used as the output of the convolutional neural network.
  • Convolutional neural network mainly learns and recognizes the characteristics of motion data. Through convolution calculation, it learns the data characteristics of the motion data detected by the sensor and the data characteristics between different sensors, and then outputs to the recursive neural network for recursive operations. .
  • S1022 Perform a recursive operation on the data features output by the convolutional neural network through the recurrent neural network to obtain a recursive calculation result as the output of the recurrent neural network.
  • the convolutional neural network is used to train and learn the motion data to obtain the characteristics of the data detected by the sensor and the relationship between the data of different sensors. Then, the recurrent neural network connects the output results of the convolutional neural network in chronological order, then performs recursive operations, and finally returns the matrix parameters of the final transformation matrix from the fully connected network.
  • the convolutional calculation is performed on the motion data through the convolutional neural network to obtain the data characteristics of the motion data ,include:
  • S11 Perform convolution processing on the motion data detected by the sensor corresponding to the second convolutional neural network through the second convolutional neural network.
  • each second convolutional neural network there are multiple second convolutional neural networks, such as N1 and N2 in FIG. 2 and FIG. 3.
  • Each second convolutional neural network is independent of each other. Since each second convolutional neural network corresponds to one sensor, each second convolutional neural network only needs to process the data detected by the corresponding sensor.
  • the sensors are accelerometer and gyroscope. Because accelerometer and gyroscope are two different types of sensors, two neural networks with independent weights, N1 and N2, are used during training. N1 and N2 are independent but have the same structure. A neural network, where N1 is used to train and learn the sensor data of the accelerometer, and N2 is used to train and learn the sensor data of the gyroscope. Specifically, in the training process, the data detected by the accelerometer is input into the convolutional neural network N1 for convolution processing, and the data detected by the gyroscope is input into the convolutional neural network N2 for convolution processing.
  • the dimension input to the second convolutional neural network is D*F
  • the convolution kernel of the first layer of the convolutional layer corresponding to the second convolutional neural network is The following layers of the second convolutional neural network (the neural network includes multiple convolutional layers) are all 1*3, where It is equal to D, which is 3 in the embodiment of the present application.
  • the first layer of convolution kernel is used to learn the relationship between different dimensional data of a single sensor, and the subsequent layers are used to learn the relationship between deeper (depth) data.
  • the first convolutional neural network N3 is used to fuse the motion data of multiple sensors through the second convolutional neural network N1 and N2 after the output data, and perform convolution calculation processing to obtain more
  • the deeper data features between the motion data detected by the two sensors are used as the output result of the entire convolutional neural network for subsequent processing by the recurrent neural network.
  • the fusing and convolution processing on the outputs of a plurality of the second convolutional neural networks through the first convolutional neural network to obtain the data feature includes: combining each of the first convolutional neural networks The data output by the second convolutional neural network is tiled into one-dimensional data, and all the one-dimensional data corresponding to the second convolutional neural network are superimposed together, and the deep convolution calculation is performed through the first convolutional neural network to obtain the Data characteristics.
  • the first convolutional neural network N3 is used to tile the data formed by multiple sensor data through the convolutional neural networks N1 and N2 into one-dimensional data and then superimpose the rows together, and then perform deep learning and deal with.
  • the first convolutional neural network includes multiple convolutional layers, and the size of the convolution kernel of the first layer is The subsequent layers of the network are all 1*3, of which K is equal to the number of sensors 2.
  • the convolutional neural network N3 is used to fuse the data of the two sensors and learn its deeper features.
  • the first convolutional neural network N3 and the second convolutional neural network N1, N2 are constructed by using multiple convolutional layers, and are provided with an activation function and a normalization layer.
  • the activation function can be The relu linear rectification activation function
  • the normalization layer can adopt a batch normalization layer (batch normalization) to ensure that the mean and variance of the input distribution are fixed within a certain range, and the training accuracy is improved.
  • the output data is also the input of the recurrent neural network, which is the data characteristics acquired by the sensor in a chronological order over a period of time.
  • the traditional method for sensors such as calculating relative displacement based on accelerometer data, it is generally a short period of time to calculate the integral of the acceleration to obtain the velocity, and then calculate the integral of the velocity to calculate the displacement.
  • All data are collected in a unit time.
  • the recurrent neural network in the embodiment of the present application is also based on a similar principle. It can learn the characteristics of integral summation from an earlier level, so as to calculate the final output result from another angle.
  • the recurrent neural network in the embodiments of the present application may adopt a stacked multi-layer LSTM network layer, specifically, two layers may be stacked.
  • An optional recurrent neural network structure is shown in FIG. 4.
  • the output of each stage in the recurrent neural network layer is sent to the fully connected network N5.
  • the fully connected network N5 is used to regress the matrix parameters of the final probe movement transformation matrix.
  • the transformation matrix form of the probe movement is:
  • the parameters that need to be trained are the angle ⁇ of the probe rotation, and the probe offsets ⁇ x and ⁇ y, and the calculated results are used as the image transformation matrix parameters for the final image transformation and stitching.
  • the above-mentioned transformation matrix is summarized and deduced based on the relative relationship of moving images.
  • an embodiment of the present application also provides a transformation matrix acquisition device, which can be used to execute the transformation matrix acquisition method of the foregoing embodiment.
  • the device includes:
  • the motion data acquisition module 501 is configured to acquire motion data detected by a sensor, where the sensor is provided on a probe used to acquire an image, and the motion data is used to indicate a movement trend of the probe in the process of acquiring an image;
  • the parameter calculation module 502 is configured to input the motion data into a neural network obtained in advance, and use the neural network to calculate matrix parameters;
  • the matrix calculation module 503 is configured to calculate a transformation matrix using the matrix parameters, and the transformation matrix is used to splice the images collected by the probe to obtain a panoramic image.
  • the motion data of the image collected by the probe is obtained, and the neural network obtained by pre-training is used to calculate and analyze the motion data to obtain the movement change of the probe, and then calculate the transformation matrix of the image, that is, indirectly calculate
  • the transformation matrix can be calculated and image splicing can be performed without using the characteristics of the image itself. It is not affected by factors such as image brightness and characteristics, which improves the accuracy of transformation matrix calculation and further improves the image splicing effect.
  • the transformation matrix obtaining apparatus of the embodiment of the present application corresponds to the transformation matrix obtaining method of the foregoing embodiment. For specific description, refer to the foregoing embodiment, which is not repeated here.
  • This embodiment provides a panoramic image stitching method, which is mainly used to stitch two or more overlapping images to form a panoramic image. As shown in Figure 6, the method includes the following steps:
  • step S601 the probe is used to detect multiple consecutive images of the target area.
  • the probe is provided with a sensor for detecting the movement data of the probe, and the probe needs to be moved in the process of capturing an image to be able to capture all areas of the target area.
  • the continuous multiple images are mainly because the images are continuous during the probe detection process, for example, video images.
  • Step S602 Obtain a transformation matrix between adjacent images in the plurality of images by using a transformation matrix acquisition method.
  • the method for obtaining the transformation matrix in the embodiment of the present application is also the method for obtaining the transformation matrix described in the above-mentioned embodiment 1.
  • the specific working principle and details please refer to the above-mentioned embodiment, which will not be repeated here.
  • Step S603 splicing the multiple images based on the acquired transformation matrix to obtain a panoramic image.
  • an embodiment of the present application also provides a panoramic image splicing device, which can be used to execute the panoramic image splicing method of the foregoing embodiment.
  • the device includes:
  • the detection module 801 is used to detect a plurality of consecutive images of the target area with a probe
  • the transformation matrix obtaining device 802 is configured to obtain a transformation matrix between adjacent images in the plurality of images.
  • the transformation matrix obtaining device 802 is the device shown in FIG. 5 in the foregoing Embodiment 1. For details, please refer to the above description.
  • the splicing module 803 is configured to splice the multiple images based on the acquired transformation matrix to obtain a panoramic image.
  • the transformation matrix acquisition method to acquire a transformation matrix between images based on the motion data detected by the sensor, multiple detected images can be spliced to obtain a panoramic image. Since the acquisition of the transformation matrix does not need to use the characteristics of the image itself, the transformation matrix can be calculated and image splicing can be performed without being affected by factors such as image brightness and characteristics, which improves the accuracy of transformation matrix calculation and further improves the image splicing effect.
  • the panoramic image splicing device of the embodiment of the present application corresponds to the panoramic image splicing method of the foregoing embodiment.
  • the panoramic image splicing device of the embodiment of the present application corresponds to the panoramic image splicing method of the foregoing embodiment.
  • the embodiment of the present application also provides a neural network training method.
  • the training method is mainly used to train the neural network described in the foregoing embodiment 1. As shown in FIG. 9, the method includes the following steps:
  • Step S901 Obtain training sample data, where the sample data includes: motion data detected by a sensor and matrix parameters corresponding to the motion data, the sensor is set on a probe for acquiring images, and the motion data is used for Represents the movement trend of the probe during image acquisition, and the matrix parameter is a parameter in the transformation matrix used to splice the panoramic image.
  • the motion data and the labeled matrix parameters are used as training sample data.
  • the training sample data can be divided into a training set and a test set.
  • the data mainly includes the motion data and the labeled corresponding matrix parameters, which are used for nerves. Network model training.
  • the embodiment of the application uses the body membrane image for training.
  • the characteristic of the body membrane image is that a fixed target can be set inside the body membrane. In the scanned image, you can clearly see the target point, as shown in Figure 10, you can determine the position of the same target point in the two images, and calculate the transformation matrix in terms of aspect.
  • the advantage of the body membrane image is that the image is clear. , The calculated transformation matrix is reliable and correct.
  • acquiring training sample data includes: acquiring body membrane images collected by the probe; using target coordinates set on adjacent body membrane images to determine the transformation matrix of two adjacent body membrane images; using least squares The matrix parameter of the transformation matrix is obtained by multiplication calculation; the movement data detected by the sensor is obtained, and the matrix parameter and the movement data are used as the training sample data.
  • the optimal matrix parameters ⁇ , ⁇ x and ⁇ y can be calculated. Thereby, the corresponding transformation matrix M is obtained.
  • Step S902 Use the training sample data to train a pre-established neural network model to obtain a neural network for obtaining the transformation matrix.
  • the data of the sensors in a fixed time interval are collected, the M of the image movement transformation in the current interval is calculated, the data is sent to the above-mentioned neural network for training, and the optimal network parameters are calculated iteratively.
  • the neural network model is trained by using the motion data and matrix parameters detected by the sensor, so that the neural network model can learn and recognize the data relationship between the motion data and the matrix parameters, and obtain the neural network for Later, the corresponding transformation matrix is identified for other motion data, and the neural network method is adopted to indirectly calculate the image change by analyzing the movement change of the probe, thereby improving the accuracy.
  • the device includes:
  • the sample acquisition module 111 is configured to acquire training sample data, the sample data including: motion data detected by a sensor and matrix parameters corresponding to the motion data, the sensor is set on a probe for acquiring images, the The motion data is used to indicate the motion trend of the probe in the process of acquiring images, and the matrix parameters are parameters in the transformation matrix used to splice the panoramic images;
  • the training module 112 is configured to use the training sample data to train a pre-established neural network model to obtain a neural network for obtaining the transformation matrix.
  • the neural network model is trained by using the motion data and matrix parameters detected by the sensor, so that the neural network model can learn and recognize the data relationship between the motion data and the matrix parameters, and obtain the neural network for Later, the corresponding transformation matrix is identified for other motion data, and the neural network method is adopted to indirectly calculate the image change by analyzing the movement change of the probe, thereby improving the accuracy.
  • the neural network training device of the embodiment of the present application corresponds to the neural network training method of the above-mentioned embodiment.
  • This embodiment also provides a computer device, such as a desktop computer, a rack server, a blade server, a tower server, or a rack server (including an independent server, or a server cluster composed of multiple servers) that can execute programs Wait.
  • the computer device 120 in this embodiment at least includes but is not limited to: a memory 121 and a processor 122 that can be communicatively connected to each other through a system bus, as shown in FIG. 12. It should be pointed out that FIG. 12 only shows the computer device 120 with components 121-122, but it should be understood that it is not required to implement all the illustrated components, and more or fewer components may be implemented instead.
  • the memory 121 (ie, readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), Read only memory (ROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM), magnetic memory, magnetic disk, optical disk, etc.
  • the memory 121 may be an internal storage unit of the computer device 120, such as a hard disk or memory of the computer device 120.
  • the memory 121 may also be an external storage device of the computer device 120, such as a plug-in hard disk, a smart media card (SMC), and a secure digital (Secure Digital, SMC) equipped on the computer device 120. SD) card, flash card (Flash Card), etc.
  • the memory 121 may also include both an internal storage unit of the computer device 120 and an external storage device thereof.
  • the memory 21 is generally used to store the operating system and various application software installed in the computer device 120, such as the program codes of the transformation matrix acquisition, the panoramic image splicing, and the neural network training method described in the embodiment.
  • the memory 121 may also be used to temporarily store various types of data that have been output or will be output.
  • the processor 122 may be a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments.
  • the processor 122 is generally used to control the overall operation of the computer device 120.
  • the processor 122 is used to run program codes or process data stored in the memory 121, for example, to implement the transformation matrix acquisition, panoramic image stitching, and neural network training methods of the embodiment.
  • This embodiment also provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), only Readable memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disks, optical disks, servers, App application malls, etc., on which computer programs are stored, The corresponding function is realized when the program is executed by the processor.
  • the computer-readable storage medium of this embodiment is used to store transformation matrix acquisition, panoramic image splicing, and neural network training devices, and when executed by a processor, realizes the transformation matrix acquisition, panoramic image splicing, and neural network training methods of the embodiment.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Radiology & Medical Imaging (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Pathology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)

Abstract

本申请公开了一种宽景拼接方法、装置及存储介质。其中，变换矩阵获取方法包括：获取传感器检测到的运动数据，其中，传感器设置在用于采集图像的探头上，运动数据用于表示探头在采集图像过程中的运动趋势；将运动数据输入到预先训练得到的神经网络中，利用神经网络计算得到矩阵参数；利用矩阵参数计算得到变换矩阵，变换矩阵用于拼接探头采集到的图像以得到宽景图像。本申请无需利用图像本身的特征，即可计算出变换矩阵并进行图像拼接，不受图像亮度和特征等因素的影响，提高了变换矩阵计算的准确性，进而提升了图像拼接效果。

Description

宽景拼接方法、装置及存储介质 技术领域
本申请涉及图像拼接技术领域，具体涉及一种宽景拼接方法、装置及存储介质。
背景技术
图像拼接技术是将数张有重叠部分的图像拼接成一幅无缝的宽景图或高分辨率图像的技术。在图像采集的过程中，由于时间不同、角度不同等因素导致在一幅图像中无法看到完整的关注区域的整体图像。例如，传统的超声探头发射的超声声波发散范围有限，而且探头的尺寸也是固定的，因此只能生成指定范围内的超声图像，而无法生成所关注区域的整体影像。在医疗领域的应用过程中，医生只能根据记忆，结合自己的经验，在大脑中形成该区域的整体图像，以便观察相邻组织的情况，因此在一定程度上影响了医疗诊断的快速性和准确性。
而针对上述这一问题，已经有相关研究通过相邻图像的常规配准技术，将超声探头移动的过程中所采集的图像拼接成一幅视野更大的图像，以便在同一幅图像中显示整个组织的结构，方便医生的诊断。
然而上述宽景拼接方法得到的宽景图像准确率较低，并且当超声探头中设置电磁定位系统时，上述实现宽景拼接的方法系统成本高，架构昂贵。
发明内容
本申请为了解决现有技术中宽景图像拼接的准确率低的技术问题，从而提供一种变换矩阵获取、宽景图像拼接、神经网络训练方法、装置、设备及存储介质。
本申请第一方面，提供了一种变换矩阵获取方法，包括如下步骤：获取传感器检测到的运动数据，其中，所述传感器设置在用于采集图像的探头上，所述运动数据用于表示所述探头在采集图像过程中的运动趋势；将所述运动数据输入到预先训练得到的神经网络中，利用所述神经网络计算得到矩阵参数；利用所述矩阵参数计算得到变换矩阵，所述变换矩阵用于拼接所述探头采集到的图像以得到宽景图像。
可选地，所述神经网络包括：卷积神经网络、递归神经网络和全连接网络；其中，所述将所述运动数据输入到预先训练得到的神经网络中，利用所述神经网络计算得到变换矩阵的参数，包括：通过所述卷积神经网络对所述运动数据进行卷积计算，得到所述运动数据的数据特征，作为所述卷积神经网络的输出；通过所述递归神经网络对所述卷积神经网络输出的数据特征进行递归运算，得到递归计算结果，作为所述递归神经网络 的输出；通过所述全连接网络对所述递归神经网络输出的递归计算结果回归计算，得到所述矩阵参数。
可选地，所述传感器为多个，所述卷积神经网络包括第一卷积神经网络和与多个所述传感器一一对应的多个第二卷积神经网络，其中，所述第一卷积神经网络的输入与多个所述第二卷积神经网络的输出连接。
可选地，所述传感器包括加速度计和陀螺仪。
可选地，所述通过所述卷积神经网络对所述运动数据进行卷积计算，得到所述运动数据的数据特征，包括：通过所述第二卷积神经网络对与所述第二卷积神经网络对应的传感器检测到的运动数据进行卷积处理；通过所述第一卷积神经网络对多个所述第二卷积神经网络的输出进行融合并进行卷积处理，得到所述数据特征。
可选地，所述通过所述第一卷积神经网络对多个所述第二卷积神经网络的输出进行融合并进行卷积处理，得到所述数据特征，包括：将每个所述第二卷积神经网络输出的数据平铺成一维数据；将所有所述第二卷积神经网络对应的一维数据叠加在一起，通过所述第一卷积神经网络进行深度卷积计算，得到所述数据特征。
可选地，所述获取传感器检测到的运动数据，包括：获取每个所述传感器检测到的待测时长的检测数据；对每个检测数据按照所述待测时长维度划分为等间隔的多段数据；对每个传感器对应的多段数据进行傅里叶变换，得到所述运动数据。
本申请第二方面，提供了一种宽景图像拼接方法，包括如下步骤：利用探头探测目标区域连续的多个图像；利用第一方面所述的变换矩阵获取方法获取所述多个图像中相邻图像之间的变换矩阵；基于获取到的变换矩阵拼接所述多个图像得到宽景图像。
本申请第三方面，提供了一种神经网络训练方法，包括如下步骤：获取训练样本数据，所述样本数据包括：传感器检测到的运动数据和与所述运动数据对应的矩阵参数，所述传感器设置在用于采集图像的探头上，所述运动数据用于表示所述探头在采集图像过程中的运动趋势，所述矩阵参数为用于拼接宽景图像的变换矩阵中的参数；利用所述训练样本数据对预先建立的神经网络模型进行训练，得到用于获取所述变换矩阵的神经网络。
可选地，获取训练样本数据，包括：获取经过所述探头采集到的体膜图像；利用设置在相邻的体膜图像上的靶点坐标确定相邻两个体膜图像的变换矩阵；利用最小二乘法计算得到所述变换矩阵的矩阵参数；获取所述传感器检测到的所述运动数据，将所述矩阵参数和所述运动数据作为所述训练样本数据。
本申请第四方面，提供了一种变换矩阵获取装置，包括：运动数据获取模块，用于获取传感器检测到的运动数据，其中，所述传感器设置在用于采集图像的探头上，所述运动数据用于表示所述探头在采集图像过程中的运动趋势；参数计算模块，用于将所述运动数据输入到预先训练得到的神经网络中，利用所述神经网络计算得到矩阵参数；矩阵计算模块，用于利用所述矩阵参数计算得到变换矩阵，所述变换矩阵用于拼接所述探头采集到的图像以得到宽景图像。
本申请第五方面，提供了一种宽景图像拼接装置，包括：探测模块，用于利用探头探测目标区域连续的多个图像；第四方面所述的变换矩阵获取装置，用于获取所述多个图像中相邻图像之间的变换矩阵；拼接模块，用于基于获取到的变换矩阵拼接所述多个图像得到宽景图像。
本申请第六方面，提供了一种神经网络训练装置，包括：样本获取模块，用于获取训练样本数据，所述样本数据包括：传感器检测到的运动数据和与所述运动数据对应的矩阵参数，所述传感器设置在用于采集图像的探头上，所述运动数据用于表示所述探头在采集图像过程中的运动趋势，所述矩阵参数为用于拼接宽景图像的变换矩阵中的参数；训练模块，用于利用所述训练样本数据对预先建立的神经网络模型进行训练，得到用于获取所述变换矩阵的神经网络。
本申请第七方面，提供了一种计算机设备，包括存储器、处理器以及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现上述任意一种方法的步骤。
本申请第八方面，提供了一种计算机可读存储介质，其上存储有计算机程序，其特征在于：所述计算机程序被处理器执行时实现上述任意一种方法的步骤。
根据本申请实施例，通过获取探头采集图像的运动数据，利用预先训练得到的神经网络对该运动数据进行计算分析，得到探头的移动变化，进而计算出图像的变换矩阵，也即是间接地计算出图像的变化，无需利用图像本身的特征，即可计算出变换矩阵并进行图像拼接，不受图像亮度和特征等因素的影响，提高了变换矩阵计算的准确性，进而提升了图像拼接效果。同时，通过探头中的传感器来获取运动数据，而无需增加电磁定位系统，达到了在提高宽景拼接准确度的同时，降低了系统成本的效果。
附图说明
为了更清楚地说明本申请具体实施方式或现有技术中的技术方案，下面将对具体实施方式或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的 附图是本申请的一些实施方式，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本申请实施例1中变换矩阵获取方法的一个具体示例的流程图；
图2为本申请实施例中神经网络架构的一个具体示例的示意图；
图3为本申请实施例中卷积神经网络的一个具体示例的示意图；
图4为本申请实施例中递归神经网络的一个具体示例的示意图；
图5为本申请实施例1中变换矩阵获取装置的一个具体示例的原理框图；
图6为本申请实施例2中宽景图像拼接方法的一个具体示例的流程图；
图7为本申请实施例中图像拼接的一个具体示例的示意图；
图8为本申请实施例2中宽景图像拼接装置的一个具体示例的原理框图；
图9为本申请实施例3中神经网络训练方法的一个具体示例的流程图；
图10为本申请实施例3中体膜图像的一个具体示例的示意图；
图11为本申请实施例3中神经网络训练装置的一个具体示例的原理框图；
图12为本申请实施例计算机设备的硬件结构示意图。
具体实施方式
下面将结合附图对本申请的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
在本申请的描述中，需要说明的是，术语“第一”、“第二”、“第三”仅用于描述目的，而不能理解为指示或暗示相对重要性。
此外，下面所描述的本申请不同实施方式中所涉及的技术特征只要彼此之间未构成冲突就可以相互结合。
实施例1
本申请实施例公开了一种变换矩阵获取方法，该方法主要用于获取图像拼接的变换矩阵，具体地，主要适用于通过设置有传感器的探头采集到的图像拼接技术，如图1所示，该方法包括如下步骤：
步骤S101,获取传感器检测到的运动数据，其中，所述传感器设置在用于采集图像的探头上，所述运动数据用于表示所述探头在采集图像过程中的运动趋势。
传感器可以是加速度计和陀螺仪，其中，加速度计用于检测探头在移动采集图像过 程中的加速度，陀螺仪则用于检测探头在移动采集图像过程中的三个方向的角度变化。具体地，加速度传感器反应了x，y，z三个方向的移动变化，陀螺仪可以计算角度的变化，这些量在一定程度上可以反应探头相对运动的趋势，可以对探头运动的位置和角度进行量化，从而可以计算出探头所扫描的图像之间的变化规律。
本申请实施例的探头可以是指设置有采集运动数据的传感器的图像采集装置，包括但不限于超声波探头其具体形式和结构不限。
步骤S102,将所述运动数据输入到预先训练得到的神经网络中，利用所述神经网络计算得到矩阵参数。
步骤S103,利用所述矩阵参数计算得到变换矩阵，所述变换矩阵用于拼接所述探头采集到的图像以得到宽景图像。
本申请实施例中，神经网络是利用包含有运动数据和对应的矩阵参数作为训练样本，预先练得到的神经网络。经过训练(本申请实施例中所述的神经网络的训练过程将在后面介绍)之后，该神经网络具备了识别运动数据与矩阵参数的关系的能力，因此，当获取到传感器采集到的运动数据之后，可以利用该神经网络计算并确定出与其对应的矩阵参数，从而利用矩阵参数组合计算得到变换矩阵。
例如，待求变换矩阵如下：
Figure PCTCN2019130603-appb-000001
其中，所涉及到的矩阵参数包括：a、b、c、d、e、f。
将传感器采集到的运动数据输入到神经网络中进行学习和训练之后，可以利用该神经网络计算出上述参数a、b、c、d、e、f，从而得到变换矩阵，用于拼接得到宽景图像。
根据本申请实施例，通过获取探头采集图像的运动数据，利用预先训练得到的神经网络对该运动数据进行计算分析，得到探头的移动变化，进而计算出图像的变换矩阵，也即是间接地计算出图像的变化，无需利用图像本身的特征，即可计算出变换矩阵并进行图像拼接，不受图像亮度和特征等因素的影响，提高了变换矩阵计算的准确性，进而提升了图像拼接效果。进一步地，通过探头中的传感器来获取运动数据，准确地计算出变换矩阵，而无需增加电磁定位系统，达到了在提高宽景拼接准确度的同时，降低了系统成本的效果。
作为一种可选的实施方式，本申请实施例设计了一个相对统一集成的神经网络结构，从传感器采集到的数据输入到该神经网络当中计算出当前时刻图像的变换矩阵M。 本申请实施例所述神经网络包括：卷积神经网络、递归神经网络和全连接网络，也即是将神经网络分为三个部分，第一部分是卷积神经网络(Convolutional Neural Networks,简称为CNN)，第二部分是递归神经网络(recursive neural network，简称为RNN)，第三部分是全连接网络(也即是回归网络)用以计算出最终的输出结果——变换矩阵M。如图2所示，卷积神经网络CNN的输出作为递归神经网络N4的输入，递归神经网络N4的输出作为全连接网络N5的输入，全连接网络N5计算得到最终的变换矩阵M。
进一步可选地，所述传感器为多个，所述传感器可以包括加速度计和陀螺仪。所述卷积神经网络包括第一卷积神经网络N3和与多个所述传感器一一对应的多个第二卷积神经网络(N1和N2)，其中，所述第一卷积神经网络的输入与多个所述第二卷积神经网络的输出连接。
上述可选的实施方式以及进一步可选的实施方式均是指本申请技术方案的一种可能的实施方式，本申请的技术方案可以采用上述实施方式来实现，也可以采用其他方式来实现，本申请对具体实现方式不做限定。
当然，本申请实施例中，传感器还可以包括其他可以检测探头运动的传感器，例如速度传感器等，其中，第二卷积神经网络与传感器的数量一一对应，第一卷积神经网络可以用于对多个第二卷积神经网络输出的数据融合处理后，做深度的学习和特征识别。本申请实施例中所述的第一卷积神经网络和第二卷积神经网络也可以称为卷积层通过设置多个层级的神经网络来训练学习和计算变换矩阵，尤其是采用卷积神经网络与传感器对应的方式，能够使得神经网络学习到更精确的特征信息，从而提高变换矩阵计算的准确性。
本申请实施例中，由于探头在移动过程中采集的数据是具有连续性的，例如T时长的检测数据，需要进行预处理，以使其能够满足神经网络进行处理和计算，本申请实施例中，所述获取传感器检测到的运动数据，包括：获取每个所述传感器检测到的待测时长的检测数据；对每个检测数据按照所述待测时长维度划分为等间隔的多段数据；对每个传感器对应的多段数据进行傅里叶变换，得到所述运动数据。
具体地，设传感器类别的数量为K，如果以加速度计和陀螺仪两种为例，K＝2，两种传感器产生的数据为X，两种传感器各自采集了T时间长的的检测数据(从1到T)，然后划分成n个等间隔的数据，其中第n批数据为
Figure PCTCN2019130603-appb-000002
对于
Figure PCTCN2019130603-appb-000003
其维度为D×U,这里D是传感器数据的维度，一般是三个维度，因此这里是D＝3；U是单个间隔的数据长度，譬如T时间内x方向采集了N个数据，划分成n个等间隔，因此U＝N/n。对于数据
Figure PCTCN2019130603-appb-000004
将对其每个维度的数据进行傅里叶变换，可以是进行快速傅里叶变换(fast Fourier transform，简称为FFT)，得到对应的频域数据
Figure PCTCN2019130603-appb-000005
该
Figure PCTCN2019130603-appb-000006
的维度为D*F,总共数据量为D*F*n，其中F＝2*f，f为当前数据的主要的f个频率，2表示需要其主要频率的系数，主要频率的系数包括正余弦分量对应的实数和虚数部分。如图2所示，采集的数据按照传感器类型和时间进行预处理后分别分组进入了相应的卷积神经网络，其中，k1表示第一个传感器，k2表示第二个传感器。
本申请实施例中，通过对传感器检测到的数据进行分割和傅里叶变换等处理，使其满足神经网络计算和识别的要求，能够对任意数据类型的传感器进行识别和计算，提高了神经网络的适用性，并提高了神经网络计算和识别的精度。
上述可选的实施方式以及进一步可选的实施方式均是指本申请技术方案的一种可能的实施方式，本申请的技术方案可以采用上述实施方式来实现，也可以采用其他方式来实现，本申请对具体实现方式不做限定。
进一步地，上述步骤S102，所述将所述运动数据输入到预先训练得到的神经网络中，利用所述神经网络计算得到变换矩阵的参数，包括：
S1021，通过所述卷积神经网络对所述运动数据进行卷积计算，得到所述运动数据的数据特征，作为所述卷积神经网络的输出。
卷积神经网络主要是对运动数据的特征进行学习和识别，通过卷积计算学习到传感器检测到的运动数据的数据特征，以及不同传感器之间的数据特征，然后输出到递归神经网络进行递归运算。
S1022，通过所述递归神经网络对所述卷积神经网络输出的数据特征进行递归运算，得到递归计算结果，作为所述递归神经网络的输出。
S1023，通过所述全连接网络对所述递归神经网络输出的递归计算结果回归计算，得到所述矩阵参数。
根据本申请实施例，卷积神经网络用于对运动数据进行训练和学习，以获得传感器检测到的数据的特征以及不同传感器之间数据的关系。然后，递归神经网络则是对卷积神经网络的输出结果按照时间顺序连接起来，然后做递归运算，最后由全连接网络回归出最终的变换矩阵的矩阵参数。
作为一种可选的实施方式，当采用多个神经网络层作为卷积神经网络时，所述通过所述卷积神经网络对所述运动数据进行卷积计算，得到所述运动数据的数据特征，包括：
S11，通过所述第二卷积神经网络对与所述第二卷积神经网络对应的传感器检测到 的运动数据进行卷积处理。
本申请实施例中，第二卷积神经网络为多个，如图2和图3中的N1和N2。各个第二卷积神经网络之间相互独立。由于每个第二卷积神经网络对应一个传感器，因此，每个第二卷积神经网络只需要处理相应的传感器检测到的数据即可。
以传感器有两个为例，进行举例说明。其中，传感器分别为加速度计和陀螺仪，由于加速度计和陀螺仪是两种不同性质的传感器，在训练的时候使用两个独立权重的神经网络N1和N2，N1和N2是各自独立但是结构相同的神经网络，其中，N1用于训练和学习加速度计的传感器数据，N2用于训练和学习陀螺仪的传感器数据。具体地，在训练过程中，加速度计检测到的数据输入到卷积神经网络N1中进行卷积处理，陀螺仪检测到的数据输入到卷积神经网络N2中进行卷积处理。如图3所示，两个结构相同的第二卷积神经网络N1和N2，通过训练N1和N2会学习到不同传感器数据的数据特征，因为加速度计和陀螺仪是两种不同性质的传感器，因此需要使用两个独立权重的神经网络训练，相应的，在神经网络使用的过程中也同理。如上述实施例所述，输入到该第二卷积神经网络维度为D*F,该第二卷积神经网络对应的卷积层的第一层的卷积核为
Figure PCTCN2019130603-appb-000007
第二卷积神经网络(该神经网络包括多层卷积层)的后面几层都为1*3，其中
Figure PCTCN2019130603-appb-000008
等于D，本申请实施例中就是3，第一层卷积核用于学习单个传感器不同维度数据之间的联系，后续层用于学习更深层(深度)的数据之间的关系。
S12，通过所述第一卷积神经网络对多个所述第二卷积神经网络的输出进行融合并进行卷积处理，得到所述数据特征。
如图2和3所示，第一卷积神经网络N3用于将多个传感器的运动数据通过第二卷积神经网络N1和N2之后输出的数据进行融合，并做卷积计算处理，得到多个传感器检测的运动数据之间更深层次的数据特征，作为整个卷积神经网络的输出结果，供后续递归神经网络进行处理。
可选地，所述通过所述第一卷积神经网络对多个所述第二卷积神经网络的输出进行融合并进行卷积处理，得到所述数据特征，包括：将每个所述第二卷积神经网络输出的数据平铺成一维数据将所有所述第二卷积神经网络对应的一维数据叠加在一起，通过所述第一卷积神经网络进行深度卷积计算，得到所述数据特征。
上述可选的实施方式以及进一步可选的实施方式均是指本申请技术方案的一种可能的实施方式，本申请的技术方案可以采用上述实施方式来实现，也可以采用其他方式来实现，本申请对具体实现方式不做限定。
如图3所示，第一卷积神经网络N3用于将多个传感器数据通过卷积神经网络N1和N2之后形成的数据平铺成一维数据然后将其行叠加在一起，再进行深度学习和处理。第一卷积神经网络包括多层卷积层，其中第一层的卷积核大小为
Figure PCTCN2019130603-appb-000009
后续几层网络都为1*3，其中
Figure PCTCN2019130603-appb-000010
为K，等于传感器的个数2，卷积神经网络N3用于融合两个传感器的数据，并学习到它更深层次的特征。
本申请实施例中，第一卷积神经网络N3和第二卷积神经网络N1、N2都采用多卷积层构建而成，并设置有激活函数和归一化层，其中，激活函数可以采用relu线性整流激活函数，归一化层可以采用批量归一化层(batch normalization)，保证输入分布的均值与方差固定在一定范围内,提高训练精度。
需要说明的是，上述实施例中所述的传感器数量和神经网络的数据仅仅是为了更清楚地描述本申请技术方案，由上述描述的原理可知，采用3个传感器以及更多的传感器，只需要对方案进行些微调整，仍属于本申请的保护范围。
本申请实施例中，经过第一卷积神经网络之后，输出的数据也即是递归神经网络的输入就是传感器在一段时间中，按照时间顺序采集的数据并学习到的数据特征。在传统的对于传感器中，譬如根据加速度计的数据计算相对位移的方法，一般都是在一小段时间内，计算加速度的积分得到速度，然后计算速度的积分就能计算出位移，此过程的输入都是单位时间内采集到的数据，本申请实施例中的递归神经网络也是基于类似的原理，可以从更早层次学习到积分求和的特征，从而在另一个角度去计算最终的输出结果。
具体地，以上述内容为例，传感器检测到的运动数据是按照传感器类型和时间依次送入上述的卷积神经网络N1和N2当中的，其输出为
Figure PCTCN2019130603-appb-000011
其中t＝1….n，表示多个时间段内传感器输入通过上述神经网络N1、N2和N3的输出结果，将其按照时间顺序连接起来，作为递归神经网络N4的输入。本申请实施例中的递归神经网络可以采用层叠多层的LSTM网络层，具体可以是层叠了两层，一种可选的递归神经网络结构如图4所示。将递归神经网络层中的每个阶段的输出送入全连接网络N5中。全连接网络N5用于回归出最终的探头移动变换矩阵的矩阵参数，探头移动的变换矩阵形式为：
Figure PCTCN2019130603-appb-000012
因此需要训练学习到的参数为探头旋转的角度θ，和探头偏移Δx和Δy，将计算出来的结果作为图像变换矩阵参数，用于最终的图像变换拼接当中。上述变换矩阵为根据 运动图像的相对关系总结和推导得到的。
上述可选的实施方式以及进一步可选的实施方式均是指本申请技术方案的一种可能的实施方式，本申请的技术方案可以采用上述实施方式来实现，也可以采用其他方式来实现，本申请对具体实现方式不做限定。
另一方面，本申请实施例还提供了一种变换矩阵获取装置，该装置可以用于执行上述实施例的变换矩阵获取方法，如图5所示，该装置包括：
运动数据获取模块501，用于获取传感器检测到的运动数据，其中，所述传感器设置在用于采集图像的探头上，所述运动数据用于表示所述探头在采集图像过程中的运动趋势；
参数计算模块502，用于将所述运动数据输入到预先训练得到的神经网络中，利用所述神经网络计算得到矩阵参数；
矩阵计算模块503，用于利用所述矩阵参数计算得到变换矩阵，所述变换矩阵用于拼接所述探头采集到的图像以得到宽景图像。
根据本申请实施例，通过获取探头采集图像的运动数据，利用预先训练得到的神经网络对该运动数据进行计算分析，得到探头的移动变化，进而计算出图像的变换矩阵，也即是间接地计算出图像的变化，无需利用图像本身的特征，即可计算出变换矩阵并进行图像拼接，不受图像亮度和特征等因素的影响，提高了变换矩阵计算的准确性，进而提升了图像拼接效果。
本申请实施例的变换矩阵获取装置与上述实施例的变换矩阵获取方法对应，具体描述参见上述实施例，这里不做赘述。
实施例2
本实施例提供一种宽景图像拼接方法，该方法主要用于对两个或者多个具有重叠的图像进行拼接，以形成宽景图像。如图6所示，该方法包括如下步骤：
步骤S601，利用探头探测目标区域连续的多个图像。
如实施例1中所述，探头上设置有用于检测探头运动数据的传感器，该探头在拍摄图像的过程中，需要进行移动以能够拍摄到目标区域的所有区域。连续的多个图像主要是由于探头在探测的过程中，图像是连续的，例如，视频图像。
步骤S602，利用变换矩阵获取方法获取所述多个图像中相邻图像之间的变换矩阵。
本申请实施例的变换矩阵获取方法也即是上述实施例1所述的变换矩阵获取方法，其具体工作原理和细节见上述实施例，这里不做赘述。
步骤S603，基于获取到的变换矩阵拼接所述多个图像得到宽景图像。
本申请实施例，通过利用上述变换矩阵获取方法基于传感器检测到的运动数据获取图像之间的变换矩阵，能够将探测的多个图像进行拼接，得到宽景图像。一种图像拼接的实例如图7所示，将图像A和图像B拼接在一起。由于变换矩阵的获取无需利用图像本身的特征，即可计算出变换矩阵并进行图像拼接，不受图像亮度和特征等因素的影响，提高了变换矩阵计算的准确性，进而提升了图像拼接效果。
另一方面，本申请实施例还提供了一种宽景图像拼接装置，该装置可以用于执行上述实施例的宽景图像拼接方法，如图8所示，该装置包括：
探测模块801，用于利用探头探测目标区域连续的多个图像；
变换矩阵获取装置802，用于获取所述多个图像中相邻图像之间的变换矩阵。
变换矩阵获取装置802即为上述实施例1中图5所示的装置，具体可以参见上面描述。
拼接模块803，用于基于获取到的变换矩阵拼接所述多个图像得到宽景图像。
本申请实施例，通过利用上述变换矩阵获取方法基于传感器检测到的运动数据获取图像之间的变换矩阵，能够将探测的多个图像进行拼接，得到宽景图像。由于变换矩阵的获取无需利用图像本身的特征，即可计算出变换矩阵并进行图像拼接，不受图像亮度和特征等因素的影响，提高了变换矩阵计算的准确性，进而提升了图像拼接效果。
本申请实施例的宽景图像拼接装置与上述实施例的宽景图像拼接方法对应，具体描述参见上述实施例，这里不做赘述。
实施例3
本申请实施例还提供了一种神经网络训练方法，该训练方法主要用于训练得到上述实施例1中所述的神经网络，如图9所示，该方法包括如下步骤：
步骤S901，获取训练样本数据，所述样本数据包括：传感器检测到的运动数据和与所述运动数据对应的矩阵参数，所述传感器设置在用于采集图像的探头上，所述运动数据用于表示所述探头在采集图像过程中的运动趋势，所述矩阵参数为用于拼接宽景图像的变换矩阵中的参数。
本申请实施例中，利用运动数据和已经标注的矩阵参数作为训练样本数据，该训练样本数据可以分为训练集和测试集，数据主要包括运动数据和标注的对应的矩阵参数，用于供神经网络模型的训练。
本申请实施例中需要标注出准确地矩阵参数，也即是标注出准确的变换矩阵，这样 训练的结果精度才能达到要求。由于变换矩阵无法通过两幅图像直接得到，本申请实施例采用了使用体膜图像用于训练，体膜图像特点在于，体膜内部可以设定固定的靶点，探头移动过程中，从体膜中扫描的图像，可以清楚的看到靶点，如图10所示，可以确定给出相同的靶点在两幅图像中的位置，以方面计算出变换矩阵，体膜图像的优点是图像清晰，计算出来的变换矩阵可靠正确。
也即是，获取训练样本数据包括：获取经过所述探头采集到的体膜图像；利用设置在相邻的体膜图像上的靶点坐标确定相邻两个体膜图像的变换矩阵；利用最小二乘法计算得到所述变换矩阵的矩阵参数；获取所述传感器检测到的所述运动数据，将所述矩阵参数和所述运动数据作为所述训练样本数据。
具体地，假设A图像靶点的坐标为Pa(x,y){1….n},则通过图像我们可以得到B图像中对应靶点的坐标Pb(x,y){1….n},则：
Figure PCTCN2019130603-appb-000013
*表示矩阵乘法。
采用最小二乘法，通过最小化实际值和计算值得误差得到：
Figure PCTCN2019130603-appb-000014
(x i,y i)和(x′ i,y′ i)分别是A和B两幅图像对应的靶点的坐标，最小化E值，通过计算对应的导数为0：
Figure PCTCN2019130603-appb-000015
可以计算出最优的矩阵参数θ、Δx和Δy。从而得到相应的变换矩阵M。
步骤S902，利用所述训练样本数据对预先建立的神经网络模型进行训练，得到用于获取所述变换矩阵的神经网络。
在探头移动的过程中，采集固定时间间隔内的传感器的数据，计算当前间隔内图像移动变换的M，将数据送入上述神经网络中训练，迭代计算出最优的网络参数。
本申请实施例中，通过利用传感器检测到的运动数据和矩阵参数来训练神经网络模型，从而使得神经网络模型能够学习和识别出运动数据与矩阵参数之间的数据关系，得到神经网络，用于后面对其他运动数据识别出相应的变换矩阵，采用了神经网络的方式，通过分析探头的移动变化，间接的计算出图像的变化，从而提高了准确性。
本申请实施例的另一方面，还提供了一种神经网络训练装置，该装置可以用于执行上述的神经网络训练方法，如图11所示，该装置包括：
样本获取模块111，用于获取训练样本数据，所述样本数据包括：传感器检测到的运动数据和与所述运动数据对应的矩阵参数，所述传感器设置在用于采集图像的探头上，所述运动数据用于表示所述探头在采集图像过程中的运动趋势，所述矩阵参数为用于拼接宽景图像的变换矩阵中的参数；
训练模块112，用于利用所述训练样本数据对预先建立的神经网络模型进行训练，得到用于获取所述变换矩阵的神经网络。
本申请实施例中，通过利用传感器检测到的运动数据和矩阵参数来训练神经网络模型，从而使得神经网络模型能够学习和识别出运动数据与矩阵参数之间的数据关系，得到神经网络，用于后面对其他运动数据识别出相应的变换矩阵，采用了神经网络的方式，通过分析探头的移动变化，间接的计算出图像的变化，从而提高了准确性。
本申请实施例的神经网络训练装置与上述实施例的神经网络训练方法对应，具体描述参见上述实施例，这里不做赘述。
实施例4
本实施例还提供一种计算机设备，如可以执行程序的台式计算机、机架式服务器、刀片式服务器、塔式服务器或机柜式服务器(包括独立的服务器，或者多个服务器所组成的服务器集群)等。本实施例的计算机设备120至少包括但不限于：可通过系统总线相互通信连接的存储器121、处理器122，如图12所示。需要指出的是，图12仅示出了具有组件121-122的计算机设备120，但是应理解的是，并不要求实施所有示出的组件，可以替代的实施更多或者更少的组件。
本实施例中，存储器121(即可读存储介质)包括闪存、硬盘、多媒体卡、卡型存储器(例如，SD或DX存储器等)、随机访问存储器(RAM)、静态随机访问存储器(SRAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、可编程只读存储器(PROM)、磁性存储器、磁盘、光盘等。在一些实施例中，存储器121可以是计算机设备120的内部存储单元，例如该计算机设备120的硬盘或内存。在另一些实施例中，存储器121也可以是计算机设备120的外部存储设备，例如该计算机设备120上配备的插接式硬盘，智能存储卡(Smart Media Card,SMC)，安全数字(Secure Digital,SD)卡，闪存卡(Flash Card)等。当然，存储器121还可以既包括计算机设备120的内部存储单元也包括其外部存储设备。本实施例中，存储器21通常用于存储安装于计算机设备120 的操作系统和各类应用软件，例如实施例所述的变换矩阵获取、宽景图像拼接、神经网络训练方法的程序代码等。此外，存储器121还可以用于暂时地存储已经输出或者将要输出的各类数据。
处理器122在一些实施例中可以是中央处理器(Central Processing Unit，CPU)、控制器、微控制器、微处理器、或其他数据处理芯片。该处理器122通常用于控制计算机设备120的总体操作。本实施例中，处理器122用于运行存储器121中存储的程序代码或者处理数据，例如实现实施例的变换矩阵获取、宽景图像拼接、神经网络训练方法。
本实施例还提供一种计算机可读存储介质，如闪存、硬盘、多媒体卡、卡型存储器(例如，SD或DX存储器等)、随机访问存储器(RAM)、静态随机访问存储器(SRAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、可编程只读存储器(PROM)、磁性存储器、磁盘、光盘、服务器、App应用商城等等，其上存储有计算机程序，程序被处理器执行时实现相应功能。本实施例的计算机可读存储介质用于存储变换矩阵获取、宽景图像拼接、神经网络训练装置，被处理器执行时实现实施例的变换矩阵获取、宽景图像拼接、神经网络训练方法。
显然，上述实施例仅仅是为清楚地说明所作的举例，而并非对实施方式的限定。对于所属领域的普通技术人员来说，在上述说明的基础上还可以做出其它不同形式的变化或变动。这里无需也无法对所有的实施方式予以穷举。而由此所引伸出的显而易见的变化或变动仍处于本申请创造的保护范围之中。

Claims (12)

  1. 一种变换矩阵获取方法，其特征在于，包括如下步骤：
    获取传感器检测到的运动数据，其中，所述传感器设置在用于采集图像的探头上，所述运动数据用于表示所述探头在采集图像过程中的运动趋势；
    将所述运动数据输入到预先训练得到的神经网络中，利用所述神经网络计算得到矩阵参数；
    利用所述矩阵参数计算得到变换矩阵，所述变换矩阵用于拼接所述探头采集到的图像以得到宽景图像。
  2. 根据权利要求1所述的变换矩阵获取方法，其特征在于，所述神经网络包括：卷积神经网络、递归神经网络和全连接网络；其中，所述将所述运动数据输入到预先训练得到的神经网络中，利用所述神经网络计算得到变换矩阵的参数，包括：
    通过所述卷积神经网络对所述运动数据进行卷积计算，得到所述运动数据的数据特征，作为所述卷积神经网络的输出；
    通过所述递归神经网络对所述卷积神经网络输出的数据特征进行递归运算，得到递归计算结果，作为所述递归神经网络的输出；
    通过所述全连接网络对所述递归神经网络输出的递归计算结果回归计算，得到所述矩阵参数。
  3. 根据权利要求2所述的变换矩阵获取方法，其特征在于，所述传感器为多个，所述卷积神经网络包括第一卷积神经网络和与多个所述传感器一一对应的多个第二卷积神经网络，其中，所述第一卷积神经网络的输入与多个所述第二卷积神经网络的输出连接。
  4. 根据权利要求3所述的变换矩阵获取方法，其特征在于，所述传感器包括加速度计和陀螺仪。
  5. 根据权利要求3或4所述的变换矩阵获取方法，其特征在于，所述通过所述卷积神经网络对所述运动数据进行卷积计算，得到所述运动数据的数据特征，包括：
    通过所述第二卷积神经网络对与所述第二卷积神经网络对应的传感器检测到的运动数据进行卷积处理；
    通过所述第一卷积神经网络对多个所述第二卷积神经网络的输出进行融合并进行卷积处理，得到所述数据特征。
  6. 根据权利要求5所述的变换矩阵获取方法，其特征在于，
    所述通过所述第一卷积神经网络对多个所述第二卷积神经网络的输出进行融合并进行卷积处理，得到所述数据特征，包括：
    将每个所述第二卷积神经网络输出的数据平铺成一维数据；
    将所有所述第二卷积神经网络对应的一维数据叠加在一起，通过所述第一卷积神经 网络进行深度卷积计算，得到所述数据特征。
  7. 根据权利要求5所述的变换矩阵获取方法，其特征在于，所述获取传感器检测到的运动数据，包括：
    获取每个所述传感器检测到的待测时长的检测数据；
    对每个检测数据按照所述待测时长维度划分为等间隔的多段数据；
    对每个传感器对应的多段数据进行傅里叶变换，得到所述运动数据。
  8. 一种宽景图像拼接方法，其特征在于，包括如下步骤：
    利用探头探测目标区域连续的多个图像；
    利用权利要求1-7任一项所述的变换矩阵获取方法获取所述多个图像中相邻图像之间的变换矩阵；
    基于获取到的变换矩阵拼接所述多个图像得到宽景图像。
  9. 一种神经网络训练方法，其特征在于，包括如下步骤：
    获取训练样本数据，所述样本数据包括：传感器检测到的运动数据和与所述运动数据对应的矩阵参数，所述传感器设置在用于采集图像的探头上，所述运动数据用于表示所述探头在采集图像过程中的运动趋势，所述矩阵参数为用于拼接宽景图像的变换矩阵中的参数；
    利用所述训练样本数据对预先建立的神经网络模型进行训练，得到用于获取所述变换矩阵的神经网络。
  10. 根据权利要求9所述的神经网络训练方法，其特征在于，获取训练样本数据，包括：
    获取经过所述探头采集到的体膜图像；
    利用设置在相邻的体膜图像上的靶点坐标确定相邻两个体膜图像的变换矩阵；
    利用最小二乘法计算得到所述变换矩阵的矩阵参数；
    获取所述传感器检测到的所述运动数据，将所述矩阵参数和所述运动数据作为所述训练样本数据。
  11. 一种计算机设备，其特征在于，包括存储器、处理器以及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现权利要求1-7、8、9-10中任一项所述方法的步骤。
  12. 一种计算机可读存储介质，其上存储有计算机程序，其特征在于：所述计算机程序被处理器执行时实现权利要求1-7、8、9-10中任一项所述方法的步骤。
PCT/CN2019/130603 2019-12-31 2019-12-31 宽景拼接方法、装置及存储介质 WO2021134506A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
PCT/CN2019/130603 WO2021134506A1 (zh) 2019-12-31 2019-12-31 宽景拼接方法、装置及存储介质
EP19958642.1A EP3901896A4 (en) 2019-12-31 2019-12-31 METHOD AND DEVICE FOR THE COMPOSITION OF DISTANCE VISION AND STORAGE MEDIUM
US17/420,762 US11983844B2 (en) 2019-12-31 2019-12-31 Panoramic stitching method, apparatus, and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2019/130603 WO2021134506A1 (zh) 2019-12-31 2019-12-31 宽景拼接方法、装置及存储介质

Publications (1)

Publication Number Publication Date
WO2021134506A1 true WO2021134506A1 (zh) 2021-07-08

Family

ID=76686316

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/130603 WO2021134506A1 (zh) 2019-12-31 2019-12-31 宽景拼接方法、装置及存储介质

Country Status (3)

Country Link
US (1) US11983844B2 (zh)
EP (1) EP3901896A4 (zh)
WO (1) WO2021134506A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117094895B (zh) * 2023-09-05 2024-03-26 杭州一隅千象科技有限公司 图像全景拼接方法及其系统
CN117764825B (zh) * 2024-02-22 2024-04-26 南京派光智慧感知信息技术有限公司 一种多隧道图像拼接算法、电子设备和可读存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105447850A (zh) * 2015-11-12 2016-03-30 浙江大学 一种基于多视点图像的全景图拼接合成方法
US20170150235A1 (en) * 2015-11-20 2017-05-25 Microsoft Technology Licensing, Llc Jointly Modeling Embedding and Translation to Bridge Video and Language
CN106920215A (zh) * 2017-03-06 2017-07-04 长沙全度影像科技有限公司 一种全景图像配准效果的检测方法
CN109584283A (zh) * 2018-11-29 2019-04-05 合肥中科离子医学技术装备有限公司 一种基于卷积神经网络的医学图像配准算法
CN109740753A (zh) * 2018-12-31 2019-05-10 浙江新铭智能科技有限公司 一种神经网络构建方法
CN110288653A (zh) * 2019-07-15 2019-09-27 中国科学院深圳先进技术研究院 一种多角度超声图像融合方法、系统及电子设备

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CA3075334C (en) * 2017-09-07 2021-08-31 Piur Imaging Gmbh Apparatus and method for determining motion of an ultrasound probe
WO2020054584A1 (ja) * 2018-09-12 2020-03-19 富士フイルム株式会社 合成処理システム、合成処理装置、及び合成処理方法

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105447850A (zh) * 2015-11-12 2016-03-30 浙江大学 一种基于多视点图像的全景图拼接合成方法
US20170150235A1 (en) * 2015-11-20 2017-05-25 Microsoft Technology Licensing, Llc Jointly Modeling Embedding and Translation to Bridge Video and Language
CN106920215A (zh) * 2017-03-06 2017-07-04 长沙全度影像科技有限公司 一种全景图像配准效果的检测方法
CN109584283A (zh) * 2018-11-29 2019-04-05 合肥中科离子医学技术装备有限公司 一种基于卷积神经网络的医学图像配准算法
CN109740753A (zh) * 2018-12-31 2019-05-10 浙江新铭智能科技有限公司 一种神经网络构建方法
CN110288653A (zh) * 2019-07-15 2019-09-27 中国科学院深圳先进技术研究院 一种多角度超声图像融合方法、系统及电子设备

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3901896A4 *

Also Published As

Publication number Publication date
US20220343460A1 (en) 2022-10-27
US11983844B2 (en) 2024-05-14
EP3901896A1 (en) 2021-10-27
EP3901896A4 (en) 2022-09-07

Similar Documents

Publication Publication Date Title
KR101908520B1 (ko) 메디컬 이미징에서 공간 및 시간 제약들을 이용하는 랜드마크 검출
CN111161153B (zh) 宽景拼接方法、装置及存储介质
CN103908300B (zh) 诊断超声成像中的针增强
CN110378966B (zh) 车路协同相机外参标定方法、装置、设备及存储介质
WO2021134506A1 (zh) 宽景拼接方法、装置及存储介质
BR112016009829B1 (pt) Sistema de imageamento diagnóstico ultrassônico que captura uma pluralidade de planos de vista padrão do coração e método para capturar uma pluralidade de planos de vista padrão do coração
KR102234461B1 (ko) ２ｄ 지도를 이용하여 거리뷰 이미지의 깊이 정보를 생성하는 방법 및 시스템
EP3532797A1 (en) Curated photogrammetry
CN108875723A (zh) 对象检测方法、装置和系统及存储介质
WO2021004261A1 (zh) 深度数据的滤波方法、装置、电子设备和可读存储介质
WO2019223123A1 (zh) 病变部位识别方法及装置、计算机装置及可读存储介质
CN108648178A (zh) 一种影像结节检测的方法及装置
WO2018002004A1 (en) Intertial device tracking system and method of operation thereof
CN110222641B (zh) 用于识别图像的方法和装置
CN109828302A (zh) 一种基于多振动传感器的震源定位方法及装置
CN107106128A (zh) 用于分割解剖目标的超声成像装置和方法
JP6558819B2 (ja) 個体計数装置、個体計数方法、およびプログラム
JP7136737B2 (ja) 三次元位置計測装置、三次元位置計測方法及びプログラム
US20230124674A1 (en) Deep learning for optical coherence tomography segmentation
JP2018521733A5 (zh)
CN108961337A (zh) 车载相机航向角标定方法和装置、电子设备以及车辆
JP2015534054A (ja) ブレード検査におけるギャップ測定に対するハフ変換アプローチ
CN108366779B (zh) 用于检测工具的装置和方法
US20210192291A1 (en) Continuous training for ai networks in ultrasound scanners
US20070167757A1 (en) Determining speed-of-sound factors in ultrasound images of a body

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2019958642

Country of ref document: EP

Effective date: 20210720

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19958642

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE