CN114912499A

CN114912499A - Deep learning-based associated imaging method and system

Info

Publication number: CN114912499A
Application number: CN202111325407.8A
Authority: CN
Inventors: 殷曼倩
Original assignee: Tianyi Digital Life Technology Co Ltd
Current assignee: Tianyi Shilian Technology Co ltd
Priority date: 2021-11-10
Filing date: 2021-11-10
Publication date: 2022-08-16

Abstract

The invention provides a deep learning-based associated imaging method and system, wherein the system comprises the following steps: the system comprises a light source, a speckle matrix and a controller, wherein the light source is used for transmitting a speckle matrix to a target to be imaged; the barrel detector is used for measuring the light intensity value of the speckle matrix after the speckle matrix projects the target; and a computing unit configured to: acquiring an initial image of a target to be imaged; dividing the initial image into a plurality of image patches; carrying out similarity clustering on the image blocks; determining a matched speckle-like matrix for each image block based on clustering; combining the speckle matrixes of each image block into a complete speckle matrix and providing the complete speckle matrix to the light source; and obtaining a reconstructed image of the target through correlation calculation based on the speckle matrix and the measured value of the barrel detector.

Description

Deep learning-based associated imaging method and system

Technical Field

The invention relates to the field of deep learning technology and associated imaging, in particular to an associated imaging method and system based on deep learning.

Background

Correlation imaging is one of the hot spot and leading edge techniques in the field of quantum optics for the last decades. In order to overcome the problems that the quality of a generated image is reduced due to the influences of cloud cover shielding, atmospheric disturbance, ocean turbulence, system noise, relative motion and the like on detection when a target object is in a shielding or complex background, and the phenomena of image distortion, image blurring and the like generally occur no matter the target object is influenced by self or objective factors, so that the identification degree of the image is low, and the anti-interference performance of imaging can be effectively improved by using a correlation imaging method.

Compared with the traditional imaging technology, the correlation imaging has non-localization property, and a reconstructed object image can be obtained by placing a detector with spatial resolution capability on a reference light path without an object to be detected and then performing coincidence calculation. Fig. 1 is a schematic scene diagram of related imaging in the prior art, as shown in fig. 1, a light source generates speckles, the speckles are projected on an object and then are received and measured by a detector without spatial resolution capability, and finally, a reconstructed image of the object is obtained through related calculation.

With the research and development of the correlation imaging technology, thermo-optic correlation imaging and computational correlation imaging are proposed, and the realization process is simpler, so that the correlation imaging can be applied to a wider field. At present, associated imaging can be quickly and effectively provided in the case that a target object is in a shielding or complex background, and the information of a measured object is acquired only by a barrel detector without spatial resolution capability, so that the associated imaging comprises various high-altitude, ground or underwater observation data, and can be applied to the fields of topographic mapping, military reconnaissance, ocean detection, high-altitude detection, medical imaging, satellite remote sensing, laser radar and the like.

However, in practical applications, when an image is reconstructed, the larger the image is, the more the number of speckles is required, the longer the reconstruction time is, and other algorithms are required to improve the imaging time and quality. In order to improve the imaging quality and resolution of the associated imaging, one generally conceivable approach is to increase the resolution of the imaging hardware device. However, the method has the disadvantages of higher cost, great technical difficulty and small lifting space in hardware improvement. It is therefore desirable to provide a solution that more effectively improves the imaging quality and resolution of the associated imaging.

Disclosure of Invention

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

In order to solve the technical problems mentioned in the background section, the invention provides a deep learning-based correlation imaging method and system, which perform speckle optimization through image segmentation and dictionary learning technology, so that an image of an unknown object can be reconstructed with a minimum number of speckles.

According to one aspect of the invention, there is provided an optimized speckle matrix training method for correlated imaging, the method comprising:

dividing a training image into a plurality of image patches;

carrying out similarity clustering on the image blocks;

performing dictionary learning on each block cluster to obtain a plurality of class dictionaries; and

an optimized speckle-like matrix corresponding to each block cluster is calculated based on the obtained class dictionary.

According to a further embodiment of the present invention, similarity clustering of image patches further comprises:

and executing a K-means clustering algorithm on the divided image blocks, and clustering a plurality of image blocks with similar image structure characteristics into block clusters.

According to a further embodiment of the present invention, the dictionary learning for each block cluster further comprises:

and selecting different dictionary learning methods for dictionary learning according to the image structure characteristic characteristics of the block clustering aiming at different block clustering.

According to a further embodiment of the present invention, calculating an optimized speckle-like matrix corresponding to each block cluster based on the obtained class dictionary further comprises:

aiming at different block clusters, different speckle optimization methods are selected according to the image structure characteristic of the block clusters to carry out speckle optimization.

According to another aspect of the invention, there is provided an associated imaging method, the method comprising:

acquiring an initial image of a target to be imaged;

dividing the initial image into a plurality of image patches;

carrying out similarity clustering on the image blocks;

determining a matched speckle-like matrix for each image block based on clustering;

combining the speckle matrixes of each image block into a complete speckle matrix; and

correlated imaging is performed with the bucket detector using the combined speckle matrix.

According to a further embodiment of the invention, correlating imaging with the combined speckle matrix and the corresponding combined bucket detector further comprises:

respectively carrying out correlated imaging by using the combination of each group of speckle matrixes and the corresponding barrel detector to obtain an image block with low resolution; and

and weighting each obtained low-resolution image block to obtain a high-resolution image.

According to a further embodiment of the present invention, determining a matching speckle-like matrix for each image patch based on clustering further comprises:

determining a closest pre-training block cluster for the block cluster to which each current image block belongs in a plurality of pre-training block clusters through mean comparison;

and taking the optimized speckle-like matrix corresponding to the closest pre-training block cluster as the speckle matrix of the current image block.

According to a further embodiment of the present invention, combining the speckle matrix of each image patch into a complete speckle matrix further comprises:

and combining the speckle matrixes according to the position of each image block in the initial image so as to obtain the complete speckle matrix.

According to a further aspect of the invention, there is provided an associated imaging system, the system comprising:

the light source is used for transmitting the speckle matrix to a target to be imaged;

the barrel detector is used for measuring the light intensity value of the speckle matrix after the speckle matrix projects the target; and

a computing unit configured to:

acquiring an initial image of a target to be imaged;

dividing the initial image into a plurality of image patches;

carrying out similarity clustering on the image blocks;

combining the speckle matrixes of each image block into a complete speckle matrix and providing the complete speckle matrix to the light source; and

and obtaining a reconstructed image of the target through correlation calculation based on the speckle matrix and the measured value of the barrel detector.

Compared with the scheme in the prior art, the associated imaging system and the associated imaging method provided by the invention at least have the following advantages:

1. the number of speckles required by a speckle matrix under high resolution and the calculation complexity are reduced through image blocking and clustering;

2. and performing multi-dictionary learning on the clusters, and performing dictionary learning and speckle optimization based on the image structure characteristics of the clusters, so that the number of speckles and the calculation complexity are further reduced, and the reconstruction definition is improved.

These and other features and advantages will become apparent from a reading of the following detailed description and a review of the associated drawings. It is to be understood that both the foregoing general description and the following detailed description are explanatory only and are not restrictive of aspects as claimed.

Drawings

So that the manner in which the above recited features of the present invention can be understood in detail, a more particular description of the invention, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only some typical aspects of this invention and are therefore not to be considered limiting of its scope, for the description may admit to other equally effective aspects.

Fig. 1 is a schematic scene diagram of correlated imaging in the prior art.

FIG. 2 is a schematic flow diagram of a deep learning based optimized speckle matrix training method according to one embodiment of the present invention.

FIG. 3 is a schematic flow diagram of an associated imaging method according to one embodiment of the invention.

FIG. 4 is a schematic block diagram of an associated imaging system according to one embodiment of the present invention.

Detailed Description

The present invention will be described in detail below with reference to the attached drawings, and the features of the present invention will be further apparent from the following detailed description.

As mentioned previously, a problem in the prior art is that as the resolution requirements of the associated imaging increase, the number of speckles required to reconstruct an image increases significantly, which results in a corresponding increase in the speckle projection time required for the associated imaging and the associated computational complexity in the reconstruction. To this end, the invention proposes that a speckle adaptive selection model can be trained based on image segmentation and dictionary learning. When the trained model is used for reconstructing an image, corresponding optimized speckles can be selected in a self-adaptive mode on the basis of image blocking, then a complete speckle matrix is spliced for correlation imaging, and the number of speckles and the calculation complexity required by the speckle matrix obtained in the mode are obviously reduced.

FIG. 2 is a schematic flow diagram of a deep learning-based optimized speckle matrix training method 200 according to one embodiment of the present invention. The method 200 begins at step 202 by dividing a training image into a plurality of image patches. The training images may be selected from a set of training images, preferably different sets of training images may be employed for different imaging scenarios. For example, for the application scenario of remote sensing imaging, the existing public NWPU VHR-10 space remote sensing data set can be used. Similarly, other domains will typically have corresponding training image sets for that domain. It will be appreciated that in a scene without a dedicated training image set, real images of the scene that meet the resolution requirements may also be collected to construct the training image set.

Blocking an image refers to dividing an original image into a plurality of small image blocks by a certain size. It will be appreciated that the smaller the patch size, the smaller the number of speckles and the amount of computation for each patch, and the greater the likelihood that there will be non-local similarities between patches. For example, in remote sensing scenes, many similar parts can be found in aerial images, such as airplanes, vehicles, ships, oil tanks, stadiums, farmlands, ports, bridges, etc., and such objects or targets all present similar shapes and image features in aerial images. Also, it will be appreciated that if the patch size is selected to be too large or too small, it may result in the objects or objects not being properly received in a patch or effectively identified. Therefore, an appropriate block size can be selected according to an actual scene.

At step 204, the image patches are similarity clustered. In one example, the image patches may be clustered using a K-means clustering algorithm. The K-means clustering algorithm selects the euclidean distance to judge whether the images are similar, and is a commonly used clustering algorithm.

Within each block cluster, any selected center image block x _i By non-locally self-similar image blocks

The weighting is obtained by the following formula:

wherein

Representing similar blocks

The weighting coefficient of (2).

During clustering, firstly, the optimal initial clustering center is determined, then the sum of squares of errors of the central image block and the similar images is calculated, whether the sum is the minimum value or not is judged, and if the sum is the minimum value, the central image block and the similar images are clustered into one class. The more similar each image block is, the better the clustering together can be made, and the image blocks grouped into one class all have strong similar structures, i.e. the image blocks in the same class are non-local similar image blocks to each other. Through this similarity clustering process, a plurality of block clusters can be obtained.

If the error is the minimum value, the error is grouped into a class, and the square sum of the errors is defined as follows:

where x is the image block, M _i Is a sample C _i Average value of (a). The above equation represents to some extent the difference between each image block in the sample and the average, J _C The representative sum of squares of errors can reflect the difference between all test samples, if the difference degree between each image block in the samples is smaller, the value is smaller, each image block is more similar, the image blocks can be better clustered together, and the image blocks clustered into one class have strong similar structures.

In step 206, dictionary learning is performed on each block cluster to obtain a plurality of class dictionaries. Dictionary Learning (Dictionary Learning) is a machine Learning method commonly used for image reconstruction, and the basic idea is to train a Learning Dictionary according to an image and then reconstruct a low-resolution image into a high-resolution image by using the learned Dictionary. Preferably, the present invention may use a sparse representation-based method, also referred to as a sparse dictionary learning method. In the sparse dictionary learning method, the reconstructed image block can be obtained by the following formula:

x＝D·α

where x denotes a reconstructed image block (assuming dimension D), D denotes a class dictionary of a block cluster corresponding to the image block (dimension D × K), and α is a class sparse matrix corresponding to the dictionary D (dimension K). According to the sparse dictionary learning method, after a plurality of image blocks belonging to a cluster are provided as training input, a class dictionary D and a class sparse matrix alpha corresponding to the cluster can be learned through the method, wherein the class dictionary D is stored. According to the dictionary learning algorithm, as will be described later, the quasi-sparse coefficient of each image block can be obtained through calculation in reconstruction, and the reconstructed image block can be obtained through x ═ D · α.

Preferably, each block cluster may employ a different dictionary learning algorithm, such as a PCA algorithm, a MOD algorithm, a K-SVD algorithm, and so forth. In other words, the dictionary learning of the present invention can be based on a multi-dictionary model. It will be appreciated that the characteristics of images of different scenes may vary greatly and that the feature structures of different parts may differ significantly even within a single image of the same scene. For example, in an aerial remote sensing image, an artificial object such as a building or a vehicle generally has a regular shape, a natural landscape generally has an irregular shape, and features such as a texture, an edge, and a structure have large differences. Under the condition, the images are partitioned and clustered, so that the image blocks with similar structural features can be effectively classified into one class, and a dictionary learning algorithm suitable for the characteristics of the class of images is selected.

At step 208, an optimized speckle-like matrix is computed based on the obtained class dictionary. According to the theory of correlated imaging, the basic mathematical expression is:

y＝Фx

where y is the observed value of the bucket detector, Φ is the speckle matrix projected by the light source, and x is the image to be reconstructed. It will be appreciated that the observed value y is a known value that can be obtained by measurement, and the speckle Φ is also known, so the image x can be solved by the above equation. However, while speckle Φ can be theoretically arbitrary, random speckle is really used in many scenes in reality, but for a particular image, a different speckle matrix can affect the effect of the reconstructed image, e.g., the error from the original image. On the contrary, the number of speckles required by different speckle matrixes may vary greatly to achieve a certain reconstruction effect. The optimization of the speckle matrix is therefore intended to achieve the required reduction accuracy with the smallest possible number of speckles.

The different effects of different speckle matrices are mainly influenced by the image characteristics. In this regard, clustering of image patches at previous steps in the present invention has classified patches with substantially the same or sufficiently similar image characteristics into a class, thus providing a good basis for seeking an optimized speckle matrix for a particular image characteristic. Currently, different speckle optimization methods exist in the prior art, and any of them can be used in the present invention. Similar to different dictionary learning methods for different clusters, different clusters may also use different speckle optimization methods to obtain an optimized speckle-like matrix. Thus, a plurality of speckle-like matrices optimized for different clusters can be obtained.

As an example, by learning a sub-dictionary D adaptively for each class _i Thereby obtaining a sparsity-like coefficient alpha _i And then, bringing the super-resolution reconstruction model into remote sensing associated imaging based on the similar sparse regularization image:

s.t. X＝T _k α _k +T _n α _n

wherein D is _i ＝Φ _i Ψ _i As class dictionary, α _i Is a sparsity-like coefficient, x _i ＝Ψ _i α _i The image to be reconstructed is obtained, lambda is a sparse regularization parameter, and eta is a regularization parameter of a non-local constraint term. λ is used to balance the fidelity term with the sparse regularization term, and η is used to balance the fidelity term with the non-local self-similar structure sparse term.

In the formula, λ _i ，η _i Can be obtained by the following formula:

wherein σ _i Alpha for similar image blocks in the ith cluster _i The standard deviation, τ, obtained ₁ ，τ ₂ For a well-defined constant, ε is a small integer to avoid λ _i ，η _i Too large.

By means of ₁ Norm to constrain a _i Sparsity of l ₂ Norm constraint non-local self-similarity error, thereby remote sensing image is fast x _i Can be obtained by coefficient coding, namely:

x _i ＝D _i α _i

meanwhile, speckles of the sampling matrix phi serving as associated imaging are optimized by dictionary learning, so that speckles with a certain compression ratio and required for imaging are obtained, and the sampling times are reduced.

Finally, a reconstructed image is obtained through a second-order correlation formula

In the formula, G represents a second order correlation function, M represents the number of speckle measurements, and y ^(m) The value of the detector is represented and,<>representing the arithmetic mean, phi _m Representing the spatial intensity distribution of the mth speckle.

FIG. 3 is a schematic flow diagram of an associated imaging method 300 in accordance with one embodiment of the present invention. The method 300 begins at step 302 by acquiring an initial image of a target to be imaged. In one example, the initial image may be an image obtained by correlated imaging using random speckle, or may be an image of the target obtained in any other way. For example, in a remote sensing scene, the initial image may be a previously historical remote sensing image or a conventional optical image of the target area. It will be appreciated that this initial image may have a lower resolution, sharpness or accuracy than is currently required, or may be locally unusable due to the presence of some occlusion, etc. Alternatively, in the case where a low-resolution initial image is used, the initial image may be further preprocessed to convert it into an image having a desired high resolution.

At step 304, the initial image is divided into a plurality of image patches. This step is similar to the image blocking step 202 described previously in connection with fig. 2, and may be divided, for example, according to the image blocking parameters (e.g., size, step size, overlap, etc.) employed during training.

At step 306, the image patches are similarity clustered. This step is similar to the image segmentation step 204 previously described in connection with fig. 1, e.g. the same clustering algorithm may be used for clustering.

At step 308, a matching speckle-like matrix is determined for each image patch based on the clustering. Through the training process described in fig. 2, speckle-like matrices for different clusters of the image block are obtained, each speckle-like matrix corresponding to a cluster. At this time, in the reconstruction process, it is necessary to determine which of the clusters formed in the training process matches or is closest to each of the clusters obtained in step 306. This determination may take many forms, and may be made, as a non-limiting example, by calculating the mean of the patches of each cluster and comparing the mean of the patches of each cluster during training to determine which cluster in the training model the current cluster best matches. Then, the class speckle matrix corresponding to the cluster to which each image patch belongs is used as the speckle matrix for that image patch.

At step 310, the speckle matrices for each image patch are combined into a complete speckle matrix. This step can be done, for example, by combining the speckle matrices according to the positional relationship of each image block, resulting in a speckle matrix corresponding to the formal correlation imaging to be performed.

At step 312, correlated imaging is performed with the combined speckle matrix and bucket detector. The process is consistent with the existing associated imaging process, namely, a light source projects a target area or a target object by using speckles, then a barrel detector receives light intensity measured values, and the target imaging is obtained through associated calculation. Optionally, the bucket detector array may also be set up accordingly when performing correlated imaging. Correspondingly, during correlated imaging, each group of speckle matrixes and the combination of the bucket detectors can be used for correlated imaging to obtain low-resolution image blocks, and then the obtained low-resolution image blocks are weighted to obtain high-resolution images, wherein the weighting coefficients can utilize the weighting coefficients obtained in the clustering process. In this way, compared with the method of directly using the combined complete speckle matrix to directly correlate the imaging calculation to obtain the high-resolution image, the method further improves the calculation complexity and the calculation amount.

Compared with the prior art, due to the use of the speckle matrix obtained by the process, each part of the speckle matrix is optimized, namely the speckle matrix formed aiming at the characteristics of the target to be imaged contains the least number of speckles under the condition of meeting the precision, so that the target reconstruction image meeting the requirements of resolution and definition can be obtained by using the least number of speckles and the least computation complexity as possible on the whole.

Alternatively, the above process may be repeatedly performed. For example, the reconstructed image obtained in step 312 may be used as an initial image in the next iteration, so as to continuously improve the image sharpness and continuously reduce the required number of speckles and the computational complexity.

FIG. 4 is a schematic block diagram of an associated imaging system 400 according to one embodiment of the invention. As shown in fig. 4, the correlated imaging system 400 may include a light source 401 for emitting a speckle matrix towards an object to be imaged, a bucket detector 402 for measuring light intensity values of the speckle matrix after it is projected through the object, and a calculation unit 403.

The calculation unit 403 may be communicatively coupled with the light source 401 and the bucket detector 402 to send the speckle matrix for projection to the light source 401 and to receive the measured light intensity values from the bucket detector 402. According to an embodiment of the invention, the calculation unit 403 may be configured to obtain an initial image of the object to be imaged, divide the initial image into a plurality of image patches, perform similarity clustering on the image patches, determine a matching speckle-like matrix for each image patch based on the clustering, combine the speckle matrices of each image patch into a complete speckle matrix and provide to the light source 401, and obtain a reconstructed image of the object by correlation calculation based on the speckle matrices and the measurement values of the bucket detector 402. Alternatively, the bucket detector 402 may be a bucket detector array composed of a plurality of bucket detectors.

What has been described above includes examples of aspects of the claimed subject matter. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the claimed subject matter, but one of ordinary skill in the art may recognize that many further combinations and permutations of the claimed subject matter are possible. Accordingly, the disclosed subject matter is intended to embrace all such alterations, modifications and variations that fall within the spirit and scope of the appended claims.

Claims

1. An optimized speckle matrix training method for correlated imaging, the method comprising:

dividing a training image into a plurality of image blocks;

carrying out similarity clustering on the image blocks;

2. The method of claim 1, wherein similarity clustering image patches further comprises:

3. The method of claim 1, wherein dictionary learning for each block cluster further comprises:

and aiming at different block clusters, selecting different dictionary learning methods according to the image structure characteristic characteristics of the block clusters to perform dictionary learning.

4. The method of claim 1, wherein computing an optimized speckle-like matrix for each block cluster based on the derived class dictionary further comprises:

5. A method of correlated imaging, the method comprising:

acquiring an initial image of a target to be imaged;

dividing the initial image into a plurality of image patches;

carrying out similarity clustering on the image blocks;

6. The method of claim 5, wherein correlating imaging with the bucket detector with the combined speckle matrix further comprises:

7. The method of claim 5, wherein similarity clustering image patches further comprises:

8. The method of claim 5, wherein determining a matching speckle-like matrix for each image patch based on clustering further comprises:

9. The method of claim 5, wherein combining the speckle matrix for each image patch into a complete speckle matrix further comprises:

and combining the speckle matrixes according to the position of each image block in the initial image to obtain the complete speckle matrix.

10. An associated imaging system, the system comprising:

a computing unit configured to:

acquiring an initial image of a target to be imaged;

dividing the initial image into a plurality of image patches;

carrying out similarity clustering on the image blocks;

and obtaining a reconstructed image of the target through correlation calculation based on the speckle matrix and the measured value of the barrel detector array.