CN110415201A - Single exposure super-resolution imaging method and device based on structure light and deep learning - Google Patents
Single exposure super-resolution imaging method and device based on structure light and deep learning Download PDFInfo
- Publication number
- CN110415201A CN110415201A CN201910697301.7A CN201910697301A CN110415201A CN 110415201 A CN110415201 A CN 110415201A CN 201910697301 A CN201910697301 A CN 201910697301A CN 110415201 A CN110415201 A CN 110415201A
- Authority
- CN
- China
- Prior art keywords
- model
- neural network
- predetermined depth
- training
- resolution image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10052—Images from lightfield camera
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Abstract
Single exposure super-resolution imaging method and device based on structure light and deep learning that the invention discloses a kind of, wherein generate composite construction optical illumination coding this method comprises: being overlapped to multiple and different traditional lightings coding；It acquires target scene to be processed and encodes the low-resolution image generated under single exposure in composite construction optical illumination；Low-resolution image is inputted default imaging model to handle, exports the corresponding super-resolution image of target scene to be processed.This method has the support of deep learning theory and structure light coding illumination theory, and implementation is simple, it is only necessary to which the quick reconstruction of super resolution image can be realized in single exposure.
Description
Technical field
The present invention relates to the super-resolution imaging technical fields calculated in camera shooting, in particular to a kind of to be based on structure light and depth
Single exposure super-resolution imaging method and device of degree study.
Background technique
The resolution ratio of optical microscopy imaging is limited by optical system diffraction characteristic, and in recent years, researcher is using now
There is imaging system to restore to pay a large amount of effort in terms of the image of higher resolution.2000, University of California San Francisco point
Doctor M.Gustafsson in school proposes Structured Illumination microscopy (Structured Illumination
Microscopy, SIM).SIM irradiates object by structure light coding lighting source (usually sinusoidal coding), in Fourier's sky
Between frequency domain, Structured Illumination encoded frequency spectrum and sample preimage frequency spectrum convolution, the optical delivery of micro-imaging process will be located at this
Information except function (Optical Transfer Function, OTF) is transferred within the scope of Observable, passes through special algorithm
The low-frequency information in high-frequency information and OTF outside reduction splicing OTF range, to go back original high resolution image.SIM imaging is differentiated
Rate reaches as high as twice of former Optical Resolution of Imaging System.2005, Gustafsson was on the basis of SIM it is further proposed that saturation
Structured Illumination microscopy (Saturated Structure Illumination Microscopy, SSIM), SSIM utilize base
Sample is irradiated in the nonlinear organization light of fluorescent molecule excitation state saturation effect, illumination light contains the higher hamonic wave point of spatial frequency
Amount, therefore the full resolution pricture for being higher than former more times of Optical Resolution of Imaging System can be extracted.
Specifically, SIM is raw by special algorithm by the low resolution coded image for acquiring the illumination of multiple structure light codings
At a full resolution pricture, traditional SIM Image Restoration Algorithm has direct method, Bayes' assessment and Gerchberg-Saxton
Algorithm (G-S algorithm) etc..
Since the image Fourier space frequency spectrum encoded through SIN function is considered as low resolution image frequency spectrum and two
The combination of the Sideband Spectrum, direct method irradiate sample by the sinusoidal coding structure light of multiple and different phases of coding mode of the same race,
The low resolution coded image of multiple groups is acquired, obtains its Fourier space frequency spectrum through two-dimensional Fourier transform, simultaneous equations solve OTF model
With the frequency spectrum outside OTF range in enclosing, and it is combined splicing and can secures satisfactory grades and distinguish image spectrum, then obtained through two-dimentional inverse Fourier transform
To high-definition picture.Direct method calculating process is simple, but needs to acquire multiple low resolution coded images to restore a Zhang Gaofen
Distinguish image, and poor robustness, the noise introduced to actual imaging process is more sensitive.
Bayes' assessment is based on Bayesian statistical model, it is known that the high-definition picture and low resolution code pattern of scene
The statistical law of picture, and by high-definition picture to the create-rule of low resolution coded image, according to Bayesian statistics mould
Type calculates the statistical model of the corresponding high-definition picture of low resolution coded image, can pass through the modes such as the Posterior Mean estimation technique
Restore high-definition picture.But Bayes' assessment is also required to acquire multiple low resolution coded images to restore a High-Resolution Map
Picture.
G-S algorithm using it is low resolution coded image and its Fourier space frequency spectrum establish target full resolution pricture airspace and
Frequency-domain constraint, iteration update target image and its Fourier space frequency spectrum, until the error arrival between iteration result is set twice
Determine threshold value.Compared to direct method and Bayes' assessment, low resolution coded image needed for G-S algorithm is less, and robustness is stronger, but by
In needing iteration to update, so real-time is poor.
In recent years, with the development of deep neural network, researcher starts to carry out the super of image with deep learning theory
Research is differentiated, Hong Kong Chinese University Tang Xiaoou professor seminar in 2016 carries out oversubscription to single image using convolutional neural networks
Distinguish recovery.Then, researcher proposes using deeper convolutional neural networks to improve resolution ratio.U.S. Twitter in 2017
Company proposes to utilize the super-resolution for generating confrontation network progress single image.It is however generally that true full resolution pricture institute
It is higher than individual low resolution image information contained amount containing information content, so only restoring high-resolution by a untreated low resolution image
Original image is ill-conditioning problem, and deep learning theory carries out super-resolution recovery based on the statistical law of image, and there is no at this
This is solved the problems, such as in matter, so being only not considered as true high-resolution by the full resolution pricture that deep neural network study restores
Image.
Summary of the invention
The present invention is directed to solve at least some of the technical problems in related technologies.
For this purpose, exposing super-resolution based on structure light and the single of deep learning an object of the present invention is to provide a kind of
Imaging method, this method have the support of deep learning theory and structure light coding illumination theory, and implementation is simple, it is only necessary to single
It is secondary to expose the quick reconstruction that super resolution image can be realized.
It is another object of the present invention to propose a kind of single exposure super-resolution based on structure light and deep learning at
As device.
In order to achieve the above objectives, one aspect of the present invention embodiment proposes a kind of single exposure based on structure light and deep learning
Light super-resolution imaging method, comprising:
S1 is overlapped multiple and different traditional lightings coding and generates composite construction optical illumination coding；
S2 acquires target scene to be processed and encodes the low resolution generated under single exposure in the composite construction optical illumination
Image；
The low-resolution image is inputted default imaging model and handled, exports the target scene to be processed by S3
Corresponding super-resolution image.
Single exposure super-resolution imaging method based on structure light and deep learning of the embodiment of the present invention, according to structure light
The Fourier space spectral characteristic of coded illumination image, analysis Structured Illumination in Fourier space frequency domain are encoded to image
It influences, to establish composite illuminating coding；Based on deep learning theory, building is suitable for what structure light coding illumination image restored
Deep neural network model；It is exposed by single composite construction pumped FIR laser, carries out super resolution image in conjunction with deep neural network
It quickly rebuilds, this method has the support of deep learning theory and structure light coding illumination theory, and implementation is simple, it is only necessary to single
It is secondary to expose the quick reconstruction that super resolution image can be realized.
In addition, single exposure super-resolution imaging side according to the above embodiment of the present invention based on structure light and deep learning
Method can also have following additional technical characteristic:
Further, in one embodiment of the invention, before step S2, further includes:
Acquisition Training scene encodes the low-resolution image training generated under single exposure in the composite construction optical illumination
Sample；
The low-resolution image training sample is trained in predetermined depth neural network model；
If the error amount between training result and the high-definition picture sample of the Training scene is less than preset threshold,
The default imaging model is set by the trained predetermined depth neural network model；
If the error amount between the training result and the high-definition picture sample of the Training scene is more than or equal to pre-
If threshold value, then the parameter of the predetermined depth neural network is updated according to error amount and the low-resolution image is trained into sample
Originally updated predetermined depth neural network model is input to be trained again.
Further, in one embodiment of the invention, the multiple different traditional lighting codings include: SIN function
Coding and/or Nonlinear fluorescence coding and/or random coded；
It is superimposed to obtain the composite construction optical illumination coding by the airspace for encoding the multiple traditional lighting, it is described multiple
The Fourier space frequency spectrum for closing Structured Illumination coding is the superposition of the multiple different traditional lighting coding frequency spectrums.
Further, in one embodiment of the invention, the predetermined depth neural network model is convolutional Neural net
The low-resolution image training sample of acquisition is inputted the predetermined depth neural network model and is trained by network model；
It is output high-definition picture by the output after the predetermined depth neural network model, high-resolution will be exported
The high-definition picture sample of image and the Training scene compares to obtain error amount；
If error amount is less than preset threshold, set described default for the trained predetermined depth neural network model
Imaging model；
If error amount is more than or equal to preset threshold, the parameter of the predetermined depth neural network is updated according to error amount
And the low-resolution image training sample is input to the updated predetermined depth neural network model and is trained again.
Further, in one embodiment of the invention, the predetermined depth neural network model is the generation pair
Anti- network model；
Generation confrontation network model includes generating model and discrimination model, and the generation model is by the low resolution
Image training sample as input, export for export high-definition picture, the generations model will export high-definition picture and
The high-definition picture sample of the Training scene compares generation and generates model error value；
The discrimination model carries out the high-definition picture sample for exporting high-definition picture and the Training scene pair
Than generating discrimination model error amount；
The generation model error value and the generation model error value weighting summation are generated into training pattern error amount；
If the training pattern error amount is less than preset threshold, the trained predetermined depth neural network model is set
It is set to the default imaging model；
If the training pattern error amount is more than or equal to preset threshold, according to training pattern error amount update
The low-resolution image training sample is simultaneously input to the updated predetermined depth by the parameter of predetermined depth neural network
Neural network model is trained again.
Further, in one embodiment of the invention, pass through low-resolution image described in collection optical system and institute
State low-resolution image training sample.
In order to achieve the above objectives, another aspect of the present invention embodiment proposes a kind of list based on structure light and deep learning
Expose super-resolution imaging device, comprising:
Constructing module generates composite construction optical illumination coding for being overlapped to multiple and different traditional lightings coding；
First acquisition module encodes under single exposure for acquiring target scene to be processed in the composite construction optical illumination
The low-resolution image of generation；
Output module is handled for the low-resolution image to be inputted default imaging model, and output is described wait locate
Manage the corresponding super-resolution image of high target scene.
Single exposure super-resolution imaging device based on structure light and deep learning of the embodiment of the present invention, according to structure light
The Fourier space spectral characteristic of coded illumination image, analysis Structured Illumination in Fourier space frequency domain are encoded to image
It influences, to establish composite illuminating coding；Based on deep learning theory, building is suitable for what structure light coding illumination image restored
Deep neural network model；It is exposed by single composite construction pumped FIR laser, carries out super resolution image in conjunction with deep neural network
It quickly rebuilds, this method has the support of deep learning theory and structure light coding illumination theory, and implementation is simple, it is only necessary to single
It is secondary to expose the quick reconstruction that super resolution image can be realized.
In addition, single exposure super-resolution imaging dress according to the above embodiment of the present invention based on structure light and deep learning
Following additional technical characteristic can also be had by setting:
Further, in one embodiment of the invention, further includes: the second acquisition module, training module, generation module
And iteration module；
Second acquisition module encodes single exposure in the composite construction optical illumination for acquiring Training scene sample
The low-resolution image training sample of lower generation；
The training module, for by the low-resolution image training sample in predetermined depth neural network model into
Row training；
The generation module, if for the error between training result and the high-definition picture sample of the Training scene
Value is less than preset threshold, then sets the default imaging model for the trained predetermined depth neural network model；
The iteration module, if between the training result and the high-definition picture sample of the Training scene
Error amount is more than or equal to preset threshold, then the parameter of the predetermined depth neural network is updated according to error amount and will be described low
Image in different resolution training sample is input to updated predetermined depth neural network model and is trained again.
Further, in one embodiment of the invention, the predetermined depth neural network model is convolutional Neural net
The low-resolution image training sample of acquisition is inputted the predetermined depth neural network model and is trained by network model；
It is output high-definition picture by the output after the predetermined depth neural network model, high-resolution will be exported
The high-definition picture sample of image and the Training scene compares to obtain error amount；
If error amount is less than preset threshold, set described default for the trained predetermined depth neural network model
Imaging model；
If error amount is more than or equal to preset threshold, the parameter of the predetermined depth neural network is updated according to error amount
And the low-resolution image training sample is input to the updated predetermined depth neural network model and is trained again.
Further, in one embodiment of the invention, the predetermined depth neural network model is the generation pair
Anti- network model；
Generation confrontation network model includes generating model and discrimination model, and the generation model is by the low resolution
Image training sample as input, export for export high-definition picture, the generations model will export high-definition picture and
The high-definition picture sample of the Training scene compares generation and generates model error value；
The discrimination model carries out the high-definition picture sample for exporting high-definition picture and the Training scene pair
Than generating discrimination model error amount；
The generation model error value and the generation model error value weighting summation are generated into training pattern error amount；
If the training pattern error amount is less than preset threshold, the trained predetermined depth neural network model is set
It is set to the default imaging model；
If the training pattern error amount is more than or equal to preset threshold, according to training pattern error amount update
The low-resolution image training sample is simultaneously input to the updated predetermined depth by the parameter of predetermined depth neural network
Neural network model is trained again.
The additional aspect of the present invention and advantage will be set forth in part in the description, and will partially become from the following description
Obviously, or practice through the invention is recognized.
Detailed description of the invention
Above-mentioned and/or additional aspect and advantage of the invention will become from the following description of the accompanying drawings of embodiments
Obviously and it is readily appreciated that, in which:
Fig. 1 is single exposure super-resolution imaging side based on structure light and deep learning according to one embodiment of the invention
Method flow chart；
Fig. 2 is the generation composite construction optical illumination coding schematic diagram according to one embodiment of the invention；
Fig. 3 is the low resolution coded image schematic diagram according to one embodiment of the invention；
Fig. 4 is the convolutional neural networks model training schematic diagram according to one embodiment of the invention；
Fig. 5 is that network model training schematic diagram is fought according to the generation of one embodiment of the invention；
Fig. 6 is building and the Irnaging procedures figure of the neural network according to one embodiment of the invention；
Fig. 7 is the Structured Illumination imaging optical path figure according to one embodiment of the invention；
Fig. 8 is single exposure super-resolution imaging dress based on structure light and deep learning according to one embodiment of the invention
Set structural schematic diagram.
Specific embodiment
The embodiment of the present invention is described below in detail, examples of the embodiments are shown in the accompanying drawings, wherein from beginning to end
Same or similar label indicates same or similar element or element with the same or similar functions.Below with reference to attached
The embodiment of figure description is exemplary, it is intended to is used to explain the present invention, and is not considered as limiting the invention.
The single exposure based on structure light and deep learning for describing to propose according to embodiments of the present invention with reference to the accompanying drawings is super
Resolution imaging method and device.
The single exposure based on structure light and deep learning proposed according to embodiments of the present invention is described with reference to the accompanying drawings first
Super-resolution imaging method.
Fig. 1 is single exposure super-resolution imaging side based on structure light and deep learning according to one embodiment of the invention
Method flow chart.
As shown in Figure 1, should single exposure super-resolution imaging method based on structure light and deep learning the following steps are included:
Step S1 is overlapped multiple and different traditional lightings coding and generates composite construction optical illumination coding.
Specifically, the spectral characteristic of the influence according to Structured Illumination to imaging process and Structured Illumination coding, to more
A different traditional lighting coding, which is overlapped, generates composite construction optical illumination coding.
It should be noted that the composite construction optical illumination coding that step S1 is generated generates default imaging model with following training
The composite construction optical illumination coding of Shi Caiyong is identical.
Further, multiple and different traditional lightings coding includes but is not limited to: SIN function coding and/or non-linear glimmering
Pumped FIR laser and/or random coded.
It is superimposed to obtain composite construction optical illumination coding, composite construction optical illumination by the airspace for encoding multiple traditional lightings
The Fourier space frequency spectrum of coding is the superposition that multiple and different traditional lightings encode frequency spectrum.
It is understood that multiple and different traditional lightings is encoded superposition, so that containing in the image of acquisition
The Fourier space spectrum information lost under traditional lighting.
As shown in Fig. 2, Structured Illumination imaging after generating coded illumination structure light irradiation sample by being imaged, it is traditional
Structure light coding mode includes but is not limited to SIN function coding, Nonlinear fluorescence coding and random coded, following sinusoidal coding
For, the airspace expression formula of the sinusoidal illumination coding of two dimension are as follows:
Wherein,For wave vector, φ1For phase.
Transformed to Fourier space frequency domain:
If composite illuminating is encoded to the superposition of three kinds of conventional sinusoidals coding:
The Fourier space frequency spectrum of composite illuminating coding are as follows:
If the Fourier space frequency spectrum of sample patterns isThe Fourier space being then imaged through composite construction optical illumination
Frequency spectrum are as follows:
Due toHigh-frequency information comprising original image sample except system OTF range, so can basisIt is extensive
Multiple super-resolution image.
As shown in figure 3, obtaining low resolution using composite structure light coded illumination Training scene through collection optical system and compiling
Code image.The training dataset of neural network can be generated using the full resolution pricture sample of a collection of Training scene according to this step.
Step S2 acquires target scene to be processed and encodes the low resolution generated under single exposure in composite construction optical illumination
Image.
The composite coding structure light single exposure that processing target scene utilizes step S1 to generate is treated, is adopted by optical system
Collect corresponding low-resolution image.
Further, before step S2 further include:
Acquisition Training scene encodes the low-resolution image training sample generated under single exposure in composite construction optical illumination
This；
Low-resolution image training sample is trained in predetermined depth neural network model；
It, will instruction if the error amount between training result and the high-definition picture sample of Training scene is less than preset threshold
Experienced predetermined depth neural network model is set as default imaging model；
If the error amount between training result and the high-definition picture sample of Training scene is more than or equal to preset threshold,
The parameter of predetermined depth neural network is updated according to error amount and is input to low-resolution image training sample updated
Predetermined depth neural network model is trained again.
Specifically, wherein target scene and Training scene can be true object or sample image.To target scene
Or Training scene is directly acquired to obtain the high-definition picture sample of target scene or Training scene.
It is understood that after obtaining the error amount between training result and the high-definition picture sample of Training scene,
By the error back propagation using network output and high-definition picture, iteration updates model parameter.It is set when error reaches
Its convergence is thought when threshold value.
Predetermined depth neural network model is including but not limited to convolutional neural networks, generation confrontation network and based on the above mind
Mutation or progressions model through network, predetermined depth neural network single exposure under composite construction optical illumination coding acquire low
Resolution image sample is compared as network inputs by the high-definition picture sample with Training scene, and iteration updates net
Network model parameter to network output and the error of high-definition picture reaches given threshold.
Convolutional neural networks each convolution kernel in network hidden layer carries out convolution algorithm to preceding layer, and convolution kernel output is made
It inputs for next stage to carry out corresponding calculation processing.
Further, predetermined depth neural network model is convolutional neural networks model, by the low-resolution image of acquisition
Training sample input predetermined depth neural network model is trained；
It is output high-definition picture by the output after predetermined depth neural network model, high-definition picture will be exported
It compares to obtain error amount with the high-definition picture sample of Training scene；
If error amount is less than preset threshold, default imaging mould is set by trained predetermined depth neural network model
Type；
If more than or equal to preset threshold, the parameter of predetermined depth neural network is updated according to error amount and will for error amount
Low-resolution image training sample is input to updated predetermined depth neural network model and is trained again.
Further, predetermined depth neural network model makes a living into confrontation network model；
Generating confrontation network model includes generating model and discrimination model, generates model for low-resolution image training sample
It as input, exports to export high-definition picture, generates the high-resolution that model will export high-definition picture and Training scene
Rate image pattern compares generation and generates model error value；
The high-definition picture sample for exporting high-definition picture and Training scene is compared generation and sentenced by discrimination model
Other model error value；
Model error value will be generated and generate model error value weighting summation and generate training pattern error amount；
If training pattern error amount is less than preset threshold, set default for trained predetermined depth neural network model
Imaging model；
If training pattern error amount is more than or equal to preset threshold, predetermined depth nerve is updated according to training pattern error amount
Low-resolution image training sample is simultaneously input to updated predetermined depth neural network model and instructed again by the parameter of network
Practice.
It is understood that the error amount of discrimination model can be only considered as a kind of implementation, for example, differentiating mould
Type is compared the high-definition picture sample of output high-definition picture and Training scene, judges the similarity of the two, In
When exporting high-definition picture and the similar or the same high-definition picture sample of Training scene, then by trained predetermined depth mind
Default imaging model is set as through network model.
As the mode of alternatively possible realization, in order to improve model imaging resolution ratio, comprehensively consider generation confrontation
Model and the respective error amount of discrimination model are generated in network model, and low-resolution image training sample is inputted and generates model
In, output high-definition picture is obtained, the high-definition picture sample for exporting high-definition picture and Training scene is carried out pair
It is by discrimination model that the high resolution graphics for exporting high-definition picture and Training scene is decent than obtaining generating model error value
Originally it compares and generates discrimination model error amount, comprehensively considered according to weight and generate model error value and discrimination model error amount,
To generate model error value and discrimination model error amount respectively with carry out being added summation again after corresponding multiplied by weight, obtain
One training pattern error amount compares this training pattern error amount with preset threshold.
Specifically, instruction low-resolution image training sample being trained in predetermined depth neural network model
Practicing result is output high-definition picture.Network in predetermined depth neural network model is different, and specific training process is not
Together.Eventually by the update iteration to predetermined depth neural network model, predetermined depth neural network model is made not meet convergence
Condition sets it to default imaging model.
Low-resolution image is inputted default imaging model and handled, it is corresponding to export target scene to be processed by step S3
Super-resolution image.
After obtaining default imaging model by training predetermined depth neural network, low-resolution image is inputted default
Imaging model is handled, and the corresponding super-resolution image of target scene to be processed is exported.
As shown in Figure 4 and Figure 5, the deep neural network that can carry out super resolution image reconstruction includes but is not limited to convolutional Neural
Network and generation confrontation network.The core of convolutional neural networks is convolutional calculation, and convolution kernel is generally two-dimensional matrix, each convolution
It checks preceding layer output characteristic pattern sliding to be multiplied, multiplied result is added, and convolution results can be through nonlinear activation unit (including but not
It is limited to line rectification unit, hyperbolic tangent function, Sigmoid function etc.) operation input next stage hidden layer or output layer.It generates
Fighting network includes to generate model and discrimination model, generates model for low resolution coded image as input and generates High-Resolution Map
Picture, discrimination model export true according to the difference judgement generation model generated between model output and true high-definition picture
Puppet, generates model and discrimination model alternating iteration updates, until discrimination model can not identify true high-definition picture and generate
Model exports image.
As shown in fig. 6, generating the training dataset of deep neural network according to composite construction pumped FIR laser, convolutional Neural is built
Network generates confrontation network even depth neural network model, using above-mentioned training dataset training neural network parameter, obtains
Network model.After forming network, the low resolution coded image of composite structure light single exposure can be inputted depth nerve net
Network, network output are high-definition picture.
The method of the embodiment of the present invention is using deep learning model as realization rate, while according to quilt under Structured Illumination
The Fourier space spectral characteristic for acquiring image establishes composite construction pumped FIR laser, realizes Structured Illumination super-resolution image
It quickly rebuilds, and builds system and carry out Proof-Of Principle.As shown in fig. 7, composite coding light structures light is produced by spatial light modulator
It is raw, it is incident upon through spectroscope and is observed sample plane, sample reflection light is received by sensor and acquired.
The single exposure super-resolution imaging method based on structure light and deep learning proposed according to embodiments of the present invention, root
According to the Fourier space spectral characteristic of structure light coding illumination image, analysis Structured Illumination in Fourier space frequency domain is encoded
Influence to image, to establish composite illuminating coding；Based on deep learning theory, building is suitable for structure light coding illumination figure
As the deep neural network model restored；It is exposed by single composite construction pumped FIR laser, carries out oversubscription in conjunction with deep neural network
Distinguish the quick reconstruction of image, this method has the support of deep learning theory and structure light coding illumination theory, implementation letter
It is single, it is only necessary to which that the quick reconstruction of super resolution image can be realized in single exposure.
The single exposure based on structure light and deep learning proposed according to embodiments of the present invention referring next to attached drawing description is super
Resolution imaging devices.
Fig. 8 is single exposure super-resolution imaging dress based on structure light and deep learning according to one embodiment of the invention
Set structural schematic diagram.
As shown in figure 8, should include: constructing module based on single exposure super-resolution imaging device of structure light and deep learning
100, the first acquisition module 200 and output module 300.
Constructing module 100 generates composite construction optical illumination coding for being overlapped to multiple and different traditional lightings coding.
First acquisition module 200 encodes under single exposure for acquiring target scene to be processed in composite construction optical illumination
The low-resolution image of generation.
Output module 300 handles for low-resolution image to be inputted default imaging model, exports target to be processed
The corresponding super-resolution image of scene.
Further, in one embodiment of the invention, further includes: the second acquisition module, training module, generation module
And iteration module；
Second acquisition module, low point generated for acquiring Training scene in the case where composite construction optical illumination encodes single exposure
Resolution image training sample；
Training module, for low-resolution image training sample to be trained in predetermined depth neural network model；
Generation module, if being less than for the error amount between training result and the high-definition picture sample of Training scene pre-
If threshold value, then default imaging model is set by trained predetermined depth neural network model；
Iteration module, if be greater than for the error amount between training result and the high-definition picture sample of Training scene etc.
In preset threshold, then the parameter of predetermined depth neural network and low-resolution image training sample is defeated is updated according to error amount
Enter to updated predetermined depth neural network model and is trained again.
Further, in one embodiment of the invention, predetermined depth neural network model is convolutional neural networks mould
The low-resolution image training sample input predetermined depth neural network model of acquisition is trained by type；
It is output high-definition picture by the output after predetermined depth neural network model, high-definition picture will be exported
It compares to obtain error amount with the high-definition picture sample of Training scene；
If error amount is less than preset threshold, default imaging mould is set by trained predetermined depth neural network model
Type；
If more than or equal to preset threshold, the parameter of predetermined depth neural network is updated according to error amount and will for error amount
Low-resolution image training sample is input to updated predetermined depth neural network model and is trained again.
Further, in one embodiment of the invention, predetermined depth neural network model makes a living into confrontation network mould
Type；
Generating confrontation network model includes generating model and discrimination model, generates model for low-resolution image training sample
It as input, exports to export high-definition picture, generates the high-resolution that model will export high-definition picture and Training scene
Rate image pattern compares generation and generates model error value；
The high-definition picture sample for exporting high-definition picture and Training scene is compared generation and sentenced by discrimination model
Other model error value；
Model error value will be generated and generate model error value weighting summation and generate training pattern error amount；
If training pattern error amount is less than preset threshold, set default for trained predetermined depth neural network model
Imaging model；
If training pattern error amount is more than or equal to preset threshold, predetermined depth nerve is updated according to training pattern error amount
Low-resolution image training sample is simultaneously input to updated predetermined depth neural network model and instructed again by the parameter of network
Practice.
It should be noted that aforementioned to single exposure super-resolution imaging embodiment of the method based on structure light and deep learning
Explanation be also applied for the device of the embodiment, details are not described herein again.
The single exposure super-resolution imaging based on structure light and deep learning proposed according to embodiments of the present invention is up to root
According to the Fourier space spectral characteristic of structure light coding illumination image, analysis Structured Illumination in Fourier space frequency domain is encoded
Influence to image, to establish composite illuminating coding；Based on deep learning theory, building is suitable for structure light coding illumination figure
As the deep neural network model restored；It is exposed by single composite construction pumped FIR laser, carries out oversubscription in conjunction with deep neural network
Distinguish the quick reconstruction of image, this method has the support of deep learning theory and structure light coding illumination theory, implementation letter
It is single, it is only necessary to which that the quick reconstruction of super resolution image can be realized in single exposure.
In addition, term " first ", " second " are used for descriptive purposes only and cannot be understood as indicating or suggesting relative importance
Or implicitly indicate the quantity of indicated technical characteristic.Define " first " as a result, the feature of " second " can be expressed or
Implicitly include at least one this feature.In the description of the present invention, the meaning of " plurality " is at least two, such as two, three
It is a etc., unless otherwise specifically defined.
In the description of this specification, reference term " one embodiment ", " some embodiments ", " example ", " specifically show
The description of example " or " some examples " etc. means specific features, structure, material or spy described in conjunction with this embodiment or example
Point is included at least one embodiment or example of the invention.In the present specification, schematic expression of the above terms are not
It must be directed to identical embodiment or example.Moreover, particular features, structures, materials, or characteristics described can be in office
It can be combined in any suitable manner in one or more embodiment or examples.In addition, without conflicting with each other, the skill of this field
Art personnel can tie the feature of different embodiments or examples described in this specification and different embodiments or examples
It closes and combines.
Although the embodiments of the present invention has been shown and described above, it is to be understood that above-described embodiment is example
Property, it is not considered as limiting the invention, those skilled in the art within the scope of the invention can be to above-mentioned
Embodiment is changed, modifies, replacement and variant.
Claims (10)
1. a kind of single exposure super-resolution imaging method based on structure light and deep learning, which is characterized in that including following step
It is rapid:
S1 is overlapped multiple and different traditional lightings coding and generates composite construction optical illumination coding；
S2 acquires target scene to be processed and encodes the low resolution figure generated under single exposure in the composite construction optical illumination
Picture；
The low-resolution image is inputted default imaging model and handled, it is corresponding to export the target scene to be processed by S3
Super-resolution image.
2. the method according to claim 1, wherein before step S2, further includes:
Acquisition Training scene encodes the low-resolution image training sample generated under single exposure in the composite construction optical illumination；
The low-resolution image training sample is trained in predetermined depth neural network model；
It, will instruction if the error amount between training result and the high-definition picture sample of the Training scene is less than preset threshold
The experienced predetermined depth neural network model is set as the default imaging model；
If the error amount between the training result and the high-definition picture sample of the Training scene is more than or equal to default threshold
Value then updates the parameter of the predetermined depth neural network and low-resolution image training sample is defeated according to error amount
Enter to updated predetermined depth neural network model and is trained again.
3. the method according to claim 1, wherein
The multiple difference traditional lighting coding includes: SIN function coding and/or Nonlinear fluorescence coding, and/or is compiled at random
Code；
It is superimposed to obtain the composite construction optical illumination coding, the composite junction by the airspace for encoding the multiple traditional lighting
The Fourier space frequency spectrum of structure optical illumination coding is the superposition that the multiple different traditional lightings encode frequency spectrum.
4. according to the method described in claim 2, it is characterized in that,
The predetermined depth neural network model is convolutional neural networks model, by the low-resolution image training sample of acquisition
This input predetermined depth neural network model is trained；
It is output high-definition picture by the output after the predetermined depth neural network model, high-definition picture will be exported
It compares to obtain error amount with the high-definition picture sample of the Training scene；
If error amount is less than preset threshold, the default imaging is set by the trained predetermined depth neural network model
Model；
If more than or equal to preset threshold, the parameter of the predetermined depth neural network is updated according to error amount and will for error amount
The low-resolution image training sample is input to the updated predetermined depth neural network model and is trained again.
5. according to the method described in claim 2, it is characterized in that,
The predetermined depth neural network model is that the generation fights network model；
Generation confrontation network model includes generating model and discrimination model, and the generation model is by the low-resolution image
Training sample as input, export for export high-definition picture, the generations model will output high-definition picture with it is described
The high-definition picture sample of Training scene compares generation and generates model error value；
The high-definition picture sample for exporting high-definition picture and the Training scene is compared life by the discrimination model
At discrimination model error amount；
The generation model error value and the generation model error value weighting summation are generated into training pattern error amount；
If the training pattern error amount is less than preset threshold, set the trained predetermined depth neural network model to
The default imaging model；
If the training pattern error amount is more than or equal to preset threshold, updated according to the training pattern error amount described default
The low-resolution image training sample is simultaneously input to the updated predetermined depth nerve by the parameter of deep neural network
Network model is trained again.
6. the method according to claim 1, wherein
Pass through low-resolution image described in collection optical system and the low-resolution image training sample.
7. a kind of single exposure super-resolution imaging device based on structure light and deep learning characterized by comprising
Constructing module generates composite construction optical illumination coding for being overlapped to multiple and different traditional lightings coding；
First acquisition module is generated for acquiring target scene to be processed in the case where the composite construction optical illumination encodes single exposure
Low-resolution image；
Output module handles for the low-resolution image to be inputted default imaging model, exports the mesh to be processed
Mark the corresponding super-resolution image of scene.
8. device according to claim 7, which is characterized in that further include: the second acquisition module, generates mould at training module
Block and iteration module；
Second acquisition module is generated for acquiring Training scene in the case where the composite construction optical illumination encodes single exposure
Low-resolution image training sample；
The training module, for instructing the low-resolution image training sample in predetermined depth neural network model
Practice；
The generation module, if the error amount between training result and the high-definition picture sample of the Training scene is small
In preset threshold, then the default imaging model is set by the trained predetermined depth neural network model；
The iteration module, if for the error between the training result and the high-definition picture sample of the Training scene
Value is more than or equal to preset threshold, then the parameter of the predetermined depth neural network is updated according to error amount and by the low resolution
Rate image training sample is input to updated predetermined depth neural network model and is trained again.
9. device according to claim 8, which is characterized in that
The predetermined depth neural network model is convolutional neural networks model, by the low-resolution image training sample of acquisition
This input predetermined depth neural network model is trained；
It is output high-definition picture by the output after the predetermined depth neural network model, high-definition picture will be exported
It compares to obtain error amount with the high-definition picture sample of the Training scene；
If error amount is less than preset threshold, the default imaging is set by the trained predetermined depth neural network model
Model；
If more than or equal to preset threshold, the parameter of the predetermined depth neural network is updated according to error amount and will for error amount
The low-resolution image training sample is input to the updated predetermined depth neural network model and is trained again.
10. device according to claim 8, which is characterized in that
The predetermined depth neural network model is that the generation fights network model；
Generation confrontation network model includes generating model and discrimination model, and the generation model is by the low-resolution image
Training sample as input, export for export high-definition picture, the generations model will output high-definition picture with it is described
The high-definition picture sample of Training scene compares generation and generates model error value；
The high-definition picture sample for exporting high-definition picture and the Training scene is compared life by the discrimination model
At discrimination model error amount；
The generation model error value and the generation model error value weighting summation are generated into training pattern error amount；
If the training pattern error amount is less than preset threshold, set the trained predetermined depth neural network model to
The default imaging model；
If the training pattern error amount is more than or equal to preset threshold, updated according to the training pattern error amount described default
The low-resolution image training sample is simultaneously input to the updated predetermined depth nerve by the parameter of deep neural network
Network model is trained again.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910697301.7A CN110415201B (en) | 2019-07-30 | 2019-07-30 | Single-exposure super-resolution imaging method and device based on structured light and deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910697301.7A CN110415201B (en) | 2019-07-30 | 2019-07-30 | Single-exposure super-resolution imaging method and device based on structured light and deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110415201A true CN110415201A (en) | 2019-11-05 |
| CN110415201B CN110415201B (en) | 2022-03-22 |
Family
ID=68364263
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910697301.7A Active CN110415201B (en) | 2019-07-30 | 2019-07-30 | Single-exposure super-resolution imaging method and device based on structured light and deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110415201B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110796603A (en) * | 2019-11-25 | 2020-02-14 | 曲阜师范大学 | High-resolution photoacoustic imaging method for deep tumor neovascularization |
| CN111292240A (en) * | 2020-01-23 | 2020-06-16 | 上海交通大学 | Magnetic resonance super-resolution imaging method based on imaging model and machine learning |
| CN112233034A (en) * | 2020-10-20 | 2021-01-15 | 常州机电职业技术学院 | Deep learning phase-shift interference fringe pattern blind denoising method for interferometry |
| CN113947565A (en) * | 2021-09-03 | 2022-01-18 | 中国科学院西安光学精密机械研究所 | Structured light illumination super-resolution imaging gene detection method based on deep learning |
| CN115144075A (en) * | 2022-06-30 | 2022-10-04 | 北京理工大学 | High-speed spectral imaging method and device |
| CN116757926A (en) * | 2023-05-22 | 2023-09-15 | 华南师范大学 | Super-resolution SIM-FRET imaging method and system based on self-supervision learning image denoising |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104197861A (en) * | 2014-08-25 | 2014-12-10 | 深圳大学 | Three-dimensional digital imaging method based on structured light gray level vector |
| US20160182889A1 (en) * | 2014-12-19 | 2016-06-23 | Datalogic ADC, Inc. | Depth camera system using coded structured light |
| CN107967697A (en) * | 2017-11-23 | 2018-04-27 | 四川大学 | Method for three-dimensional measurement and system based on colored random binary coding structured illumination |
| CN108734660A (en) * | 2018-05-25 | 2018-11-02 | 上海通途半导体科技有限公司 | A kind of image super-resolution rebuilding method and device based on deep learning |
| US20180347967A1 (en) * | 2017-06-01 | 2018-12-06 | RGBDsense Information Technology Ltd. | Method and apparatus for generating a random coding pattern for coding structured light |
| CN109685716A (en) * | 2018-12-14 | 2019-04-26 | 大连海事大学 | A kind of image super-resolution rebuilding method of the generation confrontation network based on Gauss encoder feedback |
-
2019
- 2019-07-30 CN CN201910697301.7A patent/CN110415201B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104197861A (en) * | 2014-08-25 | 2014-12-10 | 深圳大学 | Three-dimensional digital imaging method based on structured light gray level vector |
| US20160182889A1 (en) * | 2014-12-19 | 2016-06-23 | Datalogic ADC, Inc. | Depth camera system using coded structured light |
| US20180347967A1 (en) * | 2017-06-01 | 2018-12-06 | RGBDsense Information Technology Ltd. | Method and apparatus for generating a random coding pattern for coding structured light |
| CN107967697A (en) * | 2017-11-23 | 2018-04-27 | 四川大学 | Method for three-dimensional measurement and system based on colored random binary coding structured illumination |
| CN108734660A (en) * | 2018-05-25 | 2018-11-02 | 上海通途半导体科技有限公司 | A kind of image super-resolution rebuilding method and device based on deep learning |
| CN109685716A (en) * | 2018-12-14 | 2019-04-26 | 大连海事大学 | A kind of image super-resolution rebuilding method of the generation confrontation network based on Gauss encoder feedback |
Non-Patent Citations (1)
| Title |
|---|
| 刘阳阳: "基于深度神经网络的RGB-D图像识别算法研究", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110796603A (en) * | 2019-11-25 | 2020-02-14 | 曲阜师范大学 | High-resolution photoacoustic imaging method for deep tumor neovascularization |
| CN111292240A (en) * | 2020-01-23 | 2020-06-16 | 上海交通大学 | Magnetic resonance super-resolution imaging method based on imaging model and machine learning |
| CN111292240B (en) * | 2020-01-23 | 2022-01-07 | 上海交通大学 | Magnetic resonance super-resolution imaging method based on imaging model and machine learning |
| CN112233034A (en) * | 2020-10-20 | 2021-01-15 | 常州机电职业技术学院 | Deep learning phase-shift interference fringe pattern blind denoising method for interferometry |
| CN112233034B (en) * | 2020-10-20 | 2022-05-13 | 常州机电职业技术学院 | Deep learning phase-shift interference fringe pattern blind denoising method for interferometry |
| CN113947565A (en) * | 2021-09-03 | 2022-01-18 | 中国科学院西安光学精密机械研究所 | Structured light illumination super-resolution imaging gene detection method based on deep learning |
| CN113947565B (en) * | 2021-09-03 | 2023-04-18 | 中国科学院西安光学精密机械研究所 | Structured light illumination super-resolution imaging gene detection method based on deep learning |
| CN115144075A (en) * | 2022-06-30 | 2022-10-04 | 北京理工大学 | High-speed spectral imaging method and device |
| CN115144075B (en) * | 2022-06-30 | 2023-08-29 | 北京理工大学 | High-speed spectrum imaging method and device |
| CN116757926A (en) * | 2023-05-22 | 2023-09-15 | 华南师范大学 | Super-resolution SIM-FRET imaging method and system based on self-supervision learning image denoising |
| CN116757926B (en) * | 2023-05-22 | 2024-04-05 | 华南师范大学 | Super-resolution SIM-FRET imaging method and system based on self-supervision learning image denoising |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110415201B (en) | 2022-03-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110415201A (en) | Single exposure super-resolution imaging method and device based on structure light and deep learning | |
| CN107154023A (en) | Face super-resolution reconstruction method based on generation confrontation network and sub-pix convolution | |
| CN106897986B (en) | A kind of visible images based on multiscale analysis and far infrared image interfusion method | |
| EP3832359A1 (en) | Method and device for imaging of lensless hyperspectral image | |
| CN104835130A (en) | Multi-exposure image fusion method | |
| CN102063712B (en) | Multi-exposure image fusion method based on sub-band structure | |
| CN103247036A (en) | Multiple-exposure image fusion method and device | |
| Bhandari et al. | A new beta differential evolution algorithm for edge preserved colored satellite image enhancement | |
| Monnier et al. | The 2014 interferometric imaging beauty contest | |
| Ignatov et al. | Learned smartphone ISP on mobile GPUs with deep learning, mobile AI & AIM 2022 challenge: report | |
| CN114638767A (en) | Laparoscope image smoke removal method based on generation of countermeasure network | |
| CN116563693A (en) | Underwater image color restoration method based on lightweight attention mechanism | |
| CN112163998A (en) | Single-image super-resolution analysis method matched with natural degradation conditions | |
| Chen et al. | GCAM: lightweight image inpainting via group convolution and attention mechanism | |
| Guo et al. | Haze removal for single image: A comprehensive review | |
| Li et al. | Effective data-driven technology for efficient vision-based outdoor industrial systems | |
| CN105491371A (en) | Tone mapping image quality evaluation method based on gradient magnitude similarity | |
| Hassan et al. | Image quality measurement-based comparative analysis of illumination compensation methods for face image normalization | |
| Anand et al. | Architectural distortion detection in mammogram using contourlet transform and texture features | |
| CN116433518A (en) | Fire image smoke removing method based on improved Cycle-Dehaze neural network | |
| CN115311430A (en) | Training method and system of human body reconstruction model and computer equipment | |
| CN110352387A (en) | For the system and method by more depth sparse phase restoration and reconstruction holographic lensless images | |
| CN114266713A (en) | NonshadowGAN-based unmanned aerial vehicle railway fastener image shadow removing method and system | |
| CN114648604A (en) | Image rendering method, electronic device, storage medium and program product | |
| Xue et al. | Saraniqa: Self-attention restorative adversarial network for no-reference image quality assessment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |