CN110246084A - A kind of super-resolution image reconstruction method and its system, device, storage medium - Google Patents
A kind of super-resolution image reconstruction method and its system, device, storage medium Download PDFInfo
- Publication number
- CN110246084A CN110246084A CN201910409645.3A CN201910409645A CN110246084A CN 110246084 A CN110246084 A CN 110246084A CN 201910409645 A CN201910409645 A CN 201910409645A CN 110246084 A CN110246084 A CN 110246084A
- Authority
- CN
- China
- Prior art keywords
- image
- resolution
- super
- resolution image
- neural networks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000003860 storage Methods 0.000 title claims abstract description 22
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 25
- 238000004364 calculation method Methods 0.000 claims abstract description 14
- 230000015654 memory Effects 0.000 claims description 20
- 230000006870 function Effects 0.000 claims description 10
- 238000004458 analytical method Methods 0.000 claims description 9
- 238000013528 artificial neural network Methods 0.000 claims description 9
- 238000005070 sampling Methods 0.000 claims description 7
- 238000010586 diagram Methods 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 claims description 6
- 238000010191 image analysis Methods 0.000 claims description 5
- 238000013507 mapping Methods 0.000 claims description 5
- 238000004891 communication Methods 0.000 claims description 4
- 238000013075 data extraction Methods 0.000 claims 1
- 238000005096 rolling process Methods 0.000 claims 1
- 238000004422 calculation algorithm Methods 0.000 abstract description 6
- 239000000463 material Substances 0.000 abstract description 5
- 239000004615 ingredient Substances 0.000 abstract description 2
- 238000007781 pre-processing Methods 0.000 abstract description 2
- 239000010410 layer Substances 0.000 description 28
- 230000008569 process Effects 0.000 description 7
- 238000012545 processing Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 5
- 230000002708 enhancing effect Effects 0.000 description 5
- 230000001052 transient effect Effects 0.000 description 5
- 230000004913 activation Effects 0.000 description 4
- 230000009467 reduction Effects 0.000 description 4
- 238000003384 imaging method Methods 0.000 description 3
- 239000011159 matrix material Substances 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 238000009966 trimming Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000001537 neural effect Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000001815 facial effect Effects 0.000 description 1
- 238000009472 formulation Methods 0.000 description 1
- 230000005764 inhibitory process Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 239000011229 interlayer Substances 0.000 description 1
- 238000003475 lamination Methods 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 210000005036 nerve Anatomy 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000002834 transmittance Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a kind of super-resolution image reconstruction method and its system, device, storage mediums, firstly, input original low-resolution image carries out bicubic interpolation to input low-resolution image by image preprocessing；Then, pretreated image is by the sparse depth convolutional neural networks that constrain containing pond manifold, is connected with each other between image data and arrangement, learns the detail of the high frequency ingredient lacked into image, realizes the reconstruct of high-resolution clear image.The present invention can effectively reduce network parameter calculation amount while retain image material particular information.Original low-resolution image is input to the rarefaction depth convolutional neural networks model that set algorithm is trained to, realizes that Super-resolution reconstruction constitutes high-definition picture.
Description
Technical field
The present invention relates to image reconstruction technique field, especially a kind of depth sparse convolution based on the constraint of pond manifold
Neural network human face super-resolution image reconstructing method and its system, device, storage medium.
Background technique
In recent years, China has entered society high-speed development period, and people are to guarantee individual privacy, property and personal safety
Attention rate also with day be incremented by, in order to maintain public order and citizen safety, video monitoring system traffic and safety etc.
Field is widely applied.Such as in terms of scouting tracking case, police need to obtain clearly video monitoring image quality, to improve
Case-solving rate.Resolution ratio limitation and target imaging face but in actual monitored, due to video monitoring system imaging device
The interference of the factors such as remote with a distance from camera, fuzzy, noise, weather environment, it is smaller often to can only obtain resolution ratio, second-rate
Video imaging effect so that becoming highly difficult to the identification of target face.Therefore, it by super-resolution reconstruction technology, realizes
It is very crucial at high-definition picture to restore low-resolution image.
At present in the super-resolution reconstruction algorithm of mainstream, the application of depth convolutional neural networks embodies huge advantage.It is deep
The calculated performance of degree convolutional neural networks is mainly determined by the multiplication operand amount of convolution in network, also just because of convolutional Neural
The huge calculation amount of network, limits application effect；Simultaneously in view of in network pond layer (i.e. to image parameter data into
Row sampling operation) inevitably the image detail information in network data transmittance process is caused to lose, big more options use full convolution
The deep neural network structure of layer, in order to guarantee to be effectively retained image detail information, also for making super-resolution reconstruction effect more
Close to reality.But once deep neural network structure does not have pond layer, image data is instructed in the form of full convolutional layer
Practice and test, between each layer of network although input and output characteristic pattern is to maintain constant, and the calculation amount that will lead to the neural network increases
Greatly, then time loss just will increase, negative influence is caused.Although existing research and propose, the nerve in convolution operation is utilized
First and convolution kernel linear transformation is subtracted in network weight progress neural network compression realization reduction convolutional neural networks by repairing
Multiplication operand amount, but linear transformation once is carried out to neuron and convolution kernel, network sparsity can be made to disappear, nothing
Method carries out network acceleration using sparsity；Reduction neural network is realized using improved activation primitive in full convolutional network
Computation complexity between layer, but also there is no the occupancy consumption and network parameter amount that reduce network internal storage.
Summary of the invention
To solve the above problems, the purpose of the present invention is to provide a kind of super-resolution image reconstruction method and its system,
Device, storage medium can effectively reduce network parameter calculation amount while retaining image material particular information, by original low resolution
Rate image is input to the rarefaction depth convolutional neural networks model that set algorithm is trained to, and realizes that Super-resolution reconstruction is constituted
High-definition picture.
Technical solution used by the present invention solves the problems, such as it is:
In a first aspect, the embodiment of the present invention proposes a kind of super-resolution image reconstruction method, comprising:
Obtain original image；
Bicubic interpolation is carried out to original image, enhances image pixel quantity, obtains pretreatment image；
Pretreatment image is subjected to image data point by the sparse depth convolutional neural networks constrained containing pond manifold
Analysis cluster and reconstruct；
Export reconstructed image.
Further, the acquisition original image, comprising:
To monitoring video video, program is write with Matlab, shot operation is carried out to video by frame to video recording, obtained original
Image data set.
Further, the bicubic interpolation calculating is related to 16 pixels, and calculation formula is as follows:
Wherein, (i ', j ') indicate decimal partial pixel coordinate in original image, dx, dy
It is the coordinate of x-axis, y-axis direction respectively, and F (i ', j ') it indicates to pass through original image
In between 16 pixels nearest from each pixel coordinate the sum of weight convolution calculate
New pixel value, R (x) indicates interpolation expression, and calculation formula is as follows:
Image pixel quantity can be enhanced by carrying out bicubic interpolation to original image.
Further, it is described by pretreatment image by the sparse depth convolutional neural networks that are constrained containing pond manifold into
Row analysis of image data cluster and reconstruct, comprising:
Pretreatment image carries out pondization sampling, the volume of manifold function constraint by mode one and mode two to image data
The extraction of characteristic pattern, data jump connection and convolutional neural networks rarefaction respectively obtain image A and image B in lamination, scheme
Reconstructed image is obtained as A and image B are combined convolution operation again.
Further, the mode one, comprising:
Pretreatment image passes through the pond layer with manifold constraint, connection between down-sampling copy feature diagram data and again
New arrangement obtains new characteristic pattern, ignores the interpolation characteristic pattern of bicubic interpolation generation, is input to by 10 layers of convolution and jump
It connects in framework, the Feature Mapping of generation is then input to next layer of convolutional layer, obtaining vector dimension is (H/2, W/2,4),
Then rarefaction is carried out to convolutional neural networks, is upsampled to finally by sub-pixel amplifying operation is carried out to obtained vector
Target resolution generates the image A having a size of (H, W).
Further, the mode two, comprising:
Pretreatment image is first inputted to the convolutional layer with 64 filters, generate Output Size be (H, W,
64) characteristic pattern is then inputted into 10 layers of convolution and jump link architecture, and Output Size is the characteristic pattern of (H, W, 64)
After network rarefaction, by the convolutional layer of single filter, image B is generated.
Second aspect, the embodiment of the present invention also proposed a kind of super-resolution image reconstruction system, comprising:
Image acquisition unit, for obtaining original image；
Image data enhancement unit enhances image pixel quantity, obtains for carrying out bicubic interpolation to original image
Pretreatment image；
Image analysis cluster and reconfiguration unit, for pretreatment image is sparse by constraining containing pond manifold
Depth convolutional neural networks carry out analysis of image data cluster and reconstruct；
Image output unit, for exporting reconstructed image.
The third aspect, the embodiment of the present invention also proposed a kind of super-resolution image reconstruction device, comprising:
At least one processor；And
The memory being connect at least one described processor communication；Wherein,
The memory be stored with can by least one described processor execute instruction, described instruction by it is described at least
One processor executes, so that at least one described processor is able to carry out method described in first aspect present invention.
Fourth aspect, the embodiment of the present invention also proposed a kind of computer readable storage medium, described computer-readable to deposit
Storage media is stored with computer executable instructions, and the computer executable instructions are for making computer execute the present invention first
Method described in aspect.
The one or more technical solutions provided in the embodiment of the present invention at least have the following beneficial effects: that the present invention mentions
A kind of super-resolution image reconstruction method and its system, device, storage medium supplied, firstly, input original low-resolution figure
Picture carries out bicubic interpolation to input low-resolution image by image preprocessing；Then, pretreated image is by containing
There are the sparse depth convolutional neural networks of pond manifold constraint, is connected with each other between image data and image is arrived in arrangement, study
The detail of the high frequency ingredient of middle missing realizes the reconstruct of high-resolution clear image.The present invention can effectively reduce network parameter
Calculation amount retains image material particular information simultaneously.By original low-resolution image be input to set algorithm be trained to it is dilute
Thinization depth convolutional neural networks model realizes that Super-resolution reconstruction constitutes high-definition picture.The present invention can keep network sparse
Property, material particular information caused by alleviating pond layer when sampling data information disappears, and full convolution deep neural network
Bring network internal storage occupies consumption and parameter calculation amount increases bring negative influence.
Detailed description of the invention
The invention will be further described with example with reference to the accompanying drawing.
Fig. 1 a and Fig. 1 b are the flow charts of super-resolution image reconstruction method in first embodiment of the invention；
Fig. 2 is the stream that image carries out super-resolution reconstruction based on depth convolutional neural networks in first embodiment of the invention
Cheng Tu；
Fig. 3 is the schematic diagram for carrying out convolution and jump connection processing in first embodiment of the invention to image data；
Fig. 4 is to operate in first embodiment of the invention to the network rarefaction of image data convolution and jump connection framework
Flow chart；
Fig. 5 is the structure diagram of super-resolution image reconstruction system in second embodiment of the invention；
Fig. 6 is the structure diagram of super-resolution image reconstruction device in third embodiment of the invention.
Specific embodiment
This part will be described in specific embodiments of the present invention, and the preferred embodiments of the invention is shown in the accompanying drawings, attached
The effect of figure be with figure remark additionally book word segment description, enable a person to intuitively, visually understand of the invention
Each technical characteristic and overall technical architecture, but it should not be understood as limiting the scope of the invention.
In the description of the present invention, it is to be understood that, be related to orientation description, for example, above and below, front, rear, left and right etc.
The orientation or positional relationship of instruction be based on the orientation or positional relationship shown in the drawings, be merely for convenience of description the present invention and
Simplify description, rather than the device or element of indication or suggestion meaning must have a particular orientation, with specific orientation construction
And operation, therefore be not considered as limiting the invention.
In the description of the present invention, several to be meant that one or more, it is multiple to be meant that two or more, be greater than,
Be less than, more than etc. be interpreted as not including this number, it is above, following, within etc. be interpreted as including this number.If there is being described to first,
Second is only intended to for the purpose of distinguishing technical characteristic, is not understood to indicate or imply relative importance or implicitly indicates
The quantity of indicated technical characteristic or the precedence relationship for implicitly indicating indicated technical characteristic.
In description of the invention, unless otherwise restricted clearly, the words such as setting, installation, connection be shall be understood in a broad sense,
Skilled artisan can rationally determine the tool of above-mentioned word in the present invention with the particular content of combination technology scheme
Body meaning.
With reference to the accompanying drawing, the embodiment of the present invention is further elaborated.
A and Fig. 1 b referring to Fig.1, the first embodiment of the present invention provide a kind of super-resolution image reconstruction method, including
But it is not limited to following steps:
S100: original image is obtained；
S200: bicubic interpolation is carried out to original image, enhances image pixel quantity, obtains pretreatment image；
S300: pretreatment image is subjected to image by the sparse depth convolutional neural networks constrained containing pond manifold
Data analysis cluster and reconstruct；
S400: output reconstructed image.
Wherein, realize that initial data obtains by S100, S100 is specifically included: to monitoring video video, being compiled with Matlab
It programs and shot operation is carried out to video by frame to video recording, obtain low resolution image data collection.
Image data enhancing and pretreatment are realized by S200, in order to emphasize that original image is certain interested
Feature, inhibition are lost interest in feature, and rich image information content reinforces image recognition effect, and application method of the present invention is, first
Bicubic interpolation is carried out to the low-resolution image of given input, enhances image pixel quantity, the preliminary characteristic pattern that increases is differentiated
Rate.Specific functional based method is as follows:
Bicubic interpolation calculating is related to 16 pixels, (i ', j ') indicate decimal partial pixel coordinate in original image,
Dx, dy are the coordinate of x-axis, y-axis direction respectively, F (i ', j ') indicate by original input image most from each pixel coordinate
The new pixel value that the sum of weight convolution calculates between 16 close pixels.Wherein R (x) indicates interpolation expression, due to this hair
Bright research is face image super-resolution reconstruct, and the processing for facial image has certain complexity, therefore uses
One kind is calculated based on polynomial convolution sample interpolation, and formula is as follows:
Image pixel quantity can be enhanced by carrying out bicubic interpolation to original image.
Analysis of image data cluster and reconstruct are realized by S300, and the image input by image data enhancing is based on
Depth convolutional neural networks carry out super-resolution reconstruction, mainly include pond sampling process, the convolution with manifold function constraint
The extraction of characteristic pattern and data jump connection and convolutional neural networks rarefaction are reconstructed high-definition picture two in layer
Aspect.Network structure is probably as shown in Figure 2.
The image data super-resolution reconstruction main process that the present invention designs is: the low-resolution image warp being originally inputted
It crosses after the data enhancing of bicubic interpolation and convolution and connection processing is carried out to image data by two kinds of different modes, such as
Shown in Fig. 2, the characteristic pattern result images A and image B of both of which processing are combined convolution operation again, export final surpass
Resolution reconstruction image, Fig. 3 are the visual description of convolution and jump connection.
Mode one, bicubic interpolation data enhancing treated image, by the pond layer constrained with manifold, under adopt
Connection between sample copy feature diagram data with rearrange to obtain new characteristic pattern, the interpolation for ignoring bicubic interpolation generation is special
Sign figure is input to and is connected in framework by 10 layers of convolution (including conversion convolution) and jump, then that the Feature Mapping of generation is defeated
Enter to next layer of convolutional layer, obtaining vector dimension is (H/2, W/2,4), then carries out rarefaction to convolutional neural networks, most
It is upsampled to target resolution by carrying out sub-pixel amplifying operation to obtained vector afterwards, is generated having a size of the high-quality of (H, W)
Measure super-resolution image A.
Mode two, image of the original low-resolution image after bicubic interpolation processing are directly inputted to 10 layers of volume
In product and jump link architecture, more image detail informations can be provided for reconstruct high quality super-resolution image.Such as Fig. 2
Dotted portion shown in, after low-resolution image bicubic interpolation, be enter into first one have 64 filters
Convolutional layer, generate the characteristic pattern that Output Size is (H, W, 64), be then inputted into above-mentioned convolution and jump connection framework
(with one equal weight of mode), Output Size are the characteristic pattern of (H, W, 64) after network rarefaction, pass through single filter
Convolutional layer, generate high quality super-resolution image B.
High quality super-resolution image A, B of two schema creations are combined, by the convolution of a scalar filter
Layer, obtains final super-resolution reconstruction image.
Finally, realizing output high-definition picture by S400.
It is that main method elaborates below:
It include more image detail information due to being originally inputted low-resolution image, while in order to avoid full convolution
Layer network framework bring network internal storage occupies consumption and parameter calculation amount increases, and the present invention proposes there is prevalence using a kind of
Input feature vector figure is down sampled to several reduction resolution versions by readjusting location of pixels by the pond process of constraint
(no image detail information missing).
In pond, interlayer uses the manifold learning thought for being locally linear embedding into (Locally Linear Embedding),
While keeping detailed information of the data in the sampling process of pond not lose, network query function amount is reduced.As sample is empty from higher-dimension
Between be mapped to lower dimensional space after, the linear relationship in every field between sample is constant.
The coordinate of sample point xi to operation, by being found with it most in its original higher-dimension neighborhood with k nearest neighbor thought
Three close samples xj, xl, xk, it is assumed that xi can be by xj, xl, xk linear expression, it may be assumed that
xi=wijxj+wikxk+wilxl
Wherein wij, wjk, wil are weight coefficient.After LLE dimensionality reduction, so that xi is in the corresponding projection x ' i of lower dimensional space
And corresponding projection x ' i, x ' j, x ' k of xj, xl, xk maintains like linear relationship as far as possible.That is:
x'i≈wijx'j+wikx'k+wilx'l
Weighting parameter is kept to be to try in low-dimensional and higher dimensional space consistent.
The algorithm can be divided into two parts, and the first step calculates the field reconstruction coefficients w of all samples according to Domain relation,
The linear relationship in each sample and its field between sample is found out, formula is as follows:
Wherein enable Cjk=(xi-xj)T(xi-xk),
Second step be it is constant according to neighborhood reconstruction coefficients, remove the coordinate for asking each sample in lower dimensional space, formula is as follows:
Enable Cjk=(xi-xj)TZ=(z1,z2,...,zm)∈Rd'×m,(W)ij=wij, M=(I-W)T+1
In convolutional layer, adaptive Method of Sample Selection is added and realizes the analysis of face partial structurtes manifold and cluster, is used for
Restore the high-frequency information lacked in low-frequency image block.
The characteristic image block pk (k=1 ..., n) on d layers of convolutional layer corresponding position is extracted, is constituted in a manner of column vector
Set Pi=[p1, p2 ..., pn] calculates the LPP transformation matrix Ai and mapping data matrix Yi=AiTPi of Pi.For input
Low-resolution image (LR) image block, calculate its low-dimensional characteristic set, then mapping data matrix Yi in by Euclidean away from
From selecting image block corresponding with the most similar set of low-dimensional feature to gather as a training, for restoring low-frequency image
The high-frequency information lacked in block.
And then the network rarefaction operation of convolution and jump connection framework, so that retaining before executing multiplication operation
The sparsity of weight and activation primitive, and amount of calculation is substantially reduced, as shown in Figure 4.Method divides three phases to carry out:
It has an intensive drill, trimming and retraining.Have an intensive drill: we are directly in transform domain one intensive p × p kernel of training.Inversion
It changes-eliminates, transformed kernel is directly initialized and trained by backpropagation, need to keep kernel in the spatial domain or turn
Change spatial kernel；Trimming: we are by threshold value t needed for trimming rate r needed for calculating realization and all less than t by absolute value
Weight is set as zero to trim transformed kernel；Re -training: keeping the weight being trimmed to about is zero, and sparse mask is being trimmed
It is calculated during step, and keeps constant retraining in period.Formulation is as follows:
S=AT[[Prune(GgGT)]⊙[ReLU(BTdB)]]A
Wherein, GgGT L is weight gradient, and dL is the calculating gradient of input activation.
It is fixed finally in order to make the high-definition picture after reconstruct to the maximum extent close to former face database true picture Y
An adopted width residual image is r=y-x, wherein x indicates that low-resolution image, y indicate to correspond to high-resolution in face database
Image.In loss layer input residual error estimation, low-resolution image, corresponding face database image is made using mean square error function
To judge penalty values, mean square error function is indicated are as follows:
Wherein n is sample size, and θ={ W1, W2 ..., Wd, B1, B2 ..., Bd } is network parameter.
In whole network framework, ReLU activation primitive is used, for the non-linear spy of decision function and whole network
Property, allows any Approximation of Arbitrary Nonlinear Function of neural network, by all negative switch actives to returning in network rarefaction
Zero, to reduce the multiplication number in domain.Formula is as follows:
F (x)=max (0, x)
By the way that low-resolution image is reconstructed by depth convolutional neural networks, to obtain corresponding high-resolution
The output of rate image.
In addition, the second embodiment of the present invention provides a kind of super-resolution image reconstruction system referring to Fig. 5, comprising:
Image acquisition unit 110, for obtaining original image；
Image data enhancement unit 120 enhances image pixel quantity, obtains for carrying out bicubic interpolation to original image
To pretreatment image；
Image analysis cluster and reconfiguration unit 130, for pretreatment image is dilute by constraining containing pond manifold
Degree of deepening by dredging convolutional neural networks carry out analysis of image data cluster and reconstruct；
Image output unit 140, for exporting reconstructed image.
The super-resolution image reconstruction method in super-resolution image reconstruction system and first embodiment in the present embodiment
Based on identical inventive concept, therefore, the super-resolution image reconstruction system in the present embodiment it is having the same the utility model has the advantages that
Image acquisition unit 110 obtains original image；Image data enhancement unit 120 carries out bicubic interpolation, enhancing to original image
Image pixel quantity, obtains pretreatment image；Image analysis cluster and reconfiguration unit 130, which pass through pretreatment image, to be contained
The sparse depth convolutional neural networks of pond manifold constraint carry out analysis of image data cluster and reconstruct；Image output unit
140 output reconstructed images.It can effectively reduce network parameter calculation amount reserved graph simultaneously using this super-resolution image reconstruction system
As material particular information.Original low-resolution image is input to the rarefaction depth convolutional Neural that set algorithm is trained to
Network model realizes that Super-resolution reconstruction constitutes high-definition picture.
Referring to Fig. 6, the third embodiment of the present invention additionally provides a kind of super-resolution image reconstruction device, comprising:
At least one processor；
And the memory being connect at least one described processor communication；
Wherein, the memory is stored with the instruction that can be executed by least one described processor, and described instruction is described
At least one processor executes, so that at least one described processor is able to carry out such as any one in above-mentioned first embodiment
Super-resolution image reconstruction method.
The device 200 can be any type of intelligent terminal, such as mobile phone, tablet computer, personal computer etc..
Processor can be connected with memory by bus or other modes, in Fig. 6 for being connected by bus.
Memory as a kind of non-transient computer readable storage medium, can be used for storing non-transient software program, it is non-temporarily
State property computer executable program and module, such as the corresponding journey of human face recognition model construction method in the embodiment of the present invention
Sequence instruction/module.Processor is by running non-transient software program, instruction and module stored in memory, to hold
Luggage sets 200 various function application and data processing, that is, realizes the super-resolution image of any of the above-described embodiment of the method
Reconstructing method.
Memory may include storing program area and storage data area, wherein storing program area can storage program area,
Application program required at least one function；Storage data area, which can be stored, uses created data according to device 200
Deng.It can also include non-transient memory in addition, memory may include high-speed random access memory, for example, at least one
Disk memory, flush memory device or other non-transient solid-state memories.In some embodiments, the optional packet of memory
The memory remotely located relative to processor is included, these remote memories can pass through network connection to the device 200.On
The example for stating network includes but is not limited to internet, intranet, local area network, mobile radio communication and combinations thereof.
One or more of module storages in the memory, are held when by one or more of processors
When row, the super-resolution image reconstruction method in above-mentioned any means embodiment is executed, for example, executing described above first
Method and step S100 to S400 in embodiment.
The fourth embodiment of the present invention additionally provides a kind of computer readable storage medium, the computer-readable storage
Media storage has computer executable instructions, which is executed by one or more control processors, example
Such as, it is executed by a processor in Fig. 6, said one or multiple processors may make to execute in above method embodiment
A kind of super-resolution image reconstruction method, such as the method and step S100 to S400 in first embodiment.
The apparatus embodiments described above are merely exemplary, wherein the unit as illustrated by the separation member
It may or may not be physically separated, it can it is in one place, or may be distributed over multiple networks
On unit.Some or all of the modules therein can be selected to achieve the purpose of the solution of this embodiment according to the actual needs.
Through the above description of the embodiments, those of ordinary skill in the art can be understood that each embodiment party
Formula can add the mode of general hardware platform to realize by software, naturally it is also possible to pass through hardware.Those of ordinary skill in the art
It is understood that realize above-described embodiment method in all or part of the process be can be instructed by computer program it is relevant
Hardware is completed, and the program can be stored in a computer-readable storage medium, the program is when being executed, it may include
Such as the process of the embodiment of above-mentioned each method.Wherein, the storage medium can be magnetic disk, CD, read-only memory
(Read-Only Memory, ROM) or random access memory (Random Access Memory, RAM) etc..
It is to be illustrated to preferable implementation of the invention, but the invention is not limited to above-mentioned embodiment party above
Formula, those skilled in the art can also make various equivalent variations on the premise of without prejudice to spirit of the invention or replace
It changes, these equivalent deformations or replacement are all included in the scope defined by the claims of the present application.
Claims (9)
1. a kind of super-resolution image reconstruction method characterized by comprising
Obtain original image；
Bicubic interpolation is carried out to original image, enhances image pixel quantity, obtains pretreatment image；
Pretreatment image is carried out analysis of image data by the sparse depth convolutional neural networks constrained containing pond manifold to gather
Class and reconstruct；
Export reconstructed image.
2. a kind of super-resolution image reconstruction method according to claim 1, which is characterized in that the acquisition original graph
Picture, comprising:
To monitoring video video, program is write with Matlab, shot operation is carried out to video by frame to video recording, obtain original image
Data set.
3. a kind of super-resolution image reconstruction method according to claim 1, which is characterized in that the bicubic interpolation meter
Calculation is related to 16 pixels, and calculation formula is as follows:
Wherein, (i ', j ') indicate decimal partial pixel coordinate in original image, dx, dy are the coordinate of x-axis, y-axis direction, F respectively
(i ', j ') it indicates to calculate by the sum of weight convolution between 16 pixels nearest from each pixel coordinate in original image
New pixel value, R (x) indicates interpolation expression, and calculation formula is as follows:
4. a kind of super-resolution image reconstruction method according to claim 1, which is characterized in that described by pretreatment image
Analysis of image data cluster and reconstruct are carried out by the sparse depth convolutional neural networks constrained containing pond manifold, comprising:
Pretreatment image samples, in convolutional layer by the pondization that mode one and mode two carry out manifold function constraint to image data
Extraction, data jump connection and the convolutional neural networks rarefaction of characteristic pattern respectively obtain image A and image B, image A and figure
Reconstructed image is obtained as B is combined convolution operation again.
5. a kind of super-resolution image reconstruction method according to claim 4, which is characterized in that the mode one, comprising:
Pretreatment image passes through the pond layer with manifold constraint, the connection between down-sampling copy feature diagram data with rearrange
New characteristic pattern is obtained, the interpolation characteristic pattern of bicubic interpolation generation is ignored, is input to by 10 layers of convolution and jump connection frame
In structure, the Feature Mapping of generation is then input to next layer of convolutional layer, obtaining vector dimension is (H/2, W/2,4), then right
Convolutional neural networks carry out rarefaction, are upsampled to target resolution finally by sub-pixel amplifying operation is carried out to obtained vector
Rate generates the image A having a size of (H, W).
6. a kind of super-resolution image reconstruction method according to claim 4, which is characterized in that the mode two, comprising:
Pretreatment image is first inputted to the convolutional layer with 64 filters, generates the spy that Output Size is (H, W, 64)
Sign figure is then inputted into 10 layers of convolution and jump link architecture, and Output Size is that the characteristic pattern of (H, W, 64) passes through network
After rarefaction, by the convolutional layer of single filter, image B is generated.
7. a kind of super-resolution image reconstruction system characterized by comprising
Image acquisition unit, for obtaining original image；
Image data enhancement unit enhances image pixel quantity, is pre-processed for carrying out bicubic interpolation to original image
Image；
Image analysis cluster and reconfiguration unit, for rolling up pretreatment image by the sparse depth constrained containing pond manifold
Product neural network carries out analysis of image data cluster and reconstruct；
Image output unit, for exporting reconstructed image.
8. a kind of super-resolution image reconstruction device characterized by comprising
At least one processor；And
The memory being connect at least one described processor communication；Wherein,
The memory is stored with the instruction that can be executed by least one described processor, and described instruction is by described at least one
It manages device to execute, so that at least one described processor is able to carry out as the method according to claim 1 to 6.
9. a kind of computer readable storage medium, which is characterized in that the computer-readable recording medium storage has computer can
It executes instruction, the computer executable instructions are for making computer execute as the method according to claim 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910409645.3A CN110246084B (en) | 2019-05-16 | 2019-05-16 | Super-resolution image reconstruction method, system and device thereof, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910409645.3A CN110246084B (en) | 2019-05-16 | 2019-05-16 | Super-resolution image reconstruction method, system and device thereof, and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110246084A true CN110246084A (en) | 2019-09-17 |
| CN110246084B CN110246084B (en) | 2023-03-31 |
Family
ID=67884554
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910409645.3A Active CN110246084B (en) | 2019-05-16 | 2019-05-16 | Super-resolution image reconstruction method, system and device thereof, and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110246084B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110942425A (en) * | 2019-11-26 | 2020-03-31 | 贵州师范学院 | Reconstruction method and reconstruction system of super-resolution image and electronic equipment |
| CN111222446A (en) * | 2019-12-31 | 2020-06-02 | Oppo广东移动通信有限公司 | Face recognition method, face recognition device and mobile terminal |
| CN112581363A (en) * | 2019-09-29 | 2021-03-30 | 北京金山云网络技术有限公司 | Image super-resolution reconstruction method and device, electronic equipment and storage medium |
| CN113409192A (en) * | 2021-06-17 | 2021-09-17 | Oppo广东移动通信有限公司 | Super-resolution chip, super-resolution algorithm updating method and electronic equipment |
| CN113675850A (en) * | 2021-10-25 | 2021-11-19 | 山东大学 | Power grid information rapid and accurate sensing method based on nonlinear robust estimation |
| CN114170619A (en) * | 2021-10-18 | 2022-03-11 | 中标慧安信息技术股份有限公司 | Data checking method and system based on edge calculation |
| CN115546858A (en) * | 2022-08-15 | 2022-12-30 | 荣耀终端有限公司 | Face image processing method and electronic equipment |
| WO2023274391A1 (en) * | 2021-07-01 | 2023-01-05 | Beijing Bytedance Network Technology Co., Ltd. | Super resolution downsampling |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106709875A (en) * | 2016-12-30 | 2017-05-24 | 北京工业大学 | Compressed low-resolution image restoration method based on combined deep network |
| CN107480726A (en) * | 2017-08-25 | 2017-12-15 | 电子科技大学 | A kind of Scene Semantics dividing method based on full convolution and shot and long term mnemon |
| CN107507134A (en) * | 2017-09-21 | 2017-12-22 | 大连理工大学 | Super-resolution method based on convolutional neural networks |
| US20180137603A1 (en) * | 2016-11-07 | 2018-05-17 | Umbo Cv Inc. | Method and system for providing high resolution image through super-resolution reconstruction |
| US20180158177A1 (en) * | 2016-12-07 | 2018-06-07 | Idemia Identity & Security France | System for processing images |
| US20180262683A1 (en) * | 2017-03-10 | 2018-09-13 | Gopro, Inc. | Image Quality Assessment |
| US20190075301A1 (en) * | 2017-09-01 | 2019-03-07 | Apple Inc. | Machine learning video processing systems and methods |
| CN109461157A (en) * | 2018-10-19 | 2019-03-12 | 苏州大学 | Image, semantic dividing method based on multi-stage characteristics fusion and Gauss conditions random field |
-
2019
- 2019-05-16 CN CN201910409645.3A patent/CN110246084B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180137603A1 (en) * | 2016-11-07 | 2018-05-17 | Umbo Cv Inc. | Method and system for providing high resolution image through super-resolution reconstruction |
| US20180158177A1 (en) * | 2016-12-07 | 2018-06-07 | Idemia Identity & Security France | System for processing images |
| CN106709875A (en) * | 2016-12-30 | 2017-05-24 | 北京工业大学 | Compressed low-resolution image restoration method based on combined deep network |
| US20180262683A1 (en) * | 2017-03-10 | 2018-09-13 | Gopro, Inc. | Image Quality Assessment |
| CN107480726A (en) * | 2017-08-25 | 2017-12-15 | 电子科技大学 | A kind of Scene Semantics dividing method based on full convolution and shot and long term mnemon |
| US20190075301A1 (en) * | 2017-09-01 | 2019-03-07 | Apple Inc. | Machine learning video processing systems and methods |
| CN107507134A (en) * | 2017-09-21 | 2017-12-22 | 大连理工大学 | Super-resolution method based on convolutional neural networks |
| CN109461157A (en) * | 2018-10-19 | 2019-03-12 | 苏州大学 | Image, semantic dividing method based on multi-stage characteristics fusion and Gauss conditions random field |
Non-Patent Citations (1)
| Title |
|---|
| 李现国等: "基于中间层监督卷积神经网络的图像超分辨率重建", 《中国图象图形学报》 * |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112581363A (en) * | 2019-09-29 | 2021-03-30 | 北京金山云网络技术有限公司 | Image super-resolution reconstruction method and device, electronic equipment and storage medium |
| CN110942425A (en) * | 2019-11-26 | 2020-03-31 | 贵州师范学院 | Reconstruction method and reconstruction system of super-resolution image and electronic equipment |
| CN111222446A (en) * | 2019-12-31 | 2020-06-02 | Oppo广东移动通信有限公司 | Face recognition method, face recognition device and mobile terminal |
| CN113409192A (en) * | 2021-06-17 | 2021-09-17 | Oppo广东移动通信有限公司 | Super-resolution chip, super-resolution algorithm updating method and electronic equipment |
| WO2023274391A1 (en) * | 2021-07-01 | 2023-01-05 | Beijing Bytedance Network Technology Co., Ltd. | Super resolution downsampling |
| CN114170619A (en) * | 2021-10-18 | 2022-03-11 | 中标慧安信息技术股份有限公司 | Data checking method and system based on edge calculation |
| CN114170619B (en) * | 2021-10-18 | 2022-08-19 | 中标慧安信息技术股份有限公司 | Data checking method and system based on edge calculation |
| CN113675850A (en) * | 2021-10-25 | 2021-11-19 | 山东大学 | Power grid information rapid and accurate sensing method based on nonlinear robust estimation |
| CN115546858A (en) * | 2022-08-15 | 2022-12-30 | 荣耀终端有限公司 | Face image processing method and electronic equipment |
| CN115546858B (en) * | 2022-08-15 | 2023-08-25 | 荣耀终端有限公司 | Face image processing method and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110246084B (en) | 2023-03-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110246084A (en) | A kind of super-resolution image reconstruction method and its system, device, storage medium | |
| Jiang et al. | Edge-enhanced GAN for remote sensing image superresolution | |
| Wang et al. | Ultra-dense GAN for satellite imagery super-resolution | |
| CN107154023B (en) | Based on the face super-resolution reconstruction method for generating confrontation network and sub-pix convolution | |
| CN110322416A (en) | Image processing method, device and computer readable storage medium | |
| CN112991173A (en) | Single-frame image super-resolution reconstruction method based on dual-channel feature migration network | |
| Yang et al. | Image super-resolution based on deep neural network of multiple attention mechanism | |
| CN110599502B (en) | Skin lesion segmentation method based on deep learning | |
| CN111951164B (en) | Image super-resolution reconstruction network structure and image reconstruction effect analysis method | |
| CN113139585B (en) | Infrared and visible light image fusion method based on unified multi-scale dense connection network | |
| Guo et al. | Towards efficient motion-blurred public security video super-resolution based on back-projection networks | |
| CN115393191A (en) | Method, device and equipment for reconstructing super-resolution of lightweight remote sensing image | |
| Shen et al. | AFFNet: attention mechanism network based on fusion feature for image cloud removal | |
| CN114926343A (en) | Image super-resolution method based on pyramid fusion attention network | |
| Pan et al. | FDPPGAN: remote sensing image fusion based on deep perceptual patchGAN | |
| CN116739899A (en) | Image super-resolution reconstruction method based on SAUGAN network | |
| CN117788295B (en) | Super-resolution reconstruction method, system and medium for remote sensing image | |
| CN116452930A (en) | Multispectral image fusion method and multispectral image fusion system based on frequency domain enhancement in degradation environment | |
| Wang et al. | PACCDU: Pyramid attention cross-convolutional dual UNet for infrared and visible image fusion | |
| CN113034371B (en) | Infrared and visible light image fusion method based on feature embedding | |
| Kim et al. | Progressive contextual aggregation empowered by pixel-wise confidence scoring for image inpainting | |
| CN111428809B (en) | Crowd counting method based on spatial information fusion and convolutional neural network | |
| CN116523985B (en) | Structure and texture feature guided double-encoder image restoration method | |
| Sun et al. | A rapid and accurate infrared image super-resolution method based on zoom mechanism | |
| Dong et al. | Remote sensing image super-resolution via enhanced back-projection networks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |