CN110246084A - A kind of super-resolution image reconstruction method and its system, device, storage medium - Google Patents

A kind of super-resolution image reconstruction method and its system, device, storage medium Download PDF

Info

Publication number
CN110246084A
CN110246084A CN201910409645.3A CN201910409645A CN110246084A CN 110246084 A CN110246084 A CN 110246084A CN 201910409645 A CN201910409645 A CN 201910409645A CN 110246084 A CN110246084 A CN 110246084A
Authority
CN
China
Prior art keywords
image
resolution
super
resolution image
neural networks
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910409645.3A
Other languages
Chinese (zh)
Other versions
CN110246084B (en
Inventor
徐颖
关蕙欣
翟懿奎
邓文博
柯琪锐
应自炉
甘俊英
曾军英
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuyi University
Original Assignee
Wuyi University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuyi University filed Critical Wuyi University
Priority to CN201910409645.3A priority Critical patent/CN110246084B/en
Publication of CN110246084A publication Critical patent/CN110246084A/en
Application granted granted Critical
Publication of CN110246084B publication Critical patent/CN110246084B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/21Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/23Clustering techniques
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/045Combinations of networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformations in the plane of the image
    • G06T3/40Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Evolutionary Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Computational Linguistics (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a kind of super-resolution image reconstruction method and its system, device, storage mediums, firstly, input original low-resolution image carries out bicubic interpolation to input low-resolution image by image preprocessing;Then, pretreated image is by the sparse depth convolutional neural networks that constrain containing pond manifold, is connected with each other between image data and arrangement, learns the detail of the high frequency ingredient lacked into image, realizes the reconstruct of high-resolution clear image.The present invention can effectively reduce network parameter calculation amount while retain image material particular information.Original low-resolution image is input to the rarefaction depth convolutional neural networks model that set algorithm is trained to, realizes that Super-resolution reconstruction constitutes high-definition picture.

Description

A kind of super-resolution image reconstruction method and its system, device, storage medium
Technical field
The present invention relates to image reconstruction technique field, especially a kind of depth sparse convolution based on the constraint of pond manifold Neural network human face super-resolution image reconstructing method and its system, device, storage medium.
Background technique
In recent years, China has entered society high-speed development period, and people are to guarantee individual privacy, property and personal safety Attention rate also with day be incremented by, in order to maintain public order and citizen safety, video monitoring system traffic and safety etc. Field is widely applied.Such as in terms of scouting tracking case, police need to obtain clearly video monitoring image quality, to improve Case-solving rate.Resolution ratio limitation and target imaging face but in actual monitored, due to video monitoring system imaging device The interference of the factors such as remote with a distance from camera, fuzzy, noise, weather environment, it is smaller often to can only obtain resolution ratio, second-rate Video imaging effect so that becoming highly difficult to the identification of target face.Therefore, it by super-resolution reconstruction technology, realizes It is very crucial at high-definition picture to restore low-resolution image.
At present in the super-resolution reconstruction algorithm of mainstream, the application of depth convolutional neural networks embodies huge advantage.It is deep The calculated performance of degree convolutional neural networks is mainly determined by the multiplication operand amount of convolution in network, also just because of convolutional Neural The huge calculation amount of network, limits application effect;Simultaneously in view of in network pond layer (i.e. to image parameter data into Row sampling operation) inevitably the image detail information in network data transmittance process is caused to lose, big more options use full convolution The deep neural network structure of layer, in order to guarantee to be effectively retained image detail information, also for making super-resolution reconstruction effect more Close to reality.But once deep neural network structure does not have pond layer, image data is instructed in the form of full convolutional layer Practice and test, between each layer of network although input and output characteristic pattern is to maintain constant, and the calculation amount that will lead to the neural network increases Greatly, then time loss just will increase, negative influence is caused.Although existing research and propose, the nerve in convolution operation is utilized First and convolution kernel linear transformation is subtracted in network weight progress neural network compression realization reduction convolutional neural networks by repairing Multiplication operand amount, but linear transformation once is carried out to neuron and convolution kernel, network sparsity can be made to disappear, nothing Method carries out network acceleration using sparsity;Reduction neural network is realized using improved activation primitive in full convolutional network Computation complexity between layer, but also there is no the occupancy consumption and network parameter amount that reduce network internal storage.
Summary of the invention
To solve the above problems, the purpose of the present invention is to provide a kind of super-resolution image reconstruction method and its system, Device, storage medium can effectively reduce network parameter calculation amount while retaining image material particular information, by original low resolution Rate image is input to the rarefaction depth convolutional neural networks model that set algorithm is trained to, and realizes that Super-resolution reconstruction is constituted High-definition picture.
Technical solution used by the present invention solves the problems, such as it is:
In a first aspect, the embodiment of the present invention proposes a kind of super-resolution image reconstruction method, comprising:
Obtain original image;
Bicubic interpolation is carried out to original image, enhances image pixel quantity, obtains pretreatment image;
Pretreatment image is subjected to image data point by the sparse depth convolutional neural networks constrained containing pond manifold Analysis cluster and reconstruct;
Export reconstructed image.
Further, the acquisition original image, comprising:
To monitoring video video, program is write with Matlab, shot operation is carried out to video by frame to video recording, obtained original Image data set.
Further, the bicubic interpolation calculating is related to 16 pixels, and calculation formula is as follows:
Wherein, (i ', j ') indicate decimal partial pixel coordinate in original image, dx, dy
It is the coordinate of x-axis, y-axis direction respectively, and F (i ', j ') it indicates to pass through original image
In between 16 pixels nearest from each pixel coordinate the sum of weight convolution calculate
New pixel value, R (x) indicates interpolation expression, and calculation formula is as follows:
Image pixel quantity can be enhanced by carrying out bicubic interpolation to original image.
Further, it is described by pretreatment image by the sparse depth convolutional neural networks that are constrained containing pond manifold into Row analysis of image data cluster and reconstruct, comprising:
Pretreatment image carries out pondization sampling, the volume of manifold function constraint by mode one and mode two to image data The extraction of characteristic pattern, data jump connection and convolutional neural networks rarefaction respectively obtain image A and image B in lamination, scheme Reconstructed image is obtained as A and image B are combined convolution operation again.
Further, the mode one, comprising:
Pretreatment image passes through the pond layer with manifold constraint, connection between down-sampling copy feature diagram data and again New arrangement obtains new characteristic pattern, ignores the interpolation characteristic pattern of bicubic interpolation generation, is input to by 10 layers of convolution and jump It connects in framework, the Feature Mapping of generation is then input to next layer of convolutional layer, obtaining vector dimension is (H/2, W/2,4), Then rarefaction is carried out to convolutional neural networks, is upsampled to finally by sub-pixel amplifying operation is carried out to obtained vector Target resolution generates the image A having a size of (H, W).
Further, the mode two, comprising:
Pretreatment image is first inputted to the convolutional layer with 64 filters, generate Output Size be (H, W, 64) characteristic pattern is then inputted into 10 layers of convolution and jump link architecture, and Output Size is the characteristic pattern of (H, W, 64) After network rarefaction, by the convolutional layer of single filter, image B is generated.
Second aspect, the embodiment of the present invention also proposed a kind of super-resolution image reconstruction system, comprising:
Image acquisition unit, for obtaining original image;
Image data enhancement unit enhances image pixel quantity, obtains for carrying out bicubic interpolation to original image Pretreatment image;
Image analysis cluster and reconfiguration unit, for pretreatment image is sparse by constraining containing pond manifold Depth convolutional neural networks carry out analysis of image data cluster and reconstruct;
Image output unit, for exporting reconstructed image.
The third aspect, the embodiment of the present invention also proposed a kind of super-resolution image reconstruction device, comprising:
At least one processor;And
The memory being connect at least one described processor communication;Wherein,
The memory be stored with can by least one described processor execute instruction, described instruction by it is described at least One processor executes, so that at least one described processor is able to carry out method described in first aspect present invention.
Fourth aspect, the embodiment of the present invention also proposed a kind of computer readable storage medium, described computer-readable to deposit Storage media is stored with computer executable instructions, and the computer executable instructions are for making computer execute the present invention first Method described in aspect.
The one or more technical solutions provided in the embodiment of the present invention at least have the following beneficial effects: that the present invention mentions A kind of super-resolution image reconstruction method and its system, device, storage medium supplied, firstly, input original low-resolution figure Picture carries out bicubic interpolation to input low-resolution image by image preprocessing;Then, pretreated image is by containing There are the sparse depth convolutional neural networks of pond manifold constraint, is connected with each other between image data and image is arrived in arrangement, study The detail of the high frequency ingredient of middle missing realizes the reconstruct of high-resolution clear image.The present invention can effectively reduce network parameter Calculation amount retains image material particular information simultaneously.By original low-resolution image be input to set algorithm be trained to it is dilute Thinization depth convolutional neural networks model realizes that Super-resolution reconstruction constitutes high-definition picture.The present invention can keep network sparse Property, material particular information caused by alleviating pond layer when sampling data information disappears, and full convolution deep neural network Bring network internal storage occupies consumption and parameter calculation amount increases bring negative influence.
Detailed description of the invention
The invention will be further described with example with reference to the accompanying drawing.
Fig. 1 a and Fig. 1 b are the flow charts of super-resolution image reconstruction method in first embodiment of the invention;
Fig. 2 is the stream that image carries out super-resolution reconstruction based on depth convolutional neural networks in first embodiment of the invention Cheng Tu;
Fig. 3 is the schematic diagram for carrying out convolution and jump connection processing in first embodiment of the invention to image data;
Fig. 4 is to operate in first embodiment of the invention to the network rarefaction of image data convolution and jump connection framework Flow chart;
Fig. 5 is the structure diagram of super-resolution image reconstruction system in second embodiment of the invention;
Fig. 6 is the structure diagram of super-resolution image reconstruction device in third embodiment of the invention.
Specific embodiment
This part will be described in specific embodiments of the present invention, and the preferred embodiments of the invention is shown in the accompanying drawings, attached The effect of figure be with figure remark additionally book word segment description, enable a person to intuitively, visually understand of the invention Each technical characteristic and overall technical architecture, but it should not be understood as limiting the scope of the invention.
In the description of the present invention, it is to be understood that, be related to orientation description, for example, above and below, front, rear, left and right etc. The orientation or positional relationship of instruction be based on the orientation or positional relationship shown in the drawings, be merely for convenience of description the present invention and Simplify description, rather than the device or element of indication or suggestion meaning must have a particular orientation, with specific orientation construction And operation, therefore be not considered as limiting the invention.
In the description of the present invention, several to be meant that one or more, it is multiple to be meant that two or more, be greater than, Be less than, more than etc. be interpreted as not including this number, it is above, following, within etc. be interpreted as including this number.If there is being described to first, Second is only intended to for the purpose of distinguishing technical characteristic, is not understood to indicate or imply relative importance or implicitly indicates The quantity of indicated technical characteristic or the precedence relationship for implicitly indicating indicated technical characteristic.
In description of the invention, unless otherwise restricted clearly, the words such as setting, installation, connection be shall be understood in a broad sense, Skilled artisan can rationally determine the tool of above-mentioned word in the present invention with the particular content of combination technology scheme Body meaning.
With reference to the accompanying drawing, the embodiment of the present invention is further elaborated.
A and Fig. 1 b referring to Fig.1, the first embodiment of the present invention provide a kind of super-resolution image reconstruction method, including But it is not limited to following steps:
S100: original image is obtained;
S200: bicubic interpolation is carried out to original image, enhances image pixel quantity, obtains pretreatment image;
S300: pretreatment image is subjected to image by the sparse depth convolutional neural networks constrained containing pond manifold Data analysis cluster and reconstruct;
S400: output reconstructed image.
Wherein, realize that initial data obtains by S100, S100 is specifically included: to monitoring video video, being compiled with Matlab It programs and shot operation is carried out to video by frame to video recording, obtain low resolution image data collection.
Image data enhancing and pretreatment are realized by S200, in order to emphasize that original image is certain interested Feature, inhibition are lost interest in feature, and rich image information content reinforces image recognition effect, and application method of the present invention is, first Bicubic interpolation is carried out to the low-resolution image of given input, enhances image pixel quantity, the preliminary characteristic pattern that increases is differentiated Rate.Specific functional based method is as follows:
Bicubic interpolation calculating is related to 16 pixels, (i ', j ') indicate decimal partial pixel coordinate in original image, Dx, dy are the coordinate of x-axis, y-axis direction respectively, F (i ', j ') indicate by original input image most from each pixel coordinate The new pixel value that the sum of weight convolution calculates between 16 close pixels.Wherein R (x) indicates interpolation expression, due to this hair Bright research is face image super-resolution reconstruct, and the processing for facial image has certain complexity, therefore uses One kind is calculated based on polynomial convolution sample interpolation, and formula is as follows:
Image pixel quantity can be enhanced by carrying out bicubic interpolation to original image.
Analysis of image data cluster and reconstruct are realized by S300, and the image input by image data enhancing is based on Depth convolutional neural networks carry out super-resolution reconstruction, mainly include pond sampling process, the convolution with manifold function constraint The extraction of characteristic pattern and data jump connection and convolutional neural networks rarefaction are reconstructed high-definition picture two in layer Aspect.Network structure is probably as shown in Figure 2.
The image data super-resolution reconstruction main process that the present invention designs is: the low-resolution image warp being originally inputted It crosses after the data enhancing of bicubic interpolation and convolution and connection processing is carried out to image data by two kinds of different modes, such as Shown in Fig. 2, the characteristic pattern result images A and image B of both of which processing are combined convolution operation again, export final surpass Resolution reconstruction image, Fig. 3 are the visual description of convolution and jump connection.
Mode one, bicubic interpolation data enhancing treated image, by the pond layer constrained with manifold, under adopt Connection between sample copy feature diagram data with rearrange to obtain new characteristic pattern, the interpolation for ignoring bicubic interpolation generation is special Sign figure is input to and is connected in framework by 10 layers of convolution (including conversion convolution) and jump, then that the Feature Mapping of generation is defeated Enter to next layer of convolutional layer, obtaining vector dimension is (H/2, W/2,4), then carries out rarefaction to convolutional neural networks, most It is upsampled to target resolution by carrying out sub-pixel amplifying operation to obtained vector afterwards, is generated having a size of the high-quality of (H, W) Measure super-resolution image A.
Mode two, image of the original low-resolution image after bicubic interpolation processing are directly inputted to 10 layers of volume In product and jump link architecture, more image detail informations can be provided for reconstruct high quality super-resolution image.Such as Fig. 2 Dotted portion shown in, after low-resolution image bicubic interpolation, be enter into first one have 64 filters Convolutional layer, generate the characteristic pattern that Output Size is (H, W, 64), be then inputted into above-mentioned convolution and jump connection framework (with one equal weight of mode), Output Size are the characteristic pattern of (H, W, 64) after network rarefaction, pass through single filter Convolutional layer, generate high quality super-resolution image B.
High quality super-resolution image A, B of two schema creations are combined, by the convolution of a scalar filter Layer, obtains final super-resolution reconstruction image.
Finally, realizing output high-definition picture by S400.
It is that main method elaborates below:
It include more image detail information due to being originally inputted low-resolution image, while in order to avoid full convolution Layer network framework bring network internal storage occupies consumption and parameter calculation amount increases, and the present invention proposes there is prevalence using a kind of Input feature vector figure is down sampled to several reduction resolution versions by readjusting location of pixels by the pond process of constraint (no image detail information missing).
In pond, interlayer uses the manifold learning thought for being locally linear embedding into (Locally Linear Embedding), While keeping detailed information of the data in the sampling process of pond not lose, network query function amount is reduced.As sample is empty from higher-dimension Between be mapped to lower dimensional space after, the linear relationship in every field between sample is constant.
The coordinate of sample point xi to operation, by being found with it most in its original higher-dimension neighborhood with k nearest neighbor thought Three close samples xj, xl, xk, it is assumed that xi can be by xj, xl, xk linear expression, it may be assumed that
xi=wijxj+wikxk+wilxl
Wherein wij, wjk, wil are weight coefficient.After LLE dimensionality reduction, so that xi is in the corresponding projection x ' i of lower dimensional space And corresponding projection x ' i, x ' j, x ' k of xj, xl, xk maintains like linear relationship as far as possible.That is:
x'i≈wijx'j+wikx'k+wilx'l
Weighting parameter is kept to be to try in low-dimensional and higher dimensional space consistent.
The algorithm can be divided into two parts, and the first step calculates the field reconstruction coefficients w of all samples according to Domain relation, The linear relationship in each sample and its field between sample is found out, formula is as follows:
Wherein enable Cjk=(xi-xj)T(xi-xk),
Second step be it is constant according to neighborhood reconstruction coefficients, remove the coordinate for asking each sample in lower dimensional space, formula is as follows:
Enable Cjk=(xi-xj)TZ=(z1,z2,...,zm)∈Rd'×m,(W)ij=wij, M=(I-W)T+1
In convolutional layer, adaptive Method of Sample Selection is added and realizes the analysis of face partial structurtes manifold and cluster, is used for Restore the high-frequency information lacked in low-frequency image block.
The characteristic image block pk (k=1 ..., n) on d layers of convolutional layer corresponding position is extracted, is constituted in a manner of column vector Set Pi=[p1, p2 ..., pn] calculates the LPP transformation matrix Ai and mapping data matrix Yi=AiTPi of Pi.For input Low-resolution image (LR) image block, calculate its low-dimensional characteristic set, then mapping data matrix Yi in by Euclidean away from From selecting image block corresponding with the most similar set of low-dimensional feature to gather as a training, for restoring low-frequency image The high-frequency information lacked in block.
And then the network rarefaction operation of convolution and jump connection framework, so that retaining before executing multiplication operation The sparsity of weight and activation primitive, and amount of calculation is substantially reduced, as shown in Figure 4.Method divides three phases to carry out: It has an intensive drill, trimming and retraining.Have an intensive drill: we are directly in transform domain one intensive p × p kernel of training.Inversion It changes-eliminates, transformed kernel is directly initialized and trained by backpropagation, need to keep kernel in the spatial domain or turn Change spatial kernel;Trimming: we are by threshold value t needed for trimming rate r needed for calculating realization and all less than t by absolute value Weight is set as zero to trim transformed kernel;Re -training: keeping the weight being trimmed to about is zero, and sparse mask is being trimmed It is calculated during step, and keeps constant retraining in period.Formulation is as follows:
S=AT[[Prune(GgGT)]⊙[ReLU(BTdB)]]A
Wherein, GgGT L is weight gradient, and dL is the calculating gradient of input activation.
It is fixed finally in order to make the high-definition picture after reconstruct to the maximum extent close to former face database true picture Y An adopted width residual image is r=y-x, wherein x indicates that low-resolution image, y indicate to correspond to high-resolution in face database Image.In loss layer input residual error estimation, low-resolution image, corresponding face database image is made using mean square error function To judge penalty values, mean square error function is indicated are as follows:
Wherein n is sample size, and θ={ W1, W2 ..., Wd, B1, B2 ..., Bd } is network parameter.
In whole network framework, ReLU activation primitive is used, for the non-linear spy of decision function and whole network Property, allows any Approximation of Arbitrary Nonlinear Function of neural network, by all negative switch actives to returning in network rarefaction Zero, to reduce the multiplication number in domain.Formula is as follows:
F (x)=max (0, x)
By the way that low-resolution image is reconstructed by depth convolutional neural networks, to obtain corresponding high-resolution The output of rate image.
In addition, the second embodiment of the present invention provides a kind of super-resolution image reconstruction system referring to Fig. 5, comprising:
Image acquisition unit 110, for obtaining original image;
Image data enhancement unit 120 enhances image pixel quantity, obtains for carrying out bicubic interpolation to original image To pretreatment image;
Image analysis cluster and reconfiguration unit 130, for pretreatment image is dilute by constraining containing pond manifold Degree of deepening by dredging convolutional neural networks carry out analysis of image data cluster and reconstruct;
Image output unit 140, for exporting reconstructed image.
The super-resolution image reconstruction method in super-resolution image reconstruction system and first embodiment in the present embodiment Based on identical inventive concept, therefore, the super-resolution image reconstruction system in the present embodiment it is having the same the utility model has the advantages that Image acquisition unit 110 obtains original image;Image data enhancement unit 120 carries out bicubic interpolation, enhancing to original image Image pixel quantity, obtains pretreatment image;Image analysis cluster and reconfiguration unit 130, which pass through pretreatment image, to be contained The sparse depth convolutional neural networks of pond manifold constraint carry out analysis of image data cluster and reconstruct;Image output unit 140 output reconstructed images.It can effectively reduce network parameter calculation amount reserved graph simultaneously using this super-resolution image reconstruction system As material particular information.Original low-resolution image is input to the rarefaction depth convolutional Neural that set algorithm is trained to Network model realizes that Super-resolution reconstruction constitutes high-definition picture.
Referring to Fig. 6, the third embodiment of the present invention additionally provides a kind of super-resolution image reconstruction device, comprising:
At least one processor;
And the memory being connect at least one described processor communication;
Wherein, the memory is stored with the instruction that can be executed by least one described processor, and described instruction is described At least one processor executes, so that at least one described processor is able to carry out such as any one in above-mentioned first embodiment Super-resolution image reconstruction method.
The device 200 can be any type of intelligent terminal, such as mobile phone, tablet computer, personal computer etc..
Processor can be connected with memory by bus or other modes, in Fig. 6 for being connected by bus.
Memory as a kind of non-transient computer readable storage medium, can be used for storing non-transient software program, it is non-temporarily State property computer executable program and module, such as the corresponding journey of human face recognition model construction method in the embodiment of the present invention Sequence instruction/module.Processor is by running non-transient software program, instruction and module stored in memory, to hold Luggage sets 200 various function application and data processing, that is, realizes the super-resolution image of any of the above-described embodiment of the method Reconstructing method.
Memory may include storing program area and storage data area, wherein storing program area can storage program area, Application program required at least one function;Storage data area, which can be stored, uses created data according to device 200 Deng.It can also include non-transient memory in addition, memory may include high-speed random access memory, for example, at least one Disk memory, flush memory device or other non-transient solid-state memories.In some embodiments, the optional packet of memory The memory remotely located relative to processor is included, these remote memories can pass through network connection to the device 200.On The example for stating network includes but is not limited to internet, intranet, local area network, mobile radio communication and combinations thereof.
One or more of module storages in the memory, are held when by one or more of processors When row, the super-resolution image reconstruction method in above-mentioned any means embodiment is executed, for example, executing described above first Method and step S100 to S400 in embodiment.
The fourth embodiment of the present invention additionally provides a kind of computer readable storage medium, the computer-readable storage Media storage has computer executable instructions, which is executed by one or more control processors, example Such as, it is executed by a processor in Fig. 6, said one or multiple processors may make to execute in above method embodiment A kind of super-resolution image reconstruction method, such as the method and step S100 to S400 in first embodiment.
The apparatus embodiments described above are merely exemplary, wherein the unit as illustrated by the separation member It may or may not be physically separated, it can it is in one place, or may be distributed over multiple networks On unit.Some or all of the modules therein can be selected to achieve the purpose of the solution of this embodiment according to the actual needs.
Through the above description of the embodiments, those of ordinary skill in the art can be understood that each embodiment party Formula can add the mode of general hardware platform to realize by software, naturally it is also possible to pass through hardware.Those of ordinary skill in the art It is understood that realize above-described embodiment method in all or part of the process be can be instructed by computer program it is relevant Hardware is completed, and the program can be stored in a computer-readable storage medium, the program is when being executed, it may include Such as the process of the embodiment of above-mentioned each method.Wherein, the storage medium can be magnetic disk, CD, read-only memory (Read-Only Memory, ROM) or random access memory (Random Access Memory, RAM) etc..
It is to be illustrated to preferable implementation of the invention, but the invention is not limited to above-mentioned embodiment party above Formula, those skilled in the art can also make various equivalent variations on the premise of without prejudice to spirit of the invention or replace It changes, these equivalent deformations or replacement are all included in the scope defined by the claims of the present application.

Claims (9)

1. a kind of super-resolution image reconstruction method characterized by comprising
Obtain original image;
Bicubic interpolation is carried out to original image, enhances image pixel quantity, obtains pretreatment image;
Pretreatment image is carried out analysis of image data by the sparse depth convolutional neural networks constrained containing pond manifold to gather Class and reconstruct;
Export reconstructed image.
2. a kind of super-resolution image reconstruction method according to claim 1, which is characterized in that the acquisition original graph Picture, comprising:
To monitoring video video, program is write with Matlab, shot operation is carried out to video by frame to video recording, obtain original image Data set.
3. a kind of super-resolution image reconstruction method according to claim 1, which is characterized in that the bicubic interpolation meter Calculation is related to 16 pixels, and calculation formula is as follows:
Wherein, (i ', j ') indicate decimal partial pixel coordinate in original image, dx, dy are the coordinate of x-axis, y-axis direction, F respectively (i ', j ') it indicates to calculate by the sum of weight convolution between 16 pixels nearest from each pixel coordinate in original image New pixel value, R (x) indicates interpolation expression, and calculation formula is as follows:
4. a kind of super-resolution image reconstruction method according to claim 1, which is characterized in that described by pretreatment image Analysis of image data cluster and reconstruct are carried out by the sparse depth convolutional neural networks constrained containing pond manifold, comprising:
Pretreatment image samples, in convolutional layer by the pondization that mode one and mode two carry out manifold function constraint to image data Extraction, data jump connection and the convolutional neural networks rarefaction of characteristic pattern respectively obtain image A and image B, image A and figure Reconstructed image is obtained as B is combined convolution operation again.
5. a kind of super-resolution image reconstruction method according to claim 4, which is characterized in that the mode one, comprising:
Pretreatment image passes through the pond layer with manifold constraint, the connection between down-sampling copy feature diagram data with rearrange New characteristic pattern is obtained, the interpolation characteristic pattern of bicubic interpolation generation is ignored, is input to by 10 layers of convolution and jump connection frame In structure, the Feature Mapping of generation is then input to next layer of convolutional layer, obtaining vector dimension is (H/2, W/2,4), then right Convolutional neural networks carry out rarefaction, are upsampled to target resolution finally by sub-pixel amplifying operation is carried out to obtained vector Rate generates the image A having a size of (H, W).
6. a kind of super-resolution image reconstruction method according to claim 4, which is characterized in that the mode two, comprising:
Pretreatment image is first inputted to the convolutional layer with 64 filters, generates the spy that Output Size is (H, W, 64) Sign figure is then inputted into 10 layers of convolution and jump link architecture, and Output Size is that the characteristic pattern of (H, W, 64) passes through network After rarefaction, by the convolutional layer of single filter, image B is generated.
7. a kind of super-resolution image reconstruction system characterized by comprising
Image acquisition unit, for obtaining original image;
Image data enhancement unit enhances image pixel quantity, is pre-processed for carrying out bicubic interpolation to original image Image;
Image analysis cluster and reconfiguration unit, for rolling up pretreatment image by the sparse depth constrained containing pond manifold Product neural network carries out analysis of image data cluster and reconstruct;
Image output unit, for exporting reconstructed image.
8. a kind of super-resolution image reconstruction device characterized by comprising
At least one processor;And
The memory being connect at least one described processor communication;Wherein,
The memory is stored with the instruction that can be executed by least one described processor, and described instruction is by described at least one It manages device to execute, so that at least one described processor is able to carry out as the method according to claim 1 to 6.
9. a kind of computer readable storage medium, which is characterized in that the computer-readable recording medium storage has computer can It executes instruction, the computer executable instructions are for making computer execute as the method according to claim 1 to 6.
CN201910409645.3A 2019-05-16 2019-05-16 Super-resolution image reconstruction method, system and device thereof, and storage medium Active CN110246084B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910409645.3A CN110246084B (en) 2019-05-16 2019-05-16 Super-resolution image reconstruction method, system and device thereof, and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910409645.3A CN110246084B (en) 2019-05-16 2019-05-16 Super-resolution image reconstruction method, system and device thereof, and storage medium

Publications (2)

Publication Number Publication Date
CN110246084A true CN110246084A (en) 2019-09-17
CN110246084B CN110246084B (en) 2023-03-31

Family

ID=67884554

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910409645.3A Active CN110246084B (en) 2019-05-16 2019-05-16 Super-resolution image reconstruction method, system and device thereof, and storage medium

Country Status (1)

Country Link
CN (1) CN110246084B (en)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110942425A (en) * 2019-11-26 2020-03-31 贵州师范学院 Reconstruction method and reconstruction system of super-resolution image and electronic equipment
CN111222446A (en) * 2019-12-31 2020-06-02 Oppo广东移动通信有限公司 Face recognition method, face recognition device and mobile terminal
CN112581363A (en) * 2019-09-29 2021-03-30 北京金山云网络技术有限公司 Image super-resolution reconstruction method and device, electronic equipment and storage medium
CN113409192A (en) * 2021-06-17 2021-09-17 Oppo广东移动通信有限公司 Super-resolution chip, super-resolution algorithm updating method and electronic equipment
CN113675850A (en) * 2021-10-25 2021-11-19 山东大学 Power grid information rapid and accurate sensing method based on nonlinear robust estimation
CN114170619A (en) * 2021-10-18 2022-03-11 中标慧安信息技术股份有限公司 Data checking method and system based on edge calculation
CN115546858A (en) * 2022-08-15 2022-12-30 荣耀终端有限公司 Face image processing method and electronic equipment
WO2023274391A1 (en) * 2021-07-01 2023-01-05 Beijing Bytedance Network Technology Co., Ltd. Super resolution downsampling

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106709875A (en) * 2016-12-30 2017-05-24 北京工业大学 Compressed low-resolution image restoration method based on combined deep network
CN107480726A (en) * 2017-08-25 2017-12-15 电子科技大学 A kind of Scene Semantics dividing method based on full convolution and shot and long term mnemon
CN107507134A (en) * 2017-09-21 2017-12-22 大连理工大学 Super-resolution method based on convolutional neural networks
US20180137603A1 (en) * 2016-11-07 2018-05-17 Umbo Cv Inc. Method and system for providing high resolution image through super-resolution reconstruction
US20180158177A1 (en) * 2016-12-07 2018-06-07 Idemia Identity & Security France System for processing images
US20180262683A1 (en) * 2017-03-10 2018-09-13 Gopro, Inc. Image Quality Assessment
US20190075301A1 (en) * 2017-09-01 2019-03-07 Apple Inc. Machine learning video processing systems and methods
CN109461157A (en) * 2018-10-19 2019-03-12 苏州大学 Image, semantic dividing method based on multi-stage characteristics fusion and Gauss conditions random field

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180137603A1 (en) * 2016-11-07 2018-05-17 Umbo Cv Inc. Method and system for providing high resolution image through super-resolution reconstruction
US20180158177A1 (en) * 2016-12-07 2018-06-07 Idemia Identity & Security France System for processing images
CN106709875A (en) * 2016-12-30 2017-05-24 北京工业大学 Compressed low-resolution image restoration method based on combined deep network
US20180262683A1 (en) * 2017-03-10 2018-09-13 Gopro, Inc. Image Quality Assessment
CN107480726A (en) * 2017-08-25 2017-12-15 电子科技大学 A kind of Scene Semantics dividing method based on full convolution and shot and long term mnemon
US20190075301A1 (en) * 2017-09-01 2019-03-07 Apple Inc. Machine learning video processing systems and methods
CN107507134A (en) * 2017-09-21 2017-12-22 大连理工大学 Super-resolution method based on convolutional neural networks
CN109461157A (en) * 2018-10-19 2019-03-12 苏州大学 Image, semantic dividing method based on multi-stage characteristics fusion and Gauss conditions random field

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
李现国等: "基于中间层监督卷积神经网络的图像超分辨率重建", 《中国图象图形学报》 *

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112581363A (en) * 2019-09-29 2021-03-30 北京金山云网络技术有限公司 Image super-resolution reconstruction method and device, electronic equipment and storage medium
CN110942425A (en) * 2019-11-26 2020-03-31 贵州师范学院 Reconstruction method and reconstruction system of super-resolution image and electronic equipment
CN111222446A (en) * 2019-12-31 2020-06-02 Oppo广东移动通信有限公司 Face recognition method, face recognition device and mobile terminal
CN113409192A (en) * 2021-06-17 2021-09-17 Oppo广东移动通信有限公司 Super-resolution chip, super-resolution algorithm updating method and electronic equipment
WO2023274391A1 (en) * 2021-07-01 2023-01-05 Beijing Bytedance Network Technology Co., Ltd. Super resolution downsampling
CN114170619A (en) * 2021-10-18 2022-03-11 中标慧安信息技术股份有限公司 Data checking method and system based on edge calculation
CN114170619B (en) * 2021-10-18 2022-08-19 中标慧安信息技术股份有限公司 Data checking method and system based on edge calculation
CN113675850A (en) * 2021-10-25 2021-11-19 山东大学 Power grid information rapid and accurate sensing method based on nonlinear robust estimation
CN115546858A (en) * 2022-08-15 2022-12-30 荣耀终端有限公司 Face image processing method and electronic equipment
CN115546858B (en) * 2022-08-15 2023-08-25 荣耀终端有限公司 Face image processing method and electronic equipment

Also Published As

Publication number Publication date
CN110246084B (en) 2023-03-31

Similar Documents

Publication Publication Date Title
CN110246084A (en) A kind of super-resolution image reconstruction method and its system, device, storage medium
Jiang et al. Edge-enhanced GAN for remote sensing image superresolution
Wang et al. Ultra-dense GAN for satellite imagery super-resolution
CN107154023B (en) Based on the face super-resolution reconstruction method for generating confrontation network and sub-pix convolution
CN110322416A (en) Image processing method, device and computer readable storage medium
CN112991173A (en) Single-frame image super-resolution reconstruction method based on dual-channel feature migration network
Yang et al. Image super-resolution based on deep neural network of multiple attention mechanism
CN111951164B (en) Image super-resolution reconstruction network structure and image reconstruction effect analysis method
CN110599502B (en) Skin lesion segmentation method based on deep learning
CN113139585B (en) Infrared and visible light image fusion method based on unified multi-scale dense connection network
Guo et al. Towards efficient motion-blurred public security video super-resolution based on back-projection networks
CN115393191A (en) Method, device and equipment for reconstructing super-resolution of lightweight remote sensing image
Shen et al. AFFNet: attention mechanism network based on fusion feature for image cloud removal
Pan et al. FDPPGAN: remote sensing image fusion based on deep perceptual patchGAN
CN116739899A (en) Image super-resolution reconstruction method based on SAUGAN network
Wang et al. PACCDU: Pyramid attention cross-convolutional dual UNet for infrared and visible image fusion
CN113034371B (en) Infrared and visible light image fusion method based on feature embedding
Kim et al. Progressive contextual aggregation empowered by pixel-wise confidence scoring for image inpainting
CN111428809B (en) Crowd counting method based on spatial information fusion and convolutional neural network
CN116523985B (en) Structure and texture feature guided double-encoder image restoration method
Sun et al. A rapid and accurate infrared image super-resolution method based on zoom mechanism
Dong et al. Remote sensing image super-resolution via enhanced back-projection networks
CN116631064A (en) 3D human body posture estimation method based on complementary enhancement of key points and grid vertexes
Peng Super-resolution reconstruction using multiconnection deep residual network combined an improved loss function for single-frame image
Chen et al. Adaptive visual field multi-scale generative adversarial networks image inpainting base on coordinate-attention

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant