CN105072373B - Video super-resolution method and system based on bidirectional circulating convolutional network - Google Patents
Video super-resolution method and system based on bidirectional circulating convolutional network Download PDFInfo
- Publication number
- CN105072373B CN105072373B CN201510540560.0A CN201510540560A CN105072373B CN 105072373 B CN105072373 B CN 105072373B CN 201510540560 A CN201510540560 A CN 201510540560A CN 105072373 B CN105072373 B CN 105072373B
- Authority
- CN
- China
- Prior art keywords
- network
- layer
- bidirectional circulating
- convolution
- video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000002457 bidirectional effect Effects 0.000 title claims abstract description 77
- 238000000034 method Methods 0.000 title claims abstract description 34
- 125000004122 cyclic group Chemical group 0.000 claims abstract description 13
- 238000012549 training Methods 0.000 claims abstract description 10
- 230000000694 effects Effects 0.000 claims description 10
- 238000005457 optimization Methods 0.000 claims description 6
- 238000012545 processing Methods 0.000 claims description 6
- 230000001965 increasing effect Effects 0.000 claims description 4
- 238000011478 gradient descent method Methods 0.000 claims 1
- 230000006870 function Effects 0.000 description 9
- 238000012360 testing method Methods 0.000 description 9
- 230000007935 neutral effect Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 150000001875 compounds Chemical class 0.000 description 1
- 230000001143 conditioned effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 238000013442 quality metrics Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Landscapes
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The invention discloses a kind of video super-resolution method based on bidirectional circulating convolutional network, including：Establish bidirectional circulating network, including forward direction circulation sub-network sequentially in time and backward circulation sub-network, sub-network is bottom-up includes a list entries layer for each circulation, two implicit sequence layers and an output sequence layer, each sequence layer includes multiple states, corresponding to frame of video at different moments；These states, including feedforward convolution, cyclic convolution and condition convolution are connected with three kinds of convolution operations, to obtain bidirectional circulating convolutional network；Training video is sent in the bidirectional circulating convolutional network established, using stochastic gradient descent algorithm come the mean square error between high-resolution video that minimize prediction and real, so as to iteratively optimize the weight of the network, and obtain final bidirectional circulating convolutional network；The low-resolution video sequence pending to the final bidirectional circulating convolutional network mode input, obtain corresponding super-resolution result.
Description
Technical field
It is more particularly to a kind of based on bidirectional circulating convolutional network the present invention relates to computer vision and machine learning field
Video super-resolution method and system.
Background technology
With emerging in large numbers for a large amount of high definition playback equipments in recent years, how by height that low resolution Video Quality Metric is broadcasting preferably
Resolution video, i.e. super-resolution technique, it is increasingly becoming the hot research problem of computer vision field.
The work of super-resolution at present can probably be divided into two classes：1) single image super-resolution, its assume all frame of video it
Between be it is separate, then individually to each frame of video carry out super-resolution.This method ignores in video sequence one
Time dependence between important feature, i.e. frame of video.2) multiframe super-resolution, this method is during super-resolution
The time dependence of video is considered, such as the movable information in video is estimated using optic flow technique.
But in actual applications, this modeling to time dependence usually requires very high calculation cost, very big
The application of multiframe super-resolution technique is limited in degree.
The content of the invention
The invention provides a kind of video super-resolution method and system based on bidirectional circulating convolutional network.In order to effective
Time dependence characteristic in video is learnt, the present invention two-way circulation is introduced on the basis of conventional recycle neutral net
With condition convolution operation.
According to an aspect of the present invention, the present invention proposes a kind of video super-resolution based on bidirectional circulating convolutional network
Method, this method comprise the following steps：
Step 1, establish bidirectional circulating network, including forward direction circulation sub-network sequentially in time and one it is backward
Sub-network is circulated, it is bottom-up to include a list entries layer, two implicit sequence layers and one in each circulation sub-network
Individual output sequence layer, each of which sequence layer includes multiple states, corresponding to frame of video at different moments；
Step 2, with three kinds of convolution operations connect these states, three kinds of convolution operations include feedforward convolution, circulation
Convolution and condition convolution, to obtain bidirectional circulating convolutional network；
Step 3, send training video in the bidirectional circulating convolutional network established, using stochastic gradient descent algorithm come
The object function of bidirectional circulating convolutional network is minimized, so as to iteratively optimizing the weight of the network, and is obtained final two-way
Cyclic convolution network；And
Step 4, to the pending low-resolution video sequence of the final bidirectional circulating convolutional network mode input, obtain
To corresponding super-resolution result.
According to another aspect of the present invention, the invention also provides a kind of video oversubscription based on bidirectional circulating convolutional network
Resolution system, the system include：
Network establishes module, for establishing bidirectional circulating network, including forward direction circulation subnet sequentially in time
Network and a backward circulation sub-network, in each circulation sub-network, bottom-up to include a list entries layer, two hidden
Containing sequence layer and an output sequence layer, each of which sequence layer includes multiple states, corresponding to frame of video at different moments；
Link block, for connecting these states with three kinds of convolution operations, three kinds of convolution operations include feedforward and rolled up
Product, cyclic convolution and condition convolution, to obtain bidirectional circulating convolutional network；
Optimization module, for sending training video in the bidirectional circulating convolutional network established, using under stochastic gradient
Algorithm is dropped to minimize the object function of bidirectional circulating convolutional network, so as to iteratively optimize the weight of the network, and is obtained most
Whole bidirectional circulating convolutional network；And
Video processing module, for the pending low resolution of the final bidirectional circulating convolutional network mode input
Video sequence, obtain corresponding super-resolution result.
The present invention is made by introducing two-way circulation and condition convolution operation on the basis of conventional recycle neutral net
Model is more suitable for handling the video super-resolution problem containing sequential dependence relation.The model of the present invention can be largely
Lift video super-resolution effect, while two orders of magnitude faster than other multiframe super-resolution methods in speed.
Brief description of the drawings
Fig. 1 is the video super-resolution flow chart of the invention based on bidirectional circulating convolutional network.
Fig. 2 is that one embodiment of the invention solves the problems, such as the illustraton of model of video super-resolution.
Fig. 3 is the partial, detailed view in Fig. 2.
Fig. 4 is the block diagram of the video super-resolution system of the invention based on bidirectional circulating convolutional network.
Fig. 5 is the comparison figure of video super-resolution effect is best now certain methods and experimental result of the present invention.
Fig. 6 and Fig. 7 is the visualization knot of video super-resolution effect is best now certain methods and experimental result of the present invention
Fruit.
Fig. 8 is PSNR and the testing time of the certain methods that video super-resolution effect is best now and experimental result of the present invention
Comparison figure.
Embodiment
For the object, technical solutions and advantages of the present invention are more clearly understood, below in conjunction with specific embodiment, and reference
Accompanying drawing, the present invention is described in more detail.
According to an aspect of the invention, it is proposed that a kind of video super-resolution method based on bidirectional circulating convolutional network, energy
Enough it is widely used in the video super-resolution problem comprising compound movement situation.
Fig. 1 shows the flow chart of the video super-resolution method proposed by the present invention based on bidirectional circulating convolutional network.
Fig. 2 shows the bidirectional circulating convolutional network structure used in one embodiment of the invention.
As shown in figure 1, the video super-resolution method based on bidirectional circulating convolutional network comprises the following steps：
Step 1, bidirectional circulating network is established, as shown in Fig. 2 circulating sub-network including a forward direction sequentially in time
With a backward circulation sub-network.It is bottom-up to include a list entries layer in each oriented cycles sub-network, two
Implicit sequence layer and an output sequence layer, each of which sequence layer include multiple states, the video carved corresponding to different time
Frame；
Wherein, the list entries layer of the bidirectional circulating network is the sequence of frames of video of original low-resolution.In the present invention
In one embodiment, it is desirable to which all frame of video keep identical sizes, such as same length and width, but not fixed video frame
Quantity, it can be any amount；Output sequence layer is used to represent the high-resolution video frame sequence after super-resolution, defeated here
It is one-to-one with list entries to go out sequence；Implicit sequence layer then represents that some intermediate data represent that is, frame of video is from low point
Resolution is to high-resolution intermediate state.Each sequence layer state needs to set corresponding port number, such as RGB lattice
The port number of the video of formula, input layer and output layer state is arranged to 3.The port number of implicit layer state can be adjusted as needed
Save to cause the effect of the model optimal.For example, hidden layer stator channel number can be adjusted as follows：In the enough feelings of data volume
Under condition, port number more at most final result is better.But port number excessively can greatly increase computation complexity.Pass through experiment
It was found that when the port number of the first implicit layer state and the second implicit layer state is respectively 64 and 32, preferable reality can be obtained
Test result and relatively low computation complexity.
Forward direction circulates sub-network and backward circulation sub-network shares list entries layer and output sequence layer, i.e. two sub-networks
Identical low resolution input video frame is received, then associated prediction high resolution output frame of video.But each implicit sequence layer
It is not shared, and for some implies sequence layer, the state at current time is only by previous in forward direction circulation sub-network
The state at moment is influenceed, and the state for circulating current time in sub-network backward is only influenceed by the state of later moment in time, with this
Analogize on time shaft.
Step 2, include feedforward convolution, cyclic convolution and condition convolution operation with three kinds of convolution, to connect each sequence layer
State, wherein the connection of each convolution needs to set the quantity and yardstick of corresponding wave filter；
The yardstick of the wave filter can be adjusted such that the effect of the model is optimal, and wherein regulative mode is as follows：It is first
The yardstick of first selection of small is tested, and then obtains corresponding experimental result by being continuously increased yardstick, and is chosen preferably real
Test the yardstick the most final of filter scales corresponding to result.The quantity of wave filter is equal to the passage of two states of its connection
Product.The wave filter has weight, for obtaining other state values according to current state value.The weight of the wave filter is network
In parameter set in advance, similar to traditional neural network model, network can obtain institute according to object function autonomous learning
State weight.
Fig. 3 shows the preceding list entries layer to circulation sub-network and the first implicit sequence layer in moment t-1 and t state
Specific convolution situation.As shown in Fig. 2 bidirectional circulating convolutional network include one four layers before to circulation sub-network and one four layers
Backward circulation sub-network, all nodes per a line represent a sequence layer, and each node represents a moment in this layer
State.Bottom and top represent the list entries layers of two circulation sub-networks, input be all be low-resolution videoIts
Include T frame of videoI=1,2 ..., T are wherein the length or width of frame of video, and C is each frame of video
Number of active lanes.Intermediate layer is network output layer, and output is the high-resolution video predictedIt includes T frame of videoFirst implicit sequence layer of forward direction circulation sub-networkIt includes T characteristic patternFirst implicit sequence layer of backward circulation sub-networkIt includes T characteristic patternSecond implicit sequence layer of forward direction circulation sub-networkIt includes T characteristic patternSecond implicit sequence layer of backward circulation sub-networkIt includes T characteristic pattern
As shown in figure 3, the state in all sequences layer is all attached by three kinds of convolution operations：1) feedover convolution pair
Visual relevance between low resolution input video frame and its corresponding high-resolution video frame is modeled；2) cyclic convolution
Two neighboring state in same sequence layer is connected to, to learn the time dependence in video；3) condition convolution is connected to previous
State and the latter moment in the list entries layer at individual moment imply the state in sequence layer, in order to which enhancing is visually
Time dependence.Thus, the state computation that implying the state in sequence layer and output sequence layer can be closed on by them obtains：
Wherein, * represents convolution operation, and λ (x)=max (0, x) represents a nonlinear transformation,With
(orWith) the preceding wave filter that the convolution that feedovered in sub-network is circulated to (or reverse) is represented, size is respectivelyWithWherein1 HeBefore representing corresponding
Present the yardstick of convolution filter.With(orWith) represent that preceding circulated into (or reverse) circulation sub-network is rolled up
Long-pending wave filter, size are respectively n1×1×1×n1And n2×1×1×n2, wherein the yardstick for having cyclic convolution wave filter is all
1。With(orWith) represent before to (or reversely) circulation sub-network conditional convolution
Wave filter, size are respectivelyn1×1×1×n2WithWherein1
WithRepresent the yardstick of corresponding condition convolution filter. With(orWith) represent before to (or
Reversely) circulate offset parameter in sub-network.
Step 3, send training video in the bidirectional circulating convolutional network established, using stochastic gradient descent algorithm come
Minimize object function (such as the mean square error between real high-resolution video of prediction of bidirectional circulating convolutional network
Difference), so as to iteratively optimize the weight of the network, and obtain final bidirectional circulating convolutional network.
Specifically, for example, network inputs are low resolution video framesI=1,2 ..., T, net
Network output is the high-resolution video frame predictedTheir mean square error is：
For learning network parameter, the mean square error is minimized using stochastic gradient descent algorithm.Object function J on
All parametersGradientIt can be precisely calculated out by neutral net back-propagation algorithm.Obtain gradient it
Afterwards, parameter is adjusted in an iterative manner：
Wherein, ∈ represents a constant learning rate.
In fact, other more complicated optimized algorithms such as L-BFGS can also be used, but stochastic gradient algorithm's ratio
It is relatively simple, and preferable result can be obtained.
Step 4, to the pending low-resolution video sequence of the bidirectional circulating convolutional network mode input trained,
The value of output sequence layer corresponding to obtaining is the result after super-resolution.
According to another aspect of the present invention, the invention also provides a kind of video oversubscription based on bidirectional circulating convolutional network
Resolution system, the system include：Network establishes module 401, link block 402, optimization module 403 and video processing module
404。
Network establishes module 401 and is used to establish bidirectional circulating network, including forward direction circulation sequentially in time
The backward circulation sub-network of network and one, it is bottom-up to include a list entries layer in each circulation sub-network, two
Implicit sequence layer and an output sequence layer, each of which sequence layer includes multiple states, corresponding to frame of video at different moments.
Link block 402 connects these states with three kinds of convolution operations, and three kinds of convolution operations include feedforward convolution,
Cyclic convolution and condition convolution, to obtain bidirectional circulating convolutional network.
Optimization module 403 is used to optimize the bidirectional circulating convolutional network.Send training video to the bidirectional circulating established
In convolutional network, the object function of bidirectional circulating convolutional network is minimized using stochastic gradient descent algorithm, so as to iteratively
Optimize the weight of the network, and obtain final bidirectional circulating convolutional network.
The final bidirectional circulating convolutional network that video processing module 404 is used to export using optimization module 403 is come to low
Resolution video carries out super-resolution.At the video processing module 404, to the final bidirectional circulating convolutional network model
Pending low-resolution video sequence is inputted, obtains corresponding super-resolution result.
According to an embodiment of the invention, the forward direction circulation sub-network and the backward circulation sub-network include one respectively
List entries layer, two implicit sequence layers and an output sequence layer.With volume between the state of the bidirectional circulating convolutional network
Long-pending mode is attached, wherein：The state of the flanking sequence layer belonged under synchronization is attached with feedforward convolution；Belong to
Two states of the same implicit sequence layer of adjacent moment are attached with cyclic convolution；Sub-network is circulated for forward direction, currently
The state of moment list entries layer, the first implicit sequence layer and the second implicit sequence layer implies sequence with the first of previous moment respectively
Row layer, the state of the second implicit sequence layer and output sequence layer are attached with condition convolution；And for backward circulation subnet
Network, the state of current time list entries layer, the first implicit sequence layer and the second implicit sequence layer respectively with later moment in time the
One implicit sequence layer, the state of the second implicit sequence layer and output sequence layer are attached with condition convolution.
According to an embodiment of the invention, the input layer of the bidirectional circulating convolutional network is low resolution video frame sequence,
Output layer is the high-resolution video frame sequence of prediction, and the bidirectional circulating convolutional network has network weight, with according to current
Layer state obtains other layer states.
According to an embodiment of the invention, the object function of the bidirectional circulating convolutional network is prediction and real high score
Mean square error between resolution video.
According to an embodiment of the invention, each convolution connects through wave filter realization.The yardstick of wave filter be conditioned so that
Obtain that the effect of the model is optimal, and wherein regulative mode is as follows：The yardstick of selection of small first is tested, then by constantly increasing
Yardstick is added to obtain corresponding experimental result, and filter scales yardstick the most final corresponding to experimental result needed for selection.Filter
The quantity of ripple device is equal to the product of the passage of two states of its connection.The wave filter has weight, bidirectional circulating convolutional network
Learnt to obtain the weight according to object function.
In order to describe the embodiment of the present invention in detail, super-resolution is carried out to multiple videos here.Training set bag
The video of 25 YUY being widely used in the method for other video super-resolutions forms is included, and test set includes 5 videos point
It is not：Dancing, Flag, Fan, Treadmill and Turbine.All include complicated motor pattern, motion in these videos
Shake and motion blur.
It is that model is trained in a manner of video cube body to increase training data.Specifically, from 25 instructions
Practice and general 41000 video cube bodies are extracted in video, the dimension of each cube spatially is 32 × 32, temporal
Dimension is 10.It should be noted that it is not required to be tested in a manner of video cube body, because volume used in the present invention
Product operation goes for the test video of arbitrary size.
Comprise the following steps that：
Step S1, for 25 training videos, after 0 average and the processing of 1 variance is carried out, with spatial mesh size 14 and time step
8 are grown to extract the video cube body of 32 × 32 × 10 sizes, sum about 41000.
Step S2, using one four layers of bidirectional circulating convolutional network, its list entries layer, two implicit sequence layers and defeated
Go out sequence layer and include 3,64,32 and 3 passages, i.e. c=3, n respectively1=64, n2=32.The convolution filter yardstick that feedovers is as follows：
Step S3, optimizing the object function J of network using gradient descent algorithm, optimization is to carry out in an iterative manner, this
In set the maximum iteration to be that 300 can ensure to restrain.
Step S6, the low-resolution video of test is input in the model trained, and super-resolution is exported in output layer
Result after rate, the result of output is a video sequence.
As shown in Figure 5 (wherein BRCN represents the present invention), Measure Indexes are signal-to-noise ratio to quantitative experimental result
(signal-to-noise ratio, PSNR) and testing time (Time).Video super-resolution effect now is compared in the table
Best certain methods.By comparing, it can be found that the method for the present invention achieves best experimental result, and in test
Between on close to currently most fast single image super-resolution method ANR.Qualitatively visualized in addition, giving some here
As a result, as shown in Figures 6 and 7.As can be seen that the present invention can recover more image details relative to other method.In addition,
Here the comparison of PSNR and testing time are given, as shown in Figure 8, it can be seen that the present invention is significantly excellent in speed and performance
In other multiframe super-resolution methods such as 3DSKR, and better than current best single-image super-resolution method in performance
ANR。
Particular embodiments described above, the purpose of the present invention, technical scheme and beneficial effect are carried out further in detail
Describe in detail it is bright, should be understood that the foregoing is only the present invention specific embodiment, be not intended to limit the invention, it is all
Within the spirit and principles in the present invention, any modification, equivalent substitution and improvements done etc., it should be included in the guarantor of the present invention
Within the scope of shield.
Claims (14)
1. a kind of video super-resolution method based on bidirectional circulating convolutional network, it is characterised in that this method includes following step
Suddenly：
Step 1, establish bidirectional circulating network, including forward direction circulation sub-network sequentially in time and a backward circulation
Sub-network, it is bottom-up to include a list entries layer in each circulation sub-network, two implicit sequence layers and one it is defeated
Go out sequence layer, each of which sequence layer includes multiple states, corresponding to frame of video at different moments；
Step 2, with three kinds of convolution operations connect these states, three kinds of convolution operations include feedforward convolution, cyclic convolution
With condition convolution, to obtain bidirectional circulating convolutional network；
Step 3, send training video in the bidirectional circulating convolutional network established, using stochastic gradient descent algorithm come minimum
Change the object function of bidirectional circulating convolutional network, so as to iteratively optimize the weight of the network, and obtain final bidirectional circulating
Convolutional network model；And
Step 4, to the pending low-resolution video sequence of the final bidirectional circulating convolutional network mode input, obtain pair
The super-resolution result answered.
2. according to the method for claim 1, it is characterised in that the bidirectional circulating convolutional network includes a forward direction and circulated
Convolution sub-network and a backward cyclic convolution sub-network.
3. according to the method for claim 1, it is characterised in that the forward direction circulation sub-network and the backward circulation subnet
Network includes a list entries layer, two implicit sequence layers and an output sequence layer respectively.
4. according to the method for claim 1, it is characterised in that with convolution between the state of the bidirectional circulating convolutional network
Mode be attached, wherein：
The state of the flanking sequence layer belonged under synchronization is attached with feedforward convolution；
Two states for belonging to the same implicit sequence layer of adjacent moment are attached with cyclic convolution；
Sub-network, the shape of current time list entries layer, the first implicit sequence layer and the second implicit sequence layer are circulated for forward direction
State implies sequence layer with the first of previous moment respectively, the state condition convolution of the second implicit sequence layer and output sequence layer is entered
Row connection；And
For backward circulation sub-network, the shape of current time list entries layer, the first implicit sequence layer and the second implicit sequence layer
State implies sequence layer with the first of later moment in time respectively, the state condition convolution of the second implicit sequence layer and output sequence layer is entered
Row connection.
5. according to the method for claim 1, it is characterised in that the input layer of the bidirectional circulating convolutional network is low resolution
Rate sequence of frames of video, output layer are the high-resolution video frame sequence of prediction, and the bidirectional circulating convolutional network has network weight
Weight, to obtain other layer states according to current layer state.
6. according to the method for claim 1, it is characterised in that the object function of the bidirectional circulating convolutional network is prediction
The mean square error between real high-resolution video.
7. the method according to claim 11, wherein：
Each convolution connects through wave filter realization；
The yardstick of wave filter is adjusted so that the effect of the model is optimal, and wherein regulative mode is as follows：Selection of small first
Yardstick is tested, and then obtains corresponding experimental result by being continuously increased yardstick, and corresponding to experimental result needed for selection
Filter scales yardstick the most final；
The quantity of wave filter is equal to the product of the passage of two states of its connection；And
The wave filter has weight, and bidirectional circulating convolutional network learns to obtain the weight according to object function.
8. a kind of video super-resolution system based on bidirectional circulating convolutional network, it is characterised in that the system includes following mould
Block：
Network establishes module, for establishing bidirectional circulating network, including forward direction circulation sub-network sequentially in time and
One backward circulation sub-network, it is bottom-up to include a list entries layer, two implicit sequences in each circulation sub-network
Row layer and an output sequence layer, each of which sequence layer includes multiple states, corresponding to frame of video at different moments；
Link block, for connecting these states with three kinds of convolution operations, three kinds of convolution operations include feedforward convolution, followed
Ring convolution and condition convolution, to obtain bidirectional circulating convolutional network；
Optimization module, for sending training video in the bidirectional circulating convolutional network established, calculated using stochastic gradient descent
Method minimizes the object function of bidirectional circulating convolutional network, so as to iteratively optimize the weight of the network, and obtains final
Bidirectional circulating convolutional network model；And
Video processing module, for the pending low-resolution video of the final bidirectional circulating convolutional network mode input
Sequence, obtain corresponding super-resolution result.
9. system according to claim 8, it is characterised in that the bidirectional circulating convolutional network includes a forward direction and circulated
Convolution sub-network and a backward cyclic convolution sub-network.
10. system according to claim 8, it is characterised in that the forward direction circulation sub-network and backward circulation
Network includes a list entries layer, two implicit sequence layers and an output sequence layer respectively.
11. system according to claim 8, it is characterised in that with volume between the state of the bidirectional circulating convolutional network
Long-pending mode is attached, wherein：
The state of the flanking sequence layer belonged under synchronization is attached with feedforward convolution；
Two states for belonging to the same implicit sequence layer of adjacent moment are attached with cyclic convolution；
Sub-network, the shape of current time list entries layer, the first implicit sequence layer and the second implicit sequence layer are circulated for forward direction
State implies sequence layer with the first of previous moment respectively, the state condition convolution of the second implicit sequence layer and output sequence layer is entered
Row connection；And
For backward circulation sub-network, the shape of current time list entries layer, the first implicit sequence layer and the second implicit sequence layer
State implies sequence layer with the first of later moment in time respectively, the state condition convolution of the second implicit sequence layer and output sequence layer is entered
Row connection.
12. system according to claim 8, it is characterised in that the input layer of the bidirectional circulating convolutional network is low point
Resolution sequence of frames of video, output layer are the high-resolution video frame sequence of prediction, and the bidirectional circulating convolutional network has network
Weight, to obtain other layer states according to current layer state.
13. system according to claim 8, it is characterised in that the object function of the bidirectional circulating convolutional network is pre-
Mean square error between high-resolution video survey and real.
14. system according to claim 8, wherein：
Each convolution connects through wave filter realization；
The yardstick of wave filter is adjusted so that the effect of the model is optimal, and wherein regulative mode is as follows：Selection of small first
Yardstick is tested, and then obtains corresponding experimental result by being continuously increased yardstick, and corresponding to experimental result needed for selection
Filter scales yardstick the most final；
The quantity of wave filter is equal to the product of the passage of two states of its connection；And
The wave filter has weight, and bidirectional circulating convolutional network learns to obtain the weight according to object function.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510540560.0A CN105072373B (en) | 2015-08-28 | 2015-08-28 | Video super-resolution method and system based on bidirectional circulating convolutional network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510540560.0A CN105072373B (en) | 2015-08-28 | 2015-08-28 | Video super-resolution method and system based on bidirectional circulating convolutional network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105072373A CN105072373A (en) | 2015-11-18 |
| CN105072373B true CN105072373B (en) | 2018-03-27 |
Family
ID=54501642
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510540560.0A Active CN105072373B (en) | 2015-08-28 | 2015-08-28 | Video super-resolution method and system based on bidirectional circulating convolutional network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105072373B (en) |
Families Citing this family (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106067161A (en) * | 2016-05-24 | 2016-11-02 | 深圳市未来媒体技术研究院 | A kind of method that image is carried out super-resolution |
| GB2555431A (en) * | 2016-10-27 | 2018-05-02 | Nokia Technologies Oy | A method for analysing media content |
| KR102511059B1 (en) * | 2017-05-17 | 2023-03-17 | 삼성전자주식회사 | Super-resolution processing method for moving image and image processing apparatus therefor |
| CN107369189A (en) * | 2017-07-21 | 2017-11-21 | 成都信息工程大学 | The medical image super resolution ratio reconstruction method of feature based loss |
| CN109862299B (en) * | 2017-11-30 | 2021-08-27 | 北京大学 | Resolution processing method and device |
| CN108596334B (en) * | 2018-03-23 | 2021-01-01 | 大国创新智能科技(东莞)有限公司 | Data corresponding relation judging and generating method and system based on bidirectional deep learning |
| CN109102462B (en) * | 2018-08-01 | 2023-04-07 | 中国计量大学 | Video super-resolution reconstruction method based on deep learning |
| CN109118432B (en) * | 2018-09-26 | 2022-09-13 | 福建帝视信息科技有限公司 | Image super-resolution reconstruction method based on rapid cyclic convolution network |
| CN111784570A (en) * | 2019-04-04 | 2020-10-16 | Tcl集团股份有限公司 | Video image super-resolution reconstruction method and device |
| CN110060204B (en) * | 2019-04-29 | 2020-12-29 | 江南大学 | Single image super-resolution method based on reversible network |
| CN111369438B (en) * | 2020-02-28 | 2022-07-26 | 北京市商汤科技开发有限公司 | Image processing method and device, electronic equipment and storage medium |
| CN112200720B (en) * | 2020-09-29 | 2023-08-08 | 中科方寸知微(南京)科技有限公司 | Super-resolution image reconstruction method and system based on filter fusion |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1811453A1 (en) * | 2004-10-29 | 2007-07-25 | Tokyo Institute of Technology | Method for speeding up super resolution processing |
| CN102073866A (en) * | 2010-12-27 | 2011-05-25 | 清华大学 | Video super resolution method by utilizing space-time Markov random field model |
| CN102682442A (en) * | 2012-04-28 | 2012-09-19 | 东南大学 | Motion target super-resolution image reconstruction method based on optical flow field |
| EP2615579A1 (en) * | 2012-01-12 | 2013-07-17 | Thomson Licensing | Method and device for generating a super-resolution version of a low resolution input data structure |
| CN103295197A (en) * | 2013-05-21 | 2013-09-11 | 西安电子科技大学 | Image super-resolution rebuilding method based on dictionary learning and bilateral holomorphy |
-
2015
- 2015-08-28 CN CN201510540560.0A patent/CN105072373B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1811453A1 (en) * | 2004-10-29 | 2007-07-25 | Tokyo Institute of Technology | Method for speeding up super resolution processing |
| CN102073866A (en) * | 2010-12-27 | 2011-05-25 | 清华大学 | Video super resolution method by utilizing space-time Markov random field model |
| EP2615579A1 (en) * | 2012-01-12 | 2013-07-17 | Thomson Licensing | Method and device for generating a super-resolution version of a low resolution input data structure |
| CN102682442A (en) * | 2012-04-28 | 2012-09-19 | 东南大学 | Motion target super-resolution image reconstruction method based on optical flow field |
| CN103295197A (en) * | 2013-05-21 | 2013-09-11 | 西安电子科技大学 | Image super-resolution rebuilding method based on dictionary learning and bilateral holomorphy |
Non-Patent Citations (2)
| Title |
|---|
| Learning a Deep Convolutional Network for Image Super-Resolution;Chao Dong et.al.;《European Conference on Computer Vision》;20140912;全文 * |
| Video Super-Resolution Algorithm Using Bi-Directional Overlapped Block Motion Compensation and On-the-Fly Dictionary Training;Byung Cheol Song et.al.;《IEEE Transations on Ciruits and Systems for Video Technology》;20110331;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105072373A (en) | 2015-11-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105072373B (en) | Video super-resolution method and system based on bidirectional circulating convolutional network | |
| CN108734661B (en) | High-resolution image prediction method for constructing loss function based on image texture information | |
| CN106157319B (en) | The conspicuousness detection method in region and Pixel-level fusion based on convolutional neural networks | |
| Liang et al. | A fast and accurate online sequential learning algorithm for feedforward networks | |
| CN105657402B (en) | A kind of depth map restoration methods | |
| CN105095862B (en) | A kind of human motion recognition method based on depth convolution condition random field | |
| CN111046962A (en) | Sparse attention-based feature visualization method and system for convolutional neural network model | |
| CN106204499A (en) | Single image rain removing method based on convolutional neural networks | |
| CN111444878A (en) | Video classification method and device and computer readable storage medium | |
| CN110097178A (en) | It is a kind of paid attention to based on entropy neural network model compression and accelerated method | |
| CN108111860A (en) | Video sequence lost frames prediction restoration methods based on depth residual error network | |
| CN107123094B (en) | Video denoising method mixing Poisson, Gaussian and impulse noise | |
| CN112489164B (en) | Image coloring method based on improved depth separable convolutional neural network | |
| CN110569875A (en) | deep neural network target detection method based on feature multiplexing | |
| CN108537747A (en) | A kind of image repair method based on the convolutional neural networks with symmetrical parallel link | |
| CN109191511A (en) | A kind of binocular solid matching process based on convolutional neural networks | |
| CN111833400B (en) | Camera pose positioning method | |
| CN110378250A (en) | Training method, device and the terminal device of neural network for scene cognition | |
| CN104408697A (en) | Image super-resolution reconstruction method based on genetic algorithm and regular prior model | |
| KR101563569B1 (en) | Learnable Dynamic Visual Image Pattern Recognition System and Method | |
| CN110956201A (en) | Image distortion type classification method based on convolutional neural network | |
| CN106778550A (en) | A kind of method and apparatus of Face datection | |
| Camuñas-Mesa et al. | Low-power hardware implementation of SNN with decision block for recognition tasks | |
| EP4024343A1 (en) | Viewpoint image processing method and related device | |
| CN110110651B (en) | Method for identifying behaviors in video based on space-time importance and 3D CNN |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |