CN105791980B - Films and television programs renovation method based on increase resolution - Google Patents
Films and television programs renovation method based on increase resolution Download PDFInfo
- Publication number
- CN105791980B CN105791980B CN201610109909.XA CN201610109909A CN105791980B CN 105791980 B CN105791980 B CN 105791980B CN 201610109909 A CN201610109909 A CN 201610109909A CN 105791980 B CN105791980 B CN 105791980B
- Authority
- CN
- China
- Prior art keywords
- expert
- resolution video
- video frame
- parameter
- gate function
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 27
- 238000009418 renovation Methods 0.000 title claims abstract description 15
- 238000013507 mapping Methods 0.000 claims abstract description 35
- 239000000203 mixture Substances 0.000 claims abstract description 33
- 238000012549 training Methods 0.000 claims description 19
- 230000003321 amplification Effects 0.000 claims description 4
- 238000003199 nucleic acid amplification method Methods 0.000 claims description 4
- 238000005457 optimization Methods 0.000 claims description 4
- 238000005070 sampling Methods 0.000 claims description 4
- 230000009466 transformation Effects 0.000 claims description 4
- 230000000903 blocking effect Effects 0.000 claims description 3
- 239000011159 matrix material Substances 0.000 claims description 3
- 230000017105 transposition Effects 0.000 claims description 3
- 238000012886 linear function Methods 0.000 claims description 2
- 244000182264 Lucuma nervosa Species 0.000 claims 1
- 238000012804 iterative process Methods 0.000 claims 1
- 230000000694 effects Effects 0.000 abstract description 5
- 230000008569 process Effects 0.000 abstract description 3
- 238000000638 solvent extraction Methods 0.000 abstract description 2
- 230000006870 function Effects 0.000 description 51
- 238000012545 processing Methods 0.000 description 10
- 230000000007 visual effect Effects 0.000 description 6
- 230000003044 adaptive effect Effects 0.000 description 2
- 230000015556 catabolic process Effects 0.000 description 2
- 238000006731 degradation reaction Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000002708 enhancing effect Effects 0.000 description 2
- 238000012552 review Methods 0.000 description 2
- 230000001568 sexual effect Effects 0.000 description 2
- 241000220479 Acacia Species 0.000 description 1
- 235000010643 Leucaena leucocephala Nutrition 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 239000000969 carrier Substances 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000006378 damage Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000004438 eyesight Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440263—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by altering the spatial resolution, e.g. for displaying on a connected PDA
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440236—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Image Processing (AREA)
- Television Systems (AREA)
Abstract
Films and television programs of the present invention for lower resolution ratio, lower clarity, it is proposed that a kind of films and television programs renovation method based on increase resolution, concrete scheme are：First, the resolution ratio and target resolution for obtaining original video, calculate scaling；Secondly, input video is divided into set of frames by certain partitioning scheme；Then, it is converted according to pre-stored mapping relations, obtains high-resolution video frame；Finally, high-resolution video frame is combined into high-resolution video wherein, pre-stored mapping relations are obtained based on Mixture of expert model learning, the process in a computer offline complete the method for the invention have many advantages, such as adaptively it is good, speed is fast, effect is good, expansible.
Description
Technical field
The invention belongs to computer visions and image processing field, are related to films and television programs renovation method, and in particular to a kind of
Films and television programs renovation method based on increase resolution and system.
Background technology
With video acquisition, transmission, storage, the development of display technology, films and television programs constantly develop towards high-resolution.People
Appreciate video taste it is also higher and higher, constantly pursue high-resolution, high-definition films and television programs.Meanwhile high-resolution
It shows the appearance (such as 4K, 5K TV and display) of equipment, and makes it possible the universal of high-resolution films and television programs.
However on the other hand, many classical films and television programs of the remote past still have relatively low since technological means limits
Resolution ratio, lower clarity and poor visual effect.Simultaneously because of the remote past, the film holding time is longer, then
In addition various quality degradations, occurs in the destruction of the extraneous factors such as natural calamity, war, such as breakage, noise, is trembled at flicker
It moves.On the one hand people want to review classical films and television programs on the one hand have new requirement again to the quality of film.In order to full simultaneously
Sufficient people review classical films and television programs and pursue the demand of high-quality video, and films and television programs reconditioning technology comes into being.Video display are made
It is application image/video processing technique that product, which renovate its essence, is handled original video, and various quality degradations are eliminated, to carry
The visual effect of high original video.
Films and television programs are the important culture carriers of human society as books, some classical films and television programs, although the age
It is remote, but its cultureal value can not replace.Therefore, the films and television programs of the remote past to these renovated, replay just
Very important meaning is shown.Specifically, the meaning of films and television programs renovation includes the following aspects：
1. some classical films and television programs, such as documentary film, are precious historical summaries.These films and television programs are renovated,
It can preferably preserve, propagate these historical summaries.
2. a pair classical films and television programs renovate, more moderns is allowed to appreciate, is the important of progress culture and arts succession
Form.
3. a pair classical films and television programs renovate, the taking on a new look again of classical artistic work is allowed, be to artistical maximum
Respect and souvenir.
The existing method for improving films and television programs visual effect is concentrated mainly in video source modeling means, such as removal noise,
Removal is fuzzy, removes and interlocks, enhances contrast, enhancing color etc..These methods can play original video visual effect enhancing
Effect, but increase resolution is not carried out to video, therefore do not meet people inherently and classical films and television programs are renovated
Demand.
So-called increase resolution refers to by the video (or video frame) of low resolution, by certain method, quickly and effectively
One high-resolution video of generation.Its difficult point is how to break through original low-resolution video pixel quantity limitation, filling
Originally the pixel being not present should keep structure, the texture of former low-resolution video, more be closed naturally in human eye again
Reason.
Traditional increase resolution method includes mainly the method based on interpolation, based on rebuilding and based on study.Based on slotting
The method of value is by the way that existing pixel is carried out linear combination, the pixel as missing.Interpolation algorithm is simply rapid, but
It is easy appearance " mosaic " effect or excessively smooth phenomenon；Based on the algorithm of reconstruction registration weight is carried out using the similitude of multiple image
It builds, but this kind of algorithm is often simple combination multiple image, the effect is unsatisfactory；Algorithm based on study mainly utilizes one
The training data of fixed number amount, according to special algorithm train to obtain low-resolution video to high-resolution video mapping relations, this
Requirement of the class algorithm to model is relatively high, easy over-fitting or poor fitting, and operand is big, speed is slow, and practicability is not high.It can be with
It says, above-mentioned video resolution Upgrade Problem annoyings always users.
Invention content
The technology of the present invention solves the problems, such as：The present invention provides a kind of films and television programs renovation promoted based on video resolution
The films and television programs (usually less than 720P) of low resolution are passed through resolution enhancement technology, are converted to compared with high score by method and system
Resolution video (such as 1080P, 4K) realizes films and television programs renovation.
Technical solution of the invention is：The present invention concrete scheme be：First, obtain original video resolution ratio and
Target resolution calculates scaling；Secondly, input video is divided into picture frame by certain partitioning scheme；Then, according to
Pre-stored mapping relations are converted, and high-resolution video frame is obtained；Finally, high-resolution video frame is combined into high score
Resolution video.Wherein, pre-stored mapping relations model is obtained based on Mixture of expert model learning, which exists
It is completed offline in computer.Specific steps packet is as follows：
Learn mapping relations model：
(1) training video is pre-processed
(1.1) high-resolution video is chosen as training sample, and is split as high-resolution video frame；
(1.2) the high-resolution video frame obtained by step (1.1) is used into Gauss nuclear convolution
(1.3) amplification factor is calculated according to original low-resolution video and target high-resolution video, according to times of gained
Number carries out partiting row sampling, obtains corresponding low resolution video frame；
(1.4) by high-resolution video frame and sampling gained low resolution video frame divide respectively it is blocking, as training
Data.
(2) it is based on Mixture of expert model and obtains mapping relations model
(2.1) a Mixture of expert model is initialized.Mixture of expert model includes expert and gate function two parts, structure
To be tree-like, as shown in Fig. 2.Leafy node in figure in tree structure is known as expert, is responsible for carrying out mapping transformation to data；Root
Node is known as gate function, is responsible for data and selects expert appropriate.The present invention uses linear function as expert's function：
Y=Wx
Wherein W is expert's function parameter, and x and y indicate low resolution video frame block and corresponding high-resolution video respectively
Frame block.
Gate function is responsible for determining which expert of selection converts data, and in the present invention, i-th of gate function is expressed as：
Wherein, x and y indicates low resolution video frame block and corresponding high-resolution video frame block, v respectivelyiIt indicates i-th
Gate function parameter, vjIndicate that j-th of gate function parameter, K are the number of expert in Mixture of expert model, i.e. leaf in tree structure
The number of node.The initialization of Mixture of expert model specifically includes following steps：
(2.1.1) specifies the quantity K of expert；
(2.1.2) assumes the probability distribution Gaussian distributed of each expert：P (y | x, Wi)=N (y (x, Wi), σ), wherein
WiIndicate that the parameter of i-th of expert, σ are the standard deviation of Gaussian Profile.It is assumed that parameter WiDistribution also Gaussian distributed：p(Wi)
=N (0, μ), wherein μ indicate the mean value of Gaussian Profile.
(2.1.3) is clustered training data according to the quantity K of expert using k- mean algorithms, at the beginning of the parameter of each expert
Initial value Wi (0)It is appointed as slope in class, the initial value v of each gate function parameteri (0)It is appointed as cluster centre；
(2.1.4) calculates the initial value of each gate function：
Wherein x indicates low resolution video frame block, vi (0)Indicate that the initial value of i-th of gate function parameter, K are Mixture of expert
The number of expert in model, i.e., the number of leaf node in tree structure.
(2.2) training data for using step (1.4) to obtain, is iterated optimization, until iteration to Mixture of expert model
Process restrains, and finally obtained model parameter is mapping relations model.Mapping relations model includes gate function parameter and expert
Parameter.
(2.2.1) specifies allowable error ε when iteration ends；
(2.2.2) calculates the posterior probability of each gate function in epicycle iteration：
Wherein k is iterative steps, pi(y | x, Wi (k)) and pj(y | x, Wj (k)) indicate expert probability distribution, gi (k)(x, vi (k)) indicate that the kth of i-th of gate function walks iterative value.
(2.2.3) updates each expert parameter：
Wherein k is iterative steps, and X is the vector of all low resolution video frame block x compositions in training data, and Y is instruction
Practice the vector of all high-resolution video frame block y compositions in data, XTIndicate that the transposition of X, I indicate unit matrix, Hi (k+1)It indicates
The vector of the posterior probability composition of all low resolution video frame block x in+1 step of kth corresponding to i-th of expert.
(2.2.4) updates each gate function parameter：
WhereinIndicate i-th of gate function parameter in kth step iteration,I-th gate function in iteration is walked for kth
Posterior probability, x(t)Indicate t-th of low resolution video frame block.
(2.2.5) calculates the output of each gate function in epicycle iteration：
(2.2.6) calculates the likelihood probability in epicycle iteration：
Wherein, pi(y | x, Wi (k+1)) indicate expert probability distribution, p (Wi (k+1)) indicate expert parameter probability distribution.
(2.2.7) judges whether iteration restrains.When epicycle iteration likelihood probability and last round of iteration likelihood probability it
Absolute value of the difference be less than iteration ends when allowable error ε when, terminate iteration.Otherwise repeat step (2.2.2)~
(2.2.7)。
The gate function parameter v obtained at the end of iterationi, together with expert's quantity K, expert parameter Wi, expert probability distribution
The mean of a probability distribution μ of standard deviation sigma and expert parameter is stored in together as final mapping relations model in disk.
After having learnt mapping relations model and having stored, resolution ratio is carried out to video using the mapping relations model of storage and is carried
It rises：
(3) pending low-resolution video is pre-processed
(3.1) low-resolution video is split as low resolution video frame；
(3.2) low resolution video frame obtained by step (3.1) is divided into low resolution video frame block；
(4) low-resolution video is promoted to high-resolution video according to the mapping relations model that step (2) obtains, wrapped
It includes：
(4.1) the low resolution video frame block for obtaining step (3) is obtained as the input of gate function using step (2)
Mapping relations model in gate function parameter calculate the output of each gate function：
Wherein, x is the low resolution video frame block of input.K is the number of expert in Mixture of expert model, viIndicate i-th
A gate function parameter is obtained by step (2.2).
(4.2) it uses the parameter of expert's function corresponding to the maximum gate function of output valve to calculate corresponding high-resolution to regard
Frequency frame block, the wherein parameter of expert's function are obtained by step (2)；
(4.2.1) calculating obtains the serial number of maximum output value gate function：I=arg max (gi)
Wherein, giFor the output of i-th of gate function, obtained by step (4.1).
(4.2.2) calculates high-resolution video frame block using i-th of expert's function：Y=Wix
Wherein, WiFor the parameter of i-th of expert's function, y is the high-resolution corresponding to the low resolution video frame block x of input
Rate video frame block.
(4.3) increase resolution is carried out according to the step of (4.1) and (4.2) to each low resolution video frame block, obtained
To corresponding high-resolution video frame block, by all high-resolution video frame blocks according to its corresponding low resolution video frame block
Position in low resolution video frame is spliced into corresponding high-resolution video frame；
(4.4) after obtaining the corresponding high-resolution video frame of all low resolution video frames, it is combined into high-resolution
Video.
In the step (4), between video frame block, between video frame without dependence, therefore GPU processing can be used
Device, it is parallel to accelerate to handle the step.
The films and television programs renovation method based on increase resolution, can be made in the form of computer software player
With can also be integrated into hardware platform (such as set-top box, smart television) and use.
The films and television programs renovation method based on increase resolution can coordinate other screen Enhancement Methods as pre-
Processing or post-processing means, can further promote visual effect.
The advantages of the present invention over the prior art are that：From visual effect, obtained by implementing the present invention program
The high-resolution video details that arrives is complete, edge clear, Acacia crassicarpaA are good, and fast and stable.Specifically, the features of the present invention
Including：
1. adaptive.The present invention program adaptive polo placement scaling multiple, is suitable for different increase resolution demands.
2. speed is fast.Since dependence being not present between sequence of frames of video, processing speed can be improved by parallel processing.
In addition, the processing procedure of algorithm is that sequence of frames of video is carried out Linear Mapping transformation, mapping parameters used can pre-save
In memory, processing speed can be further increased.
3. effect is good.Used mapping parameters are to be based on Mixture of expert model during the present invention carries out increase resolution
What study obtained, break and has divided the drawbacks of being detached with submodel study in traditional increase resolution algorithm based on study.Together
When, statistical robust sexual clorminance and the accurate sexual clorminance based on learning algorithm are combined, avoids and is based on learning algorithm in the past
The drawbacks of mass data information cannot be utilized, while precision more higher than simple statistical method is also achieved, even for this
The higher video of status resolution can also obtain good effect and faster processing speed.
4. expansible.Since dependence being not present between sequence of frames of video, the technological means such as application GPU acceleration can be passed through
Realize that parallel processing improves processing speed.In addition, algorithm proposed by the invention may be directly applied to image resolution ratio promotion
Field.
Description of the drawings
Fig. 1 is the flow chart of the films and television programs renovation method of the present invention based on increase resolution.
Fig. 2 is Mixture of expert model structure schematic diagram of the present invention.
Video frame is divided into video frame block schematic diagram by Fig. 3 to be of the present invention.
Specific implementation mode
The method of the invention is illustrated with an example in detailed description below.
One original resolution is by the films and television programs renovation method according to the present invention based on increase resolution
The process that the video of 768*432 is promoted to 3072*1728 includes the following steps：
(1) training video is pre-processed
(1.1) a high-resolution films and television programs are selected, the video flowing of films and television programs is read in using Video processing software, it will
Each frame in video flowing saves as video frame, and in the present embodiment, films and television programs length is 1200 seconds, and frame rate is 25 frames/second,
Gained video frame sum is：1200*25=15000；
(1.2) the use of mean value is 0 to the video frame obtained by step (1.1), the Gauss nuclear convolution that standard deviation is 1；
(1.3) original low-resolution video resolution and target resolution are obtained, amplification factor is calculated according to the two.It is original
Resolution ratio is 768*432, target resolution 3072*1728, and amplification factor is：3072/768=4.It accordingly, will be after convolution
Video frame is down sampled to the 1/4 of original size, obtains corresponding low resolution video frame.
(1.4) each width low resolution video frame of gained by existing segmentation standard, being divided into size is
10 × 10 pixels are not overlapped fritter, as shown in figure 3, simultaneously therefrom choosing 1,000,000 piece is used as training data.
(2) it is based on Mixture of expert model and obtains mapping relations model
(2.1) Mixture of expert model is initialized
(2.1.1) specifies the quantity K of expert.In the present embodiment, K=100 is taken；
(2.1.2) specifies the parameter σ and μ of the probability distribution of expert and the probability distribution of expert parameter, in the present embodiment, take
σ=0.32, μ=0.58；
(2.1.3) is using k- mean algorithms by training data according to the quantity K clusters of expert, the W of each experti (0)Parameter
It is initialized as slope in class, gate function parameter vi (0)It is initialized as cluster centre；
(2.1.4) calculates the initial value of each gate function according to the following formula：
Wherein x indicates low resolution video frame block, vi (0)Indicate that the initial value of i-th of gate function parameter, K are Mixture of expert
The number of expert in model, i.e., the number of leaf node in tree structure.
(2.2) training data for using step (1.4) to obtain, the Mixture of expert model obtained to step (2.1) change
Generation optimization：
(2.2.1) specifies allowable error ε when iteration ends.In the present embodiment, the permitted mistake of modulus type iteration ends
Poor ε=0.005.
(2.2.2) calculates the posterior probability of each gate function in epicycle iteration：
Wherein k is iterative steps, pi(y | x, Wi (k)) and pj(y | x, Wj (k)) indicate expert probability distribution, gi (k)(x, vi (k)) indicate that the kth of i-th of gate function walks iterative value.
(2.2.3) updates each expert parameter：
Wherein k is iterative steps, and X is the vector of all low resolution video frame block x compositions in training data, and Y is instruction
Practice the vector of all high-resolution video frame block y compositions in data, XTIndicate that the transposition of X, I indicate unit matrix, Hi (k+1)It indicates
The vector of the posterior probability composition of all low resolution video frame block x in+1 step of kth corresponding to i-th of expert.
(2.2.4) updates each gate function parameter：
WhereinIndicate i-th of gate function parameter in kth step iteration,I-th gate function in iteration is walked for kth
Posterior probability, x(t)Indicate t-th of low resolution video frame block.
(2.2.5) calculates the output of each gate function in epicycle iteration：
(2.2.6) calculates the likelihood probability in epicycle iteration：
Wherein, pi(y | x, Wi (k+1)) indicate expert probability distribution, p (Wi (k+1)) indicate expert parameter probability distribution.
(2.2.7) judges whether iteration restrains.When epicycle iteration likelihood probability and last round of iteration likelihood probability it
Absolute value of the difference be less than iteration ends when allowable error ε when, terminate iteration.Otherwise repeat step (2.2.2)~
(2.2.7)。
The gate function parameter v obtained at the end of iterationi, together with expert's quantity K, expert parameter Wi, expert probability distribution
The mean of a probability distribution μ of standard deviation sigma and expert parameter is stored in together as final mapping relations model in disk.Its
In, vi, K, σ, μ be known as the gate function parameter of mapping relations model, WiThe referred to as expert parameter of mapping relations model.
(3) pending low-resolution video is pre-processed
(3.1) pending low-resolution video is split as low resolution video frame, in the present embodiment, films and television programs are long
Degree is 2000 seconds, and frame rate is 25 frames/second, and gained video frame sum is：2000*25=50000；
(3.2) low resolution video frame obtained by step (3.1) is divided into 10 × 10 video frame block, as shown in Figure 3；
(4) low-resolution video is mapped as high-resolution video, including：
(4.1) the low resolution video frame block for obtaining step (3) is obtained as the input of gate function using step (2)
Mapping relations model in gate function parameter calculate the output of each gate function：
Wherein, x is the low resolution video frame block of input.K is the number of expert in Mixture of expert model, viIndicate i-th
A gate function parameter is obtained by step (2.2).
(4.2) expert's function parameter corresponding to the maximum gate function of output valve is used to calculate corresponding high-resolution video
Frame block；
(4.2.1) calculating obtains the serial number of maximum output value gate function：I=arg max (gi).Wherein, giIt is i-th
The output of function is obtained by step (4.1).
(4.2.2) calculates high-resolution video frame block using i-th of expert's function：
Y=Wix
Wherein, WiFor i-th of expert's function parameter, obtained by step (2).X is the low resolution video frame block of input,
Its size is that 10 × 10, y is high-resolution video frame block after increase resolution, and size is 40 × 40.
(4.3) increase resolution is carried out according to the step of (4.1) and (4.2) to each low resolution video frame block, obtained
To corresponding high-resolution video frame block, by all high-resolution video frame blocks according to its corresponding low resolution video frame block
Position in low resolution video frame is spliced into corresponding high-resolution video frame；
(4.4) after obtaining the corresponding high-resolution video frame of all low resolution video frames, it is combined into high-resolution
Video.
Claims (3)
1. a kind of films and television programs renovation method based on increase resolution, it is characterised in that：Including study mapping relations model and
Increase resolution two parts are carried out according to mapping relations model；
Wherein, study mapping relations model includes following two steps：
(1) training video is pre-processed, including：
(1.1) high-resolution video is chosen as training sample, and is split as high-resolution video frame；
(1.2) the high-resolution video frame obtained by step (1.1) is used into Gauss nuclear convolution；
(1.3) amplification factor is calculated according to original low-resolution video and target high-resolution video, according to the multiple of gained into
Row partiting row sampling obtains corresponding low resolution video frame；
(1.4) by high-resolution video frame and sampling gained low resolution video frame divide respectively it is blocking, as training data；
(2) it is based on Mixture of expert model and obtains mapping relations model, including：
(2.1) a Mixture of expert model is initialized, is included the following steps：
1. the quantity K of specified expert；
2. assuming the probability distribution Gaussian distributed of each expert：P (y | x, Wi)=N (y (x, Wi), σ), wherein x indicates low point
Resolution video frame block, y indicate high-resolution video frame block, WiIndicate that the parameter of i-th of expert, σ are the standard deviation of Gaussian Profile；
It is assumed that parameter WiDistribution also Gaussian distributed：p(Wi)=N (0, μ), wherein μ indicate the mean value of Gaussian Profile；
3. using k- mean algorithms by training data according to the quantity K clusters of expert, the initial value W of the parameter of each experti (0)Refer to
It is set to slope in class, the initial value v of each gate function parameteri (0)It is appointed as cluster centre；
4. calculating the initial value of each gate function：
Wherein vi (0)Indicate that the initial value of i-th of gate function parameter, K are the number of expert in Mixture of expert model, i.e. tree structure
The number of middle leaf node；
(2.2) training data for using step (1.4) to obtain, is iterated optimization, until iterative process to Mixture of expert model
Convergence, finally obtained model parameter is mapping relations model；Mapping relations model includes gate function parameter and expert parameter；
It is described optimization is iterated to model to include the following steps：
1. allowable error ε when specified iteration ends；
2. calculating the posterior probability of each gate function in epicycle iteration：
Wherein k is iterative steps, pi(y | x, Wi (k)) and pj(y | x, Wj (k)) indicate expert probability distribution, gi (k)(x, vi (k)) table
Show the kth step iterative value of i-th of gate function；
3. updating each expert parameter：
Wherein k is iterative steps, and X is the vector of all low resolution video frame block x compositions in training data, and Y is training number
The vector of all high-resolution video frame block y compositions, X inTIndicate that the transposition of X, I indicate unit matrix, Hi (k+1)Expression kth+
The vector of the posterior probability composition of all low resolution video frame block x in 1 step corresponding to i-th of expert；
4. updating each gate function parameter：
WhereinIndicate i-th of gate function parameter in kth step iteration,The posteriority that i-th of gate function in iteration is walked for kth is general
Rate, x(t)Indicate t-th of low resolution video frame block；
5. calculating the output of each gate function in epicycle iteration：
6. calculating the likelihood probability in epicycle iteration：
Wherein, pi(y | x, Wi (k+1)) indicate expert probability distribution, p (Wi (k+1)) indicate expert parameter probability distribution；
7. judging whether iteration restrains；When the absolute value of the difference of the likelihood probability of the likelihood probability and last round of iteration of epicycle iteration
When allowable error ε when less than iteration ends, terminate iteration；Otherwise repeat step 2.~7.；
The gate function parameter v obtained at the end of iterationi, together with expert's quantity K, expert parameter Wi, expert probability distribution standard
The mean of a probability distribution μ of poor σ and expert parameter are stored in together as final mapping relations model in disk；
It includes following two steps to carry out increase resolution according to mapping relations model：
(3) pending low-resolution video is pre-processed, including：
(3.1) pending low-resolution video is split as low resolution video frame；
(3.2) low resolution video frame is divided blocking；
(4) low-resolution video is promoted to high-resolution video according to the mapping relations model that step (2) obtains, including：
(4.1) the low resolution video frame block for obtaining step (3) uses step as the input of Mixture of expert model gate function
(2) the gate function parameter in the mapping relations model obtained calculates the output of each gate function；
(4.2) parameter of expert's function corresponding to the maximum gate function of output valve is used to calculate corresponding high-resolution video frame
Block, the wherein parameter of expert's function are obtained by step (2)；
(4.3) increase resolution is carried out according to the step of (4.1) and (4.2) to each low resolution video frame block, obtained pair
The high-resolution video frame block answered, by all high-resolution video frame blocks according to its corresponding low resolution video frame block low
Position in resolution video frame is spliced into corresponding high-resolution video frame；
(4.4) after obtaining the corresponding high-resolution video frame of all low resolution video frames, it is combined into high-resolution video.
2. the films and television programs renovation method according to claim 1 based on increase resolution, it is characterised in that step (2.1)
The Mixture of expert model includes expert and gate function two parts；
Expert is responsible for carrying out mapping transformation to data, and the mapping transformation in the present invention uses linear function as expert's function：
Y=Wx
Wherein W is expert parameter, and x and y indicate low resolution video frame block and corresponding high-resolution video frame block respectively；
Gate function is responsible for determining which expert of selection converts data, and i-th of gate function is expressed as in the present invention：
Wherein, viIndicate i-th of gate function parameter, vjIndicate that j-th of gate function parameter, K are of expert in Mixture of expert model
Number.
3. the films and television programs renovation method according to claim 1 based on increase resolution, it is characterised in that：Step
(4.2) parameter using expert's function corresponding to the maximum gate function of output valve calculates corresponding high-resolution video frame
Block includes the following steps：
1. calculating obtains the serial number of maximum output value gate function：I=arg max (gi)
Wherein, giFor the output of i-th of gate function, obtained by step (4.1)；
2. calculating high-resolution video frame block using i-th of expert's function：Y=Wix
Wherein, WiFor the parameter of i-th of expert's function, y is that the high-resolution corresponding to the low resolution video frame block x of input regards
Frequency frame block.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610109909.XA CN105791980B (en) | 2016-02-29 | 2016-02-29 | Films and television programs renovation method based on increase resolution |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610109909.XA CN105791980B (en) | 2016-02-29 | 2016-02-29 | Films and television programs renovation method based on increase resolution |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105791980A CN105791980A (en) | 2016-07-20 |
| CN105791980B true CN105791980B (en) | 2018-09-14 |
Family
ID=56403789
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610109909.XA Active CN105791980B (en) | 2016-02-29 | 2016-02-29 | Films and television programs renovation method based on increase resolution |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105791980B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109151339B (en) * | 2018-08-27 | 2021-08-13 | 武汉有文心文化传媒有限公司 | Method for synthesizing characters in recommendation video and related products |
| CN109981991A (en) * | 2019-04-17 | 2019-07-05 | 北京旷视科技有限公司 | Model training method, image processing method, device, medium and electronic equipment |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8520736B2 (en) * | 2009-04-14 | 2013-08-27 | Fastvdo, Llc | Real-time superresolution and video transmission |
| CN101639937B (en) * | 2009-09-03 | 2011-12-14 | 复旦大学 | Super-resolution method based on artificial neural network |
| CN104778671B (en) * | 2015-04-21 | 2017-09-22 | 重庆大学 | A kind of image super-resolution method based on SAE and rarefaction representation |
-
2016
- 2016-02-29 CN CN201610109909.XA patent/CN105791980B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN105791980A (en) | 2016-07-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105744357B (en) | A kind of reduction network video bandwidth occupancy method based on online increase resolution | |
| US11055828B2 (en) | Video inpainting with deep internal learning | |
| US10685282B2 (en) | Machine-learning based video compression | |
| CN109087273B (en) | Image restoration method, storage medium and system based on enhanced neural network | |
| US10902571B2 (en) | Automated image synthesis using a comb neural network architecture | |
| CN108734661B (en) | High-resolution image prediction method for constructing loss function based on image texture information | |
| CN112543317B (en) | Method for converting high-resolution monocular 2D video into binocular 3D video | |
| CN110570377A (en) | group normalization-based rapid image style migration method | |
| CN110717868B (en) | Video high dynamic range inverse tone mapping model construction and mapping method and device | |
| CN106127689B (en) | Image/video super-resolution method and device | |
| CN110533579B (en) | Video style conversion method based on self-coding structure and gradient order preservation | |
| US20170109873A1 (en) | Image enhancement using self-examples and external examples | |
| CN109191460B (en) | Quality evaluation method for tone mapping image | |
| CN106204472A (en) | Video image deblurring method based on sparse characteristic | |
| JP2021525401A (en) | Image generation network training and image processing methods, equipment, electronics, and media | |
| CN107945255A (en) | A kind of virtual actor's facial expression driving method and system | |
| CN113129212B (en) | Image super-resolution reconstruction method and device, terminal device and storage medium | |
| WO2020228347A1 (en) | Superpixel-based three-dimensional object model generation method, system, and storage medium | |
| CN105791980B (en) | Films and television programs renovation method based on increase resolution | |
| CN110428382A (en) | A kind of efficient video Enhancement Method, device and storage medium for mobile terminal | |
| Zhang et al. | Deep tone mapping network in HSV color space | |
| CN114339030A (en) | Network live broadcast video image stabilization method based on self-adaptive separable convolution | |
| Xu et al. | Deep video inverse tone mapping | |
| CN102542536A (en) | Image quality strengthening method based on generalized equilibrium model | |
| Ye et al. | Real3d-portrait: One-shot realistic 3d talking portrait synthesis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |