CN107945108A - Method for processing video frequency and device - Google Patents
Method for processing video frequency and device Download PDFInfo
- Publication number
- CN107945108A CN107945108A CN201610895061.8A CN201610895061A CN107945108A CN 107945108 A CN107945108 A CN 107945108A CN 201610895061 A CN201610895061 A CN 201610895061A CN 107945108 A CN107945108 A CN 107945108A
- Authority
- CN
- China
- Prior art keywords
- video code
- code flow
- resolution
- quantization
- video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 66
- 238000012545 processing Methods 0.000 title claims abstract description 25
- 238000013139 quantization Methods 0.000 claims abstract description 206
- 238000013507 mapping Methods 0.000 claims description 36
- 230000008569 process Effects 0.000 claims description 30
- 238000012549 training Methods 0.000 claims description 19
- 230000008859 change Effects 0.000 claims description 9
- 230000009466 transformation Effects 0.000 claims description 6
- 239000000523 sample Substances 0.000 description 115
- 230000006835 compression Effects 0.000 description 7
- 238000007906 compression Methods 0.000 description 7
- 230000005540 biological transmission Effects 0.000 description 4
- 238000013527 convolutional neural network Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000007667 floating Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000008520 organization Effects 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000011002 quantification Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000006866 deterioration Effects 0.000 description 1
- 238000005315 distribution function Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 210000003127 knee Anatomy 0.000 description 1
- 230000035800 maturation Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000012488 sample solution Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The present embodiments relate to a kind of method for processing video frequency and device, this method includes：The first video code flow is received, wherein, the resolution ratio of first video code flow is first resolution, and first video code flow includes the quantitative information of first video code flow；The quantization error of first video code flow is obtained according to the quantitative information of first video code flow；Super-resolution SR is carried out according to the database code stream sample and the quantization error of first video code flow to prestore to first video code flow to rebuild to obtain the second video code flow, wherein, the resolution ratio of second video code flow is second resolution, and the second resolution is more than the first resolution.Method for processing video frequency and device provided in an embodiment of the present invention, it is contemplated that the quantization error of video code flow so that the quality higher for the high-resolution video code stream that SR is rebuild.
Description
Technical field
The present invention relates to technical field of video processing, more particularly to a kind of method for processing video frequency and device.
Background technology
With the progressively maturation of high definition ultra high-definition industrial chain, high definition city has all been met to terminal again from record and broadcast to transmission
The necessary condition of fieldization, in the processing procedure of image or video, image or video are before terminal is shown, it is necessary to by oversubscription
Resolution (Super Resolution, SR) is rebuild.
SR, which is rebuild, to be referred to improve the resolution ratio of image or video using software approach, to meet high score at this stage
Resolution requirement.Existing SR method for reconstructing, three layers are trained based on acquired high-resolution and the feature of low-resolution image block
Depth network obtains Nonlinear Mapping relation, is single-frame images SR according to Nonlinear Mapping relation and rebuilds.Or learnt by training
Obtain a depth convolutional neural networks (Convolutional between low resolution and high-definition picture end to end
Neural Network, CNN) mapping relations, the SR reconstructions of single-frame images are done according to depth CNN mapping relations.
But existing SR reconstruction techniques, the image deterioration that coded quantization is brought when have ignored compressed video bit stream, is decoding
The important information of quantization error has been abandoned during reconstruction, has caused the loss of SR reconstruction qualities.
The content of the invention
The embodiment of the present invention provides a kind of method for processing video frequency and device, and the high-resolution that SR is rebuild can be caused to regard
Frequency code current mass is more preferable.
In a first aspect, an embodiment of the present invention provides a kind of method for processing video frequency, this method includes：Receive the first video codes
Stream, wherein, the resolution ratio of first video code flow is first resolution, and first video code flow includes described first and regards
The quantitative information of frequency code stream；The quantization that first video code flow is obtained according to the quantitative information of first video code flow misses
Difference；According to the database code stream sample and the quantization error of first video code flow to prestore to first video code flow into
Row super-resolution SR rebuilds to obtain the second video code flow, wherein, the resolution ratio of second video code flow is second resolution, institute
State second resolution and be more than the first resolution.
Specifically, method for processing video frequency provided in an embodiment of the present invention, it is contemplated that the quantization error of video code flow, SR are rebuild
The quality higher of obtained high-resolution video code stream.
In a kind of possible embodiment, before the first video code flow is received, further include：Obtain and quantify noise mode
Type, wherein, the quantization noise model is used to obtain quantization error according to quantitative information；It is described according to first video code flow
Quantitative information obtain the quantization error of first video code flow, including：According to the quantitative information of first video code flow
The quantization error of first video code flow is obtained with the quantization noise model.
Specifically, the embodiment of the present invention establishes quantization noise model, when being rebuild to low-resolution video code stream SR, examines
Consider corresponding quantization error, improve the quality for the high-resolution video code stream that SR is rebuild.
In a kind of possible embodiment, database code stream sample and the quantization error pair that the basis prestores
The first video code flow SR rebuilds to obtain the second video code flow, including：According to the quantization error pair of first video code flow
The first video code flow decoding, obtains multiple first medians；According to the database code stream sample to prestore to described more
A first median carries out SR and rebuilds to obtain multiple second medians；The multiple second median is encoded to the second video codes
Stream.
In a kind of possible embodiment, the quantitative information includes quantization parameter QP information and the discrete cosine quantified
Convert DCT coefficient information.The quantization error according to first video code flow decodes first video code flow, obtains
Multiple first medians, including:To the first video code flow entropy decoding, QP information and the quantization of the first video code flow are obtained
DCT coefficient information；According to the DCT coefficient information of quantization of the QP information of first video code flow to first video code flow
Inverse quantization operation is carried out, obtains the first DCT coefficient after multiple inverse quantizations；According to the first DCT systems after the multiple inverse quantization
The quantization error of number and first video code flow obtains the second DCT coefficient after multiple inverse quantizations；To the multiple inverse quantization
The second DCT coefficient afterwards carries out inverse discrete cosine transformation IDCT operations, obtains the multiple first median.
In a kind of possible embodiment, the database code stream sample to prestore described in the basis is to the multiple first
Before median progress SR rebuilds to obtain multiple second medians, further include：Prestored according to the quantization noise model to described
Database code stream sample decoding, obtain low resolution median sample and high-resolution median sample, wherein, the data
Storehouse code stream sample includes low resolution code stream sample and high-resolution code stream sample；To the low resolution median sample and height
Resolution ratio median sample is trained study, obtains the low resolution median sample and the high-resolution median sample
Mapping relations between this；The database code stream sample to prestore described in the basis carries out the multiple first median SR weights
Build to obtain multiple second medians, including：SR is carried out according to the mapping relations to the multiple first median to rebuild to obtain
Multiple second medians.
Specifically, database sample provided in an embodiment of the present invention includes low-resolution video code stream sample and high-resolution
Video code flow sample.Further according to quantization noise model to low-resolution video code stream sample and high-resolution video code stream sample solution
Code, and more accurate mapping is closed between training study low-resolution video code stream sample and high-resolution video code stream sample
System.So that the high-resolution video code stream that SR is rebuild is more nearly ideal value.
In a kind of possible embodiment, the acquisition quantization noise model, including：According to the QP information and quantization
DCT coefficient information obtain quantization noise model.
Second aspect, an embodiment of the present invention provides a kind of video process apparatus, which includes：Receiving unit, is used for
The first video code flow is received, wherein, the resolution ratio of first video code flow is first resolution, in first video code flow
Include the quantitative information of first video code flow；Quantization error unit, for being believed according to the quantization of first video code flow
Breath obtains the quantization error of first video code flow；SR reconstruction units, for according to the database code stream sample that prestores and
The quantization error of first video code flow carries out super-resolution SR to first video code flow and rebuilds to obtain the second video codes
Stream, wherein, the resolution ratio of second video code flow is second resolution, and the second resolution is more than described first and differentiates
Rate.
In a kind of possible embodiment, further include：Quantization noise model acquiring unit, quantifies noise mode for obtaining
Type, wherein, the quantization noise model is used to obtain quantization error according to quantitative information；The quantization error unit, it is specific to use
The quantization of first video code flow is obtained in the quantitative information according to first video code flow and the quantization noise model
Error.
In a kind of possible embodiment, the SR reconstruction units are specifically used for：According to first video code flow
Quantization error decodes first video code flow, obtains multiple first medians；According to the database code stream sample to prestore
This carries out SR to the multiple first median and rebuilds to obtain multiple second medians；The multiple second median is encoded to
Second video code flow.
In a kind of possible embodiment, the quantitative information includes quantization parameter QP information and the discrete cosine quantified
Convert DCT coefficient information.The SR reconstruction units are specifically used for：To the first video code flow entropy decoding, the first video is obtained
The QP information of code stream and the DCT coefficient information quantified；According to the QP information of first video code flow to first video codes
The DCT coefficient information of the quantization of stream carries out inverse quantization operation, obtains the first DCT coefficient after multiple inverse quantizations；According to described more
The quantization error of the first DCT coefficient and first video code flow after a inverse quantization obtains the 2nd DCT after multiple inverse quantizations
Coefficient；Inverse discrete cosine transformation IDCT operations are carried out to the second DCT coefficient after the multiple inverse quantization, obtain the multiple the
One median.
In a kind of possible embodiment, which further includes：Code stream sample decoding unit, for according to the quantization
Noise model decodes the database code stream sample to prestore, obtains low resolution median sample and high-resolution median
Sample, wherein, the database code stream sample includes low resolution code stream sample and high-resolution code stream sample；Training study is single
Member, for being trained study to the low resolution median sample and high-resolution median sample, obtains described low point
Mapping relations between resolution median sample and the high-resolution median sample；The SR reconstruction units are specifically used for：
SR is carried out according to the mapping relations to the multiple first median to rebuild to obtain multiple second medians.
In a kind of possible embodiment, the quantization noise model acquiring unit, specifically for being believed according to the QP
Breath and the DCT coefficient information quantified obtain quantization noise model.
Based on above-mentioned technical proposal, method for processing video frequency and device provided in an embodiment of the present invention, to low-resolution video
When code stream carries out SR reconstructions, consider the quantization error of low-resolution video code stream, reduce the mass loss that quantizing process is brought.
So that the corresponding high-resolution video code stream that SR is rebuild is more nearly ideal value, SR reconstruction qualities are improved.
Brief description of the drawings
To describe the technical solutions in the embodiments of the present invention more clearly, below will be to needed in the embodiment
Attached drawing is briefly described, it should be apparent that, drawings in the following description are only some embodiments of the present invention, for ability
For the those of ordinary skill of domain, without creative efforts, it can also be obtained according to these attached drawings other attached
Figure.
Fig. 1 is processing system for video Organization Chart provided in an embodiment of the present invention；
Fig. 2 is a kind of method for processing video frequency flow diagram provided in an embodiment of the present invention；
Fig. 3 is database sample training learning method flow diagram provided in an embodiment of the present invention；
Fig. 4 is video process apparatus Organization Chart provided in an embodiment of the present invention.
Embodiment
To make the purpose, technical scheme and advantage of the embodiment of the present invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, the technical solution in the embodiment of the present invention is clearly and completely described, it is clear that described embodiment is
Part of the embodiment of the present invention, instead of all the embodiments.Based on the embodiments of the present invention, those of ordinary skill in the art
All other embodiments obtained without making creative work, belong to the scope of protection of the invention.
For ease of the understanding to the embodiment of the present invention, below by drawings and examples, technical scheme is done
It is described in further detail.
Video coding or the video decoding that the embodiment of the present invention refers to can be operated according to video compression standard.For example,
Compression of digital video form H.264 standard and high efficiency Video coding (High Efficiency Video Coding, HEVC)
Standard.Wherein, HEVC standard can be described as H.265 standard again.It is to be understood that the technology of the present invention is not limited to any specific encoding and decoding mark
Accurate or technology.
It should be noted that video code flow includes DCT coefficient, the quantization parameter (Quantization quantified
Parameter, QP) and the information such as prediction mode, decoder can be according to information such as the DCT coefficient of quantization, QP by video code flow
It is decoded as multiple images frame.
It is understood that the cataloged procedure of video code flow includes：To pre- in the pixel data interframe and/or frame of image block
Survey, dct transform, quantization, entropy coding and code stream output and etc..The spatial domain value of pixel is transformed to frequency domain by dct transform operation
Value, quantization operation are compressed frequency domain pixels value.Wherein, dct transform operates to obtain original DCT coefficient, to original DCT coefficient
Carry out the DCT coefficient after quantization operation is quantified.The decoding process of video code flow includes entropy decoding, inverse quantization, inverse DCT
(Inverse DCT, IDCT) conversion, interframe and/or infra-frame prediction recovery image block pixel data and etc..Correspondingly, entropy solution
After code operation, the information such as DCT coefficient, QP and prediction mode for being quantified.The DCT coefficient after quantization is carried out according to QP anti-
Quantization operation, obtains the DCT coefficient after inverse quantization.DCT coefficient after inverse quantization is subjected to IDCT operations, obtains spatial domain pixel
Value.
It should be noted that since quantizing process is many-to-one mapping, it is thus possible to can introduce and quantify in quantizing process
Error, lost effective information.Therefore, the DCT coefficient after inverse quantization has a certain amount error compared to original DCT coefficient,
The image block directly decoded using the DCT coefficient of inverse quantization might have certain mass loss compared to original picture block.
The spatial domain pixel value of image block before coding can be denoted as floating number, correspondingly, such as consider that quantization error has certain quality
Loss, the spatial domain pixel value of the image block decoded are denoted as integer value.
Existing database sample includes low-resolution image sample storehouse and high-definition picture sample storehouse, is learnt by training
Mapping relations between obtained low-resolution image sample storehouse and high-definition picture sample storehouse.Here mapping relations, refer to
Be the unique corresponding high-definition picture of a low-resolution image in database sample.Further, when SR is rebuild,
According to the mapping relations of database sample, the low-resolution image of reception is rebuild, obtains corresponding high-definition picture.
Fig. 1 is processing system for video Organization Chart provided in an embodiment of the present invention.As shown in Figure 1, processing system for video includes：
Video process apparatus 100.Video process apparatus 100 receives low-resolution video code stream, and low-resolution video code stream is carried out
SR processing, exports high-resolution video code stream.
Video process apparatus 100 is by the low-resolution video code stream decoding of reception, in decoding process, considers to quantify to miss
Difference, obtains corresponding multiple first medians.Video process apparatus 100 carries out SR reconstructions to each first median, obtains every
Corresponding second median of a first median.Multiple second medians are encoded to corresponding high score by video process apparatus 100
Resolution video code flow simultaneously exports.
The embodiment of the present invention establishes a quantization noise model, which can estimate that dependent quantization information is corresponding and quantify to miss
Difference.When video process apparatus 100 is to the low-resolution video code stream decoding of reception, according to quantization noise model, consider to quantify to miss
Difference, obtains multiple first medians of corresponding consideration quantization error so that the slave original of multiple first medians decoded
First integer value is changed into being more nearly the floating point values of actual value.
It should be noted that such as directly video code flow is decoded, multiple spatial domain pixel values will be obtained without considering quantization error
For the picture frame of integer.The embodiment of the present invention is on this basis, it is contemplated that quantization error, obtained numerical value are accurate floating-point
Number, therefore median is denoted as, these medians and the pixel value of the picture frame without considering quantization error correspond.
The database sample of the embodiment of the present invention is low-resolution video code stream sample and high-resolution video code stream sample,
Before SR of the embodiment of the present invention is rebuild, video process apparatus 100 needs to regard low-resolution video code stream sample and high-resolution
Frequency code stream sample decodes respectively, while in decoding process, considers quantization error, obtain corresponding low resolution median sample
With high-resolution median sample.Low resolution median sample and high-resolution median sample are obtained by training study
Mapping relations.And SR reconstructions are carried out to each first median according to the mapping relations, it is corresponding to obtain each first median
Second median.Finally, multiple second medians are encoded to corresponding high-resolution video code stream and exported.
It is understood that treat that the low-resolution image of SR reconstructions and the resolution ratio of high-definition picture can be a variety of.
For example, high definition market is very big to the high-definition picture of 4K resolution ratio (3840 × 2160) or the demand of video at present.Meanwhile mesh
The video resolution of preceding transmission is mostly 1080P resolution ratio (1920 × 1080).Therefore need the 1080P low resolution figures of transmission
Picture or video SR are shown again after being redeveloped into 4K high-definition pictures or video.The embodiment of the present invention is high using low resolution as 1080P
Exemplified by resolution ratio is 4K, illustrate.But it is not intended to limit the invention embodiment.Video reconstruction provided in an embodiment of the present invention
Method is equally applicable to rebuild the video of other resolution ratio.
Method for processing video frequency provided in an embodiment of the present invention and system, by establishing quantization noise model, to database
Code stream sample considers quantization error when decoding, and obtains more accurate median sample.Further obtain more accurately mapping
Relation.And introduce quantization noise model in SR reconstruction process so that treat that the low resolution median that SR is rebuild is more accurate.Into one
Step ground, the corresponding high-resolution median rebuild by more accurate mapping relations to low resolution median SR,
So that the quality higher of the high-resolution video of final output.
It is understood that processing system for video provided in an embodiment of the present invention may also include source device and destination dress
Put.Wherein, source device is used to produce low-resolution video code stream and be sent to video process apparatus 100, and destination device can solve
The high-resolution video code stream that code and display video process apparatus 100 are sent.It should be noted that video process apparatus 100
Also can be integrated with destination device, that is to say, that destination device may include low-resolution video code stream being redeveloped into
The module of high-resolution video code stream, instead of the function of performing video process apparatus 100.
Specifically, source device and destination device may include a wide range of devices, for example, comprising desktop computer, moving
The hand-held sets such as dynamic computing device, notebook (for example, on knee) computer, tablet PC, set-top box, smart phone, TV,
Camera, display device, digital media player, video game console, car-mounted computer, or its fellow.
Below in conjunction with the accompanying drawings 2, the scheme that embodiment that the present invention will be described in detail provides.Fig. 2 is provided in an embodiment of the present invention
A kind of method for processing video frequency flow diagram, subject of implementation is video process apparatus in embodiments of the present invention.As shown in Fig. 2,
The embodiment specifically includes following steps：
Step S101, receives the first video code flow, wherein, the resolution ratio of first video code flow is first resolution,
First video code flow includes the quantitative information of first video code flow.
It should be noted that low-resolution video code stream can be collectively referred to as the first video code flow.Second resolution video code flow
It can be collectively referred to as high-resolution video code stream.
Wherein, it is further comprising the steps of before the first video code flow is received：Quantization noise model is obtained, wherein, it is described
Quantization noise model is used to obtain quantization error according to quantitative information.
It should be noted that the distribution of the quantitative information estimation quantizing noise in code stream can be utilized in dequantization step
Model, then obtains more accurate decoded median according to distributed model, and quality caused by reducing quantizing process is damaged
Lose.
Preferably, the quantitative information includes quantization parameter QP information and the discrete cosine transform (Discrete quantified
Cosine Transform, DCT) coefficient information.
The acquisition quantization noise model, including：Obtain quantization according to the QP information and the DCT coefficient information quantified and make an uproar
Acoustic model.
Specifically, if the DCT coefficient that (k, l) a dct transform obtains in cataloged procedure is denoted as Y [k, l], then quantified
Journey is real value quantized intervalThe DCT coefficient for being mapped to (k, l) a quantization after single real number quantifies is Yq[k,
L]=Q [Y [k, l]].Wherein,For row k, at l column positions, the DCT coefficient before the corresponding quantization of i-th of quantification gradation
Value；I be quantification gradation sequence number, quantification gradation, that is, QP；K, l are respectively the row k of the image block of current dct transform, l row
Location label.
It should be noted that the numerical value reflection compression degree of QP.QP numerical value is smaller, and quantization is thinner, and compression degree is lower, code
Rate is higher.QP numerical value is bigger, and quantization is more coarse, and compression degree is higher, and code check is lower.
In high code check, since QP is smaller, compression degree is relatively low.It is uniformly distributed at this point it is possible to think that quantizing noise is obeyed,
And uncorrelated to the DCT signals of input, i.e., quantization error is evenly distributed in quantization boundary, at this time, (k, l) a DCT
The quantization error of coefficientFor：
In low bit- rate, since QP is larger, compression degree is higher, and quantized interval is larger.The elder generation DCT coefficient is needed at this time
Test information and be added to the accurate estimation that noise is carried out in model, DCT coefficient is frequency-region signal, according to Lam et al. to DCT coefficient
The research of distribution, laplace model are well suited for, at this time, the quantization error of (k, l) a DCT coefficientFor：
Wherein, γ is distributed a normalized constant, p for guaranteeY[k,l](y) be Lam et al. propose laplacian distribution
Function, y are located at scope for coefficient valueInterior gradual change integrated value.
The value range of quantization parameter QP is generally [0~t].When QP is minimized the most fine quantization of 0 interval scale, code check
Highest；When QP is maximized the most coarse quantization of t interval scales, code check is minimum.In the embodiment of the present invention, we are by under two kinds of code checks
Model united by a weighting function λ：OrderThe quantization error of DCT coefficient after (k, l) a inverse quantization
σ2[k, l] can be summarized as：
It should be noted that the formula shown in above-mentioned formula (3), is quantizing noise mould provided in an embodiment of the present invention
Type.
It should be noted that the maximum t that luminance coding corresponds to QP is 51.The maximum t that chroma coder corresponds to QP is 39.
In addition, the value of QP maximums, can also be varied from according to the change of standard.
Step S102, the quantization that first video code flow is obtained according to the quantitative information of first video code flow miss
Difference.
Preferably, described first is obtained according to the quantitative information of first video code flow and the quantization noise model to regard
The quantization error of frequency code stream.
Specifically, can according to the quantization noise model shown in above-mentioned formula (3), and in the first video code flow quantization letter
Breath, obtains the quantization error σ of the DCT coefficient after (k, l) a inverse quantization2[k,l]。
Step S103, according to the database code stream sample and the quantization error of first video code flow to prestore to described
First video code flow carries out SR and rebuilds to obtain the second video code flow, wherein, the resolution ratio of second video code flow is second point
Resolution, the second resolution are more than the first resolution.
Preferably, first video code flow is decoded according to the quantization error of first video code flow, obtained multiple
First median；The multiple first median progress SR is rebuild to obtain according to the database code stream sample to prestore multiple
Second median；The multiple second median is encoded to the second video code flow.
Specifically, to the first video code flow entropy decoding, obtain the quantization of the first video code flow DCT coefficient information and
QP information；Carried out according to the DCT coefficient information of quantization of the QP information of first video code flow to first video code flow
Inverse quantization operation, obtains the first DCT coefficient after multiple inverse quantizations；According to the first DCT coefficient after the multiple inverse quantization and
The quantization error obtains the second DCT coefficient after multiple inverse quantizations；The second DCT coefficient after the multiple inverse quantization is carried out
Inverse discrete cosine transformation IDCT is operated, and obtains the multiple first median.
Further, the DCT coefficient to (k, l) a quantization is Yq[k, l]] inverse quantization operation is carried out, obtain (k, l)
The first DCT coefficient after a inverse quantization is denoted asMissed according to the quantization of the first DCT coefficient after (k, l) a inverse quantization
Difference, obtains the second DCT coefficient after multiple inverse quantizations
Wherein,For the DCT coefficient after the modified more accurate inverse quantization of quantization error, do not consider to quantify
DCT coefficient after the inverse quantization of error is
The embodiment of the present invention carries out idct transform to the accurate DCT coefficient by quantization noise model estimation, by DCT coefficient
Spatial domain value is converted to from frequency domain value, obtains more accurate spatial domain median, this median is continuous floating number, is not original
The integer come.More accurate mapping relations are finally established using quantization noise model, and then obtain more accurate high-resolution
Rate median and corresponding high-resolution video code stream.
Preferably, the database code stream sample to prestore described in the basis carries out SR reconstructions to the multiple first median
Before obtaining multiple second medians, further include：According to the quantization noise model to the database code stream sample to prestore
Decoding, obtains low resolution median sample and high-resolution median sample, wherein, the database code stream sample includes low
Resolution ratio code stream sample and high-resolution code stream sample；To the low resolution median sample and high-resolution median sample
Study is trained, the mapping obtained between the low resolution median sample and the high-resolution median sample is closed
System.Specifically it can refer to shown in Fig. 3.
It should be noted that when considering to decode the quantization error of video code flow sample, will be floated accordingly
Point median sample, these floating-point median samples and does not consider the picture frame that pixel value that quantization error decodes is integer
Sample is corresponding.Therefore, it is necessary to which training learns the mapping relations between these corresponding floating-point medians before SR reconstructions are carried out.
Mapping relations between floating-point median do not consider low-resolution image sample and height that quantization error decodes compared to before
Mapping relations between image in different resolution sample are more accurate.
Fig. 3 is database sample training learning method flow diagram provided in an embodiment of the present invention.As shown in figure 3, bag
Step S201 is included to step S202：
Step S201, decodes low-resolution video code stream sample, and considers that low resolution regards according to quantization noise model
The quantization error of frequency code stream sample, obtains accurate low resolution median sample.High-resolution video code stream sample is decoded,
And the quantization error of high-resolution video code stream sample is considered according to quantization noise model, obtain accurate high-resolution median
Sample.
It should be noted that the embodiment of the present invention is to low-resolution video code stream sample and corresponding high-resolution video code
Stream sample decoding process all does the quantizing noise estimation of above-mentioned formula (3).Quantization noise model is utilized in decoding process so that solution
The low resolution median sample and high-resolution median sample that code obtains are floating point values, obtain more accurate database sample
This.
Step S202, is trained study to low resolution median sample and high-resolution median sample, obtains low
Mapping relations between resolution ratio median sample and high-resolution median sample.
Specifically, training study may include that depth CNN learns.Training the destination of study is output low-resolution video code stream
Mapping relations between sample and high-resolution video code stream sample.
Further, the database code stream sample to prestore described in the basis carries out the multiple first median SR weights
Build to obtain multiple second medians, including：SR is carried out according to the mapping relations to the multiple first median to rebuild to obtain
Multiple second medians.Finally, the multiple second median is encoded to the second video code flow.
Method for processing video frequency and device provided in an embodiment of the present invention, using low-resolution video code stream as input, pass through
After adding quantizing noise estimation solution to model code process, accurate low resolution median is generated.This median is subjected to SR weights
Build, obtain high-resolution median, then encoded to obtain output high-resolution video code stream.
Method for processing video frequency and device provided by the invention, the training sample of deep learning are pre-processed so that sample
This value is more accurate.The present invention takes full advantage of the quantitative information that the low-resolution video code stream of input is provided, according to quantization
The distribution situation of noise, before more accurately having estimated quantization using a unified model to the quantizing noises of different code checks
DCT coefficient, so convert to obtain more accurate pixel spatial domain median by inverse DCT.Meanwhile according to quantization noise model
Sample value after processing is more accurate, and study obtains more accurate mapping relations, to more accurately decoding obtained pixel
Spatial domain median carries out SR reconstructions according to more accurate mapping relations so that the high-resolution video code stream matter that SR is rebuild
Amount is more preferable.
Fig. 4 is video process apparatus Organization Chart provided in an embodiment of the present invention.As shown in figure 4, including：Receiving unit 401,
Quantization error unit 402, SR reconstruction units 403, quantization noise model acquiring unit 404, code stream sample decoding unit 405 and
Training unit 406.
The receiving unit 401 of video process apparatus provided in an embodiment of the present invention is used to receive the first video code flow, wherein,
The resolution ratio of first video code flow is first resolution, and first video code flow includes first video code flow
Quantitative information.
Quantization error unit 402 is used to obtain first video code flow according to the quantitative information of first video code flow
Quantization error.
SR reconstruction units 403 are used to be missed according to the quantization of the database code stream sample and first video code flow that prestore
Difference carries out super-resolution SR to first video code flow and rebuilds to obtain the second video code flow, wherein, second video code flow
Resolution ratio be second resolution, the second resolution is more than the first resolution.
Preferably, quantization noise model acquiring unit 404 is used to obtain quantization noise model, wherein, the quantizing noise
Model is used to obtain quantization error according to quantitative information.
The quantization error unit 402 is specifically used for being made an uproar according to the quantitative information and the quantization of first video code flow
Acoustic model obtains the quantization error of first video code flow.
Preferably, the SR reconstruction units 403 are specifically used for：According to the quantization error of first video code flow to described
First video code flow decodes, and obtains multiple first medians；According to the database code stream sample to prestore to the multiple
One median carries out SR and rebuilds to obtain multiple second medians；The multiple second median is encoded to the second video code flow.
Preferably, the quantitative information includes quantization parameter QP information and the discrete cosine transform coefficient information quantified.
Preferably, the SR reconstruction units 403 are specifically used for：To the first video code flow entropy decoding, obtain first and regard
The QP information of frequency code stream and the DCT coefficient information quantified；According to the QP information of first video code flow to first video
The DCT coefficient information of the quantization of code stream carries out inverse quantization operation, obtains the first DCT coefficient after multiple inverse quantizations；According to described
The quantization error of the first DCT coefficient and first video code flow after multiple inverse quantizations obtains second after multiple inverse quantizations
DCT coefficient；Inverse discrete cosine transformation IDCT operations are carried out to the second DCT coefficient after the multiple inverse quantization, are obtained described more
A first median.
It should be noted that the specific implementation of SR reconstruction units 403, can refer to the introduction in above-mentioned Fig. 2.
Preferably, code stream sample decoding unit 405 is used for according to the quantization noise model to the database to prestore
Code stream sample decodes, and obtains low resolution median sample and high-resolution median sample, the database code stream sample bag
Include low resolution code stream sample and high-resolution code stream sample.
Training unit 406 is used to instruct the low resolution median sample and high-resolution median sample
Practice study, obtain the mapping relations between the low resolution median sample and the high-resolution median sample.
It should be noted that the specific implementation of code stream sample decoding unit 405 and training unit 406, can
With reference to the introduction in above-mentioned Fig. 2 and Fig. 3.
The SR reconstruction units 403 are specifically used for：SR is carried out to the multiple first median according to the mapping relations
Reconstruction obtains multiple second medians.
Preferably, the quantization noise model acquiring unit 404, specifically for according to the QP information and the DCT quantified
Coefficient information obtains quantization noise model.
It should be noted that the function of above-mentioned each unit, it is intended to realize each step in preceding method embodiment.Its
In, video process apparatus provided in an embodiment of the present invention may also include more or fewer units, so as to realizing that the present invention is real
The method for applying example.
The 1080P resolution video code streams sample of database sample provided in an embodiment of the present invention including different code checks, no
With the 4K resolution video code stream samples of code check.The training module of deep learning SR method for reconstructing directly trains input code flow sample
(such as 1080P resolution videos code stream) arrives the mapping relations of output code flow sample (such as 4K resolution videos code stream).It is defeated in training
Enter code stream into the mapping relations of output code flow, consider that quantization error decodes video code flow to obtain median so that trained
It is more accurate to low resolution median sample and the mapping relations of high-resolution median sample.
Method for processing video frequency and device provided in an embodiment of the present invention, deep learning SR is added by the estimation of quantizing noise
The training module of reconstruction, quantization error is just considered in the learning training stage, is fundamentally solved and is trained learn to obtain to reflect
The drawbacks of relation is inaccurate is penetrated, reduces the mass loss that quantizing process is brought.And in the low resolution rebuild to pending SR
Rate video code flow decoding stage considers corresponding quantization error so that the corresponding high-resolution video code stream that SR is rebuild
Ideal value is more nearly, improves SR reconstruction qualities.
Method for processing video frequency and device provided in an embodiment of the present invention, can also be applied to image or video restoration, image
The fields such as video deblurring.Need change data storehouse sample, input and output are the video before restoring and after restoring, or deblurring it
Video after preceding and deblurring.
Those skilled in the art are it will be appreciated that in said one or multiple examples, work(described in the invention
It is able to can be realized with hardware, software, firmware or their any combination.When implemented in software, can be by these functions
It is stored in computer-readable medium or is transmitted as one or more instructions on computer-readable medium or code.
Computer-readable medium includes computer storage media and communication media, and wherein communication media includes being easy to from a place to another
Any medium of one place transmission computer program.It is any that storage medium can be that universal or special computer can access
Usable medium.
Above-described embodiment, has carried out the purpose of the present invention, technical solution and beneficial effect further
Describe in detail, it should be understood that the foregoing is merely the embodiment of the present invention, be not intended to limit the present invention
Protection domain, all any modification, equivalent substitution, improvement and etc. on the basis of technical scheme, done should all
It is included within protection scope of the present invention.
Claims (12)
- A kind of 1. method for processing video frequency, it is characterised in that the described method includes：The first video code flow is received, wherein, the resolution ratio of first video code flow is first resolution, first video codes Stream includes the quantitative information of first video code flow；The quantization error of first video code flow is obtained according to the quantitative information of first video code flow；According to the database code stream sample and the quantization error of first video code flow to prestore to first video code flow Super-resolution SR is carried out to rebuild to obtain the second video code flow, wherein, the resolution ratio of second video code flow is second resolution, The second resolution is more than the first resolution.
- 2. according to the method described in claim 1, it is characterized in that, before the first video code flow is received, further include：Quantization noise model is obtained, wherein, the quantization noise model is used to obtain quantization error according to quantitative information；The quantitative information according to first video code flow obtains the quantization error of first video code flow, including：The amount of first video code flow is obtained according to the quantitative information of first video code flow and the quantization noise model Change error.
- 3. according to the method described in claim 2, it is characterized in that, database code stream sample that the basis prestores and described Quantization error carries out SR to first video code flow and rebuilds to obtain the second video code flow, including：First video code flow is decoded according to the quantization error of first video code flow, obtains multiple first medians；SR is carried out according to the database code stream sample to prestore to the multiple first median to rebuild to obtain in multiple second Between be worth；The multiple second median is encoded to the second video code flow.
- 4. according to the method described in claim 3, it is characterized in that, the quantitative information includes quantization parameter QP information and quantization Discrete cosine transform coefficient information；The quantization error according to first video code flow is to first video codes Stream decoding, obtains multiple first medians, including:To the first video code flow entropy decoding, the DCT coefficient information for obtaining the QP information of the first video code flow and quantifying；Inverse is carried out according to the DCT coefficient information of quantization of the QP information of first video code flow to first video code flow Change operation, obtain the first DCT coefficient after multiple inverse quantizations；Multiple inverses are obtained according to the quantization error of the first DCT coefficient after the multiple inverse quantization and first video code flow The second DCT coefficient after change；Inverse discrete cosine transformation IDCT operations are carried out to the second DCT coefficient after the multiple inverse quantization, obtain the multiple the One median.
- 5. according to the method described in claim 3, it is characterized in that, the database code stream sample to prestore described in the basis is to institute Multiple first medians are stated before SR rebuild to obtain multiple second medians, further include：The database code stream sample to prestore is decoded according to the quantization noise model, obtains low resolution median sample With high-resolution median sample, wherein, the database code stream sample includes low resolution code stream sample and high-resolution code Flow sample；Study is trained to the low resolution median sample and high-resolution median sample, obtains the low resolution Mapping relations between median sample and the high-resolution median sample；The database code stream sample to prestore described in the basis carries out the multiple first median SR and rebuilds to obtain multiple the Two medians, including：SR is carried out according to the mapping relations to the multiple first median to rebuild to obtain multiple second medians.
- 6. according to the method described in claim 4, it is characterized in that, the acquisition quantization noise model, including：Quantization noise model is obtained according to the QP information and the DCT coefficient information quantified.
- 7. a kind of video process apparatus, it is characterised in that described device includes：Receiving unit, for receiving the first video code flow, wherein, the resolution ratio of first video code flow is first resolution, First video code flow includes the quantitative information of first video code flow；Quantization error unit, for obtaining the quantization of first video code flow according to the quantitative information of first video code flow Error；SR reconstruction units, the database code stream sample and the quantization error of first video code flow to prestore for basis is to institute The first video code flow progress super-resolution SR is stated to rebuild to obtain the second video code flow, wherein, the resolution of second video code flow Rate is second resolution, and the second resolution is more than the first resolution.
- 8. device according to claim 7, it is characterised in that further include：Quantization noise model acquiring unit, for obtaining quantization noise model, wherein, the quantization noise model is used for according to amount Change information and obtain quantization error；The quantization error unit, specifically for the quantitative information according to first video code flow and the quantization noise model Obtain the quantization error of first video code flow.
- 9. device according to claim 8, it is characterised in that the SR reconstruction units are specifically used for：According to described first The quantization error of video code flow decodes first video code flow, obtains multiple first medians；According to the number to prestore SR is carried out according to storehouse code stream sample to the multiple first median to rebuild to obtain multiple second medians；By in the multiple second Between value be encoded to the second video code flow.
- 10. device according to claim 9, it is characterised in that the quantitative information includes quantization parameter QP information and amount The discrete cosine transform coefficient information of change；The SR reconstruction units are specifically used for：To the first video code flow entropy decoding, The DCT coefficient information for obtaining the QP information of the first video code flow and quantifying；According to the QP information of first video code flow to institute The DCT coefficient information for stating the quantization of the first video code flow carries out inverse quantization operation, obtains the first DCT systems after multiple inverse quantizations Number；Multiple inverses are obtained according to the quantization error of the first DCT coefficient after the multiple inverse quantization and first video code flow The second DCT coefficient after change；Inverse discrete cosine transformation IDCT operations are carried out to the second DCT coefficient after the multiple inverse quantization, Obtain the multiple first median.
- 11. device according to claim 9, it is characterised in that further include：Code stream sample decoding unit, for being decoded according to the quantization noise model to the database code stream sample to prestore, Low resolution median sample and high-resolution median sample are obtained, wherein, the database code stream sample includes low resolution Rate code stream sample and high-resolution code stream sample；Training unit, for being trained to the low resolution median sample and high-resolution median sample Practise, obtain the mapping relations between the low resolution median sample and the high-resolution median sample；The SR reconstruction units are specifically used for：SR is carried out according to the mapping relations to the multiple first median to rebuild to obtain Multiple second medians.
- 12. device according to claim 10, it is characterised in that the quantization noise model acquiring unit, is specifically used for The quantization noise model is obtained according to the QP information and the DCT coefficient information quantified.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610895061.8A CN107945108A (en) | 2016-10-13 | 2016-10-13 | Method for processing video frequency and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610895061.8A CN107945108A (en) | 2016-10-13 | 2016-10-13 | Method for processing video frequency and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107945108A true CN107945108A (en) | 2018-04-20 |
Family
ID=61928477
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610895061.8A Withdrawn CN107945108A (en) | 2016-10-13 | 2016-10-13 | Method for processing video frequency and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107945108A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113491134A (en) * | 2018-12-28 | 2021-10-08 | 多玩国株式会社 | Image transmission/reception system, data transmission/reception system, transmission/reception method, computer program, image transmission system, image reception device, transmission system, and reception device |
| WO2023133888A1 (en) * | 2022-01-17 | 2023-07-20 | 深圳市大疆创新科技有限公司 | Image processing method and apparatus, remote control device, system, and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101551902A (en) * | 2009-05-15 | 2009-10-07 | 武汉大学 | A characteristic matching method for compressing video super-resolution based on learning |
| CN101577825A (en) * | 2009-05-15 | 2009-11-11 | 武汉大学 | Interactive quantized noise calculating method in compressed video super-resolution |
| CN103475876A (en) * | 2013-08-27 | 2013-12-25 | 北京工业大学 | Learning-based low-bit-rate compression image super-resolution reconstruction method |
-
2016
- 2016-10-13 CN CN201610895061.8A patent/CN107945108A/en not_active Withdrawn
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101551902A (en) * | 2009-05-15 | 2009-10-07 | 武汉大学 | A characteristic matching method for compressing video super-resolution based on learning |
| CN101577825A (en) * | 2009-05-15 | 2009-11-11 | 武汉大学 | Interactive quantized noise calculating method in compressed video super-resolution |
| CN103475876A (en) * | 2013-08-27 | 2013-12-25 | 北京工业大学 | Learning-based low-bit-rate compression image super-resolution reconstruction method |
Non-Patent Citations (1)
| Title |
|---|
| LU TAO 等: "Super-resolution for compressed video based on DCT side information", 《2009 ASIA-PACIFIC CONFERENCE ON COMPUTATIONAL INTELLIGENCE AND INDUSTRIAL APPLICATIONS》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113491134A (en) * | 2018-12-28 | 2021-10-08 | 多玩国株式会社 | Image transmission/reception system, data transmission/reception system, transmission/reception method, computer program, image transmission system, image reception device, transmission system, and reception device |
| US11800185B2 (en) | 2018-12-28 | 2023-10-24 | Dwango Co., Ltd. | Image transmission and reception system, data transmission and reception system, transmission and reception method, computer program, image transmission system, image reception device, transmission system, reception device |
| WO2023133888A1 (en) * | 2022-01-17 | 2023-07-20 | 深圳市大疆创新科技有限公司 | Image processing method and apparatus, remote control device, system, and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220353534A1 (en) | Transform Kernel Selection and Entropy Coding | |
| US9407915B2 (en) | Lossless video coding with sub-frame level optimal quantization values | |
| KR101266667B1 (en) | Dual-mode compression of images and videos for reliable real-time transmission | |
| US11102477B2 (en) | DC coefficient sign coding scheme | |
| CN104378636B (en) | A kind of video encoding method and device | |
| CN111741302B (en) | Data processing method and device, computer readable medium and electronic equipment | |
| US10506256B2 (en) | Intra-prediction edge filtering | |
| US9681140B2 (en) | Device and a method for encoding an image and corresponding decoding method and decoding device | |
| US10021398B2 (en) | Adaptive tile data size coding for video and image compression | |
| CN109922340A (en) | Image coding/decoding method, device, system and storage medium | |
| CN107945108A (en) | Method for processing video frequency and device | |
| CN104581173A (en) | Soft decoding verification model platform | |
| JP7125559B2 (en) | Video stream adaptive filtering for bitrate reduction | |
| US20200404273A1 (en) | Transforms for large video and image blocks | |
| EP3026912A1 (en) | Method and device for encoding and decoding a HDR picture and a LDR picture using illumination information | |
| CN111491166A (en) | Dynamic compression system and method based on content analysis | |
| CN110692247A (en) | Composite motion compensated prediction | |
| WO2022120829A1 (en) | Image encoding and decoding methods and apparatuses, and image processing apparatus and mobile platform | |
| WO2023000182A1 (en) | Image encoding, decoding and processing methods, image decoding apparatus, and device | |
| EP3035685A1 (en) | A device and a method for encoding an image and corresponding decoding method and decoding device | |
| KR102398232B1 (en) | Method and apparatus for decoding a video signal with reference picture filtering | |
| CN116830574A (en) | Palette mode coding with specified bit depth precision | |
| CN115866244A (en) | Image coding method and device | |
| CN117939157A (en) | Image processing method, device and equipment | |
| CN111147865A (en) | Pre-decoding method before video coding |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20180420 |