CN1777287A - Image decoding apparatus, image decoding program, image decoding method, image encoding apparatus, image encoding program, and image encoding method - Google Patents

Image decoding apparatus, image decoding program, image decoding method, image encoding apparatus, image encoding program, and image encoding method Download PDF

Info

Publication number
CN1777287A
CN1777287A CNA2005101204670A CN200510120467A CN1777287A CN 1777287 A CN1777287 A CN 1777287A CN A2005101204670 A CNA2005101204670 A CN A2005101204670A CN 200510120467 A CN200510120467 A CN 200510120467A CN 1777287 A CN1777287 A CN 1777287A
Authority
CN
China
Prior art keywords
mentioned
motion vector
unit
image
image signal
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CNA2005101204670A
Other languages
Chinese (zh)
Inventor
铃木芳典
文仲丞
陈朝庆
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NTT Docomo Inc
Original Assignee
NTT Docomo Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by NTT Docomo Inc filed Critical NTT Docomo Inc
Priority to CN201010113362.3A priority Critical patent/CN101854544B/en
Publication of CN1777287A publication Critical patent/CN1777287A/en
Pending legal-status Critical Current

Links

Images

Landscapes

  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Television Systems (AREA)

Abstract

The invention provides an image decoding device, an image decoding program, an image decoding method, an image encoding device, an image encoding program and an image encoding method, wherein, the image decoding device comprises a video data decoding unit for receiving and decoding video encoding data to obtain a plurality of reproduction images; an auxiliary data decoding unit for receiving and decoding auxiliary data to obtain auxiliary dynamic information; a definition expansion unit, which generates a dynamic vector that shows the space-time correspondence among the correspondence based on the auxiliary dynamic information obtained by the auxiliary data decoding unit, and generates a high-definition image, with the space definition thereof being higher than the reproduction images by using the generated dynamic vector and the reproduction images obtained by the video data decoding unit.

Description

Picture decoding apparatus, program, method, and picture coding device, program, method
Technical field
The present invention relates to picture decoding apparatus, image decoding program, picture decoding method, picture coding device, image encoding program and method for encoding images.
Background technology
Known from a plurality of low picture rich in detail of being regenerated by video data encoder is decoded (below, " low clear " is designated as LR) the generation HD image (below, " high definition " is designated as HR) super clear technology (below, " super clear " is designated as SR) (for example, with reference to " C.A.Segall et al.; " High-Resolution Images from Low-Resolution Compressed Vidio "; IEEE Signal Processing Magazine; in May, 2003, pp.37-48 " (hereinafter referred to as " non-patent literature 1 ")).
In the SR technology, the relationship modeling of a plurality of LR images and 1 HR image, handle Given information and supposition information by statistical ground, can generate the HR image from a plurality of LR images.Fig. 1 represents the model between LR image and the HR image.In this model, suppose the original image LR image 104 that generates multiframe (L opens) from 1 original image HR image 101.In this supposition, for generate original image LR image 104-1,104-2 ..., 104-L, for original image HR image 101 be suitable for dynamic model 201-1,201-2 ..., 201-L.At this moment,, according to low pass filter with fall the sampling pattern 202 of sampling, implement sampling and handle if for the HR image, then generate original image LR image 104-1,104-2 ..., 104-L.Here, if the regeneration LR image 102-1, the 102-2 that generate by the decoded video coded data ..., 102-L and original image LR image 104-1,104-2 ..., the difference of 104-L be assumed to quantizing noise 103-1,103-2 ..., 103-L, the original image HR image f_k (x of frame k then, z), 1≤x≤2M, and the regeneration LR image y_l of 1≤z≤2N and frame l (m, n), 1≤m≤M, the relation of 1≤n≤N can be according to 1 modeling of following formula.
Y_l=AHC (d_lk) f_k+e_l ... (formula 1)
In the formula, l is the integer from 1 to L, C (d_lk) is the matrix of the dynamic model between the HR image of frame k and frame l, AH is that (H is the matrix of 4MN * 4MN of the Filtering Processing of expression HR image for the matrix of sampling pattern, A is the matrix of sampling that falls of MN * 4MN), e_l represents the quantizing noise of the regeneration LR image of frame l.
Like this, certain regeneration LR image of video data encoder and 1 HR image can enoughly be represented dynamic model that the space-time between the LR-HR image is corresponding and in the signal model modeling of the noise that takes place from the HR image to the deterioration process of LR image.Thereby in order to generate 1 HR image from a plurality of regeneration LR images, the cost function of the guess value of dynamic model or signal model is estimated in the method definition of use statistical, finds the solution nonlinear optimization process.In this optimization process,, ask the multidate information (SR multidate information) of space-time correspondence and separating of HR image between expression LR image-HR image for a plurality of LR images each.
Method as optimization process, coordinate decent method is for example arranged (with reference to " H.He; L.P.Kondi; " MAP Based Resolution Enhancement of Video SequencesUsing a Huber-Markov Random Field Image Prior Model, Proc.OfIEEE International Conference on Image Processing Vol.II, (Spain) "; in September, 2003; pp.933-936 " (below, be called " non-patent literature 2 ")).In the method, at first, handle generation virtual HR image (being accompanied by the temporary transient HR image of the optimization that has utilized repeated treatments) by interpolation from regeneration LR image.The HR image is fixed, utilized cost function, ask the corresponding multidate information of space-time between expression virtual HR image and a plurality of LR images.Then, the multidate information of obtaining is fixed, utilized cost function to upgrade virtual HR image.And then, virtual HR image is fixed, upgrade multidate information.Carry out this processing repeatedly, make and separate convergence.
Summary of the invention
In existing super clear technology, expand the processing in the definition that generates the higher image of definition from a plurality of images, because the coding noise of low definition image and take a sample unclear, and, be difficult to hang down accurately the detection of dynamic between picture rich in detail and the HD image as the influence of the uncertainty etc. of the model of supposition.In addition, in order to carry out detection of dynamic or the optimization process between the image, in handling, the definition expansion needs huge operand.
The present invention finishes for solving above-mentioned problem, its purpose is to provide the operand that alleviates in the image definition expansion processing, can improve picture decoding apparatus, image decoding program, picture decoding method, picture coding device, image encoding program and the method for encoding images of the detection of dynamic precision between the image simultaneously.
Picture decoding apparatus of the present invention possesses: the receiver, video coded data is decoded, and obtains the video data decoding unit of a plurality of reproduced pictures; Receive auxiliary data and decode, obtain the auxiliary data decoding unit of auxiliary multidate information; According to the auxiliary multidate information that obtains by above-mentioned auxiliary data decoding unit, generate the motion vector of the space-time correspondence between the above-mentioned a plurality of reproduced pictures of expression, use motion vector that is generated and a plurality of reproduced pictures that obtain by above-mentioned video data decoding unit, the definition expansion unit of the HD image that span definition is higher than above-mentioned a plurality of reproduced pictures.
If according to above-mentioned picture decoding apparatus, then generate motion vector according to auxiliary multidate information, use the motion vector and a plurality of reproduced picture that are generated, the HD image that span definition is higher than these a plurality of reproduced pictures.
In addition, in above-mentioned picture decoding apparatus, preferably adopt following form.That is, in above-mentioned picture decoding apparatus, preferably constitute auxiliary multidate information and comprise auxiliary motion vector, the definition expansion unit is used as motion vector to auxiliary motion vector.
In addition, in above-mentioned picture decoding apparatus, preferably adopt auxiliary multidate information to comprise auxiliary motion vector, the definition expansion unit uses a plurality of reproduced pictures to detect middle motion vector, generates the structure of motion vector by the addition process of this centre motion vector and auxiliary motion vector.
In addition, in above-mentioned picture decoding apparatus, preferably adopt auxiliary multidate information to comprise auxiliary motion vector, the definition expansion unit is the initial motion vector of auxiliary motion vector as motion vector, by using a plurality of reproduced pictures to upgrade initial motion vector, generate the structure of motion vector.
And then, in above-mentioned picture decoding apparatus, preferably adopt auxiliary motion vector to comprise as the motion vector formation condition that is used to generate the required condition of motion vector, the definition expansion unit generates the structure of motion vector from a plurality of reproduced pictures according to the motion vector formation condition.
Picture coding device of the present invention possesses: the image sampling unit that HD image is transformed to low picture rich in detail; The a plurality of low picture rich in detail compressed encoding that generates by above-mentioned image sampling unit, generate the video data encoding unit of coding video frequency data; The coding video frequency data decoding that generates by above-mentioned video data encoding unit, the video data decoding unit of the low picture rich in detail that obtains regenerating; Use above-mentioned HD image or above-mentioned low picture rich in detail, be created on the auxiliary multidate information generation unit of needed auxiliary multidate information in the generation of above-mentioned motion vector; According to the auxiliary multidate information that generates by above-mentioned auxiliary multidate information generation unit, generate expression and hang down the motion vector of the space-time correspondence between the picture rich in detail by a plurality of regeneration that above-mentioned video data decoding unit obtains, use the motion vector and the low picture rich in detail of above-mentioned a plurality of regeneration that are generated, generate the definition expansion unit of regeneration HD image; The auxiliary data coding unit of encoding as auxiliary data by the auxiliary multidate information of above-mentioned auxiliary multidate information generation unit generation.
If according to above-mentioned picture coding device, then use HD image or low picture rich in detail, be created on needed auxiliary multidate information in the generation of motion vector, generate motion vector according to the auxiliary multidate information that is generated, use the motion vector and the low picture rich in detail of a plurality of regeneration that are generated to generate the regeneration HD image, simultaneously, auxiliary multidate information is encoded as auxiliary data.
In addition, picture decoding apparatus of the present invention possesses: the received code data, carry out the entropy decoding, obtain as the coded data decoding unit that is used to generate the motion vector formation condition of motion vector conditions needed and represents the coded message of prediction error image signal; Be used to preserve the video memory of decoded image signal; According to the motion vector formation condition that is obtained by above-mentioned coded data decoding unit, the decoded image signal from be kept at above-mentioned video memory generates the motion vector generation unit of motion vector; Use is by the motion vector of above-mentioned motion vector generation unit generation and the prediction image signal generation unit of above-mentioned decoded image signal generation forecast picture signal; The coded message decoding that obtains by above-mentioned coded data decoding unit, obtain the decoding unit of prediction error image signal; Prediction image signal that generates by above-mentioned prediction image signal generation unit and the prediction error image signal plus that obtains by above-mentioned decoding unit, restore decoded image signal, this decoded image signal is kept at preservation unit in the above-mentioned video memory.
If according to above-mentioned picture decoding apparatus, then be used to generate the motion vector formation condition of motion vector conditions needed according to conduct, decoded image signal from be kept at video memory generates motion vector, uses the motion vector and the decoded image signal generation forecast picture signal that are generated.On the other hand, after the coded message decoding of expression prediction error image signal has been obtained the prediction error image signal, the prediction image signal of above-mentioned generation and above-mentioned prediction error image signal plus, restore decoded image signal, this decoded image signal is kept in the video memory.
In addition, picture coding device of the present invention possesses: the input unit that received image signal is imported; Be used to preserve the video memory of decoded image signal; According to the received image signal by the input of above-mentioned input unit, decision is as the motion vector formation condition decision unit of the motion vector formation condition that is used to generate the motion vector conditions needed; According to the motion vector formation condition by above-mentioned motion vector formation condition decision unit decision, the decoded image signal from be kept at above-mentioned video memory generates the motion vector generation unit of motion vector; Motion vector and above-mentioned decoded image signal that use is generated by above-mentioned motion vector generation unit, the prediction image signal generation unit of generation forecast picture signal; Prediction error image signal generation unit from the prediction image signal generation forecast error image signal that generates by the received image signal of above-mentioned input unit input with by above-mentioned prediction image signal generation unit; The prediction error image signal encoding that is generated by above-mentioned prediction error image signal generation unit, the coded message that obtains coded message obtains the unit; Obtain the coded message decoding that obtains the unit by above-mentioned coded message, the local solution code element of the prediction error image signal that obtains decoding; From prediction image signal that generates by above-mentioned prediction image signal generation unit and the decoding prediction error image signal restoring decoded image signal that obtains by above-mentioned local solution code element, this decoded image signal is kept at preservation unit in the above-mentioned video memory; Determining the motion vector formation condition of unit decision by above-mentioned motion vector formation condition and obtaining the coded message that obtains the unit by above-mentioned coded message and carry out entropy coding, generate the coded data generation unit of coded data.
If according to above-mentioned picture coding device, then according to the received image signal of being imported, decision is as the motion vector formation condition that is used to generate the motion vector conditions needed, according to the motion vector formation condition that is determined, decoded image signal from be kept at video memory generates motion vector, use the motion vector and the decoded image signal that are generated, the generation forecast picture signal.On the other hand, from received image signal and the above-mentioned prediction image signal generation forecast error image signal of being imported that generates, the prediction error image signal encoding that is generated is obtained coded message, and resulting coded message is decoded obtains decoding the prediction error image signal.And, prediction image signal and the above-mentioned decoding prediction error image signal restoring decoded image signal that obtains from being generated are kept at this decoded image signal in the video memory, simultaneously, above-mentioned motion vector formation condition and coded message are carried out entropy coding, generate coded data.
In addition, picture decoding apparatus of the present invention can adopt following form.
Picture decoding apparatus of the present invention can adopt following structure, and the feature of this structure is to possess: the receiver, video coded data is decoded, and obtains the video data decoding unit of a plurality of reproduced pictures; Receive auxiliary data and decode, obtain the auxiliary data decoding unit of auxiliary multidate information; Generate the corresponding motion vector of space-time between a plurality of reproduced pictures that expression obtains by above-mentioned video data decoding unit and 1 HD image, use the motion vector and the above-mentioned a plurality of reproduced picture that are generated, the definition expansion unit of the HD image that span definition is higher than above-mentioned a plurality of reproduced pictures, above-mentioned definition expansion unit implements to generate the motion vector generation processing of above-mentioned motion vector repeatedly according to auxiliary multidate information that is obtained by above-mentioned auxiliary data decoding unit and the HD image that had before generated, the HD image that generates HD image from the motion vector that generated and above-mentioned a plurality of reproduced picture generates to be handled.
In above-mentioned, the definition expansion unit generates in the repeated treatments of processing and HD image generation processing at each motion vector, also can adopt the auxiliary multidate information of each basis to implement motion vector and generate the structure of handling, auxiliary multidate information is included in the specific times information of the specific times in the repeated treatments that the generation of expression motion vector is handled and the HD image generation is handled, the definition expansion unit also can adopt the motion vector at the represented specific times of specific times information to generate in the processing, implements to generate the structure of handling based on the motion vector of auxiliary multidate information.
Picture decoding apparatus of the present invention can adopt following structure, and this structure possesses: the receiver, video coded data is decoded, and obtains the video data decoding unit of a plurality of reproduced pictures; Be used to preserve the video memory that definition expands the HD image that finishes; Generate the motion vector of the space-time correspondence between the above-mentioned a plurality of reproduced pictures of expression, use motion vector that is generated and a plurality of reproduced pictures that obtain by above-mentioned video data decoding unit, the 1st HD image that span definition is higher than above-mentioned a plurality of reproduced pictures is used the 1st HD image generated and is kept at the definition expansion unit that HD image in the above-mentioned video memory generates the 2nd HD image; The image that the 1st or the 2nd HD image that is generated by above-mentioned definition expansion unit is kept in the video memory is preserved the unit.
In addition, picture decoding apparatus of the present invention can adopt following structure, and this structure possesses: the receiver, video coded data is decoded, and obtains the video data decoding unit of a plurality of reproduced pictures; Receive auxiliary data and decode, obtain the auxiliary data decoding unit of auxiliary multidate information; Be used to preserve the video memory that definition expands the HD image that finishes; Generate the motion vector of the space-time correspondence between the above-mentioned a plurality of reproduced pictures of expression, use motion vector that is generated and a plurality of reproduced pictures that obtain by above-mentioned video data decoding unit, the 1st HD image that span definition is higher than above-mentioned a plurality of reproduced pictures, according to the auxiliary multidate information that obtains by above-mentioned auxiliary data decoding unit, use the 1st HD image generated and be kept at the definition expansion unit that HD image in the above-mentioned video memory generates the 2nd HD image; The image that the 1st or the 2nd HD image that is generated by above-mentioned definition expansion unit is kept in the above-mentioned video memory is preserved the unit.
In addition, picture decoding apparatus of the present invention can adopt following structure, and the feature of this structure is to possess: the receiver, video coded data is decoded, and obtains the video data decoding unit of a plurality of reproduced pictures and regeneration motion vector; Receive auxiliary data and decode, obtain the auxiliary data decoding unit of needed auxiliary multidate information in the correction of above-mentioned regeneration motion vector; Generate the motion vector of the space-time correspondence between a plurality of reproduced pictures of expression, use motion vector that is generated and a plurality of reproduced pictures that obtain by above-mentioned video data decoding unit, the definition expansion unit of the HD image that span definition is higher than above-mentioned a plurality of reproduced pictures, above-mentioned definition expansion unit is according to the regeneration motion vector of the auxiliary multidate information correction that is obtained by the above-mentioned auxiliary data decoding unit initial motion vector as above-mentioned motion vector, by using above-mentioned a plurality of reproduced picture to upgrade above-mentioned initial motion vector, generate above-mentioned motion vector.
In addition, the present invention removes beyond the invention that can record and narrate as described above to picture decoding apparatus and picture coding device, can also record and narrate the invention into picture decoding method, image decoding program, method for encoding images and image encoding program as following.This is example and product form difference, but serves the same role and effect.
The feature of picture decoding method of the present invention is to possess: the receiver, video coded data is decoded, and obtains the video data decoding step of a plurality of reproduced pictures; Receive auxiliary data and decode, obtain the auxiliary data decoding step of auxiliary multidate information; According to the auxiliary multidate information that in above-mentioned auxiliary data decoding step, obtains, generate the motion vector of the space-time correspondence between the above-mentioned a plurality of reproduced pictures of expression, use motion vector that is generated and a plurality of reproduced pictures that obtain in above-mentioned video data decoding step, the definition of the HD image that span definition is higher than above-mentioned a plurality of reproduced pictures expands step.
The feature of method for encoding images of the present invention is to possess: the image sampling step that HD image is transformed to low picture rich in detail; The a plurality of low picture rich in detail compressed encoding that in above-mentioned image sampling step, generates, generate the video data encoding step of coding video frequency data; The decoding of the coding video frequency data that in above-mentioned video data encoding step, generates, the video data decoding step of the low picture rich in detail that obtains regenerating; Use above-mentioned HD image or above-mentioned low picture rich in detail, be created on that the auxiliary multidate information of needed auxiliary multidate information generates step in the generation of above-mentioned motion vector; According to generating the auxiliary multidate information that generates in the step at above-mentioned auxiliary multidate information, be created on the motion vector of the space-time correspondence between the low picture rich in detail of a plurality of regeneration that obtain in the above-mentioned video data decoding step, the definition that the low picture rich in detail of the motion vector that use is generated and above-mentioned a plurality of regeneration generates the regeneration HD image expands step; Generate the auxiliary data coding step that the auxiliary multidate information that generates in the step is encoded as auxiliary data at above-mentioned auxiliary multidate information.
In addition, the feature of picture decoding method of the present invention is to possess: the received code data are carried out the entropy decoding, obtain as the coded data decoding step that is used to generate the motion vector formation condition of motion vector conditions needed and represents the coded message of prediction error image signal; According to the motion vector formation condition that obtains in above-mentioned coded data decoding step, the motion vector that the decoded image signal from be kept at video memory generates motion vector generates step; Use generates motion vector and the above-mentioned decoded image signal that generates in the step at above-mentioned motion vector, and the prediction image signal of generation forecast picture signal generates step; The coded message that obtains in above-mentioned coded data decoding step is decoded, obtain the decoding step of prediction error image signal; Generating prediction image signal that generates in the step and the prediction error image signal plus that in above-mentioned decoding step, obtains at above-mentioned prediction image signal, restore decoded image signal, this decoded image signal is kept at preservation step in the above-mentioned video memory.
The feature of method for encoding images of the present invention is to possess: the input step that received image signal is imported; According to the received image signal of importing in above-mentioned input step, decision is as the motion vector formation condition deciding step of the motion vector formation condition that is used to generate the motion vector conditions needed; According to the motion vector formation condition that determines in above-mentioned motion vector formation condition deciding step, the motion vector that the decoded image signal from be kept at video memory generates motion vector generates step; Use generates motion vector and the above-mentioned decoded image signal that generates in the step at above-mentioned motion vector, and the prediction image signal of generation forecast picture signal generates step; From received image signal of importing above-mentioned input step and the prediction image signal that generates in above-mentioned prediction image signal generation step, the prediction error image signal of generation forecast error image signal generates step; Generating the prediction error image signal encoding that generates in the step at above-mentioned prediction error image signal, the coded message that obtains coded message obtains step; Obtain the coded message decoding that obtains in the step in above-mentioned coded message, the local decoding step of the prediction error image signal that obtains decoding; From generating prediction image signal that generates the step and the decoding prediction error image signal restoring decoded image signal that in above-mentioned local decoding step, obtains, this decoded image signal is kept at preservation step in the above-mentioned video memory at above-mentioned prediction image signal; The motion vector formation condition that determines in above-mentioned motion vector formation condition deciding step with obtain the coded message that obtains in the step in above-mentioned coded message and carry out entropy coding, the coded data that generates coded data generates step.
Image decoding program of the present invention is to make the computer that is arranged in the picture decoding apparatus play the effect of following each unit, and these unit are: the receiver, video coded data is decoded, and obtains the video data decoding unit of a plurality of reproduced pictures; Receive auxiliary data and decode, obtain the auxiliary data decoding unit of auxiliary multidate information; According to the auxiliary multidate information that obtains by above-mentioned auxiliary data decoding unit, generate the motion vector of the space correspondence between the above-mentioned a plurality of reproduced pictures of expression, use motion vector that is generated and a plurality of reproduced pictures that obtain by above-mentioned video data decoding unit, the definition expansion unit of the HD image that span definition is higher than above-mentioned a plurality of reproduced pictures.
Image encoding program of the present invention is to make the computer that is arranged in the picture coding device play the effect of following each unit, and these unit are: the image sampling unit that HD image is transformed to low picture rich in detail; The a plurality of low picture rich in detail compressed encoding that generates by above-mentioned image sampling unit, generate the video data encoding unit of coding video frequency data; The coding video frequency data decoding that generates by above-mentioned video data encoding unit, the video data decoding unit of the low picture rich in detail that obtains regenerating; Use above-mentioned HD image or above-mentioned low picture rich in detail, be created on the auxiliary multidate information generation unit of needed auxiliary multidate information in the generation of above-mentioned motion vector; According to the auxiliary multidate information that generates by above-mentioned auxiliary multidate information generation unit, generate expression and hang down the motion vector of the space-time correspondence between the picture rich in detail by a plurality of regeneration that above-mentioned video data decoding unit obtains, use the motion vector and the low picture rich in detail of above-mentioned a plurality of regeneration that are generated, generate the definition expansion unit of regeneration HD image; The auxiliary data coding unit of encoding as auxiliary data by the auxiliary multidate information of above-mentioned auxiliary multidate information generation unit generation.
In addition, image decoding program of the present invention is to make the computer that is arranged in the picture decoding apparatus play the effect of following each unit, these unit are: the received code data, carry out the entropy decoding, obtain as the coded data decoding unit that is used to generate the motion vector formation condition of motion vector conditions needed and represents the coded message of prediction error image signal; Be used to preserve the video memory of decoded image signal; According to the motion vector formation condition that is obtained by above-mentioned coded data decoding unit, the decoded image signal from be kept at above-mentioned video memory generates the motion vector generation unit of motion vector; Use is by the motion vector of above-mentioned motion vector generation unit generation and the prediction image signal generation unit of above-mentioned decoded image signal generation forecast picture signal; The coded message decoding that obtains by above-mentioned coded data decoding unit, obtain the decoding unit of prediction error image signal; Prediction image signal that generates by above-mentioned prediction image signal generation unit and the prediction error image signal plus that obtains by above-mentioned decoding unit, restore decoded image signal, this decoded image signal is kept at preservation unit in the above-mentioned video memory.
In addition, image encoding program of the present invention is to make the computer that is arranged in the picture coding device play the effect of following each unit, and these unit are: the input unit that received image signal is imported; Be used to preserve the video memory of decoded image signal; According to the received image signal by the input of above-mentioned input unit, decision is as the motion vector formation condition decision unit of the motion vector formation condition that is used to generate the motion vector conditions needed; According to the motion vector formation condition by above-mentioned motion vector formation condition decision unit decision, the decoded image signal from be kept at above-mentioned video memory generates the motion vector generation unit of motion vector; Motion vector and above-mentioned decoded image signal that use is generated by above-mentioned motion vector generation unit, the prediction image signal generation unit of generation forecast picture signal; Prediction error image signal generation unit from the prediction image signal generation forecast error image signal that generates by the received image signal of above-mentioned input unit input with by above-mentioned prediction image signal generation unit; The prediction error image signal encoding that is generated by above-mentioned prediction error image signal generation unit, the coded message that obtains coded message obtains the unit; Obtain the coded message decoding that obtains the unit by above-mentioned coded message, the local solution code element of the prediction error image signal that obtains decoding; From prediction image signal that generates by above-mentioned prediction image signal generation unit and the decoding prediction error image signal restoring decoded image signal that obtains by above-mentioned local solution code element, this decoded image signal is kept at preservation unit in the above-mentioned video memory; Determining the motion vector formation condition of unit decision by above-mentioned motion vector formation condition and obtaining the coded message that obtains the unit by above-mentioned coded message and carry out entropy coding, generate the coded data generation unit of coded data.
If according to above-described the present invention, then improve the precision of the detection of dynamic between the image, improve the image quality of regeneration HD image.In addition, owing to reduce the treating capacity of News Search, therefore reduce the operand in the image definition expansion processing.
Description of drawings
Fig. 1 represents the relation of low picture rich in detail of a plurality of regeneration and HD image.
The kind of the multidate information that Fig. 2 explanation is relevant with auxiliary data of the present invention.
Fig. 3 illustrates the overall structure of code device of the present invention.
Fig. 4 illustrates the structure of encoding process of the present invention.
Fig. 5 illustrates that the definition of having used encoding process of the present invention expands processing.
Fig. 6 illustrates the overall structure of decoding device of the present invention.
Fig. 7 illustrates that the definition of having used decoding processing of the present invention expands processing.
Fig. 8 represents the data structure of auxiliary data of the present invention.
Fig. 9 represents encoding process flow process of the present invention.
Figure 10 has represented to use the definition of auxiliary data of the present invention to expand handling process.
Figure 11 represents decoding processing flow process of the present invention.
Figure 12 illustrates to preserve and is used for being handled or the data storage medium of the program that picture decoding is handled by the image encoding of computer realization the invention process form.
Figure 13 illustrates other example of the structure of encoding process of the present invention.
Figure 14 illustrates that the definition of having used encoding process of the present invention expands other example of handling.
Figure 15 illustrates that the definition of having used decoding processing of the present invention expands other example of handling.
Figure 16 illustrated block matching method.
Figure 17 illustrates the News Search in the decoding processing.
Figure 18 illustrates the structure of the video coding processing of having used encoding process of the present invention.
Figure 19 illustrates the structure of the video decode processing of having used decoding processing of the present invention.
Figure 20 illustrates the structure of video data encoder.
Figure 21 illustrates other example of encoding process flow process of the present invention.
Figure 22 illustrates other example of decoding processing flow process of the present invention.
Embodiment
To Figure 12 example of the present invention is described with reference to Fig. 2.
Fig. 2 explanation is included in the motion vector in the interior data of various multidate informations.In addition, Fig. 3 represents the structure of decoding device of the present invention to Fig. 5, and Fig. 6 and Fig. 7 represent the structure of decoding device of the present invention.Fig. 8 represents the data format structures of the auxiliary data among the present invention.In addition, Fig. 9 illustrates handling process and the process of decoding flow process that the handling process of coding, super picture rich in detail generate respectively to Figure 11.Figure 12 illustrates and preserves the data storage medium that is used for by the program of processing of computer realization image encoding or picture decoding processing.
Auxiliary data among the present invention has auxiliary multidate information in advance, as described later, in auxiliary multidate information, low clear multidate information (LR multidate information), the super clear multidate information of correction (revising the SR multidate information), high definition multidate information (HR multidate information) are arranged.In addition, below, simply be LR suitably " low clear ", simply be HR " high definition ", " super clear " simply described for SR.In addition, definition is described as " HD image (HR image) " than " low picture rich in detail (LR image) " high image.
The code device 10 of the invention process form at first, is described.
Fig. 3 represents the overall structure of the code device 10 of the invention process form.Code device 10 possesses image sampling unit 302, piece cutting unit 303, coding processing unit 304, codec processing unit 305, data storage 308, frame memory 307, data storage 309 and definition transformed/coded unit 306.
Original image HR image 101 is transformed to the definition original image LR image 104 lower than original image HR image possessing low pass filter and falling in the image sampling unit 302 of sample process unit.The original image LR image 104 that has been transformed is divided into encoding block in piece cutting unit 303, for example, to the lower right, be input to coding processing unit 304 according to raster scan order from the upper left side of image.Coding processing unit 304 is carried out moving picture to input block, is compressed into video data encoder 120.In addition, coding processing unit 304 outputs to codec processing unit 305 to video data encoder 120.Codec processing unit 305 is by decoding video data encoder 120, generates regeneration LR image 102 and decoding multidate information (below, be called " DEC multidate information ") 108.In addition, coding processing unit 304 is owing to have local codec processing unit in inside, so codec processing unit 305 also can be substituted by the local codec processing units in the coding processing unit 304.
Coding processing unit 304 and codec processing unit 305 output to frame memory 307, data storage 308 and data storage 309 to the regeneration LR image 102, multidate information (DEC multidate information) 108 and the quantization parameter 114 that generate respectively.When frame memory 307, data storage 308 and data storage 309 are preserved regeneration LR image 102, DEC multidate information 102 and quantization parameter 114 respectively, they are outputed to definition transformed/coded unit 306.In addition, piece is cut apart, the detailed process of encoding process and (part) decoding processing for example is documented in " MPEG-4 Video Verification Model version18.0 ", Output document of MPEG Pisa Meeting, among the January 2001 (below, as a reference 1).
DEC multidate information 108 is made of type of prediction, motion vector (below, the motion vector in the DEC multidate information as " DECMV "), encodes after having determined respectively at each encoding block.
Use Fig. 2 (a), the type of prediction and the DECMV of DEC multidate information is described.In type of prediction, have and use motion vector to carry out the middle model of dynamic prediction and do not follow motion vector and the pixel of using the coding of present frame to finish is carried out the internal schema of spatial prediction.And then, in middle model, have at LR image 910 usefulness of present frame and show constantly the forward direction dynamic prediction that finishes coding with the past LR image 920a of frame carries out time prediction as the reference image, carry out the back of time prediction with the finish LR image 920b of frame of the coding in future as the reference image to dynamic prediction, carry out time prediction with these two kinds of images respectively as the reference image, handle the bi-directional predicted of synthetic predicted picture by interpolation.In Fig. 2 (a), 922a represents the prediction piece of forward prediction, 921a represents the DECMV of forward direction, 922b represents the prediction piece of back forecast, 921b represent the back to DECMV, 924a and 924b represent to predict piece before the bi-directional predicted interpolation that 923a and 923b represent that bi-directional predicted forward direction DECMV and back are to DECMV.
Then, use Fig. 4 and Fig. 5 that definition transformed/coded unit 306 is described.Definition transformed/coded unit 306 possesses definition and expands processing unit 310, auxiliary data generation unit 351, auxiliary data coding/rate controlled unit 311, frame memory 315.In addition, auxiliary data generation unit 351 possesses low clear dynamic compensation unit 312, super clear dynamic compensation unit 313, high definition dynamic compensation unit 314.Low clear dynamic compensation unit 312 is generated as auxiliary data to LR multidate information 109 (aftermentioned), super clear dynamic compensation unit 313 is generated as supplementary revising SR multidate information 111 (aftermentioneds), and high definition dynamic compensation unit 314 is generated as auxiliary data to HR multidate information 112 (aftermentioned).
Definition transformed/coded unit 306 as input, carries out local definition expansion processing with a plurality of regeneration LR images, DEC multidate information (comprising DECMV) and the quantization parameter that generate in coding processing unit 304 and codec processing unit 305.In addition, in definition transformed/coded unit 306, definition expands processing unit 310 and expands processing generation regeneration HR image 106 by local definition, from outside input original image HR image 101 and original image LR image 104.Use these images and information, definition transformed/coded unit 306 generates auxiliary definition and expands the auxiliary data of handling, and in auxiliary data coding/rate controlled unit 311, implements the encoding process (that is, generating auxiliary data 113) of auxiliary data.
In this example, in the generation of auxiliary data 113, use regeneration HR image 106, SR multidate information (super clear multidate information) 110, quantization parameter 114, original image HR image 101 and original image LR image 104.In addition, so-called super clear multidate information means the corresponding multidate information of space-time between expression regeneration HR image and a plurality of LR images.
With reference to Fig. 4, the internal structure of definition transformed/coded unit 306 is described.Use relates to and has added the processing that 3 frames that show moment front and back amount to the information enforcement definition transformed/coded unit 306 of 7 frames on the frame of implementing the definition expansion.That is, after the frame before 3 frames has been encoded, implement definition and expand processing.
The encoding process that definition in the code device 10 in this example expands processing and auxiliary data is divided into 7 steps.Below, according to processing sequence explanation action.
In the 1st step, in low clear dynamic compensation unit 312, use original image LR image 104, DEC multidate information 108 is modified to high-precision LR multidate information 109.The LR multidate information is made of the piece positional information of LR image and auxiliary motion vector (below, the motion vector in the LR multidate information as " LRMV ").The enforcement definition that is received at low clear dynamic compensation unit 312 expands the input of 3 regeneration of the total LR image 102 of the regeneration LR image (being the reference picture of dynamic prediction) that has added its 2 frames in front and back on the regeneration LR image of handling, corresponding 3 original image LR images 104 and DEC multidate information 108 when video coding, LR multidate information 109 is outputed to auxiliary data coding/rate controlled unit 311 and definition expansion processing unit 310.
Use Fig. 2 (b), the LR multidate information is described.In the LR multidate information, have in the piece that does not have DECMV new add auxiliary motion vector (LRMV) though type and DECMV have the type that its value is changed to different auxiliary power-assist state vector (LRMV).
In adding the type of LRMV,, promptly regenerate at the reference picture of the original image LR image 910 of implementing the frame that definition expands and preceding frame and to implement News Search between the LR image 920a for the piece 915a that does not have DECMV.And, evaluation of estimate (for example, the sum of squares of deviations of the pixel in the piece) is retrieved as LRMV for minimum motion vector.In Fig. 2 (b), the piece 926a on the regeneration LR image 920a of preceding frame becomes the evaluation of estimate minimum, retrieves corresponding motion vector LRMV925a.In addition, under the situation of evaluation of estimate greater than pre-set threshold of minimum, the motion vector that is judged as its piece is invalid, does not carry out the interpolation of LR multidate information.Under the situation of evaluation of estimate less than threshold value of minimum, the motion vector that retrieves is outputed to auxiliary data coding/rate controlled unit 311 as the LR multidate information 109 of LRMV and definition expands processing unit 310.
On the other hand, in changing to the type of LRMV,, between the original image LR image 920b of the original image LR image 910 of implementing the frame that definition expands and reference picture, implement dynamically retrieval for piece 915b with DECMV.And detecting evaluation of estimate (for example, the sum of squares of deviations of the pixel in the piece) becomes minimum motion vector.In Fig. 2 (b), the piece 926b on the LR image 920b of rear frame becomes the evaluation of estimate minimum, detects corresponding motion vector 925b.This motion vector and DECMV are compared, under the situation of difference value, outputing to auxiliary data coding/rate controlled unit 311 with detected motion vector as the LR multidate information 109 of LRMV and definition expands processing unit 310 greater than pre-set threshold.
As described later, the primary data use of the multidate information (SR multidate information) that a plurality of LR images that DECMV is gone out with the SR technology for detection as expression are corresponding with space-time between the HR image.The approaching more reality of this primary data dynamically, can reduce the time that in the detection of SR multidate information, spends more.Thereby, by using the decoding multidate information is revised the low clear multidate information that is generated, can reduce the operation time in the definition expansion processing.
In the 2nd step, expand in the processing unit 310 in definition, implement the generation of regeneration HR image 106 and SR multidate information 110 and handle.The regeneration LR image 102 (with reference to regeneration LR image) that definition expands processing unit 310 inputs 3 frames before and after implementing to have added on the regeneration LR image 102 that definition expands is totally 7 regeneration LR images, the DEC multidate information 108 and the LR multidate information 109 that use in its coding, generates regeneration LR image 106 and SR multidate information 110.
Fig. 5 represents that definition expands the internal structure of processing unit 310.At first, in primary data generation unit 405, generate the primary data that is used to carry out definition expansion processing.Specifically, primary data generation unit 405 as input, calculates the primary data that is expanded the motion vector of the SR multidate information 110 that processing unit 310 detects by definition to DEC multidate information 108 and LR multidate information 109.
Here, the SR multidate information is described.The SR multidate information is by the frame number of regeneration LR image and motion vector (below, the motion vector in the SR multidate information as " SRMV ") formation.As illustrating in the background technology, handle for the definition of implementing to have used the SR technology expands, for 6 with reference to each pixel on the regeneration HR images, needing to detect with regeneration HR image is motion vector (SRMV) during with reference to image.In addition, 1 pixel on the original image LR image can generate with falling to sample by implementing low-pass filtering in a plurality of pixels on original image HR image.
Use Fig. 2 (c), SRMV is described.In Fig. 2 (c), 1 lattice are represented 1 pixel.In addition, if pay close attention to 6 with reference to the pixel 927 on 1 regeneration LR image 920 of regeneration LR images, then pixel 927 is supposed to be equivalent to the block of pixels 942 that 8 pixels with pixel 927 corresponding pixels 941 and its periphery constitute on regeneration HR image 940.At this moment, can be by implementing low-pass filtering and the predict pixel 945 of falling sample calculation pixel 927 in the block of pixels 944 that constitutes by detected 9 pixels of 9 motion vectors of block of pixels 942 usefulness from regeneration HR image.Thereby the error that the SRMV943 of pixel 927 becomes pixel 927 and predict pixel 945 is 9 minimum motion vectors.
In the primary data generation unit 405 in this example, for 6 initial values with reference to needed 9 SRMV in the prediction of 1 pixel on all pixels calculating regeneration LR images on the regeneration LR image.As shown in Figure 1, the LR image generates by implementing low-pass filtering and fall sampling on the HR image.Therefore, in order to determine 1 corresponding with reference on all pixels on the regeneration LR image and the HR image of regenerating, for the pixel on the image (with reference to the HR image) that is enlarged into HR picture size with reference to regeneration LR image, can detect the initial value of the corresponding points of regeneration HR image as SRMV.That is, in the prediction of 1 pixel on regeneration LR image among the initial SRMV of needed 9 block of pixels 944, the MV that overlaps with the initial SRMV of adjacent pixels on the regeneration LR image becomes identical value.
Here, if the regeneration LR image 920 of Fig. 2 (c) is thought of as the frame of the tight front of regeneration HR image 940, then Fig. 2 (a) and regeneration LR image 920a (b) are corresponding with image 920, and it is corresponding that regeneration LR image 910 and the definition of regeneration HR image 940 expand preceding LR image.In addition, by along search (from the direction of image 920a) in the other direction to image 910 regeneration LR image 920a the DECMV or the LRMV of the regeneration LR image 910 during as the reference image, ask the pixel on the image 920a and the corresponding points of image 910.At this moment, for the pixel that does not have consistent corresponding points, handle the calculating corresponding points by the spatial interpolation of motion vector.And then, the motion vector of the LR image as unit suitable with corresponding points is extended for the motion vector of HR image as unit.
Then, by being DECMV or the LRMV of the regeneration LR image 920a during with reference to image with the regeneration LR image of the tight previous frame of image 920a, ask pixel on the regeneration LR image of tight previous frame of image 920a and the corresponding points of image 920a along search in the other direction.At this moment, for the pixel that does not have consistent corresponding points, ask corresponding points by the spatial interpolation processing of motion vector.And then, from the corresponding points between the tight previous frame of the corresponding points between image 910 and the image 920a and image 920a and image 920a, the pixel on the regeneration LR image of the tight previous frame of computed image 920a and the corresponding points of image 910 are extended for the motion vector of the LR image as unit suitable with corresponding points the motion vector of HR image as unit.By implementing this processing with reference to regeneration LR image continuously along the direction of leaving from regeneration HR image 940, the primary data of generation SRMV retrieval for whole 6.
Then, generate regeneration HR image 106 at super picture rich in detail synthesis unit 410.The primary data of the SRMV search that super picture rich in detail synthesis unit 410 generates 7 regeneration LR images 102 with at primary data generation unit 405 and quantization parameter 114 are as importing, the optimization of the SR multidate information 110 that enforcement is undertaken by dynamic retrieval unit 411 and infer the repeated treatments of the optimization of the regeneration HR image 106 that carries out unit 412 by coding noise, output SR multidate information 110 and regeneration HR image 106 (detailed process of optimization of having used repeated treatments is for example with reference to non-patent literature 1).
In the 3rd step, in super clear dynamic compensation unit 313, use SR multidate information 110 original image to be modified to high-precision correction SR multidate information 111.It is the input of original image HR image 101 and SR multidate information 110 that 6 original images with reference to 7 of the totals of regeneration LR images have been accepted to have added on 6 regeneration LR images that expand process object with reference to the original image LR images 104 of regeneration LR images, in definition in super clear dynamic compensation unit 313, outputs to definition expansion processing unit 310 and auxiliary data coding/rate controlled unit 311 revising SR multidate information 111.
Revise the SR multidate information by the piece positional information on the regeneration LR image, with reference to frame number, piece size, auxiliary motion vector (below, revising motion vector in the SR multidate information) formation as " correction SRMV ".The piece size is by being summarised in several pixel portion together coding, is used to reduce the purpose of the size of code of auxiliary data.The quantity that belongs to the correction SRMV that revises the SR multidate information is of a size of under the situation of 1 * 1 pixel at piece be 9, under the situation of N * N pixel, become (2 * N+1) * (2 * N+1) is individual.
In super clear dynamic compensation unit 313, use 6 original image LR images and original image HR image, detect 6 with reference to pixel on the regeneration LR image and the SRMV between the original image HR image.And then, under the situation of difference value greater than predetermined threshold value of the object pixel on the original image LR image and its predict pixel, do not carry out handling based on low-pass filtering and the sampling of falling sampling, between original image LR image, detect SRMV.Detected SRMV is compared with the difference value of the SRMV that is imported with mean value at several blocks of having specified, under the situation of difference value, the mean value of detected SRMV and inscape data are exported as revising SR multidate information 111 greater than threshold value.In addition, even under the situation of difference value of SRMV less than threshold value, under the situation of the sum of squares of deviations greater than threshold value of the piece of the predict pixel when the SRMV that has been suitable for detected SRMV and has been imported and the difference value of the pixel on the original image LR image, also the mean value of detected SRMV and inscape data are output as correction SR multidate information 111.Revise SRMV according to this, improve regeneration LR image with carried out between the HR image that definition expands corresponding points infer precision, improve the image quality of the HR image of regenerating.In addition, owing to can shorten the time that spends in the detection of SRMV, therefore can shorten the operation time in the generation of super picture rich in detail.
In the 4th step, expand in the processing unit 310 in definition, adjust regeneration HR image 106 and SR multidate information 110 once more.Definition expansion processing unit 310 is the regeneration LR image 102 of 3 frames of regeneration LR image 102 and front and back thereof of implementing the definition expansion and revise the 111 conduct inputs of SR multidate information, upgrades and also exports regeneration HR image 106 and SR multidate information 110.Specifically, in super picture rich in detail synthesis unit 410, after SRMV being replaced into correction SRMV, SR multidate information 110 and regeneration HR image 106 (detailed process of optimization of having used repeated treatments is for example with reference to non-patent literature 1) are upgraded in the optimization of the SR multidate information 110 that enforcement is undertaken by dynamic retrieval unit 411 and infer the repeated treatments of the optimization of the regeneration HR image 106 that carries out unit 412 by coding noise.
In the 5th step, in high definition dynamic compensation unit 314, use the regeneration HR image (with reference to the HR image) and the original image HR image that generate preceding 3 frames that finish, generate the multidate information of the image quality that is used for further improvement regeneration HR image.High definition dynamic compensation unit 314 is a plurality of original images with reference to HR image 107, regeneration HR image 106 and regeneration HR image that original image HR image 101 is as input, outputing to definition expansion processing unit 310 and auxiliary data coding/rate controlled unit 311 with reference to the HR multidate information 112 between HR image 107 and the regeneration HR image 106.
The HR multidate information is by the piece positional information on the reference HR image, constitute with reference to frame number, piece size, auxiliary motion vector (below, " motion vector " in the HR multidate information as HRMV).
Use Fig. 2 (d) that HRMV is described.In Fig. 2 (d), expression is for the upgrading with reference to the piece on the HR image 950 952 of the tight previous frame of piece 946 usefulness on the regeneration HR image 940 time, and the space motion vector between the piece 951 of same position on piece 946 and the space on the reference HR image 950 and the piece 952 that upgrades becomes the situation of HRMV954.The piece size is being encoded by a plurality of pixel portion are summarised in together, uses under the purpose of the size of code of minimizing auxiliary data.
In high definition dynamic compensation unit 314, at first,, compare original image HR image and regeneration HR image for preassigned multiple block, the sum of squares of deviations of the difference value of detection piece pixel is greater than the piece of predetermined threshold value.Then, from the piece of original image HR image contract detection position, with a plurality of be object search and the sum of squares of deviations of the difference value that extracts piece is the piece position of minimum with reference to the HR image.And, under the situation of the sum of squares of deviations less than threshold value of piece that the result who searches for obtains and the difference value that extracts piece, export corresponding HR multidate information 112.According to the HR multidate information of the information of utilizing this original image HD image, use and to have expanded the high-quality of definition in the past with reference to HD image, can revise the image quality of regeneration HD image, improve the image quality of regeneration HR image.
In the 6th step, expand the image quality improving of implementing regeneration HR image 106 in the processing unit 310 in definition and handle.Definition expands processing unit 310 with reference to HR image 107 and the 112 conduct inputs of HR multidate information, upgrades and output regeneration HR image 106.Specifically, in the dynamic compensation unit 421 of the image sharpening unit 420 of Fig. 5,, extract the piece image singly, repair unit 422 in image quality and the piece images that extract are synthesized on the regeneration HR image from reference HR image 107 based on HR multidate information 112.Implement this processing for all HR multidate informations, the regeneration HR image 106 that output has been upgraded.In synthetic method, use with original regeneration HR image on the weighting interpolation of corresponding blocks.
In the 7th step, the supplementary that handle is generated by auxiliary data generation unit 351 in auxiliary data coding/rate controlled unit 311 is LR multidate information 109, correction SR multidate information 111 and HR multidate information 112 codings, generate auxiliary data 113, to decoding device 20 outputs.
Fig. 8 represents the data format with 1 image-related auxiliary data of HR of regenerating.The auxiliary data 113 that becomes the coded object that is undertaken by auxiliary data coding/rate controlled unit 311 is from being used to retrieve initial beginning code 701 beginnings of 1 frame auxiliary data partly.In the beginning code, use the unique code that its datagraphic does not take place in auxiliary data.Synchronizing code 707 is the unique code that are used for distinguishing at every kind of data type described later the auxiliary data of 1 frame, is right after at the beginning code to omit later on.The 705 usefulness variable-length codes encode (about variable-length codes, with reference to patent documentation 1) from data type 702 to motion vector.
Piece positional information 703 is illustrated in reference to the location of pixels on frame number and the image (in LR multidate information and correction SR multidate information is the LR image, is the HR image in the HR multidate information).In addition, be under the situation of LR multidate information in data type, with reference to frame number because according to the decision of DEC multidate information, so only the information of location of pixels is encoded.
704 expressions of piece dimension information are the size of above-mentioned location of pixels as the piece of left upper end.Motion vector density information 708 is for above-mentioned scope, the pixel separation of the auxiliary motion vector of presentation code.Thereby, remove pixel separation and be 0 situation, the auxiliary motion vector of promptly encoding in the piece is beyond 1 the situation, by encircling 712 a plurality of auxiliary motion vectors codings repeatedly.Motion vector is encoded the vector value of LRMV, correction SRMV or HRMV according to the order of horizontal composition, vertical composition.In fact conduct of the motion vector of coding and the differential vector of predicting motion vector.
In LRMV, for the piece that does not have DECMV, the median of the motion vector composition of 3 pieces of adjacency (the median prediction of motion vector is with reference to list of references 1), for the piece that has DECMV, the vector value of DECMV becomes predicted value.In revising SRMV and HRMV, for the motion vector of same data type, the median of the motion vector composition of adjacent 3 pieces becomes predicted value.
Auxiliary data coding/rate controlled unit 311 reduces amount of information according to priority under the many situations of information encoded amount.Make under the preferential situation of high speed, according to LR multidate information, correction SR multidate information, the order of HR multidate information is preferential, and is under the preferential situation of the image quality that makes regeneration HR image, preferential according to the order of HR multidate information, correction SR multidate information, LR multidate information.In addition, in with a kind of data type, make evaluation of estimate (LR multidate information: with the difference value of DECMV, revise the SR multidate information: with the difference value of SRMV, the HR multidate information: the sum of squares of deviations of the pixel unit between the corresponding blocks on the piece of reference SR image contract and the HR image of regenerating) big piece is preferential.
The decoding device 20 of the invention process form then, is described.
Fig. 6 represents the overall structure of the decoding device 20 of the invention process form.Decoding device 20 possesses codec processing unit 501, definition expands processing unit 502, frame memory 503, data storage 504, data storage 505, frame memory 508 and auxiliary data decoding/separative element 531.
At first, video data encoder 120 is decoded as regeneration LR image 102 at codec processing unit 501.The regeneration LR image 102 of having decoded is kept in the frame memory 503, the multidate information (DEC multidate information) 108 of having decoded is kept in the data storage 504, the quantization parameter 114 of having decoded is kept in the data storage 505, exports according to the request of expanding processing unit 502 from the definition place.The detailed process of decoding processing for example is documented in " Text ofISO/IEC 14496-2 Third Edition ", among the March 2003 (below, as a reference 2).Definition expands processing unit 502 and as input, generates the HR image 106 of regenerating the supplementary of regeneration LR images 102, DEC multidate information 108, quantization parameter 114, auxiliary data 113 decodings (by 531 decodings of auxiliary data decoding/separative element and the LR multidate information 109 that has separated, revise SR multidate information 111 and HR multidate information 112) and with reference to HR image 107 (from the regeneration HR image that generates in the past of frame memory 508 outputs).
The definition of representing Fig. 7 expands the internal structure and the auxiliary data decoding separative element 531 of processing unit 502.The auxiliary data 113 that definition expands processing unit 502 request input regeneration LR images 102, DEC multidate information 108, quantization parameter 114, decoded and with reference to HR image 107 (the regeneration HR image that has generated), and at this moment, regeneration LR image and DEC multidate information need amount to 7 information that frame is relevant with 3 frames that added before and after showing constantly in implementing the frame that definition expands, perhaps with reference to the information of 3 frames parts of HR image front.That is, after the regeneration LR picture decoding of 3 frame fronts, implement definition expansion processing.
Definition in the decoding device 20 of the invention process form expands processing and is divided into 3 steps.Below, according to processing sequence explanation action.
In the 1st step, carry out the decoding of LR multidate information 109 and the primary data of SRMV search and generate.At first, from the data that definition expands the auxiliary data 113 separation LR multidate informations 109 of process object frame, carry out length-changeable decoding at auxiliary data decoding/separative element 531.Then, primary data generation unit 405 as input, generates the primary data of SRMV retrieval to the DEC multidate information 108 of the LR multidate information 109 of having decoded and 7 frames parts.Owing in Fig. 5, illustrated, therefore cast out explanation about the action of primary data generation unit 405 here.
In the 2nd step, revise the decoding of SR multidate information 111 and the generation of regeneration HR image 106.At first, expand the data of the auxiliary data 113 of picture frame being separated correction SR multidate information 111 from definition, carry out length-changeable decoding by auxiliary data decoding/separative element 531.Then, super picture rich in detail synthesis unit 510 as input, generates regeneration HR image 106 to the primary data of 111,7 regeneration of the correction SR multidate information LR image 102 of having decoded, SRMV search and quantization parameter 114.Specifically, in dynamic retrieval unit 511, according to revise SR multidate information 111 the primary data correction of SRMV search after, optimization by the SRMV that implements to be undertaken by dynamic retrieval unit 511 and infer the repeated treatments of the optimization of the regeneration HR image 106 that carries out unit 512 by coding noise makes regeneration HR image 106 convergences (about the detailed process of the optimization of having used repeated treatments for example with reference to non-patent literature 1).Wherein, about with revise the correction of SR multidate information SRMV be the high value of precision owing to can be estimated as, therefore only be limited to the inching in the following real number value scope of integer pixel.
In the 3rd step, carry out the decoding of HR multidate information 112 and the image quality improving of regeneration HR image 106 and handle.At first, expand the data that the auxiliary data 113 of picture frame is separated HR multidate information 112 from definition, carry out length-changeable decoding by auxiliary data decoding/separative element 531.Then, image sharpening unit 520 uses HR multidate information 112 and implements image quality improving with reference to HR image 107 and handle.Specifically, singly extract piece image based on HR multidate information 112 from reference HR image 107 by dynamic compensation unit 521, repair unit 522 by the piece image that extracts is synthesized on the regeneration HR image 123 that synthetic processing unit 510 is generated by super picture rich in detail in image quality, upgrade regeneration HR image 106.Implement this processing for all HR multidate informations, the regeneration HR image 106 that output has been upgraded.In synthetic method, the weighting interpolation of the corresponding blocks on use and the original regeneration HR image.
Fig. 9 represents to implement encoding process flow process of the present invention.In addition, therefore the detailed process of each step of Fig. 9 only illustrates the flow process of handling here owing to the explanation with Fig. 3, Fig. 4, Fig. 5 repeats.Encoding process begins after 601, handling in 602, and by handling, be original image HR image transform original image LR image based on low-pass filtering and the sampling of falling sampling.Handling in 603, conversion original image LR image when carrying out video coding, generate regeneration LR image and DEC multidate information by local decoding processing.Handling in 604, using original image LR image, at least a portion of DEC multidate information is being modified to high-precision LR multidate information.Handling in 605, using the DEC multidate information and the LR multidate information of a plurality of frames, generating the primary data of SRMV retrieval.Handling in 606, expanding by definition and handle, generating regeneration HR image and SR multidate information from a plurality of regeneration LR images.In step 607, use original image HR image and original image LR image, being modified to high-precision correction SR multidate information in a part of handling the SR multidate information that generates in 606.Handling in 608, SRMV is replaced into the correction SRMV that generates in 607 handling, implement definition once more and expand processing, upgrade regeneration HR image and SR multidate information.Handling in 609, using with reference to the HR image, detect use with reference to the HR image enhancement as the image quality of the regeneration HR image of object with reference to HR image and the multidate information (HR multidate information) of regenerating between the HR image.In step 610, use and to handle in 609 detected HR multidate information and, to improve the image quality of regeneration HR image with reference to the HR image.Handling in 611,, generating auxiliary data handling LR multidate information, the correction SR multidate information of handling generation in 607 that generates in 604 and handling the HR multidate information coding that generates in 609.After the processing 611 that is through with, finish encoding process (handling 612).
Figure 10 represents that the super picture rich in detail of implementing in the decoding processing of the present invention generates handling process.In addition, therefore the detailed process of each step of Figure 10 only illustrates the flow process of handling here owing to the explanation with Fig. 7 repeats.After super resolution chart picture generates processing beginning 801,, the LR multidate information is decoded handling in 802.Handling in 803, using LR multidate information and a plurality of frame DEC multidate information of having decoded in 802, generating the primary data of SRMV search handling.Handling in 804, revising the decoding of SR multidate information.Handling in 805, when handling the correction SR multidate information of having decoded in 804 and being set at the primary data of SRMV search, in the scope that the renewal of revising the SR multidate information is limited to below the integer pixel under such condition, search SRMV is from the regeneration LR image generation regeneration HR image of a plurality of frames.Handling in 806, the HR multidate information is decoded.Handling in 807, based on handling the HR multidate information of having decoded in 806, from the image quality of reference HR image enhancement regeneration HR image.After the processing 807 that is through with, finish super resolution chart picture and generate processing (handling 808).
Figure 11 represents to implement decoding processing flow process of the present invention.In addition, therefore the detailed process of each step of Figure 11 only illustrates the flow process of handling here owing to the explanation with Fig. 6 and Fig. 7 repeats.Begin after 901 in decoding processing,, coding video frequency data is decoded, generate regeneration low picture rich in detail, DEC multidate information and quantization parameter handling in 902.Then, the decoding handling the auxiliary data of having implemented in 903 to be encoded generates the LR multidate information, revises SR multidate information and HR multidate information.Then, handling in 904, using and handling the LR multidate information of having decoded in 903 and the DEC multidate information of a plurality of frames, generating the primary data of SRMV search.Handling in 905, when handling the correction SR multidate information of having decoded in 903 and being set at the primary data of SRMV search, in the scope that the renewal of revising the SR multidate information is limited to below the integer pixel, search for SRMV under such condition, generate regeneration HR image from the regeneration LR image of a plurality of frames.Handling in 906, based on handling the HR multidate information of having decoded in 903, from the image quality of reference HR image enhancement regeneration HR image.After the processing 906 that is through with, finish decoding processing (handling 907).
Figure 12 is used to illustrate and uses the image encoding of having preserved above-mentioned example to handle or the mediums such as floppy disk of the program that picture decoding is handled, situation about being implemented by computer system.
Outward appearance, profile construction and floppy disk that Figure 12 (b) expression is watched from the front of floppy disk, Figure 12 (a) expression is as the example of the physical format of the floppy disk of medium main body.Floppy disk FD is installed in shell F inside, and on the surface of this floppy disk, circular concentric ground forms from the inwardly all a plurality of magnetic track Tr of periphery, and each magnetic track is divided into 16 sector Se along angle direction.Thereby in the floppy disk of having preserved said procedure, on assigned region on the above-mentioned floppy disk FD, record is as the data of said procedure.
In addition, Figure 12 (c) expression is used for carrying out the structure of the record regenerating of said procedure on floppy disk FD.When said procedure being recorded among the floppy disk FD, write data through floppy disk as said procedure from computer system Cs.In addition, under the situation of by the program in the floppy disk above-mentioned coding or decoding device being constructed in computer system, from the floppy disk read routine, be sent to computer system by floppy disk.
In addition, in the above description, use floppy disk to be illustrated as data medium, and use CD also can carry out equally.In addition, recording medium is not limited to these, and IC-card, boxlike ROM etc. then just can similarly implement so long as can logging program.In addition,, comprise possessing CPU, carry out by the DVD player of the processing of software implementation or control, set-top box, portable phone etc. as computer.
Example of the present invention more than has been described, and also can have carried out following distortion, every kind of form is included among the present invention.
(1) relates to the variation of the utilization part of function
Do not need complete all constitute LR multidate information that auxiliary data of the present invention is a supplementary, revise SR multidate information, HR multidate information, only using the auxiliary multidate information of its part, generate under the situation of HD image from low picture rich in detail, also can access same effect.
Promptly, even using definition than the both sides of the original image HR image of original image LR figure image height and original image LR image or wherein under the situation of side generation auxiliary data of the present invention, also can improve the precision of the detection of dynamic between the image in picture decoding apparatus and the picture coding device, improve the image quality of regeneration HD image.In addition, owing to reduce the treating capacity of the News Search of picture decoding apparatus and picture coding device, therefore can reduce the operand in the image definition expansion processing.
Specifically, if according to picture decoding apparatus of the present invention and picture coding device, then when employing only constitutes auxiliary data to correction SR multidate information, adopt when revising SR multidate information and HR multidate information and constitute auxiliary data, adopt under any situation when revising a SR multidate information, HR multidate information and LR multidate information and constitute supplementary and can both realize raising pixel quality as described above and reduce operand.In addition, under the situation of the structure of taking not use the auxiliary multidate information of a part, in the auxiliary data generation unit 351 of picture coding device 10, the generation of not carrying out the corresponding multidate information of multidate information auxiliary with it just can realize.
In addition, also can implement synthetic processing of super resolution chart picture of Fig. 5, Fig. 7 even without SRMV search primary data.Thereby, not implement under the situation that primary data generates and the LR multidate information is encoded, correction SR multidate information of the present invention and HR multidate information also are effective.
And then, handle the regeneration HR image that also can implement by the synthetic processing generation of the super picture rich in detail of Fig. 5, Fig. 7 even without the image quality improving of handling the regeneration HR image that carries out by image sharpening.Thereby under the situation of the coding of not implementing image sharpening processing and HR multidate information, LR multidate information of the present invention and correction SR multidate information also are effective.
In addition, even be not always from the decoded picture of video coded data, but generate under the situation of the higher super picture rich in detail of definition from a plurality of images of obtaining by unit such as cameras or a plurality of images of being kept in advance the device such as hard disk, auxiliary data of the present invention also is effective.In this case, though there is not the DEC multidate information, it is effective revising SR multidate information and HR multidate information.
(2) relate to the variation of the change of functional definition
The method that the piece with reference on the HR image that extracts in image sharpening is handled is synthesized in the regeneration HR image is not limited to the synthetic processing of weighting.Even under the situation of the part of displacement regeneration HR image, HR multidate information of the present invention also is effective in the piece that extracts.
In addition, the kind of the low pass filter when the HR image transform is the LR image without limits.In the explanation of Fig. 2 (c), the branches of filter all is taken as 3 branches in horizontal/, and the filter or the different filter of coefficient of long branches also are effective.In addition, illustrated 1 corresponding situation of pixel on 9 pixels and the LR image on the HR image, but be not limited to this corresponding.Specifically, if since the pixel of pairing HR image be 1 pixel then can generate 1 pixel on the LR image, therefore also can lack the part of the pixel in the zone that filter exerts an influence.And then, in Fig. 2 (c), by the even column that falls the HR image of the pixel given up of sampling after and the pixel of even number line, but be not limited to this situation as the filtering arrangement.Even on by the HR image of low-pass filtering treatment the situation of the sample of real number position as the pixel sample on the LR image under, the present invention also is effective.
And then, the method that SRMV search primary data generation method is not limited to illustrate in Fig. 5.Not being the method along the direction search that leaves from regeneration HR image, also is effective according to the method for motion vector being calibrated according to frame period.
(3) relate to the variation of the coding method of auxiliary data
The data format that becomes the auxiliary data of coded object of the present invention is not limited to the form of Fig. 8.In addition, the Forecasting Methodology of motion vector also is not limited to the method that illustrated in Fig. 8.
In addition, in the data format of Fig. 8, replacement as reference frame number information, is carried out Methods for Coding to the piece positional information to location of pixels, image being carried out piece cut apart, also is effective there being the upper left side of having no way of to begin to carry out Methods for Coding according to the auxiliary multidate information that raster scan order is encoded.At this moment, might not need the piece dimension information.
And then, in the data format of Fig. 8, replace the piece positional information as reference frame number information, and then, location of pixels information, piece dimension information, motion vector density information are carried out Methods for Coding, image is carried out piece cut apart, beginning according to raster scan order also is effective from the upper left side having or not motion vector to carry out Methods for Coding.
In addition, in the data format of Fig. 8, the information of data type is encoded, also can consider to remove and repeat to encircle 713, the situation of data type being encoded for each piece in the auxiliary data information at each frame.In this form,, under the situation of the auxiliary multidate information of auxiliary data retrieval specific pixel, be effective therefore in hope owing to add synchronous code at the auxiliary multidate information of each piece.
And then the encoding precision of motion vector is unqualified.For example, be documented in list of references 2 or " Text of ISO/IEC 14496-10 Advanced Video Coding 3 RdEdition ", the high-precision motion vector among the September 2004 (below, as a reference 3) also is effective.
In addition, in the explanation of Fig. 8, carry out the coding of the inscape of auxiliary data, but be not limited to this method according to variable-length codes.It also is effective being documented in arithmetic coding method in the patent documentation 3 etc.
(4) relate to the variation of the inscape of auxiliary data
The inscape of auxiliary data is not limited to the record of above-mentioned example.
In addition, the auxiliary motion vector information of Fig. 8 also comprises the situation that does not have corresponding motion vector between 2 pixel images that is illustrated in.The corresponding fully one to one situation of pixel in any 2 pixels of video order is extremely rare, and it is effective that expression does not have the information of corresponding points.
And then the auxiliary motion vector information of Fig. 8 also comprises and not being according to vector value, but according to the situation of the News Search range information of the scope of piece dimension information appointment.In this case, omit the motion vector density information.Specify according to this hunting zone, can reduce the detection time of motion vector.
In addition, the auxiliary motion vector information of Fig. 8 be not vector value, but also is effective under the situation of the dynamic parameter of expression rotation, amplification, distortion etc.Be documented in the list of references 1 about dynamic parameter detailed contents such as (affine transformation parameter, projective transformation parameters).
And then, in Fig. 2 (b), the type of prediction of LR multidate information is defined as the type of prediction or the default value of DEC multidate information, yet is not limited thereto.In this case, in the LR multidate information, comprise type of prediction (forward prediction/back forecast/bi-directional predicted etc.).In addition, under the situation of coding/decoding mode of dynamic prediction of a plurality of reference frame that has been suitable for the use be documented in the patent documentation 3, in the DEC multidate information, comprise with reference to frame number.And then, as like that, under the situation of the coding/decoding mode of the piece size that can select the enforcement dynamic prediction from polytype, in the DEC multidate information, also comprise the piece size in record in the patent documentation 3.In this case, similarly in the LR multidate information, also comprise with reference to frame number and piece size.
In addition, need not obtain SRMV for all pixels on the regeneration LR image.According to the influence of containing etc., under can not situation, do not use this pixel to be optimized processing and can generate effective regeneration HR image yet by regeneration HR image detection.
And then the piece positional information of revising the SR multidate information is not that adopting with regeneration LR image is the value of benchmark, is that the value of benchmark also is effective but adopt with regeneration HR image.In this case, (for all pixels in the piece motion vector is encoded) when dynamic density information is 1, the pixel count in the piece is consistent with the quantity of the correction SRMV of pixel unit.
In addition, the shape of the piece of auxiliary multidate information also can be an arbitrary shape.In this case, also can encode shape information.In the coding method of shape information, the method for arithmetic coding has for example been arranged in the use shown in the patent documentation 2.
(5) relate to the variation of the method for dynamically inferring
In above-mentioned, between a plurality of original image LR images and original image HR image, implement to revise the detection of SRMV, and owing to be not to use original image LR image to be to use the method for HR image enforcement also to improve the precision of SRMV, so effect is also very high.In this case, the location of pixels of piece positional information becomes the value on the HR image.
In addition, in above-mentioned, between a plurality of regeneration LR images and regeneration HR image, implement the detection of SRMV, and owing to be not to use the LR image of regenerating to be to use the precision that also improves SRMV with reference to the method for HR image enforcement, so effect is also very high.
(6) relate to integrally-built variation
In above-mentioned, the coding/decoding method of video order is taked to be the method in list of references 1 and the list of references 2, but be not limited to these methods.
In addition, in above-mentioned, the definition extending method has been described and has inferred model with non-patent literature 1 and non-patent literature 2, and generating the technology of high-definition image from a plurality of low definition images, owing to can be suitable for the coding of supplementary of the present invention and utilize the high image quality of this coding to handle, therefore be not limited to this method.
And then, above-mentioned in, the quantity of the regeneration LR image that uses is taken as 7, and owing under quantity in addition, also can implement the present invention in definition expand to be handled, therefore quantitatively unqualified.In addition, also unqualified with reference to the quantity of HR image.
The definition introduced in this specification expands that to handle be that relation the high-definition image of 1 the unknown and a plurality of known low picture rich in detail is set up formula, infer the HD image of the best satisfy these formula and the technology of multidate information, non-patent literature 1 and non-patent literature 2 are to use the method for statistical to infer the example of technology of the high order vector of the best that satisfies evaluation function.As document " Sung Cheol Park et al; " Super-Resolution ImageReconstruction:A Technical Overview "; IEEE Signal ProcessingMagazine; May 2003 " (below, as a reference 4) shown in like that, in the definition expansion is handled, the whole bag of tricks is arranged, and the auxiliary data of this specification can be suitable for all methods under the situation of using the relation between motion vector performance high-definition image and a plurality of low picture rich in detail.As the method beyond non-patent literature 1 or 2, the method that the simultaneous equations found the solution is arranged, the method of perhaps having used the projected frame framework (projections onto convexsets) to convex set (for example, " A.M.Tekalp; M.K.Ozkan and M.I.Sezan; " High-resolution image reconstruction from low-resolution imagesequences and space Varying image restoration "; in Proc.IEEE Int.Conf.Acoustics; Speech and Signal Processing (ICASSP); SanFransisco, CA., vol.3, Mar.1992, pp.169-172 "; (below, as patent documentation 5)) etc.In the HD image that is generated, for as known a plurality of low definition images, the definition in space raises, and has in the contraposition and the synthetic feature of handling the radio-frequency component that does not take place in (for example, inlaying) that are created on the image between a plurality of low definition images.
In above-mentioned, infer in the processing of unit 412, as input, also can implement the present invention to quantization parameter 114 and infer in the processing at the coding noise that does not need quantization parameter at the coding noise of Fig. 5 and Fig. 7.Therefore, the quantization parameter that has or not definition to expand in handling does not influence enforcement the present invention.
In above-mentioned, constitute DEC multidate information 108, but be not limited to this inscape according to type of prediction, motion vector (DECMV).For example, as list of references 3, in the mode of a plurality of regeneration LR images, owing to need predict that the piece selection with reference to frame number, therefore is also contained in the DEC multidate information with reference to frame number at each as the reference image.
(7) the generation method of auxiliary data
In the explanation of the super clear dynamic compensation unit 313 in above-mentioned Fig. 4, under the situation of difference value greater than predetermined threshold value of the object pixel on the original image LR image and its predict pixel, replace with original image HR image to what in the detection of revising SRMV, use with reference to image from original image LR image, but the utilization of original image HR image is not limited to this method.For example, also can not use original image LR image, and use original image HR image for all pixels.In addition, about revising the output condition of SR information 111, in the explanation of Fig. 4, be defined as at mean value detected SRMV being compared with the difference value that is included in the SRMV in the SR information 110 in each specified multiple block, export under at difference value, but be not limited to this method greater than the threshold value situation.For example, also can not the difference value of MV, but detected SRMV, predicted value when being suitable for SRMV and the difference value of the pixel on the original image are used as output condition.In addition, the size Selection method of block also is not limited to a kind of method.And then, in the explanation of Fig. 4,, still be not limited to this method the piece mean value of the correction SRMV of output as detected SRMV.For example, also can not be the mean value that detects MV but the interior SRMV of piece as steady state value, detect with block unit.
About the auxiliary data system of selection in low clear dynamic compensation unit and the high definition dynamic compensation unit, equally also unqualified in a kind of method.
And then priority and evaluation method about the selection that relates to the auxiliary multidate information in the auxiliary data coding/rate controlled unit also are not limited to the method for representing in the explanation of Fig. 4.For example, even under the situation of having used the evaluation method of considering size of code, it also is effective having utilized the definition expansion processing of auxiliary data of the present invention.
(8) example of variation (5)
In super clear synthesis unit 410, use many regeneration LR images 102 to generate the SR multidate information 110 of regenerating between HR image 106 and a plurality of LR of regeneration image, and shown in variation (4) or (5), by using dynamically inferring between the HR image, can expect the precision of inferring that improves the SR multidate information and revise the SR multidate information.Therefore, use Figure 13, Figure 14 and Figure 15 explain the example of dynamically inferring between the HR image.In addition, here, about variation (1), only needing the situation of SR multidate information with definition expansion processing is that example describes.
Figure 13 represents the structure of definition transformed/coded unit 306, the i.e. variation of Fig. 4.It is the processing units that generate regeneration HR image 106 and SR multidate information 110 from a plurality of regeneration LR images 102 that definition expands processing unit 310, and Figure 14 illustrates its internal structure.
Figure 14 becomes the variation of Fig. 5.As from the input/output terminal of figure as can be known, it is different with Fig. 4, Fig. 5 with the structure of super clear dynamic compensation unit 313 that definition expands processing unit 310.That is, in the present invention, use the method for the definition expansion processing of SR information to be not limited to Fig. 5, the generation method of revising SR information also is not limited to Fig. 4.In the explanation of Fig. 2 (c), SR information as the corresponding multidate information of space-time between expression regeneration HR image and a plurality of LR images.Therefore, in the example of Fig. 2 (c), the branches decision (among Fig. 2 (4), being 9) of the low pass filter that when the HR image transform is the LR image, uses with the several certificates of bar of 1 corresponding SRMV of pixel on the LR image (revising SRMV).Yet, the structure of SRMV is as expression in variation (4) and (5), unqualified in the structure of Fig. 2 (c), in the present embodiment, in order to become the corresponding information of space-time between expression regeneration HR image and a plurality of HR images, SR information is as in explanation in the variation (4) like that, becomes 1 with 1 corresponding SRMV of pixel (correction SRMV) on the HR image.
Under the situation of the difference of considering 2 examples from the viewpoint of dynamic model, the SRMV of Fig. 2 (c) comprises dynamic model 201 and the sampling pattern 202 of Fig. 1, expression original image HR image 101 is corresponding with the space-time of a plurality of original image LR images 102, to this, the SRMV of present embodiment represents the motion vector of dynamic model 201, and promptly original image HR image 101 is corresponding with the space-time of a plurality of original image HR images.Because original image HR image and original image LR image are unknown, therefore in Fig. 2 (c), from the virtual HR image and the regeneration LR image generation SR information of virtual generation.To this, in the present embodiment, remove beyond the virtual HR image, also generate virtual HR image for a plurality of regeneration LR images, generate SR information by dynamically inferring between the HR.Thereby, because the generation method difference of 2 example SR information, though therefore according to identical dynamic model, resulting difference as a result.If according to present embodiment, then think by suitably selecting virtual HR image, can improve the quality and the processing speed of regeneration HR image.In addition, be to revise SRMV, so think that the importing effect of auxiliary data is also than Fig. 2 (c) height because the motion vector between the original image HR image applied flexibly.
In the present embodiment, the local definition of Figure 13 expands the super picture rich in detail synthesis unit 410 that processing unit 310 is equivalent to Fig. 5.Expand in the processing unit 310 in local definition, identical with Fig. 5, a plurality of regeneration LR images 102 as importing, but before being input to detection of dynamic unit 411, are transformed to enlarged image 118 to them in image magnification processing 406.Processing about image magnification processing 406, unqualified in the present invention, can consider simple linear interpolation processing, use the spline interpolation of B-spline function to handle, and then, can consider that the image repair model that utilizes non-patent literature 1 to put down in writing improves the method for image quality etc. in by interpolation processing enlarged image.
In definition expand to be handled, by carrying out the processing that definition expands processing unit 310 repeatedly, the situation of the high image quality of the HR image of seeking to regenerate was a lot.In this case, expand the regeneration LR image of object for definition, at first (the 1st repeated treatments), in the expanded view of image augmentation processing unit 406 generations as 118 as virtual HR image 119, be input to News Search unit 411.Then, in having generated the 2nd later repeated treatments of virtual regeneration HR image,, be input to News Search unit 411 being chosen as virtual HR image 119 with reference to HR image 107 (virtual regeneration HR image) by replacing enlarged image 118 at switch 407.For with reference to regeneration LR image, before the 1st repeated treatments, expanded sometimes and handled generation regeneration HR image (with reference to HR image 107) by definition.In this case, at switch 407, be chosen as virtual HR image with reference to HR image 107.Like this, by utilizing, can expect to improve inferring precision or shortening and handle such effect operation time of the SR information 110 that generates by News Search unit 411 with reference to HR image 107.
In News Search unit 411,, generate SR information 110 by dynamically inferring between 2 HR images.The SR information 110 that is generated is input to super clear dynamic compensation unit 311.In super clear dynamic compensation unit 311, use original image that SR multidate information 110 is modified to high-precision correction SR multidate information 111.In the present embodiment, super clear dynamic compensation unit 313 is received in the input of opening original image HR image 101 and SR multidate information 110 with reference to the total (L+1) of the regeneration HR image that has added definition expansion process object on the corresponding original image HR of the regeneration LR image image with a plurality of (L opens), detects and revises SRMV.And, for big zone of SRMV and poor (perhaps the predicated error of SRMV when being suitable for SRMV and having revised SRMV is poor) of revising SRMV or the zone that in the detection of the SRMV of the best, needs a large amount of operation times, generate and revise SR multidate information 111, output to definition and expand processing unit 310 and auxiliary data coding/rate controlled unit 311.In addition, as shown in the variation (7), the generation method of the correction SR multidate information in the super clear dynamic compensation unit 313 is not limited to a kind of method.Revising the SR multidate information for example can consider by the piece positional information on the reference HR image (the image that is enlarged into the HR size with reference to regeneration LR image), with reference to frame number, piece size, revise the information that SRMV constituted.The piece size is being encoded by several pixel portion are summarised in together, is used to reduce the purpose of the size of code of auxiliary data.The quantity that belongs to the correction SRMV that revises the SR multidate information becomes greater than 1 individual smaller or equal to N * N when the piece size is N * N pixel.The quantity that revises SRMV can be expressed in decoding one side by add the information of motion vector density information 708 grades in revising motion vector information.
Definition expands processing unit 310 and uses correction SR multidate information 111, upgrades SR multidate informations 110 with News Search unit 411.Infer in the unit 412 at coding noise, use to implement regeneration LR image 102 and L that definition expands and open regeneration LR image 102, and the SR multidate information 110 that has upgraded, virtual regeneration HR image 106 generated.Like this, in the present embodiment, definition expands processing and super clear dynamic compensation is handled by carrying out repeatedly, optimizes regeneration HR image 106, SR multidate information 110 and revises SR multidate information 111.As optimization method, can consider to determine the size of code of auxiliary data, adjust and revise SR multidate information 111 and make that the error of regeneration HR image 106 and original image HR image becomes minimum method etc. under its size of code, yet the application does not limit optimization method.In addition, after the optimization of revising SR multidate information 111, need generate regeneration HR image once more, make to generate identical regeneration HR image 106 with decoding one side in coding one side according to the usability methods of revising the SR multidate information.The correction SR multidate information of having optimized becomes auxiliary data 113 by being encoded by auxiliary data coding/rate controlled unit 311, transmits to decoding device.
In the present embodiment, also as shown in the variation (1), even be not always from the decoded picture of video coded data, but generate from a plurality of images under the situation of the higher image of definition, the present invention also is effective.In addition, shown in variation (4), need not calculate SRMV,, not use the dynamic data of its pixel to generate regeneration HR image 106 for the pixel of not finding corresponding points for all pixels.Thereby in the present embodiment also as illustrating in the variation (4), as revising the SR multidate information, it is effective that the information of the dynamic data of object pixel is not used in the transmission expression.
Figure 15 represents that the definition of present embodiment expands the internal structure and the auxiliary data decoding/separative element 531 of processing unit 502.In the present embodiment, the definition of Figure 15 expands the synthetic processing unit 510 of super picture rich in detail that processing unit 502 is equivalent to Fig. 7.
Definition expands that processing unit 502 uses regeneration LR images 102, the correction SR multidate information 111 of having decoded and with reference to HR image 107 (the regeneration HR image that has generated), generates regeneration HR image 106 and SR multidate information 110.At first, from the definition expansion auxiliary data 113 of picture frame is separated correction SR multidate informations 111, carry out variable-length codes by auxiliary data decoding/separative element 531.Then, definition expands processing unit 502 by image magnification processing 406 generation enlarged images 118.As shown in Figure 14, select virtual HR image 119 according to predetermined order from enlarged image 118 with reference to HR image 107.Then, use a plurality of virtual HR images 119 and revise SR multidate information 111, generate SR multidate information 110 and regeneration HR image 106.Specifically, the generation of the SRMV that is undertaken by dynamic retrieval unit 511 by repeated treatments and be optimized by the generation that coding noise is inferred the regeneration HR image 106 that carries out unit 512.
In the present embodiment, between the HR image, carry out the generation of SR multidate information 110, and under situation about not existing with reference to HR image (the 1st repeated treatments), omit the processing of image magnification processing, implements dynamically to infer between regeneration LR image, interpolation processing or zero degree maintenance method by the motion vector value are enlarged into SRMV.Therefore, in this example, according to the number of times of repeated treatments, the inscape of revising the SR multidate information is sometimes promptly revised meaning or the quantity difference of SRMV.In addition, it is also conceivable that not being by the News Search between the original image, but by the detected motion vector of the News Search between the common reproduced picture as revising the SR multidate information, situation about transmitting for the operand that reduces decoding one side.
About revising the method for utilizing of SR multidate information, can consider some kinds of methods, be not limited to a kind of method in this application.As utilizing the method for revising SRMV, can consider not carry out the News Search of SRMV and be suitable for the method for revising SRMV, after being suitable for correction SRMV, the method of adjusting once more by News Search, then, from determine the method for final SRMV by detected SRMV of enforcement News Search and correction SRMV.About adjusting once more, can consider in fact to have considered the difference of the regeneration LR image that in the generation of regeneration HR image, uses, adjust the situation of precision that the situation or improve that makes regeneration HR image become high-quality situation, the correction SRMV that transmits with block unit is improved as the SRMV of pixel unit is revised the various MV of raising such as situation of SRMV pixel precision etc.About method, can consider to revise SRMV and be the situation by implementing the detected SRMV of News Search and the differential vector of final SRMV or revise the situation etc. that SRMV and mean value by the detected SRMV of enforcement News Search are final SRMV from the final motion vector of 2 motion vectors decision.Promptly, both can be to revise the SR multidate information to comprise correction SRMV, this correction SRMV is used as the form of motion vector, can also be to revise the SR multidate information to comprise correction SRMV, use a plurality of reproduced pictures to detect middle motion vector, handle by this centre motion vector and the addition of revising SRMV, generate motion vector.In addition, revise the SR multidate information and comprise correction SRMV,, also can generate motion vector by using a plurality of reproduced pictures to upgrade initial motion vector revising the initial motion vector of SRMV as motion vector.
In addition, corresponding about the repeated treatments when utilize revising the SR multidate information also can be considered several different methods.If roughly distinguish, the method that is suitable for the method for identical correction SRMV and only is suitable for is arranged then in the specific times of repeated treatments in the repeated treatments of all number of times.About the latter,, and then can consider in identical zone or piece, to transmit situation for the different correction SRMV of the number of occurrence according to the reason that is used to reduce operand etc.
In this variation, code device, decoding device have been described, and, equally also can be out of shape about Fig. 9～handling process shown in Figure 11.In this case, though diagram not, the method by above-mentioned explanation is implemented in the generation of the virtual HR image 119 shown in the super clear processing 805 of Figure 10.In addition, the image encoding of this variation is handled or picture decoding is handled as illustrated in fig. 12, can be by the computer system enforcement based on program.
(9) relate to the variation of utilization with reference to the HR image
In Fig. 5 and Fig. 7, utilize HR multidate information 112 to implement the image quality sharpenings and handle, even and in high definition dynamic compensation unit 314, do not use the method for HR supplementary, also can implement the image quality sharpening and handle.In this case, use and a plurality ofly infer the regeneration HR image 123 of unit 412 (512) output and expand pixel data on the virtual regeneration HR image of object images by the definition that the image quality repair process has been revised, detect HRMV by dynamic compensation unit 421 (521) with reference to HR image 107, coding noise.At this moment, can consider the pixel data that finishes by the correction that utilizes on the regeneration HR image to improve pixel precision.For example, implementing under the situation of correcting process the pixel that can in search, use the renewal of the upper end of the current block on the regeneration HR image that has upgraded and left end to finish according to raster scan order with block unit.Repair the image quality of utilizing detected HRMV improvement regeneration HR image 106 in the unit 422 (522) in image quality.In addition, as shown in the variation (2), in image sharpening is handled, use method to be not limited to a kind of with reference to the image quality of HR image enhancement regeneration HR image.Remove the pixel portion of 2 images (the HR image that generates from reference HR image 107 and HRMV and virtual regeneration HR image 1123) be weighted beyond the synthetic method, be replaced into the method for the pixel on the HR image that is generated with reference to the HR image in addition, perhaps use a plurality of methods of seeking to optimize regeneration HR image with reference to the SRMV between HR image 107 and the virtual HR image etc.And then here, revising by the method for dynamic compensation unit 421 (521) detected HRMV with HR multidate information 112 also is effective as the method that improves the performance that the image quality sharpening handles.In this case, being included in motion vector in the HR multidate information 112 (revising HRMV) becomes by the difference motion vector between dynamic compensation unit 421 detected HRMV and the final HRMV.In addition, method as image quality sharpening processing, prepare to use the HRMV that in Fig. 5 and Fig. 7, illustrate method, use the method for the correction HRMV that has illustrated here and detect the method etc. of HRMV by dynamic compensation unit 421 (521), comprise that in the inscape of HR multidate information its method of selecting information also is effective as the method for the treatment effeciency of raising image quality sharpening processing.
In addition, in above-mentioned, infer in the unit at the coding noise of Fig. 5, Fig. 7, Figure 14 and Figure 15, use a plurality of regeneration LR images 102 and SR multidate information 110 to implement the optimization of regeneration HR image 123 (in Figure 14 and Figure 15, being 106), and for there being the frame that generates the regeneration HR image that finishes, it also is effective replacing regeneration LR image 102 to use with reference to HR image 107.In this case, regeneration HR image 107 coding noise that is input to Fig. 5 and Figure 14 is inferred in the coding noise amending unit 512 of unit 412 and Fig. 7 and Figure 15.In addition, in this variation, it is also contemplated that the model that has omitted sampling pattern 202 among Fig. 1.About dynamic model 201, can utilize the SRMV between the virtual HR image of in the explanation of Figure 14 and Figure 15, representing.
(10) relate to the distortion that utilizes method of the inscape of auxiliary data
Do not need all to transmit the data of inscape,, can remove from the inscape that transmits for information in coding one side and the unique decision of decoding one side.For example, in the information such as feature of using the image that has jointly in coding one side and decoding one side, can stipulate to omit and transmit its composition data under the situation of a part of inscape.In the combination of the data of inscape,, also can remove from the inscape that is transmitted for unwanted data.For example,, do not need the transmission block positional information,, do not need to transmit auxiliary motion vector yet according to situation when being suitable for expression whether during at the method for the auxiliary motion vector of each block movement.In addition, as illustrating in the variation (4), in the generation of regeneration HR image under the invalid situation, do not need to transmit yet and revise SRMV at the SRMV that represents zone arbitrarily or piece.And then, not the method that the rule of acquiescence according to coding one side and decoding one side suppresses to transmit a part of data of inscape in addition, but comprise that in inscape expression carries out the pattern information which kind of data transmits, the method for the inscape that transmitted of expression expressly.
As the method that transmits auxiliary data, the method that the data of inscape is divided into sequence unit, frame unit, sheet unit's (constituting by a plurality of), the transmission of block unit hierarchical also is the effective ways that are used to reduce size of code.That is, can reduce size of code by the data that transmit inscape according to the task layering of its assignment.In this case, the information that is transmitted in upper level need not transmit in the next.For example, under situation, in the auxiliary multidate information in block unit, do not need to transmit this information as the supplementary transmission block size of frame unit.In this case, also the data of the common element that transmits with block unit are segmented in several transmission types (combinations that the inscape data are different) or the transmission group (inscape classification of Data), represent expressly that in decoding one side the method with the corresponding pattern information of combination of transmission type or transmission group is effective.And then, go back the method for handlebar pattern information as the information transmission of sheet unit or frame unit, we can say that the method for being inclined to as the variation of the image under control reflecting regional or the frame unit is effective.
As other assistance data delivery methods, data based its locality of handlebar inscape is divided into several groups, transmits expression and whether changes the respectively method of the information of the value of the inscape of group.Can think the size of code that can reduce auxiliary data according to this method.For the data of not change group, pre-determine and be useful in the firm value that transmits in front or the regulation of default value.The data of the inscape of the group of change transmit as auxiliary data.
As shown in the variation (4), LR multidate information, the inscape of revising SR multidate information and HR multidate information are unqualified.For example, also can consider to transmit the LR multidate information, revise the method for the kind (data type 702) of moving information of SR and HR multidate information with block unit.In addition, as in variation (8) or illustrating in above-mentioned, express the combination of data of inscape of auxiliary multidate information or the pattern information of transfer approach and be also contained in the variation of inscape.The transfer approach of this pattern information is not limited to revise the SR multidate information, also can be suitable for for LR multidate information or HR multidate information.
And then, the method of utilizing method that is illustrated in the correction SR multidate information shown in the variation (8) as the data of inscape expressly also is effective, use original image, according to the information that requires in coding one side, the processing in the side of can implementing effectively to decode.The method of utilizing about this auxiliary multidate information also is not limited to revise the SR multidate information, also can be suitable equally for the method for utilizing of LR multidate information or HR multidate information.In utilizing the information of method, comprise that the usability methods of adjusting auxiliary multidate information (does not carry out the News Search in the decoder, the auxiliary multidate information of reflection, perhaps with the auxiliary multidate information of decoder adjustment) time method of adjustment (generate the motion vector of trickleer unit, adjust the pixel precision of motion vector).In addition, also be included in the corresponding information (in all repeated treatments, be suitable for and assist multidate information, be only suitable in the processing of the specific number of occurrence) of the expression that illustrate in the variation (8) and repeated treatments or represent the information of the specific times of repeated treatments.As the method for utilizing of the auxiliary multidate information of expression, it is also conceivable that by the information of utilizing purpose (reduce operand, improve the quality of regeneration HR image) of the auxiliary multidate information of expression is sent to receive a side that change receives the method for the processing in the side.
On the other hand,, it is also conceivable that, do not transmit and the method for expression motion vector quantity, perhaps the method that determines uniquely according to the piece dimension information to receiving a side for motion vector density information 707 as other example.
And then, about the LR multidate information, though and have DECMV but its value is changed to the method for different LRMV because the method for the new LRMV of interpolation is arranged in the piece that does not have DECMV, therefore it is also conceivable that not to be from the DECMV generation but to transmit this LRMV expressly and utilize information.In this case, by in inscape, comprising the piece size, can provide precision higher multidate information for definition expands processing.
As the variation of HR multidate information, at first, the method for dynamically inferring is arranged.As being (variation (9)) enforcement between regeneration HR image, the adaptation that also is to use original image HR image to implement is selected, by the data of dynamic presuming method as the inscape of HR multidate information are transmitted, can seek to reduce the size of code of HR multidate information and optimize regeneration HR image quality.In addition, about image quality improving repair process (using processing) with reference to the image quality of HR image enhancement regeneration HR image, owing to the weighting synthetic method is also arranged or is replaced into some kinds of candidates such as method, therefore by the information that transmits the expression synthetic method expressly can expect the to regenerate image quality improving of HR image with reference to the HR image.
Also can consider distortion about the SR multidate information.For example, SRMV is the expression LR image data corresponding with the space-time of HR image in Fig. 5, and in Figure 14, is between the expression HR image or the data of the space-time correspondence between the LR image, the technique of expression difference.As sharpness information,, can carry out and the local corresponding image quality improving of feature, the further improvement that brings image quality by transmitting this difference expressly with the frame unit or the unit that determines.Therefore,, replace revising SRMV and transmit,, also can improve the accuracy of detection of SRMV even do not transmit SRMV by on the inscape of revising SR information, adding this information.In addition, as Figure 14 or as shown in Figure 15, among the candidate of the virtual HR image that in the detection of SRMV, uses, enlarged image is arranged and, can select any with reference to the HR image.By on the inscape of revising SR information, add the kind that is used for selecting expressly virtual HR image with reference to image information, can expect to improve the accuracy of detection of SRMV.At this moment do not need to transmit correction SRMV yet.
About revising the variation of the moving information of SR, remove the above-mentioned sharpness information (MV of the MV of LR image level or HR image level) that it is also conceivable that correction SRMV in addition that illustrates.By transmitting this data, can represent to be adapted to the definition of regional local feature expressly, in addition, it is also conceivable that by representing effective repeated treatments number of times expressly, after specified repeated treatments number of times, to receiving the situation that a side transmits does not need to search for once more this regional SRMV.According to this information, can suppress invalid News Search and handle.
(11) application of auxiliary data
Hereto, be transmission and the auxiliary multidate information that the center has illustrated auxiliary multidate information with auxiliary motion vectors such as correction SRMV.In this case, receive a side, can generate more high-quality regeneration HR image in reception one side by useful motion vector is notified to.On the other hand, if be conceived to the motion vectors such as SRMV that in definition expands processing unit, generate,, for example be used to implement the method for News Search and condition and implement receiving a side according to predetermined rule then in the generation conditions needed of motion vector.Below, be that example describes with the News Search.In this News Search, the corresponding several different methods of various features with image is arranged, and the motion vector that searches out to receiving under the situation that a side transmits, can use former data sending the best approach and the condition that side decision is used for News Search.Yet, as definition expands processing, carry out under the situation of News Search at the pixel data that uses decoding to finish, need determine its method or condition in reception one side that does not have former data.Therefore,, be illustrated in coding one side here, implement to be used for the selected of the Perfected process of News Search and condition with initial data, this information as auxiliary multidate information to the method that receives side transmission.This method has the effect of the precision that improves motion vector and the effect that shortens the needed time in the detection of motion vector owing to also use original digital image data, therefore can think to improve the quality of regeneration HR image and the effective ways of efficient.In addition,, need become identical regeneration HR image with reception one adnation, therefore need and receive method and the condition that a side has News Search jointly in coding one side in coding one side in order to utilize regeneration HR image in the processing afterwards.In order to realize having jointly, have in coding one side and receive the method that a side is carried out the method for News Search in advance or is predetermined condition, and as shown here, by transmitting, can reduce image quality with corresponding operand of the locality of image or raising HR image as auxiliary multidate information.
About the method for News Search and the kind or the quantity of condition (below, be called the News Search conditional information), unqualified in this manual.About the example of its kind, hunting zone or searching method etc. are arranged, details is narrated in the back.The generation method of explanation News Search conditional information among Figure 13.Though do not illustrate, yet in this case, think that regeneration HR image 106 also outputs to super clear dynamic compensation unit 313.Super clear dynamic compensation unit 313 is included in the candidate of News Search conditional information and revises in the SR multidate information 111, is input to definition and expands processing unit 310.Expand in the processing unit 310 in definition,, generate SR multidate information 110 and regeneration HR image 106 according to being included in the News Search conditional information of revising in the SR multidate information 111.Super clear dynamic compensation unit 313 uses regeneration HR image 106 and original image HR images, estimates News Search conditional information (for example, evaluation of estimate be absolute value difference and).Candidate for a plurality of News Search conditional informations implements this processing, and selecting evaluation result is optimal conditions information.In addition, the determining method of the generation method of News Search conditional information is not limited to this method.For example, it or not the comparison of regeneration HR image and original image HR image, but SRMV that generates in definition expands processing unit and the correction SRMV111 that generates in super clear dynamic compensation unit 313 are compared, selecting its difference also is effective for the method for hour News Search conditional information.In this case, do not need to super clear dynamic compensation unit 313 output regeneration HR images 106.
About the transfer approach of News Search conditional information, equally with motion vector information can consider some kinds of methods, being divided into the method that frame unit or block unit layering transmit also is the effective ways that are used to reduce size of code.About the data that transmitted, can consider directly to transmit the method for data, perhaps prepare some kinds of candidates, transmit the method for selection information.Transmit the degree of freedom height of the method for numerical value, but the increase that will bring size of code on the other hand.Therefore, can think the method that when transmitting, can use with batch units such as sequence unit or frame units.Under situation about transmitting, be suitable for the method for from the candidate, selecting basically with block unit or pixel unit.
Here, the method that is suitable for has been described in the definition expansion that utilizes auxiliary motion vector and News Search conditional information is handled.And as this variation, might comprise under the situation of News Search conditional information at auxiliary multidate information, as the processing of the part in the image-region, need to implement different processing according to the auxiliary motion vector and the News Search conditional information that can utilize.
Comprise auxiliary motion vector and do not comprise under the situation of News Search conditional information that definition expands the auxiliary motion vector that the processing unit utilization has been restored at auxiliary multidate information, detect the SRMV (HRMV) of its zone (piece), generate regeneration HR image.Utilize the detailed process of method owing to illustrated, therefore here omit the utilizing in the method for correction SRMV of variation (8).Restored method as auxiliary motion vector, as as shown in the explanation of Fig. 8, generally be the method for difference motion vector addition that prediction motion vector of obtaining according to predetermined method and decoding by auxiliary data are obtained, be not limited in this application as shown in the variation (3).For example, under the motion vector of having decoded is situation by the detected SRMV of predetermined method (HRMV) and the difference motion vector of final SRMV (HRMV), even because direct coding its size of code also seldom, therefore can omit prediction processing.In addition, about prepare multiple auxiliary motion vector utilize method the time method for selecting, have by selected information is transmitted as auxiliary multidate information, can transmit the method for usability methods expressly, perhaps according to the method for the unique decision of condition (for example, the data value of the inscape in the auxiliary multidate information) etc.
Assisting multidate information to comprise the News Search conditional information and do not wrapping under the situation of auxiliary motion vector, definition expands processing unit according to the News Search method or the condition of having decoded, detect the SRMV (HRMV) of its zone (piece), in the generation of regeneration HR image, utilize.At this moment, in the News Search conditional information of having decoded, do not comprise under the situation of necessary all information,, be suitable for predetermined value for its information.For example, in the hunting zone littler,,, has the effect that can reduce the needed operand of News Search by sending the information of hunting zone with regard under enough conditions than predetermined value.
In addition, comprise the News Search conditional information and do not comprise other example of assisting motion vector as auxiliary multidate information, can consider that the conditional information according to predetermined News Search detects after the SRMV (HRMV), according to the method for the detected SRMV of conditional information correction (HRMV) that has decoded, perhaps according to the method for the News Search conditional information correction of having decoded by the detected SRMV of repeated treatments (HRMV) of front etc.For example, be used to revise the little hunting zone of SRMV (HRMV), have the effect of balance of obtaining operand and search performance etc. by transmission.In addition, about have a plurality of News Search conditional informations utilize the candidate of method the time method for selecting, have by selected information is transmitted as auxiliary multidate information, can transmit the method for usability methods expressly, perhaps according to the method for the unique decision of condition (for example, the data value of the inscape in the auxiliary multidate information) etc.
Comprise under the situation of auxiliary motion vector and News Search conditional information having simultaneously from determine the method for final SRMV (HRMV) according to the detected SRMV of News Search conditional information (HRMV) and the auxiliary motion vector that restored at auxiliary multidate information.In the example in this case, comprise that auxiliary motion vector is according to the situation of the detected SRMV of News Search conditional information (HRMV) with the differential vector of final SRMV (HRMV).For example, replace with big area unit such as frame unit or sheet unit, need the motion vector of the above precision of block unit, by only reducing size of code encoding from the difference value of inferred value for the News Search conditional information.In the zone of the mis-behave of the motion vector prediction of the motion vector in having used the dispersed big neighboring region of motion vector is effective.In addition, detect roughly motion vector with seldom operand,, also have the effect of the operand that can reduce News Search by itself and the such order of difference motion vector addition according to the News Search conditional information.
Other example that comprises auxiliary motion vector and News Search conditional information as auxiliary multidate information simultaneously, auxiliary motion vector that with good grounds News Search conditional information correction has been restored is the method for the auxiliary motion vector of having revised as final motion vector.If according to this method, then,, can be modified to the more motion vector of narrow zone (piece or pixel) to the auxiliary motion vector that is transmitted according to the News Search conditional information for the auxiliary motion vector of zone (piece) transmission widely.Therefore, can reduce size of code.And then, also have after SRMV (HRMV) having been revised with auxiliary multidate information, adjust the method for revising SRMV (HRMV) once more according to the News Search conditional information.
About whether auxiliary motion vector and News Search conditional information being arranged, can be considered as method that pattern information such shown in the first half of variation (10) represents expressly etc.In addition, about being included in parameters such as method in the News Search conditional information or condition, can reduce size of code by using layering to transmit respectively.
Because also have auxiliary multidate information to comprise that neither auxiliary motion vector does not comprise the situation of News Search conditional information yet, therefore explanation definition in this case expands the example of the order of handling.One of them is the situation that auxiliary multidate information comprises motion vector invalid information in the generation of regeneration HR image in its zone of expression (piece).In this case, definition expands the News Search that processing unit is not implemented the SRMV (HRMV) of its zone (piece), does not use its regional SRMV (HRMV) to generate regeneration HR image.If with above-mentioned non-patent literature 1 (model of Fig. 1) is example, then have for Matrix C (d_lk), replace becoming the motion vector that can not use, use the motion vector of neighbor, the method that the motion vector that generates by the interpolation processing substitutes, perhaps for matrix H, be on 0 the basis making with becoming the corresponding filter coefficient of the motion vector that can not use, adjust the method for the pixel corresponding filter coefficient related with becoming the motion vector that can not use.In addition, as comprising that neither auxiliary motion vector does not comprise other situation of News Search conditional information yet, has auxiliary multidate information to comprise that the renewal of the SRMV (SRMV) of its zone (piece) is treated to the situation of the number of times of effective repeated treatments.In this case, definition expands processing unit in the later repeated treatments of specified number of times, need not search for its regional SRMV (HRMV) once more, implements the generation of regeneration HR image.
Below, be illustrated in the example of necessary condition in the News Search.
A) dynamic search method
In Fig. 2, use BMA to be illustrated as dynamic testing method, and in the News Search method, there is gradient method (for example, patent No. 3271369 (offering 6 as a reference)), several different methods such as model matching method (for example the spy opens flat 1-228384 (as a reference 8)) pixel matching method (for example, No. the 2934151st, patent (as a reference 7)).The validity of these methods is because according to the feature of pixel inner region and difference, therefore by as the condition of News Search to the suitable method of decoding one side notice, can expect to improve the performance of detection of dynamic of a side of decoding.
B) hunting zone and search center
When dynamically exporting, if searching image integral body then need huge operand, therefore design hunting zone is usually searched in its scope.Desirable output area is because according to the feature of image-region and difference, so its condition is also very big for exporting the influence that the result brings.Thereby,, can not have the News Search of waste by transmitting suitable hunting zone to decoding one side expressly.In addition, by the center of setting speed scope suitably, can reduce the hunting zone.Thereby,, can improve the treatment effeciency of following News Search in decoding one side by transmit the method for decision search center expressly to decoding one side.As the method at decision News Search center, the method for the dynamic amount that utilization goes out from the News Search results presumption of neighboring region is for example arranged or with dynamic amount 0 method that is search center etc.The example of expression piece coupling among Figure 16.Among the figure, a100 as present frame, a101 as the object search piece, a200 as reference frame, with the spatially consistent a201 of piece a101 as the piece on the reference frame.A202 is the motion vector of the adjacent block of a101, uses when being used to set the search center a204 of piece a101.Hunting zone a206 be set in search center a204 around, by in the hunting zone, searching for, detect motion vector a203 and prediction piece a205.As the motion vector that is used to determine search center, the left side of piece a101, go up, the motion vector of upper right 3 pieces is as the candidate, the frequent motion vector that constitutes from the median of each composition that uses in the News Search in encoder.
C) search block size
About implementing the unit of News Search, also different according to the size that feature suited of image-region.For example, under the situation of piece coupling, prepare multiple search block size,, improve the precision of News Search by transmitting the search block size to decoding one side with sequence unit, frame unit, sheet unit or block unit (bigger) than search block size.In addition, the situation that is not according to block unit but arbitrary shape is carried out News Search as unit is also arranged.In this case, transmit the method etc. of shape information (triangular plate etc.) or Region Segmentation.
D) News Search method
When carrying out News Search, if since in the hunting zone search whole then need a large amount of operands, therefore can consider the situation of sampling search in the hunting zone.Thereby,, can adjust operand and search performance by the function that transmits the method for News Search expressly to decoding one side is set.As the News Search method, have and in the hunting zone, do not have the full search of searching for with omitting, search for to count pixel separation in length and breadth, dwindle dynamic tree-like search etc. according to such order around the position that retrieves with narrower pixel separation search.In addition, not in the hunting zone, once to search for, but, determine search center according to its result with the big big hunting zone of piece size search, also be effective with the such hierarchical search in the littler hunting zone of little piece size search as the method that reduces operand.In addition, in this hierarchical search, need transmit hunting zone and search block size according to the number of plies.
E) search order
Search order during about the enforcement News Search, the method of searching in according to the scope of raster scan order in News Search to the bottom right from upper left is arranged, perhaps the some kinds of methods such as method of outwards searching for according to the spiral order from the center of dynamic hunting zone.By transmitting search order to decoding one side expressly, make to obtain same result in coding one side and decoding one side.In addition, at a high speed the situation that the condition of ending search is set is arranged in order to realize.About this search termination condition (threshold value of predicated error etc.),, also can reduce the operation time that needs in the News Search of decoding unit by representing to decoding one side expressly.
F) dynamic retrieval precision
About the dynamic retrieval precision in the News Search, in fact in standard modes such as MPEG, use multiple precision such as 1 pixel precision, 1/2 pixel precision, 1/4 pixel precision, and then 1/8 pixel precision.By transmitting this search precision, also can seek to optimize operation time and image quality to decoding one side.In addition, about the generation method of real number pixel,, can seek to optimize image quality by transmitting its method to decoding one side.
G) evaluation function
Evaluation function when carrying out News Search, the absolute value of the piece absolute value that uses predictive error signal and, the identical sum of squares of deviations, predictive error signal and and the several different methods such as evaluation of estimate that calculate from the motion vector size of code.About evaluation function, select in coding one side with former data, transmit to decoding one side, can seek to optimize operation time and News Search performance.In addition, also have dynamic mode (parallel Move Mode, affine transformation pattern, projective transformation pattern), News Search method various conditions such as (forward directions, reverse).
Hereto, be treated to prerequisite with the definition expansion, show conditions needed is encoded/transmitted in the generation of motion vector a method, and the order of conditions needed is receiving a side and can be useful in ad lib in the device or software of generation motion vector in receiving the generation that a side is transmitted in motion vector.For example, in video coding that decoding one adnation becomes etc., can be suitable for motion vector.Figure 16 explanation is in the video coding mode, and the pixel data that using decodes finishes is in the method for decoding one side search motion vector.The preceding frame that a200 represents to decode and finishes, the present frame of a100 presentation code object.A100 encodes according to raster scan order to the bottom right from upper left with block unit, and among the figure, piece (7 pieces) coding/decoding in the zone of expression a103 finishes.When the motion vector of search block a101, from the decoded number pixel data formation model a108 that decoding finishes regional, the sum of errors that detects in the model is minimum regional a208.At this moment, detect a203 as motion vector, piece a205 becomes the prediction piece of piece a101.In coding one side, the error block coding of encoding block a101 with prediction piece a205, motion vector is not encoded.In decoding one side, carry out News Search according to the condition identical and detect motion vector with coding one side.Then, by according to the prediction piece that generates at the detected motion vector of decoding one side and the error block addition of having decoded, ask the regeneration value of encoding block.Thereby, in decoding one side, in comprising the video coding that generates the information processing related, implement the condition of News Search in coding one side decision like this with former data with motion vector, by this condition is transmitted to decoding one side, can improve the performance of the News Search in coding one side.About the coding method of conditions needed in the generation of motion vector, the transfer approach of layering is effective.Figure 20 represents the data configuration of general video coding.General video data is by the block header b14 of the encoding condition of the block unit of sheet title b13, expression motion vector or the Forecasting Methodology etc. of the encoding condition of the frame title b12 of the encoding condition of the sequence-header b11 of the encoding condition of expression sequence integral body, expression frame unit, expression sheet unit and comprise that the blocks of data b15 of the coded data of predictive error signal constitutes.According to its locality, by being divided into 4 kinds of heading messages, needed various conditions in the generation of motion vector encode, improve code efficiency.
Figure 18 and Figure 19 are illustrated in the code device of video coding of the motion vector that decoding one adnation becomes and the example of decoding device.The structure of Figure 18 presentation code device.Present frame a100 is divided into encoding block a101 at piece cutting unit c102.Encoding block a101 is input to News Search conditional decision unit c112 and subtracter c103.News Search conditional decision unit c112 outputs to News Search unit c114 to the candidate c115 of conditions needed in the generation of motion vector.In addition, about the condition of selecting with sequence unit or frame unit in the conditions needed in motion vector generates, use original image to select in advance in News Search condition unit.As system of selection, for example, implemented to use the News Search of original image to handle by candidate for multiple condition, can select suitable condition.News Search unit c114 according to conditions needed c115 in the generation of motion vector, detects motion vector a203 from preceding frame a200 and model a108 that frame memory c111 input decoding finishes.Dynamic compensation unit c113 extracts and the corresponding prediction piece of motion vector a203 c120 from the preceding frame a200 that the decoding of frame memory c111 finishes, and outputs to News Search conditional decision unit c112.News Search conditional decision unit c112 handle compares with a plurality of candidates corresponding prediction piece c120 and the input block a101 of conditions needed in the generation of motion vector, the decision predicated error absolute value and be the minimum prediction piece a205.In addition, at this moment condition is outputed to News Search conditional compilation unit c120 as conditions needed c117 in the generation of motion vector.News Search conditional compilation unit c120 outputs to the entropy coding unit to the coding of conditions needed in the generation of motion vector to information encoded.Unqualified about coding method, as mentioned above, can use layered structure or divide in groups method, limit the method for using the inscape that pattern information encodes, directly transmit numerical value method, prepare method that the candidate of multiple coded message selects, the differential coding of next predicted value of inferring since adjacent block etc.
Prediction piece 205 outputs to subtracter c103 and adder c108.In subtracter c103, calculate input block a101 and the error block c104 that predicts piece a205, output to error block coding unit c105.Error block coding unit c105 carries out the encoding process of error block, and error coding data c106 is outputed to entropy coding unit c110 and error block decoding unit c107.Error block decoding unit c107 restores regeneration error block c118 to the error coding data decode, outputs to adder c108.Adder c108 handles with the addition of prediction piece c205 by regeneration error block c118 and generates reconstituting piece c109, is synthesized on the reproduced picture of the present frame in the frame memory.At last, entropy coding unit c110 synthesizes outputting encoded data c116 to error coding data c106 and the information c119 and the various heading message that are illustrated in conditions needed in the generation of motion vector.
Figure 19 represents the structure of decoding device.Coded data c116 is decoded into decoded data d102 in the entropy decoding unit, be separated into error coding data c106 and be illustrated in the information c119 of conditions needed in the generation of motion vector by separative element d103.The information c119 that is illustrated in conditions needed in the generation of motion vector is decoded into conditions needed c117 in the generation of motion vector by News Search condition decoder unit d109.Preceding frame a200 and model a108 that News Search unit d107 decodes and finishes from the d106 input of frame memory according to conditions needed c117 in the generation of motion vector, detect motion vector a203.Dynamic compensation unit d108 extracts and the corresponding prediction piece of motion vector a203 c205 from the preceding frame a200 that the decoding of frame memory d111 finishes, and outputs to adder d105.Error block decoding unit d104 restores regeneration error block c118 to the error coding data decode, outputs to adder d105.Adder d105 handles with the addition of prediction piece c205 by regeneration error block c118 and generates reconstituting piece c109, is synthesized on the reproduced picture of the present frame in the frame memory.
In the example of video coding, can consider also that piece has motion vector jointly and the situation of conditions needed in the generation of motion vector.In this case, according to conditions needed in the generation of motion vector, the motion vector that the decoder correction has been decoded.In addition, in other example, the motion vector that generates according to conditions needed the generation of motion vector from decoder and the motion vector of having decoded generate final motion vector.In this case, the decoding motion vector is thought the difference motion vector of the motion vector that generated by decoder and final motion vector.Like this, receive a side, in the device or software that generate motion vector, can be suitable for the method for conditions needed and motion vector in receiving the generation that a side is transmitted in motion vector together.
In this variation, code device, decoding device have been described, and can be out of shape too for the coding/decoding handling process.In addition, the image encoding of this variation is handled or picture decoding is handled as illustrated in fig. 12 like that, can pass through computer system enforcement based on program.
The encoding process flow process and the processing of the piece in the decoding processing flow process of this variation implemented in expression among Figure 21 and Figure 22.Coding/decoding for sequence-header, frame title omits explanation, in encoding process, in the conditions needed, the information that transmits with frame unit and sequence unit is selected according to their unit in motion vector generates.As the method for selecting, as shown in the explanation of Figure 18 like that, the News Search of original image has been used in utilization.In decoding processing, the coded data of sequence-header, frame title decoding back is preserved, when carrying out the decoding processing of each piece, use.
The piece processing of the encoding process flow process of this variation is implemented in explanation among Figure 21.After block encoding begins to handle c201, in handling c202, import next encoding block.In handling c203, be selected in the candidate of conditions needed in the generation of motion vector.Handling in 204, according to this condition, as shown in Figure 17, the image and the model that use decoding to finish detect motion vector.In handling c205, obtain and the corresponding prediction piece of motion vector from the image that decoding finishes.In handling c206, estimate encoding block and prediction piece the predicated error absolute value and.About a plurality of candidates of conditions needed in the generation of motion vector, handle c203 repeatedly to handling c206, select predicated error absolute value and motion vector formation condition hour and predict piece.In handling c207, encoding block and each pixel of prediction piece are subtracted each other the generated error piece.In handling c208, error block is encoded (for example, discrete cosine transform, quantification).In handling c209, error block is decoded (for example, the inverse quantization of quantization transform coefficient and inverse discrete cosine transformation).In handling c210, the error block and the addition of prediction piece, recovery decoding block of having decoded.In handling c211, the information of conditions needed in coded message of error block (quantization transform coefficient) and the generation that is illustrated in motion vector selected in handling c206 is carried out entropy coding, generate coded data, end block encoding process in handling c212.
The piece processing of the decoding processing flow process of present embodiment is implemented in explanation in Figure 22.Piece decoding begin handle d201 after, in handling d202, input and the corresponding coded data of next decoding block.In handling d203, decoded data is carried out the entropy decoding, obtain the coded message of conditions needed and error block in the generation of motion vector.In handling d204, according to this condition, as shown in figure 17, the image and the model that use decoding to finish detect motion vector.In handling d205, obtain and the corresponding prediction piece of motion vector from the image that decoding finishes.In handling d206, the coded message of error block decoding (for example, the inverse quantization of quantization parameter and inverse discrete cosine transformation).In handling d207,, restore decoding block, end block encoding process in processing d208 the error block and the addition of prediction piece of having decoded.
Remove beyond the motion vector, situation for the information relevant such as the generation method that becomes reference frame, predictive mode (single directional prediction, bi-directional predicted), prediction piece at decoding one adnation (generating the methods of 1 prediction piece from 2 prediction pieces) with motion vector, be used to generate these information conditions needed in the decision of coding one side, by this information being sent to decoding one side, can improve the generation performance of its information.In addition, also comprise the condition that is used to revise these information that once generates.

Claims (22)

1. picture decoding apparatus is characterized in that possessing:
The receiver, video coded data is decoded, and obtains the video data decoding unit of a plurality of reproduced pictures;
Receive auxiliary data and decode, obtain the auxiliary data decoding unit of auxiliary multidate information;
Based on the auxiliary multidate information that obtains by above-mentioned auxiliary data decoding unit, generate the motion vector of the space-time correspondence between the above-mentioned a plurality of reproduced pictures of expression, use motion vector that is generated and a plurality of reproduced pictures that obtained by above-mentioned video data decoding unit, span definition is higher than the definition expansion unit of the HD image of above-mentioned a plurality of reproduced pictures.
2. picture decoding apparatus according to claim 1 is characterized in that:
Above-mentioned auxiliary multidate information comprises auxiliary motion vector,
Above-mentioned definition expansion unit is used as above-mentioned motion vector to above-mentioned auxiliary motion vector.
3. picture decoding apparatus according to claim 1 is characterized in that:
Above-mentioned auxiliary multidate information comprises auxiliary motion vector,
Above-mentioned definition expansion unit use above-mentioned a plurality of reproduced picture detect in the middle of motion vector, the addition process by this centre motion vector and above-mentioned auxiliary motion vector generates above-mentioned motion vector.
4. picture decoding apparatus according to claim 1 is characterized in that:
Above-mentioned auxiliary multidate information comprises auxiliary motion vector,
Above-mentioned definition expansion unit upgrades above-mentioned initial motion vector to the initial motion vector of above-mentioned auxiliary motion vector as above-mentioned motion vector by using above-mentioned a plurality of reproduced picture, generates above-mentioned motion vector.
5. picture decoding apparatus according to claim 1 is characterized in that:
Above-mentioned auxiliary multidate information comprises as in order to generate the motion vector formation condition of above-mentioned motion vector conditions needed,
Above-mentioned definition expansion unit generates above-mentioned motion vector based on above-mentioned motion vector formation condition from above-mentioned a plurality of reproduced pictures.
6. picture coding device is characterized in that possessing:
HD image is transformed to the image sampling unit of low picture rich in detail;
The a plurality of low picture rich in detail compressed encoding that generates by above-mentioned image sampling unit, generate the video data encoding unit of coding video frequency data;
The coding video frequency data decoding that generates by above-mentioned video data encoding unit, the video data decoding unit of the low picture rich in detail that obtains regenerating;
Use above-mentioned HD image or above-mentioned low picture rich in detail, be created on the auxiliary multidate information generation unit of needed auxiliary multidate information in the generation of above-mentioned motion vector;
Based on the auxiliary multidate information that is generated by above-mentioned auxiliary multidate information generation unit, generate expression and hang down the motion vector of the space-time correspondence between the picture rich in detail by a plurality of regeneration that above-mentioned video data decoding unit obtains, use the motion vector and the low picture rich in detail of above-mentioned a plurality of regeneration that are generated, generate the definition expansion unit of regeneration HD image;
The auxiliary data coding unit of encoding as auxiliary data by the auxiliary multidate information of above-mentioned auxiliary multidate information generation unit generation.
7. picture decoding apparatus is characterized in that possessing:
The received code data are carried out the entropy decoding, obtain as the coded data decoding unit for motion vector formation condition that generates the motion vector conditions needed and the coded message of representing the prediction error image signal;
Be used to preserve the video memory of decoded image signal;
Based on the motion vector formation condition that is obtained by above-mentioned coded data decoding unit, the decoded image signal from be kept at above-mentioned video memory generates the motion vector generation unit of motion vector;
Motion vector and above-mentioned decoded image signal that use is generated by above-mentioned motion vector generation unit, the prediction image signal generation unit of generation forecast picture signal;
The coded message that is obtained by above-mentioned coded data decoding unit is decoded, obtain the decoding unit of prediction error image signal;
Prediction image signal that generates by above-mentioned prediction image signal generation unit and the prediction error image signal plus that obtains by above-mentioned decoding unit, restore decoded image signal, this decoded image signal is kept at preservation unit in the above-mentioned video memory.
8. picture coding device is characterized in that possessing:
The input unit that received image signal is imported;
Be used to preserve the video memory of decoded image signal;
Based on the received image signal of being imported by above-mentioned input unit, decision is as the motion vector formation condition decision unit for the motion vector formation condition that generates the motion vector conditions needed;
Based on determining the motion vector formation condition that the unit determined by above-mentioned motion vector formation condition, the decoded image signal from be kept at above-mentioned video memory generates the motion vector generation unit of motion vector;
Motion vector and above-mentioned decoded image signal that use is generated by above-mentioned motion vector generation unit, the prediction image signal generation unit of generation forecast picture signal;
From received image signal of importing by above-mentioned input unit and the prediction image signal that generates by above-mentioned prediction image signal generation unit, the prediction error image signal generation unit of generation forecast error image signal;
The prediction error image signal encoding that is generated by above-mentioned prediction error image signal generation unit, the coded message that obtains coded message obtains the unit;
Obtain the coded message decoding that obtains the unit by above-mentioned coded message, the local solution code element of the prediction error image signal that obtains decoding;
From prediction image signal that is generated by above-mentioned prediction image signal generation unit and the decoding prediction error image signal that obtains by above-mentioned local solution code element, restore decoded image signal, this decoded image signal is kept at preservation unit in the above-mentioned video memory;
Determining the motion vector formation condition that the unit determined by above-mentioned motion vector formation condition and obtaining the coded message that obtains the unit by above-mentioned coded message and carry out entropy coding, generate the coded data generation unit of coded data.
9. picture decoding apparatus is characterized in that possessing:
The receiver, video coded data is decoded, and obtains the video data decoding unit of a plurality of reproduced pictures;
Receive auxiliary data and decode, obtain the auxiliary data decoding unit of auxiliary multidate information;
Generate expression by the corresponding motion vector of space-time between the resulting a plurality of reproduced pictures in above-mentioned video data decoding unit and 1 HD image, use the motion vector and the above-mentioned a plurality of reproduced picture that are generated, the definition expansion unit of the HD image that span definition is higher than above-mentioned a plurality of reproduced pictures
Above-mentioned definition expansion unit is implemented to generate the HD image of handling, generating HD image from the motion vector that generated and above-mentioned a plurality of reproduced picture based on the motion vector that is generated above-mentioned motion vector by resulting auxiliary multidate information of above-mentioned auxiliary data decoding unit and the previous HD image that generates repeatedly and is generated and handle.
10. picture decoding apparatus according to claim 9 is characterized in that:
Above-mentioned definition expansion unit generates in the repeated treatments of processing and HD image generation processing at above-mentioned motion vector, implements above-mentioned motion vector generation based on above-mentioned auxiliary multidate information at every turn and handles.
11. picture decoding apparatus according to claim 9 is characterized in that:
Above-mentioned auxiliary multidate information is included in specific time specific times information in the repeated treatments that the above-mentioned motion vector generation of expression is handled and the HD image generation is handled,
Above-mentioned definition expansion unit generates in the processing at the motion vector of the represented specific times of above-mentioned specific information, implements to generate processing based on the motion vector of above-mentioned auxiliary multidate information.
12. a picture decoding apparatus is characterized in that possessing:
The receiver, video coded data is decoded, and obtains the video data decoding unit of a plurality of reproduced pictures;
Be used to preserve the video memory that definition expands the HD image that finishes;
Generate the motion vector of space-time correspondence between the above-mentioned a plurality of reproduced pictures of expression, use motion vector that is generated and a plurality of reproduced pictures that obtain by above-mentioned video data decoding unit, the 1st HD image that span definition is higher than above-mentioned a plurality of reproduced pictures is used the 1st HD image generated and is kept at the definition expansion unit that HD image in the above-mentioned video memory generates the 2nd HD image;
The image that the 1st or the 2nd HD image that is generated by above-mentioned definition expansion unit is kept in the above-mentioned video memory is preserved the unit.
13. a picture decoding apparatus is characterized in that possessing:
The receiver, video coded data is decoded, and obtains the video data decoding unit of a plurality of reproduced pictures;
Receive auxiliary data and decode, obtain the auxiliary data decoding unit of auxiliary multidate information;
Be used to preserve the video memory that definition expands the HD image that finishes;
Generate the motion vector of the space-time correspondence between the above-mentioned a plurality of reproduced pictures of expression, use motion vector that is generated and a plurality of reproduced pictures that obtain by above-mentioned video data decoding unit, the 1st HD image that span definition is higher than above-mentioned a plurality of reproduced pictures, based on the auxiliary multidate information that obtains by above-mentioned auxiliary data decoding unit, use the 1st HD image generated and be kept at the definition expansion unit that HD image in the above-mentioned video memory generates the 2nd HD image;
The image that the 1st or the 2nd HD image that is generated by above-mentioned definition expansion unit is kept in the above-mentioned video memory is preserved the unit.
14. a picture decoding apparatus is characterized in that possessing:
The receiver, video coded data is decoded, and obtains the video data decoding unit of a plurality of reproduced pictures and regeneration motion vector;
Receive auxiliary data and decode, obtain the auxiliary data decoding unit of needed auxiliary multidate information in the correction of above-mentioned regeneration motion vector;
Generate the motion vector of the space-time correspondence between a plurality of reproduced pictures of expression, use motion vector that is generated and a plurality of reproduced pictures that obtain by above-mentioned video data decoding unit, the definition expansion unit of the HD image that span definition is higher than above-mentioned a plurality of reproduced pictures
Above-mentioned definition expansion unit is based on the regeneration motion vector of the auxiliary multidate information correction that is obtained by the above-mentioned auxiliary data decoding unit initial motion vector as above-mentioned motion vector, by using above-mentioned a plurality of reproduced picture that above-mentioned initial motion vector is upgraded, generate above-mentioned motion vector.
15. a picture decoding method is characterized in that possessing:
The receiver, video coded data is decoded, and obtains the video data decoding step of a plurality of reproduced pictures;
Receive auxiliary data and decode, obtain the auxiliary data decoding step of auxiliary multidate information;
Based on the auxiliary multidate information that in above-mentioned auxiliary data decoding step, obtains, generate the motion vector of the space-time correspondence between the above-mentioned a plurality of reproduced pictures of expression, use motion vector that is generated and a plurality of reproduced pictures that obtain in above-mentioned video data decoding step, the definition that span definition is higher than the HD image of above-mentioned a plurality of reproduced pictures expands step.
16. a method for encoding images is characterized in that possessing:
HD image is transformed to the image sampling step of low picture rich in detail;
The a plurality of low picture rich in detail compressed encoding that in above-mentioned image sampling step, generates, generate the video data encoding step of coding video frequency data;
The decoding of the coding video frequency data that in above-mentioned video data encoding step, generates, the video data decoding step of the low picture rich in detail that obtains regenerating;
Use above-mentioned HD image or above-mentioned low picture rich in detail, be created on that the auxiliary multidate information of needed auxiliary multidate information generates step in the generation of above-mentioned motion vector;
Based on generating the auxiliary multidate information that generates in the step at above-mentioned auxiliary multidate information, generation is illustrated in the motion vector of the space-time correspondence between the low picture rich in detail of a plurality of regeneration that obtain in the above-mentioned video data decoding step, and the definition that the low picture rich in detail of the motion vector that use is generated and above-mentioned a plurality of regeneration generates the regeneration HD image expands step;
Generate the auxiliary data coding step that the auxiliary multidate information that generates in the step is encoded as auxiliary data at above-mentioned auxiliary multidate information.
17. a picture decoding method is characterized in that possessing:
The received code data are carried out the entropy decoding, obtain as the coded data decoding step for motion vector formation condition that generates the motion vector conditions needed and the coded message of representing the prediction error image signal;
Based on the motion vector formation condition that obtains in above-mentioned coded data decoding step, the motion vector that the decoded image signal from be kept at video memory generates motion vector generates step;
Use generates motion vector and the above-mentioned decoded image signal that generates in the step at above-mentioned motion vector, and the prediction image signal of generation forecast picture signal generates step;
The coded message that obtains in above-mentioned coded data decoding step is decoded, obtain the decoding step of prediction error image signal;
Generating prediction image signal that generates in the step and the prediction error image signal plus that in above-mentioned decoding step, obtains at above-mentioned prediction image signal, restore decoded image signal, this decoded image signal is kept at preservation step in the above-mentioned video memory.
18. a method for encoding images is characterized in that possessing:
The input step that received image signal is imported;
Based on the received image signal of being imported in above-mentioned input step, decision is as the motion vector formation condition deciding step for the motion vector formation condition that generates the motion vector conditions needed;
Based on the motion vector formation condition that is determined in above-mentioned motion vector formation condition deciding step, the motion vector that the decoded image signal from be kept at video memory generates motion vector generates step;
Use generates motion vector and the above-mentioned decoded image signal that generates in the step at above-mentioned motion vector, and the prediction image signal of generation forecast picture signal generates step;
From received image signal of importing above-mentioned input step and the prediction image signal that generates in above-mentioned prediction image signal generation step, the prediction error image signal of generation forecast error image signal generates step;
Generating the prediction error image signal encoding that generates in the step at above-mentioned prediction error image signal, the coded message that obtains coded message obtains step;
Obtain the coded message decoding that obtains in the step in above-mentioned coded message, the local decoding step of the prediction error image signal that obtains decoding;
From generating prediction image signal that is generated the step and the decoding prediction error image signal that in above-mentioned local decoding step, obtains at above-mentioned prediction image signal, restore decoded image signal, this decoded image signal is kept at preservation step in the above-mentioned video memory;
The motion vector formation condition that is determined in above-mentioned motion vector formation condition deciding step with obtain the coded message that obtains in the step in above-mentioned coded message and carry out entropy coding, the coded data that generates coded data generates step.
19. an image decoding program is characterized in that:
Make the effect of following each unit of computer performance that is arranged in the picture decoding apparatus, these unit are:
The receiver, video coded data is decoded, and obtains the video data decoding unit of a plurality of reproduced pictures;
Receive auxiliary data and decode, obtain the auxiliary data decoding unit of auxiliary multidate information;
Based on the auxiliary multidate information that obtains by above-mentioned auxiliary data decoding unit, generate the motion vector of the space correspondence between the above-mentioned a plurality of reproduced pictures of expression, use motion vector that is generated and a plurality of reproduced pictures that obtained by above-mentioned video data decoding unit, span definition is higher than the definition expansion unit of the HD image of above-mentioned a plurality of reproduced pictures.
20. an image encoding program is characterized in that:
Make the effect of following each unit of computer performance that is arranged in the picture coding device, these unit are:
HD image is transformed to the image sampling unit of low picture rich in detail;
The a plurality of low picture rich in detail compressed encoding that generates by above-mentioned image sampling unit, generate the video data encoding unit of coding video frequency data;
The coding video frequency data decoding that generates by above-mentioned video data encoding unit, the video data decoding unit of the low picture rich in detail that obtains regenerating;
Use above-mentioned HD image or above-mentioned low picture rich in detail, be created on the auxiliary multidate information generation unit of needed auxiliary multidate information in the generation of above-mentioned motion vector;
Based on the auxiliary multidate information that is generated by above-mentioned auxiliary multidate information generation unit, generate expression and hang down the motion vector of the space-time correspondence between the picture rich in detail by a plurality of regeneration that above-mentioned video data decoding unit obtains, use the motion vector and the low picture rich in detail of above-mentioned a plurality of regeneration that are generated, generate the definition expansion unit of regeneration HD image;
The auxiliary data coding unit of encoding as auxiliary data by the auxiliary multidate information of above-mentioned auxiliary multidate information generation unit generation.
21. an image decoding program is characterized in that,
Make the effect of following each unit of computer performance that is arranged in the picture decoding apparatus, these unit are:
The received code data are carried out the entropy decoding, obtain as the coded data decoding unit for motion vector formation condition that generates the motion vector conditions needed and the coded message of representing the prediction error image signal;
Be used to preserve the video memory of decoded image signal;
Based on the motion vector formation condition that is obtained by above-mentioned coded data decoding unit, the decoded image signal from be kept at above-mentioned video memory generates the motion vector generation unit of motion vector;
Motion vector and above-mentioned decoded image signal that use is generated by above-mentioned motion vector generation unit, the prediction image signal generation unit of generation forecast picture signal;
The coded message that is obtained by above-mentioned coded data decoding unit is decoded, obtain the decoding unit of prediction error image signal;
Prediction image signal that generates by above-mentioned prediction image signal generation unit and the prediction error image signal plus that obtains by above-mentioned decoding unit, restore decoded image signal, this decoded image signal is kept at preservation unit in the above-mentioned video memory.
22. an image encoding program is characterized in that,
Make the effect of following each unit of computer performance that is arranged in the picture coding device, these unit are:
The input unit that received image signal is imported;
Be used to preserve the video memory of decoded image signal;
Based on the received image signal of being imported by above-mentioned input unit, decision is as the motion vector formation condition decision unit for the motion vector formation condition that generates the motion vector conditions needed;
Based on determining the motion vector formation condition that the unit determined by above-mentioned motion vector formation condition, the decoded image signal from be kept at above-mentioned video memory generates the motion vector generation unit of motion vector;
Motion vector and above-mentioned decoded image signal that use is generated by above-mentioned motion vector generation unit, the prediction image signal generation unit of generation forecast picture signal;
From received image signal of importing by above-mentioned input unit and the prediction image signal that generates by above-mentioned prediction image signal generation unit, the prediction error image signal generation unit of generation forecast error image signal;
The prediction error image signal encoding that is generated by above-mentioned prediction error image signal generation unit, the coded message that obtains coded message obtains the unit;
Obtain the coded message decoding that obtains the unit by above-mentioned coded message, the local solution code element of the prediction error image signal that obtains decoding;
From prediction image signal that is generated by above-mentioned prediction image signal generation unit and the decoding prediction error image signal that obtains by above-mentioned local solution code element, restore decoded image signal, this decoded image signal is kept at preservation unit in the above-mentioned video memory;
Determining the motion vector formation condition that the unit determined by above-mentioned motion vector formation condition and obtaining the coded message that obtains the unit by above-mentioned coded message and carry out entropy coding, generate the coded data generation unit of coded data.
CNA2005101204670A 2004-11-19 2005-11-18 Image decoding apparatus, image decoding program, image decoding method, image encoding apparatus, image encoding program, and image encoding method Pending CN1777287A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201010113362.3A CN101854544B (en) 2004-11-19 2005-11-18 Picture decoding apparatus, program, method and picture coding device, program, method

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
JP2004336463 2004-11-19
JP2004336463 2004-11-19
JP2005299326 2005-10-13

Related Child Applications (2)

Application Number Title Priority Date Filing Date
CNA2008101753991A Division CN101437162A (en) 2004-11-19 2005-11-18 Image decoding apparatus, image decoding method, image encoding apparatus, image encoding program, and image encoding method
CN201010113362.3A Division CN101854544B (en) 2004-11-19 2005-11-18 Picture decoding apparatus, program, method and picture coding device, program, method

Publications (1)

Publication Number Publication Date
CN1777287A true CN1777287A (en) 2006-05-24

Family

ID=36766543

Family Applications (2)

Application Number Title Priority Date Filing Date
CNA2008101753991A Pending CN101437162A (en) 2004-11-19 2005-11-18 Image decoding apparatus, image decoding method, image encoding apparatus, image encoding program, and image encoding method
CNA2005101204670A Pending CN1777287A (en) 2004-11-19 2005-11-18 Image decoding apparatus, image decoding program, image decoding method, image encoding apparatus, image encoding program, and image encoding method

Family Applications Before (1)

Application Number Title Priority Date Filing Date
CNA2008101753991A Pending CN101437162A (en) 2004-11-19 2005-11-18 Image decoding apparatus, image decoding method, image encoding apparatus, image encoding program, and image encoding method

Country Status (2)

Country Link
JP (2) JP5689291B2 (en)
CN (2) CN101437162A (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102823242A (en) * 2010-01-22 2012-12-12 汤姆森特许公司 Methods and apparatus for sampling-based super resolution vido encoding and decoding
CN104023242A (en) * 2007-04-09 2014-09-03 株式会社Ntt都科摩 Image prediction/encoding device, image prediction/encoding method, image prediction/decoding device, and image prediction/decoding method
US9338477B2 (en) 2010-09-10 2016-05-10 Thomson Licensing Recovering a pruned version of a picture in a video sequence for example-based data pruning using intra-frame patch similarity
CN102148983B (en) * 2010-02-08 2016-09-21 杨占昆 A kind of method of solving over-high occupancy of high-resolution image resource
US9544598B2 (en) 2010-09-10 2017-01-10 Thomson Licensing Methods and apparatus for pruning decision optimization in example-based data pruning compression
US9813707B2 (en) 2010-01-22 2017-11-07 Thomson Licensing Dtv Data pruning for video compression using example-based super-resolution

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108520503B (en) * 2018-04-13 2020-12-22 湘潭大学 Face defect image restoration method based on self-encoder and generation countermeasure network

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH02171092A (en) * 1988-12-23 1990-07-02 Fujitsu Ltd Inter-frame coding decoding system
JPH07336695A (en) * 1994-06-13 1995-12-22 Oki Electric Ind Co Ltd Method for encoding and decoding moving picture
JP3377893B2 (en) * 1995-10-25 2003-02-17 沖電気工業株式会社 Video encoding / decoding device
JP3325445B2 (en) * 1995-12-22 2002-09-17 株式会社日立製作所 Video signal processing device
JP3823370B2 (en) * 1996-04-15 2006-09-20 ソニー株式会社 Video encoding method, decoding method and apparatus
JP4250237B2 (en) * 1998-11-10 2009-04-08 キヤノン株式会社 Image processing apparatus, method, and computer-readable storage medium
US6285804B1 (en) * 1998-12-21 2001-09-04 Sharp Laboratories Of America, Inc. Resolution improvement from multiple images of a scene containing motion at fractional pixel values
US6289052B1 (en) * 1999-06-07 2001-09-11 Lucent Technologies Inc. Methods and apparatus for motion estimation using causal templates
JP4412062B2 (en) * 2004-06-07 2010-02-10 ソニー株式会社 Moving picture conversion apparatus, moving picture conversion method, and computer program
EP1610560A1 (en) * 2004-06-24 2005-12-28 Deutsche Thomson-Brandt Gmbh Method and apparatus for generating and for decoding coded picture data

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104023242A (en) * 2007-04-09 2014-09-03 株式会社Ntt都科摩 Image prediction/encoding device, image prediction/encoding method, image prediction/decoding device, and image prediction/decoding method
CN104023242B (en) * 2007-04-09 2017-07-07 株式会社Ntt都科摩 Image prediction encoding device and method, image prediction/decoding device and method
CN102823242A (en) * 2010-01-22 2012-12-12 汤姆森特许公司 Methods and apparatus for sampling-based super resolution vido encoding and decoding
CN102823242B (en) * 2010-01-22 2016-08-10 汤姆森特许公司 Based on sampling super-resolution Video coding and the method and apparatus of decoding
US9602814B2 (en) 2010-01-22 2017-03-21 Thomson Licensing Methods and apparatus for sampling-based super resolution video encoding and decoding
US9813707B2 (en) 2010-01-22 2017-11-07 Thomson Licensing Dtv Data pruning for video compression using example-based super-resolution
CN102148983B (en) * 2010-02-08 2016-09-21 杨占昆 A kind of method of solving over-high occupancy of high-resolution image resource
US9338477B2 (en) 2010-09-10 2016-05-10 Thomson Licensing Recovering a pruned version of a picture in a video sequence for example-based data pruning using intra-frame patch similarity
US9544598B2 (en) 2010-09-10 2017-01-10 Thomson Licensing Methods and apparatus for pruning decision optimization in example-based data pruning compression

Also Published As

Publication number Publication date
JP2012085341A (en) 2012-04-26
JP2011041329A (en) 2011-02-24
CN101437162A (en) 2009-05-20
JP5313326B2 (en) 2013-10-09
JP5689291B2 (en) 2015-03-25

Similar Documents

Publication Publication Date Title
CN1193620C (en) Motion estimation method and system for video coder
CN1202650C (en) Image processing method, image processing device, and data storage medium
CN1161989C (en) Editing device, editing method, splicing device splicing method, encoding device and encoding method
CN1187988C (en) Motion compensating apparatus, moving image coding apparatus and method
CN1076932C (en) Method and apparatus for coding video signal, and method and apparatus for decoding video signal
CN1178509C (en) Image encoder and image encoding method, image decoder and image decoding method, and image processor and image processing method
CN101854544A (en) Image decoding apparatus, image decoding method, image encoding apparatus, image encoding program, and image encoding method
CN1910931A (en) Video encoding method and device, video decoding method and device, program thereof, and recording medium containing the program
CN1777287A (en) Image decoding apparatus, image decoding program, image decoding method, image encoding apparatus, image encoding program, and image encoding method
CN1784011A (en) Image coding apparatus and image coding method
CN1288914C (en) Image coding and decoding method, corresponding devices and application
CN1557099A (en) Image encoding device, image encoding method, image decoding device, image decoding method, and communication device
CN1898964A (en) Motion image coding apparatus, and control method and program of the apparatus
CN1366776A (en) Decoder and decoding method, recorded medium and program
CN1256046A (en) Image transmitting method, image processing method, image processor, data storage medium
CN1808469A (en) Image searching device and method, program and program recording medium
CN1788496A (en) Image processing device, image processing method, information processing device, information processing method, information recording device, information recording method, information reproduction dev
CN1270541C (en) Decoder and method thereof, coding device and method thereof, image processing system and method thereof
CN1843039A (en) System and method for encoding and decoding enhancement layer data using descriptive model parameters
CN1240225C (en) Picture coding device and picture coding method
CN1195949A (en) Picture editing apparatus and picture editing method
CN1320825C (en) Image decoding method, image decoding apparatus
CN1311692C (en) Apparatus and method for checking dynamic vector
CN1453936A (en) Image coding apparatus and image coding method
CN1816129A (en) Recording apparatus

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C02 Deemed withdrawal of patent application after publication (patent law 2001)
WD01 Invention patent application deemed withdrawn after publication

Open date: 20060524