CN101026762B - Moving image prediction unit and method - Google Patents

Abstract

Obtain the character etc. of a kind of scene content according to dynamic image and significance level, movement, carry out the dynamic image coding mode of high-precision prediction.Have: the multiple memory blocks storing reference image; Can be corresponding with the arbitrary deformation parameter of the movement of the predicted area of image of performance, according to this deformation parameter, utilize the motion compensating device of the arbitrary data generation forecast image on above-mentioned multiple memory block; And the memory updating control device of the content of the memory block of in above-mentioned multiple memory block more than 1 can be upgraded at any time in interval.

Description

Moving image prediction unit and method

The application be applicant Mitsubishi Electric Corporation in the denomination of invention that on January 22nd, 1998 submits to be the divisional application of the Chinese patent application 200310124564.8 of " moving image coding device and method ".

Technical field

The present invention relates to the moving picture encoding/code translator such as using in the portable and fixed image communication machine of image communication etc. in video telephone, video conference etc., dynamic image coding/code translator in the image such as digital VTR, video server storages, tape deck etc., with the prediction of the dynamic image in the dynamic image coding/translator of independent software or the installation of DSP (Digital Signal Processor) form of firmware etc.

Background technology

Video coding/decoding the reference pattern (Verification Model, hereinafter referred to as VM) of the MPEG-4 (Moving Picture Experts Group Phase-4) carrying out standardized work in ISO/IEC JTC1/SC29/WG11 can be enumerated as the predictive coding in existing dynamic image coding/decoded mode/decoding example.VM is with the carrying out of the standardized work of MPEG-4, and the content of mode changes, but imagines VM Version5.0 here, and only table is VM below.

VM be dynamic image order is taken as time/spatially get the aggregate of the image object of arbitrary shape, with each image object for unit carries out the mode of coding/decoding.Video data structure in VM has been shown in Figure 29.In VM, the dynamic image target comprising time shaft is called Video Objedt (VO), by representing the state of VO in each moment, the pictorial data becoming coding unit is called Video Objedt Plane (VOP).When VO time/spatially there is level time, between VO and VOP, be provided with the unit being called Video Objedt Layer (VOL) especially, performance VO in hierarchical structure.Each VOP is divided into shape information and structural information.But, in dynamic image order when VO is 1, each VOP just with frame synonym.At this moment, there is not shape information, only carry out the coding/decoding of structural information.

As shown in figure 30, VOP has the structure be made up of the performance primary data of shape information and the structured data of performance structure information.Each data are respectively defined as the aggregate by the block of 16 × 16 compositions of sample (original block, macro block).Each sample in original block is existing with 8 bit tables.Macro block, with the luminance signal of 16 × 16 samples, comprises the color difference signal corresponding with it.Process according to dynamic image sequentially built VOP data is carried out beyond this code device.

Figure 31 represents the structure of the VOP code device adopting VM coding/decoding mode.In this figure, P1 is the former VOP data of input, P2 is the original block of the shape information of performance VOP, P3a be transmit input former VOP data shape information have useless switch, P4 is shape coding portion original block being carried out to compressed encoding, P5 is the original block data of compression, P6 is partial decode original block, P7 is structured data (macro block), P8 is mobility detect portion, P9 is moving parameter, P10 is motion compensation portion, P11 is predictive image candidate, P12 is prediction mode selection portion, P13 is prediction mode, P14 is predictive image, P15 is predictive error signal, P16 is structured coding portion, P17 is structured coding information, P18 is partial decode predictive error signal, P19 is partial decode macro block, P20 is sprite memory updating portion, P21 is VOP memory, P22 is sprite memory, P23 is variable-length code multi-way multiplex portion, P24 is buffer, P25 is coding stream.

In addition, Figure 32 shows the flow process of the work of this code device of brief description.

In the code device shown in Figure 31, first former VOP data P1 is divided into original block P2 and macro block P7 (step PS2, step PS3), and original block P2 is sent to shape coding portion P4, and macro block P7 is sent to mobility detect portion P8.Shape coding portion P4 is the processing block (step PS4) of the data compression carrying out original block P2, and the present invention is not the compression method about shape information, thus this process detailed description is omitted.。

The output of shape coding portion P4 is compression original block data P5 and partial decode original block P6, the former is fed to variable-length code multi-way multiplex portion P23, and the latter is given mobility detect portion P8, motion compensation portion P10, prediction mode selection portion P12 and structured coding portion P16 respectively.

Mobility detect portion P8 (step PS5) receives macro block P7, utilizes the reference pictorial data and partial decode original block P6 that store in VOP memory P21, detects the mobile vector of each macro block local.Here, mobile vector is an example of moving parameter.The partial decode image of encoded VOP is stored into VOP memory P21.After the end-of-encode of each macro block, just upgrade the content of VOP memory P21 successively with this partial decode image.Mobility detect portion P8 also has the structured data receiving whole former VOP simultaneously in addition, utilizes the reference pictorial data and partial decode primary data that store in sprite memory P22, detects the function of the bending parameters of the overall situation.Hereinafter will describe sprite memory P22 in detail.

Motion compensation portion P10 (step PS6) utilizes the moving parameter P9 and partial decode original block P6 that are detected by mobility detect portion P8, generation forecast image candidate P11.Secondly, in prediction mode selection portion P 12, with predictive error signal electric power and original signal electric power, determine the prediction mode P13 that this macro block is last and predictive image P14 (step PS7).By prediction mode selection portion P12 carry out be both frame interior coding/intermediate frame is encoded in the judgement of any coding.

In structured coding portion P16, according to prediction mode P13, after prediction DCT (Discrete Cosine Transform), quantization, the quantization DCT coefficient that obtains, or directly itself give variable-length coding unit multiplexed portion P23 predictive error signal P15 or former macro block, carry out encode (step PS8, step PS9).The data of reception, according to predetermined grammer and variable-length code, are transformed into bit stream by variable-length code multi-way multiplex portion P23, carry out multiplexed (step PS10).Quantization DCT coefficient, through inverse guantization (IQ), inverse DCT, after becoming partial decode predictive error signal P18, obtains partial decode macro block P19 (walking PS11) with predictive image P14 after being added.Partial decode macro block P19 is written into VOP memory P21 and sprite memory P22, for the prediction (step PS12) of later VOP.

Below, describe in detail carry out predicting part, particularly prediction mode and motion compensation, sprite memory P22 and VOP memory P21 renewal control.

(1) prediction mode in VM

As shown in figure 33, in VM, usually there are 4 kinds of VOP type of codings, all types of middle prediction mode represented with zero can be selected to each macro block.Adopt during I-VOP and do not predict completely, all carry out frame interior coding.P-VOP can carry out the prediction from the VOP in past.B-VOP can use in the past and the VOP in future in prediction.

Above prediction is the prediction all undertaken by mobile vector.On the other hand, Sprite-VOP is the prediction that can use sprite memory.So-called sprite refers to so a kind of pattern space, namely in VOP unit, detects following formula

x’＝(ax+by+c)/(gx+hy+1)

y’＝(dx+ey+f)/(gx+hy+1)

Shown bent ginseng manifold

$\stackrel{\→}{\mathrm{\α}}=(a,b,c,d,e,f,g,h)$

(→ represent vector.As follows)

Take this as a foundation, the pattern space generated by mixing VOP successively, and be stored into sprite memory P22.

Here, (x, y) is the pixel location of former VOP on two-dimensional coordinate, and (x ', y ') is the pixel location in the sprite memory corresponding with (x, y) according to bending parameters.In each macro block of Sprite-VOP, use this bent ginseng manifold uniformly, determine (x ', y ') in sprite memory, can predict by generation forecast image.Strictly speaking, " dynamically sprite " predicted in sprite and for while predicting in decoding side by " static sprite " synthesized approx for the purpose of VOP be distinguishing.But in following Figure 34 ~ Figure 37, use dynamic sprite by the meaning of " sprite ".

Above mobile vector for predicting and bending parameters is detected with mobility detect portion P8.They are referred to as moving parameter P9.

(2) motion compensation portion

Motion compensation portion P10 such as gets the internal structure shown in Figure 34.In the figure, P26 is bending parameters, and P27 is mobile vector, and P28 is overall motion compensation portion, and P29 is motion compensation portion, local, and P30 is the predictive image candidate determined by bending parameters, and P31 is the predictive image candidate determined by mobile vector.The predictive image candidate P30 determined by bending parameters and the predictive image candidate P31 that determined by mobile vector is generically and collectively referred to as predictive image candidate P11.

Step PS14 in Figure 35 diagrammatically illustrates the workflow of motion compensation portion P10 to step PS21.

In motion compensation portion P10, the bending parameters P26 utilizing the VOP that detected each macro block P7 by mobility detect portion P8 overall or the mobile vector P27 of macroblock unit, generation forecast image candidate P11.In overall motion compensation portion P28, carry out the motion compensation using bending parameters P26, in motion compensation portion, local P29, carry out the motion compensation using mobile vector P27.

When I-VOP, motion compensation portion P10 does not work (step PS14 is to walking PS21).Beyond I-VOP, local motion compensation portion P29 works, and utilizes mobile vector P27, takes out predictive image candidate (PR1) (step PS15) from the VOP partial decode image in the past in VOP memory P21.When I-VOP, only use this predictive image candidate (PR1).

In step PS16, when B-VOP, again in motion compensation portion, local P29, mobile vector P27 is utilized to take out predictive image candidate (PR2) (step PS17) from the VOP partial decode image in the future in VOP memory P21, be averaged after the predictive image candidate that VOP partial decode image from past and future obtains being added simultaneously, obtain predictive image candidate (PR3) (step PS18).

In addition, even directly prediction is (according to the prediction being equivalent to ITU-T and advising the prediction mode of the B frame in H.263 coded system.The vector of B frame is made according to the vector of the P-VOP becoming group.At this, detailed description is omitted), generation forecast image candidate (PR4) (step PS19) too.In Figure 34, the predictive image candidate P31 determined by mobile vector is also the general name of part or all from above-mentioned predictive image candidate PR1 to PR4.

When neither I-VOP, neither B-VOP, but during Sprite-VOP, mobile vector is utilized to take out predictive image candidate (PR1) from VOP memory, simultaneously in step PS20, in overall motion compensation portion P28, use bending parameters P26, take out predictive image candidate P30 from sprite memory P22.

Motion compensation portion P28 is according to bending parameters P26 for the overall situation, calculates the address that the predictive image candidate in sprite memory P22 exists, and takes out and prediction of output image candidate P30 according to this address from sprite memory P22.Local motion compensation portion P29, according to mobile vector P27, calculates the address that the predictive image in VOP memory P21 exists, and takes out and prediction of output image candidate P31 according to this address from VOP memory P21.

In prediction mode selection portion P12, to these predictive images candidate P11 and the frame interior code signal comprising structured data P7 evaluate, the predictive image candidate selecting predictive error signal electric power minimum and prediction mode.

(3) memory updating

Carry out memory updating by memory updating portion P20 to control (step PS12), carry out the renewal of VOP memory P21 and sprite memory P22.The renewal of these memory contents is independently carried out with the prediction mode P13 selected in macroblock unit.

The internal structure of memory updating portion P20 is shown in Figure 36, represents that the flow process of work of memory updating portion P20 is shown in step PS22 in Figure 37 to walking PS28.

In Figure 36, P32 is the VOP type of coding from outside supply, P33 represents whether use sprite to predict, be the sprite Forecasting recognition symbol from outside supply, P34 is the Bu Laite coefficient from outside supply used in the prediction of sprite memory, P35 is switch, P36 is switch, and P37 is sprite combining unit, and P38 is sprite deformation process portion, P39 is VOP memory updating signal, P40 sprite memory updating signal.

First, utilize sprite Forecasting recognition to accord with P33 to check whether to specify in this VO or VOL and use sprite (step PS22), when not using sprite, check B-VOP (step PS27), if B-VOP, do not carry out the renewal of VOP memory P21 completely.When I-VOP or P-VOP, by partial decode macro block P19, VOP memory 21 (step PS28) is write to each macro block.

On the other hand, in the inspection of step PS22, when using sprite, after first carrying out the renewal of VOP memory P21 similar to the above (step PS23, step PS24), carry out the renewal of sprite memory P22 according to following order.

Bending (the step PS25) of a) sprite

First, in sprite deformation process portion P38, bending parameters is utilized

$\stackrel{\→}{\mathrm{\α}}=(a,b,c,d,e,f,g,h)$

Make region on sprite memory (when the moment of this VOP is t, with using region identical as the VOP area of initial point for the position on sprite memory)

$M(\stackrel{\→}{R},t-1)$

Bending (distortion).

B) the Bu Laite coefficient (step PS26) of sprite

Utilize the bending image that above-mentioned result a) obtains, in sprite combining unit P37, obtain new sprite memory areas according to following formula.

$\stackrel{\→}{M}(R,t)=(1-\mathrm{\α})\·\mathrm{Wb}[M(\stackrel{\→}{R},t-1),\stackrel{\→}{\mathrm{\α}}]+\mathrm{\α}\·V0(\stackrel{\→}{r},t)$

In formula

α: Bu Laite FACTOR P 34

the bending image that the result of above-mentioned (α) obtains

the position of partial decode VOP and pixel value during moment t.

But the region not belonging to VOP in partial decode macro block is regarded as

$V0(\stackrel{\→}{r},t)=0$

Bu Laite factor alpha provides with VOP unit, thus partial decode VOP no matter VOP district content how, press proportion according to α, synthesize in sprite memory in the lump.

In prediction mode in above this existing coded system, when carrying out the prediction of dynamic image target, the memory only carrying out dynamic vector detection and the memory only carrying out bending parameters detection can only be used due to maximum a picture picture, so the reference image that can be used for predicting can only be used with extremely limited method, forecasting efficiency can not be improved fully.

In addition, in the system of at the same time multiple dynamic image target being encoded, owing to only comprising the reference image of the history of the dynamic image target representing prediction itself in these memories, so define the change with reference to image, the dependency relation between dynamic image target can not be utilized to predict simultaneously.

Moreover, even if when carrying out the renewal of memory, due to the content with the internal structure of dynamic image target, character and history etc. independently alterable storage, so can not by for the useful important knowledge of prediction dynamic image target fully stored in memory, there is the problem can not seeking to improve forecasting efficiency.

The present invention completes to solve the problem, its object is to provide such one prediction mode flexibly, namely when carrying out coding/decoding to pictorial data, multiple memory is set, consider the internal structure, character etc. of dynamic image order, by the history of dynamic image order effectively stored in multiple memory, improve the efficiency of prediction and coding/decoding, can predict between multiple dynamic image target simultaneously.

Disclosure of an invention

The invention discloses a kind of moving image estimating system predicting dynamic image, it has: multiple memories of the pictorial data of reference during Storage Estimation; Input represents the parameter of the movement of predicted image region, according to this parameter, utilize the pictorial data stored in above-mentioned multiple memory, the motion compensation portion of generation forecast image, and predictive image generating unit, it has the memory updating portion of the pictorial data stored at least one memory upgraded at any time in above-mentioned multiple memory, for coding, the information relevant according to the usage frequency of described memory during prediction, carries out the code assign ment with reference to memory numbering.

The invention also discloses a kind of method predicting dynamic image, it comprises following operation: during prediction with reference to pictorial data stored in the operation of multiple memory; Input represents the operation of the parameter of the movement of predicted area of image; According to above-mentioned parameter, utilize the operation of the pictorial data generation forecast image stored in above-mentioned multiple memory; And the operation of the pictorial data stored at least one memory upgrading in above-mentioned multiple memory at any time, for coding, the information relevant according to the usage frequency of described memory during prediction, carries out the code assign ment with reference to memory numbering.

The simple declaration of accompanying drawing

Fig. 1 is the structure chart of the dynamic image coding device represented in example of the present invention.

Fig. 2 is the workflow diagram of the dynamic image coding device represented in example of the present invention.

Fig. 3 is the structure chart in the motion compensation portion of the dynamic image coding device represented in example of the present invention.

Fig. 4 is the workflow diagram representing motion compensation portion.

Fig. 5 is the structure chart in the memory updating portion of the dynamic image coding device represented in example of the present invention.

Fig. 6 is the workflow diagram representing memory updating portion.

Fig. 7 is the structure chart in the motion compensation portion of the dynamic image coding device represented in example of the present invention.

Fig. 8 is the workflow diagram in the motion compensation portion represented in Fig. 7.

Fig. 9 is the structure chart in the motion compensation portion of the dynamic image coding device represented in example of the present invention.

Figure 10 is the workflow diagram in the motion compensation portion represented in Fig. 9.

Figure 11 is the structure chart of the dynamic image coding device represented in example of the present invention.

Figure 12 is the structure chart in the motion compensation portion of the dynamic image coding device represented in example of the present invention.

Figure 13 is the workflow diagram in the motion compensation portion represented in Figure 12.

Figure 14 is the structure chart in the memory updating portion of the dynamic image coding device represented in example of the present invention.

Figure 15 is the workflow diagram in the memory updating portion represented in Figure 14.

Figure 16 is the structure chart of the dynamic image coding device represented in example of the present invention.

Figure 17 is the structure chart of the dynamic image coding device represented in example of the present invention.

Figure 18 is the structure chart of the dynamic image coding device represented in example of the present invention.

Figure 19 is the structure chart of the dynamic image coding device represented in example of the present invention.

Figure 20 is the schematic diagram of the bit stream 21 representing example 1 of the present invention.

Figure 21 is the schematic diagram of the bit stream 21 representing example 2 of the present invention.

Figure 22 is the schematic diagram of the bit stream 21 representing example 3 of the present invention.

Figure 23 is the schematic diagram of the bit stream 21 representing example 6 of the present invention.

Figure 24 is the structure chart of the moving image coding device represented in example of the present invention.

Figure 25 is the structure chart in the motion compensation portion of the moving image coding device represented in example of the present invention.

Figure 26 is the workflow diagram representing motion compensation portion.

Figure 27 is the key diagram that interpolation process is described.

Figure 28 is the workflow diagram in the memory updating portion of the moving image coding device represented in example of the present invention.

Figure 29 is the key diagram of the video data structure represented in VM coded system.

Figure 30 is the key diagram representing VOP data structure.

Figure 31 is the structure chart of the structure representing VM code device.

Figure 32 is the workflow diagram of the code device represented in Figure 31.

Figure 33 is the key diagram representing the types of forecast corresponding with VOP type of coding.

Figure 34 is the structure chart in the motion compensation portion represented in the code device shown in Figure 31.

Figure 35 is the workflow diagram in the motion compensation portion represented in Figure 34.

Figure 36 is the structure chart in the memory updating portion represented in the code device shown in Figure 31.

Figure 37 is the workflow diagram in the memory updating portion represented in Figure 36.

The optimal morphology carried out an invention

Example 1

Fig. 1 is the structured flowchart of the code device in that represent example 1 and following example.In this figure, 1 is input dynamic image signal, 2 is structured datas, 3 is mobility detect portions, 4 is moving parameters, 5 is motion compensation portions, 6 is predictive image candidates, 7 is prediction mode selection portions, 8 is prediction mode, 9 is predictive images, 10 is predicated error images, 11 is structured coding portions, 12 is quantization DCT coefficient, 13 be partial decode predicated error image, 14 is partial decode images, 15 is memory updating portions, 16 is memory a, 17 is memory b, 18 is memory c, 19 is variable-length code multi-way multiplex portions, 20 is transmit buffer, 21 is bit streams.80 is scene change detection portions, and 81 is timers.Wherein motion compensation portion 5 and memory updating portion 15 are the predictive image generating units 100 realizing prediction mode.In addition, memory a, b, c is memory areas 200.The part do not related in this example in figure illustrates in example afterwards.Fig. 2 is the flow chart of the workflow representing this code device.

In this example, according to the significance level that the feature of the images such as the amount of movement colourity to input dynamic image is foundation, use multiple such as 3 memories respectively, and be provided with the structure upgrading arbitrary memory (region) the such as content of memory a with arbitrary time span.In addition, dynamic image order is the order of incoming frame unit.

(1) input signal

As mentioned above, above-mentioned code device will represent in the frame input unit of the image in each moment in dynamic image order, and frame is divided into the macro block (step S1) becoming coding unit of one of predicted area of image example.

(2) memory-aided method is made respectively

The image translating code in the past or the image fixedly provided in advance are stored in memory, but in this example, according to the significance level in the part district in frame, use 3 memories respectively as follows.

Memory a: store the area of image (=static or evenly movement, and the area of image of the background of even structure) that significance level is little.

Memory b: store the medium area of image of significance level (=subject moves, mobile less area of image).

Memory c: store the large area of image of significance level (=subject moves, mobile large area of image).

The area of image that the significance level stored in memory a is little can consider the middle background images districts occurred such as the scene of video conference.In addition, comprising the subject with certain minute movement, being also equivalent to comprise the background area in the scene making the overall evenly movement of picture by operating gamma camera.If replace these to move, in advance from being called that the junior unit of macro block obtains the movement of frame entirety, it can be used as the movement of macro block, is then effective.This can do like this, namely obtains the deformation parameter of the bending parameters in the sprite being equivalent to illustrate in the prior embodiment, the deformation parameter of frame entirety is regarded as the moving parameter of macro block.Moving parameter just can move in parallel parameter (=with mobile vector synonym), also can be comprise the affine parameter of distortion, far and near method moving parameter etc.Here, mobile vector is one of moving parameter example.

The medium area of image of the significance level stored in memory b can be considered such as not make a speech in the scene of video conference, personage's area of image of just health movement, the object tagma that in picture, attention level is low.The area of image that the significance level stored in memory c is large can consider that the spokesman in the scene of such as video conference waits the object tagma that attention level is the highest.

The district stored in memory b, c owing to having the action of subject itself, so certainly will consider, in each macro block, there is different moving parameters.At this moment moving parameter just can move in parallel parameter (=with mobile vector synonym), also can be comprise the affine parameter of distortion, far and near method moving parameter etc.

(3) mobility detect (step S2)

Mobility detect portion 3 in this example removes the difference of that mobile vector in conventional example and bending parameters, can detect the arbitrary deformation parameter of whole 3 memories in macroblock unit.In addition, mobility detect portion 3 has the overall moving parameter measuring ability of the deformation parameter detecting frame entirety with memory a, and detects the local moving parameter measuring ability of deformation parameter of each macro block with memory a to c.

(4) motion compensation (step S3)

The internal structure in the motion compensation portion 5 in this example is shown in Fig. 3.In the figure, 22 is predictive image storage address calculating parts, and 23 is predictive image storage addresss, and 24 is memory reading units, and 25 is the reference memory index signals from outside supply.Here, use memory a, b, c is used to refer to reference to memory index signal 25.Describe the flow process of the work in this motion compensation portion 5 step S11 shown in Figure 4 to walking S16.

First, if I (inside)-frame, do not carry out motion compensation (step S11).If not I-frame, then the overall moving parameter generation forecast image candidate (step S12 is to walking S15) together with the local moving parameter from each memory will detected by mobility detect portion 3.Specifically, according to moving parameter 4 in predictive image storage address calculating part 22, calculate the predictive image storage address 23 existed by the predictive image candidate in the memory indicated with reference to memory index signal 25, according to predictive image storage address 23, memory reading unit 24 is taken out and prediction of output image candidate 6 from the memory of correspondence.

In this example, because overall moving parameter or local moving parameter are all based on same mode of texturing, so generate any predictive image jointly can use mobile motion compensation portion 5 in Fig. 3.In addition, with (step S15) during overall moving parameter generation forecast image candidate 6, often memory a is used as with reference to memory.

(5) selection (step S4) of prediction mode

Prediction mode in this example has following several:

A () uses the mode of memory a;

B () uses the mode of memory b;

C () uses the mode of memory c;

D () uses the mode of frame interior code signal.

Prediction mode selection portion 7 is as shown in conventional example, comprise the whole predictive image candidate 6 and frame interior code signal that are generated by motion compensation portion 5, select the minimum predictive image candidate 6 of the electric power (amplitude) of predictive error signal to export as predictive image 9.In addition, also the prediction mode 8 corresponding with selected predictive image 9 is exported.Containing representing that information selected by the memory predicting the memory of selected predictive image 9 in the information of this prediction mode 8.Prediction mode 8 is fed to variable-length code multi-way multiplex portion 19, by the code length of distributing, encodes, as forecast memory indication information 800 in bit stream 21.

(6) renewal of memory

The renewal of memory is controlled by memory updating portion 15.The internal structure in the memory updating portion 15 in this example is shown in Fig. 5.In the figure, 26 is memory a activity (will below describe in detail) when upgrading, 27 is that memory a upgrades judging part, 28 is with reference to memory selection portion, 29,30 is switches, and 31 is pictorial data of more new memory a, and 32 is pictorial data of more new memory b, 33 is pictorial data of more new memory c, and 34 is global prediction pictorial data of more new memory a.In addition, the workflow of memory updating is shown in Fig. 6.

Memory updating in this example carries out in the following order.If memory updating portion 15 have input partial decode image 14, then upgrade in judging part 27 at memory a and judge the more new memory a necessity (step S17) for the frame belonging to this partial decode image 14 in reference to memory selection portion 28 according to prediction mode 8, select memory (the step S18 used in prediction, S19), use the pictorial data 31 of the memory a upgrading this partial decode image 14, the more pictorial data 32 of new memory b, the more pictorial data 33 of new memory c, a kind of data in the global prediction pictorial data 34 of more new memory a, according to the reference image in the memory selected by following Policy Updates.In addition, by predicting unit (macroblock unit), this memory updating is carried out to each frame.

(1) each frame of memory b, c upgrades (step S20, S21)

Partial decode image 14 is write the memory b or memory c that use in the prediction of this image.

(2) arbitrary frame of memory a upgrades (step S22, S23)

Judged result 1000 is upgraded, only to arbitrary frame or with the arbitrary time interval, by the memory a used in the prediction of this image of partial decode image 14 write according to the memory a in step S17.

The content of memory a can be described as the history of the time-independent area of image of background images etc., and except the content of non-area changes a lot along with mobile grade of the change of scene or the large of picture entirety, otherwise the content of memory just there is no need renewal.

As mentioned above, if having the behaviour area of subject etc. with frame is that unit successively upgrades, and the content of memory a carries out the structure that upgrades with the longer time interval, then can effectively according to the movement of subject, prediction background images vaguely.

In this example, according to above viewpoint, be provided with the structure of the renewal carrying out memory a with the arbitrary time interval.Specifically, such as can consider:

A. according to the size of overall moving parameter, when mobile large time picture all content is upgraded together, the method for mobile hour not more fresh content;

B. not by the restriction in the time interval between frame, the method for picture all content is upgraded together every certain regular hour;

When c. detecting that scene changes, arbitrarily judgment standard is upgraded by the method etc. that picture all content upgrades together to the frame after scene change.

In this example, the data becoming above any renewal judgment standard are like this referred to as the activity 26 that memory a upgrades.First, memory a upgrades the content (step S22) that judging part 27 activity 26 judges whether more new memory a.When above-mentioned any renewal judgment standard a, the overall moving parameter value detected by mobility detect portion 3 becomes activity, when upgrading arbitrarily judgment standard b, time stamp from this frame of timer 81 is equivalent to activity 26, when upgrading arbitrarily judgment standard c, the mark detected from the notice scene change of scene change test section 80 output is equivalent to activity 26.

Conclude will the content of more new memory a time, the content of partial decode image 14 exported by as global prediction pictorial data 34, the content (step S23) of alterable storage a.When not concluding the content of more new memory a, do not carry out the renewal of memory a.

In addition, the memory a in this frame upgrades judged result 1000 can carry out any renewal same with in decoding side and by demultiplexing in bit stream 21, give decoding side.

Figure 20 is the schematic diagram of the bit stream 21 representing this embodiment.

In fig. 20, situation about being transferred after frame data are encoded in order is schematically shown.Heading message is added in the beginning of each frame data as the additional information of frame unit.In addition, in heading message, memory a upgrades judged result 1000 by demultiplexing, and is transmitted to decoding side.Is the macro block data forming this frame after heading message.The forecast memory indication information 800 of the memory representing this macro block data of prediction is comprised in the inside of macro block data.In code translator, according to the forecast memory indication information 800 of each macro block data, the memory of particular prediction image, generation forecast image.

In addition, although not shown, also memory b lastest imformation and/or memory c lastest imformation can be upgraded together with judged result 1000 with memory a or replace this memory a to upgrade judged result 1000 and be transported to decoding side.

Utilize above-mentioned code device, according to the content of dynamic image order, supply uses the structure of multiple memory effectively respectively, can improve forecasting efficiency.Namely, when carrying out the prediction of dynamic image order, according to perhaps character in dynamic image order, use multiple memory respectively, the prediction that to carry out with arbitrary deformation parameter be foundation, even if so comprise complicated movement, also the effective moving image estimating conformed to local image property can be carried out, can forecasting efficiency be improved, can the quality that can ensure coding image be formed, the code device of the traffic volume of coded data can be reduced again.In addition, carry out in the code translator of decoding in prediction mode according to the present invention to the bit stream after coding, also can form by same prediction mode.

In this example, describe the device that each frame is encoded, even but the device that the dynamic image target (VOP) with arbitrary shape is encoded, also can obtain same effect.

In addition, in this example, it is the code device of unit that an example as predicted area of image describes with macro block, even but according to the device that the image cell of the variable block of the shape waiting the combination of image cell or multiple fixed dimension block to form by the local image with arbitrary shape etc. is encoded to image, also same effect can be obtained.

In addition, in this example, employ and utilize the overall moving parameter of memory a to detect, but without it, even only use the structure of local mobility detect, also can certainly be suitable for.When not carrying out overall mobility detect, as prediction mode, do not need the judgement information transmitting the overall situation/local prediction.

In addition, in this example, be provided with prestore generate according to the content of dynamic image order reference pictorial data, do not carry out the memory that upgrades in an encoding process, use it for prediction.

In addition, in this example, describe after memory a, b, c subregion is separately stored, more news is carried out by any one memory in 15 pairs, memory updating portion memory a, b, c, but when memory a, b, c have part or all and memory image of image, two memories of memory updating portion 15 more in new memory a, b, c or all memory.Such as, when memory a be the frame memory of the reference pictorial data of storage 1 frame, memory b is when being not the dynamic sprite memory with successively upgrading with the static sprite memory successively upgraded, memory c, because data fixing in advance store as with reference to pictorial data by static sprite memory, memory b upgrades so can not be stored device update section 15, but when memory a and memory c stores the reference pictorial data of the same area, memory updating portion 15 simultaneously more new memory a and memory c.Like this, memory a, b, c are when repeated storage is with reference to pictorial data, and memory updating portion 15 upgrades the region of repetition respectively.

Situation also with above in the example of the following stated is identical.

In addition, in this example, describe the situation of use 3 memories a, b, c, even if but use during any two memories and also have no relations.

In addition, code translator can also be formed by the predictive image generating unit 100 of the motion compensation portion 5 had with illustrate in this example, the identical component in memory updating portion 15.When for code translator, motion compensation portion does not need to generate whole 3 predictive image candidates, only generates the predictive image relevant with the moving parameter translating code.

Example 2

Secondly, be given in the example of the structure difference only making motion compensation portion 5 in the code device formed shown in Fig. 1 and the code translator formed, and structure and the working condition of the motion compensation portion 5a of example 2 are described.

The internal structure of the motion compensation portion 5a of this example is shown in Fig. 7.In the figure, 35 is with reference to memory determination portion.In addition, the flow process of the detailed operation situation describing this motion compensation portion 5a is shown in Fig. 8.

First, if I-frame, then do not carry out motion compensation (step S24).If not I-frame, then with reference to the value of memory determination portion 35 according to moving parameter 4, determine with reference to memory (step S25).The effective moving parameter value scope (will describe in detail) distributing to each memory a, b, c is remain below with reference to memory determination portion 35, by comparing with the value of moving parameter 4, judge which memory block this moving parameter 4 refers to, export the reference memory index signal 25a of recognition memory a, b, c.

So-called effectively moving parameter value scope, refer to such as when detecting mobile vector, assuming that its exploration scope is ± 15 pixels, then effective exploration scope of each memory is: in ± 0 to 3 pixel coverage, memory a is used for prediction, in ± 4 to 8 pixel coverages, memory b is used for prediction, in ± 9 to 15 pixel coverages, memory c is used for prediction.But, when with overall moving parameter generation forecast image, regarding as memory a with reference to memory, so when only using local moving parameter, just starting with reference to memory determination portion 35.Like this, the specific memory for predicting of value of what is called mobile vector, refers to and moves little based on background images and attention level is higher image moves prerequisite more so carries out specific.Like this, with the value of mobile vector specific for predict memory time, do not need to prediction mode carry out coding transmit.

Secondly, along with selected reference memory index signal 25a, generation forecast image candidate 6 (step S26 is to walking S30).Specifically, according to moving parameter 4 in predictive image storage address calculating part 22, calculate the predictive image storage address 23 existed by the predictive image candidate 6 in the memory indicated with reference to memory index signal 25a, memory reading unit 24, according to predictive image storage address 23, is taken out and prediction of output image candidate 6 from the memory of correspondence.

In this example, due to overall moving parameter or local moving parameter be all based on same mode of texturing, so generate any predictive image jointly can use the motion compensation portion 5a shown in Fig. 7.In addition, with (step S31) during overall moving parameter taking-up predictive image candidate 6, often memory a is used as with reference to memory.

Although effectively moving parameter value region also can be certain value region concerning each dynamic image order, such as, also can change by each frame.When changing by each frame, effective moving parameter value region of each memory in this frame in bit stream, is given decoding side by demultiplexing, can carry out same memory select in decoding side.

Figure 21 is the schematic diagram of the bit stream 21 representing this example.

The heading message be attached in dynamic image sequential cell is had in the beginning of this bit stream.This heading message has effective moving parameter value region indication information of each memory.Like this, by specifying effective moving parameter value region indication information in the beginning of dynamic image order, in code translator, can perform to this dynamic image order the prediction mode that have employed effective moving parameter value region in certain numerical value region.

When changing effective moving parameter value region to each frame, be attached in the heading message in frame unit the indication information that also can comprise effective moving parameter value region.

Utilize the above-mentioned code device with the structure of motion compensation portion 5a, according to the mobile degree in the local of frame, supply uses the structure of multiple memory effectively respectively, can improve forecasting efficiency.

In this example, describe the device that each frame is encoded, even but the device that the dynamic image target (VOP) with arbitrary shape is encoded, also can obtain same effect.

In addition, in this example, describing with macro block is the code device of unit, even but according to the device that the image cell of the variable block of the shape be made up of the image cell of local image etc. or the combination of multiple fixed dimension block with arbitrary shape etc. is encoded to image, also same effect can be obtained.

In addition, in this example, employ and utilize the overall moving parameter of memory a to detect, but without it, even only use the structure of local mobility detect, also can certainly be suitable for.When not carrying out overall mobility detect, as prediction mode, do not need the judgement information transmitting the overall situation/local prediction.

In addition, code translator can also be formed by the predictive image generating unit 100 of the identical component of motion compensation portion 5a had and illustrate in this example.When for code translator, motion compensation portion only generates the predictive image relevant with the moving parameter translating code.

Example 3

Secondly, be given in the example of the structure difference only making motion compensation portion 5 in the code device formed shown in Fig. 1 and the code translator formed, and structure and the working condition of motion compensation portion 5b are described.Mobility detect portion 3a in this example exports amount of parallel movement, affine parameter, far and near method parameter respectively as moving parameter 4a.

In addition, assuming that the memory a in this example is the frame memory with reference to image of storage 1 frame sign, memory b is static sprite memory, and memory c is dynamic sprite memory.

The internal structure of the motion compensation portion 5b of this example is shown in Fig. 9.In the figure, 36 is amount of parallel movement (=mobile vectors), 37 is affine parameters, 38 is far and near method parameters, 39 is the predictive image storage address calculating parts based on amount of parallel movement, 40 is the predictive image storage address calculating parts based on affine parameter, and 41 based on the predictive image storage address calculating part of far and near method parameter.In addition, the flow process of the detailed operation situation describing this motion compensation portion 5b is shown in Figure 10.

First, if I-frame, then do not carry out motion compensation (step S33).During if not I-frame, then predictive image storage address calculating part 39 to 41 is according to the value of each moving parameter 4a, computational prediction video memory address 23 (step S34).

Each predictive image storage address calculating part carries out address computation according to the anamorphose mode distributing to corresponding each memory.In this example, will move in parallel for memory a, be used for memory b by with rotation to a certain degree or amplification, the affine parameter that the simple distortion such as to reduce, the far and near method parameter with the complicated movement of 3 dimensions will be used for memory c.The following transform of these modes of texturing represents.

[moving in parallel]

Amount of parallel movement (a, b):

x’＝x+a

y’＝y+b

[affine transformation]

Affine parameter (a, b, c, θ):

x’＝a(cosθ)x+a(sinθ)y+b

y’＝a(-sinθ)x+a(cosθ)y+c

[far and near method conversion]

Far and near method parameter (a, b, c, d, e, f):

x’＝(ax+by+c)/(gx+hy+1)

y’＝(dx+ey+f)/(gx+hy+1)

In formula, (x, y) on two-dimensional coordinate is the pixel location of former macro block, and (x ', y ') is the pixel location according to each parameter in the memory corresponding with (x, y).That is, according to these parameters, (x ', y ') position is on a memory calculated.Utilize this structure, can predict by the memory that is all applicable to of the character of movement from each macro block.Utilize the predictive image storage address 23 calculated according to each moving parameter 36,37,38, memory reading unit 24 is taken out and prediction of output image candidate 6 (step S 35 to step S 39) from the memory of correspondence.

In addition, the type of the mode of texturing of each memory in this frame as movement detection method indication information by demultiplexing in bit stream 21, give decoding side, same motion compensation can be carried out in decoding side.

Figure 22 is the schematic diagram of the bit stream 21 representing this example.

Movement detection method indication information is comprised in the heading message that the beginning of dynamic image order is additional.In code device, the type of the mode of texturing used in each memory can change, and will represent that the movement detection method indication information of its corresponding relation gives code translator as the heading message of dynamic image order.The type of the mode of texturing used in each memory just can be identified like this in code translator.

In code translator, the type of the mode of texturing of this identification dynamically corresponds to each memory.

Utilize the above-mentioned code device with the structure of motion compensation portion 5b, according to the mobile character in the local of frame, supply uses the structure of multiple memory effectively respectively, can improve forecasting efficiency.

In this example, describe the device that each frame is encoded, even but the device that the dynamic image target (VOP) with arbitrary shape is encoded, also can obtain same effect.

In addition, in this example, describing with macro block is the code device of unit, even but according to the device that the image cell of the variable block of the shape be made up of the image cell of local image etc. or the combination of multiple fixed dimension block with arbitrary shape etc. is encoded to image, also same effect can be obtained.

In addition, in this example, employ and utilize the overall moving parameter of memory a to detect, but without it, even only use the structure of local mobility detect, also can certainly be suitable for.When not carrying out overall moving parameter and detecting, as prediction mode, do not need the judgement information transmitting the overall situation/local prediction.

In addition, in this example, give the situation using memory a, b, c, even but only use the situation of memory a and b or only use the situation of memory a and c or only use the situation of memory b and c also to have no relations.

In addition, code translator can also be formed by the predictive image generating unit 100 of the identical component of motion compensation portion 5b had and illustrate in this example.When for code translator, motion compensation portion only generates the predictive image relevant with the moving parameter translating code.

Example 4

Secondly, the dynamic image order that multiple, such as 2 the different dynamic image target mixing that illustrate to have shape information exist is object, to the example of the device that these dynamic image targets are encoded in the lump.Figure 11 shows the structure of the code device in this example.

In this figure, 42 is input imagery frames, and 43 is target separation units, 44a, 44b is target data, 45a, 45b is shape block, 46a, 46b is switch, 47a, 47b is shape coding portion, 48a, 48b is compression shape blocks of data, 49a, 49b is partial decode shape block, 50a, 50b is structured data (macro block), 51a, 51b is mobility detect portion, 52a, 52b is moving parameter, 53a, 53b is motion compensation portion, 54a, 54b is predictive image candidate, 55a, 55b is prediction mode selection portion, 56a, 56b is prediction mode information, 57a, 57b is predictive image, 58a, 58b is predictive error signal, 59a, 59b is structured coding portion, 60a, 60b is pressure texture data, 61a, 61b is partial decode predictive error signal, 62a, 62b is partial decode macro block, and 63 is memory updating portions, and 64 is memory a, and 65 is memory b, and 66 is memory c, and 67 is memory d, and 68 is memory e, and 69 is memory f, 70a, 70b is variable-length coding unit, and 71 is multiplexed portions, 72 buffers, 73 bit streams, and 94 is storage parts, and 88a is the A target code portion to A target code, and 88b is the B target code portion to B target code.Target code portion 88a, 88b have the same internal structure be made up of commaterial.

This code device input imagery frame 42, it is divided into the target of coding unit in target separation unit 43.The process of target separation unit 43 can adopt arbitrary method.

Target shape information is fed to shape coding portion 47a, 47b with the form of shape block 45a, 45b and encodes, and gives variable-length coding unit 70a, 70b as compression shape blocks of data 48a, 48b.

Mobility detect portion 51a, 51b and VM coded system are the same, consider partial decode shape block 49a, 49b, carry out the detection of moving parameter.Moving parameter detects and with whole memory a to f, can carry out in macroblock unit.

But, as principle, for the target A carrying out encoding in target A coding unit 88a, use memory a to c, for the target B carrying out encoding in target B coding unit 88b, use memory d to f.

In addition, as the kind of movement, remove the difference of mobile vector and bending parameters, the whole memories for storage part 94 can detect arbitrary deformation parameter in macroblock unit.

According to each moving parameter 52a, 52b in motion compensation portion 53a, 53b, after generating whole predictive image candidate 54a, 54b, obtain predictive image 57a, 57b and prediction mode information 56a, 56b at prediction mode selection portion 55a, 55b.Get the difference of original signal or structured data 50a, 50b and predictive image 57a, 57b, obtain predictive error signal 58a, 58b, it is fed to variable-length coding unit 70a, 70b after encoding in structured coding portion 59a, 59b.In addition, predictive error signal 61a, 61b of carrying out partial decode are added with predictive image 57a, 57b, become partial decode macro block 62a, 62b, according to the instruction in memory updating portion, are written into memory a to f.

In above-mentioned A/B target code portion 88a, 88b, by the A/B target data of encoding in multiplexed portion 71 by demultiplexing in 1 bit stream 73, carried by buffer 72.

Below centered by motion compensation portion 53a, 53b of carrying out groundwork, the prediction in this example is described.

Motion compensation portion 53a, 53b in this example according to moving parameter 52a, 52b of being detected by mobility detect portion 51a, 51b, generation forecast image candidate.The internal structure of motion compensation portion 53a is shown in Figure 12.In addition, the workflow of the motion compensation portion 53a in A target code portion 88a is shown in Figure 13.In addition, motion compensation portion 53b forms too, works equally.

In fig. 12,74a is B target reference judging part, and 75a is that B target is with reference to cue mark.

Here, moving parameter 52a also comprises the memorizer information detected.The same with example 1, utilize forecast memory address computation portion 22a, memory reading unit 24a according to parameter value, generation forecast image candidate (step S44 is to walking S49).In addition, B target according to the memorizer information of the reference in moving parameter 52a, judges the memory (step S43) whether using target B in the prediction of this macro block with reference to judging part 74a.

Whether judged result exports as B target with reference to cue mark 75a with reference to judging part 74a by B target, can only regenerate separately with itself memory a, b, c, so by demultiplexing in bit stream 73, give decoding side for judging in this target of decoding side.Can control like this during coding, namely according to the signal 85a from outside supply, just can carry out the independent regeneration of this target through decoding side of being everlasting, when detecting moving parameter, can control like this, be i.e. the memory (a, the b, c) that predict of a use itself.

Utilize the code device of the structure with above-mentioned motion compensation portion 53a, 53b, according to the mobile character in the local of frame, supply uses the structure of multiple memory effectively respectively, can improve forecasting efficiency.

In addition, in this example, describe to the device that target is encoded in macroblock unit, even but carry out the device of encoding according to unit such as the variable blocks of the shape be made up of the local image or the combination of multiple fixed dimension block with arbitrary shape, also can obtain same effect.

In addition, can also use and form code translator with the motion compensation portion identical component of 53a, 53b illustrated in this example.When for code translator, 53, motion compensation portion generates the predictive image relevant with the moving parameter translating code.In addition, if formed like this, namely according to bit stream, carry out decoding to being equivalent to the position of another target with reference to cue mark 75a, 75b, and can identify whether the target in decoding can regenerate separately, so just inerrably can carry out decoding regeneration to target data.

Example 5

Secondly, illustrate and correspond to dynamic image target over time, the example of the code device that the quantity of memory and capacity can be made at random to change.In example 5, consider the code device after the structure in the memory updating portion 15 in the code device of structure shown in Fig. 1 being changed.

Figure 14 shows the internal structure of the memory updating portion 15a in this example.In the figure, 76 is that memory launches judging part, and 77 is that memory launches index signal, and 78 is that memory is shunk back judging part, and 79 is that memory is shunk back index signal.Figure 15 shows the workflow (step S51 is to walking S63) of memory updating portion 15a.

When significantly different visual of the history of the dynamic image order occurred due to scene change etc. and store in memory, in the prediction after scene change, only with often declining with reference to forecasting efficiency time visual of containing in original memory.In this case, if the change of scene can be detected by the change test section 80 of scene, frame interior coding etc. is carried out to the frame after scene change, and it can be used as new adding stored in memory with reference to image, then can improve later forecasting efficiency.

In addition, the capacity due to the memory that can add has the limit of physical property, so have the structure of shrinking back gradually with reference to the part used hardly when predicting in image that can make on memory.Make it to have such structure, namely according to prediction mode 8, the frequency used when being measured each memory block prediction of memory a, b, c by memory updating portion 15a, by the structure of storage area little for frequency from the expansion of use district in memory updating portion.If adopt this example, such as, when with software sharing code device, limited RAM (random access memory) resource can be effectively utilized.

According to above-mentioned viewpoint, the memory updating portion 15a in this example has according to the time dependent state of dynamic image order, or memory block is increased, or makes the function of shrinking back with reference to the memory block of image containing less using during prediction.

Memory a is identical with example 1, upgrades in judging part 27 determine whether can upgrade (step S50) at memory a, by partial decode image 14 write memory a (step S56, step S57) during renewal.In addition, according to prediction mode 8, by each memory of partial decode image 14 write (step S51 is to walking S55).

The renewal of these memory contents is that the judgement launching according to memory/shrink back is carried out.Launching in judging part 76 at memory, the activity 26 used when upgrading according to memory a, judging whether to increase memory a (or memory b, capacity (walking S58 to walking S60) c).If conclude can increase due to scene change etc., just launch index signal 77 instruction memory with memory and launch.In addition, shrink back in judging part 78 at memory, the counting of the memory block in predicting according to prediction mode 8, only to the memory block in the prediction below stipulated number, shrink back (step S61 is to walking S63) of index signal 79 instruction memory of shrinking back with memory.

Utilize the code device with the structure of above-mentioned memory updating portion 15a, can change in time along with dynamic image order and effectively predict, supply the structure of temporally being distributed necessary memory block during prediction simultaneously, can forecasting efficiency be improved, can memory resource be effectively utilized simultaneously.

In addition, in this example, describe the device that each frame is encoded, even but the device that the dynamic image target (VOP) with arbitrary shape is encoded, also can obtain same effect.

In addition, in this example, describe in macroblock unit the device that frame is encoded, even but the device of encoding is carried out according to image cells such as the variable block districts of the shape be made up of the image cell of local image etc. or the combination of multiple fixed dimension block with arbitrary shape, also can obtain same effect.

In addition, can also use and form code translator with the memory updating portion identical component of 15a illustrated in this example.

Example 6

In above-mentioned each example, describe the situation changing the memory used in macro block unit-prediction, but also can adopt the memory changing and use in frame or the prediction of dynamic image object element, carry out the structure predicted.Therefore, do not need corresponding frame by frame or the memory of the dynamic image object element coding information of being correlated with and information (comprising prediction mode 8) should be selected to encode by the memory of macroblock unit coding, can effectively encode.

Such as in the code device shown in Fig. 1 of example 1, due to the memory used in macroblock unit prediction can be switched, so the additional information representing and employ which memory of macroblock unit in prediction must be transmitted.Unlike this, in this example, be limited in frame or dynamic image target by the switch unit of the memory by this use and predict, so can effectively reduce the additional information that should send macroblock unit to.Figure 23 shows the difference of the transmission bit stream 21 of the example 1 shown in Figure 20 and the transmission bit stream 21 of this example.Example shown in Figure 23 represents the situation changing the memory used in frame unit prediction, and forecast memory indication information 800 is included in the heading message of frame unit.Example shown in Figure 23 is effective often localized variation that is constant, macro block level is little at the image property of such as dynamic image order.In addition, can form to as above coding after bit stream carry out decoding, make the code translator of frame or dynamic image object regeneration.

Example 7

In above-mentioned each example, can also form like this, be about to from multiple memory arbitrary 2 memories (such as memory a and b) 2 predictive image candidates taking out be added, and using the image after average as in predictive image candidate 6 or use as predictive image 9.In addition, can form to as above coding after bit stream carry out decoding, make the code translator of frame or dynamic image object regeneration.

Example 8

In the code device shown in above-mentioned each example, can also form like this, namely detect regulation in advance to become the complexity in the space of the area of image carrying out predicting unit, watch the amount of characteristic parameter of degree etc. attentively, using they as determine prediction mode, more new memory time deposit index utilize.

Such as, imagine a kind of and comprise complicated movement, carry out reaching the inconvenient dynamic image of coding allowing quality in given encoding amount.In this case, determine the significance level of each predicted area of image (macro block, arbitrary shape area of image, arbitrary shape block etc.), the district low for significance level can reduce quality to a certain extent, reduce encoding amount, this part encoding amount is distributed to the high district of significance level, can overall qualities be improved.Resemble the present invention to switch multiple memory in the arbitrary moment like this and carry out in the device predicted, by detecting the amount of characteristic parameter of the significance level representing predicted area of image, determine the using method of memory according to this dynamic state of parameters, the prediction being more suitable for image property can be carried out.Such as, as shown in figure 16, setting area significance level test section 95, is used for detecting the amount of characteristic parameter in each district, determines significance level.District's significance level test section 95 sends district's significance level to prediction mode selection portion 7a, simultaneously according to district's significance level, quantization parameter is sent to structured coding portion 11a.Be judged as by significance level test section 95 district that significance level is high, predict by the most complicated mode in the multiple prediction mode prepared.Such as, utilize the reference image from each memory a, b, c, according to the mobility model of various complexity, obtain moving parameter and predictive image, in prediction mode selection portion 7a, comprise combination in any (be added after the average etc.) prediction mode that selection forecasting efficiency is the highest of these predictive images.At this moment, the whole memory updatings relatively used in prediction are with reference to image.And then in structured coding portion 11a, utilize the differential little quantization parameter of quantification to encode.On the other hand, in the district that significance level is low, utilize easy prediction mode (only detecting by the amount of parallel movement of a memory) to predict, utilize the differential large quantization parameter of quantification to encode, to become the little code amount had nothing to do with the amplitude of obtained predictive error signal.By carrying out above control, even if the low district of significance level reduce somewhat quality, but the high district of significance level is by high-precision forecast, can guarantee quality, can improve all qualities by given code amount.

Example 9

Utilizing multiple memory, carry out dynamic image order forming like this in the device of predictive coding, namely in each moment of dynamic image order, detect the parameter of the quantity of spendable code or the scene variable quantity in this moment (scene change detects), the characteristic quantity of the predicted area of image described in example 8 or significance level etc., the deposit index when prediction mode that can use these values in the picture prediction in this moment with electing or the selection of reference memory block.Such as, as shown in figure 17, so a kind of apparatus structure can be considered, namely the frame significance level test section 96 pressing frame unit determination significance level is set, detect the presence or absence etc. that this frame is hidden relative to the variable quantity (detecting the scene change from scene change test section 80) of front frame, the appearance of fresh target, consider, from the code amount that can use in the frame transmitting buffer 20 notice, to determine last frame significance level.Accordingly, when the frame that significance level is high, utilize the whole prediction mode and reference memory block that prepare, improve forecasting efficiency to greatest extent, when the frame that significance level is low, limit usage forecastings mode or reference memory block, simplify coded treatment, reduce treating capacity.In addition, it is also conceivable to not predict when scene changes, and all carry out the device of frame interior coding.Moreover, if used in the lump with district's significance level test section 95 of illustrating in example 8, then can carry out quite thin quality control.By carrying out above control, even if the low frame of significance level reduce somewhat quality, but the high frame of significance level ensure that quality due to high-precision forecast, can improve all qualities by supplied code amount.

The idea of this example also can be applicable to following situation: the quantity of handling procedure or operable memory, when being encoded by mobile software, is applied the resource that can utilize to greatest extent, effectively carried out coded treatment.The treating capacity of the low frame of significance level can be reduced, can overall processing speed be improved.

Example 10

Structure as shown in figure 11 can be adopted, with multiple memory, the dynamic image order by multiple dynamic image target configuration is being predicted, in the device of coding, detecting can as the gross yards amount used in order, the quantity of the code that can use in each moment of each dynamic image target, or the variable quantity (appearance of target of dynamic image target in this moment, hidden etc.), the grade of the significance level of each dynamic image target in scene/watch attentively degree, the parameter of the characteristic quantity of the predicted area of image described in example 8 or 9 or significance level etc., by these values with elect this time the prediction of dynamic image target of being somebody's turn to do in the prediction mode that can use or with reference to deposit index during memory block.

Such as, as shown in figure 18, so a kind of apparatus structure can be considered, namely the significance level test section 97a ~ 97n corresponding with each target 1 ~ n is set, detect represent target the variable quantity in each moment or target appearance, hidden etc. with presence or absence of parameter, consider the occupation rate of whole buffer 72x of coded data and the occupation rate of the virtual bumper 72a ~ 72n of each target that store target complete simultaneously, determine the significance level of target in each moment.Such as, when there is novel district as the result of the part overlap of other target etc. in target, this has a great impact later forecasting efficiency, even if so control when not having enough and to spare in the virtual bumper of target to a certain extent, also will considered to improve significance level, make coding image excellent etc.The significance level detected by significance level test section 97a ~ 97n is fed to target 1 ~ N coding unit 98a ~ 98n, when the target that significance level is high, utilize the whole prediction mode and reference memory block that prepare, improve forecasting efficiency to greatest extent, when the target that significance level is low, limit usage forecastings mode or with reference to memory, simplify coded treatment, reduce treating capacity.In addition, carry out the device of encoding making target in real time after frame is separated, when making the content marked change of this target due to the appearance of new target or the hidden of existing target etc., it is also conceivable to not predict, all carry out the device of frame interior coding.Moreover, if used in the lump with district's significance level test section 95 of illustrating in example 8 in target 1 ~ N coding unit 98a ~ 98n, then can carry out the quality control thinner than each predicted target area unit in target.By carrying out above control, even if the low target of significance level reduce somewhat quality, but the high target of significance level ensure that quality due to high-precision forecast, can improve all qualities by supplied code amount.

Example 11

In addition, as shown in figure 19, the information of forecasting coding unit 91 of the code assign ment (coding) of the coded message (with reference to memory numbering etc.) of carrying out phone predicts is set sometimes.

Such structure can be adopted: predicting dynamic image order or dynamic image target with multiple memory a, b, c, in the device of encoding, according to the usage frequency of memory during prediction, grade is added to multiple memory, and this grade in an encoding process added by the change of dynamic ground, according to the grade of multiple memories during prediction, carry out the code assign ment of the coded message (with reference to memory numbering etc.) of the phone predicts in above-mentioned information of forecasting coding unit 91.

Such as in the code device shown in Figure 19, counting memory a, b, c being respectively set in memory updating portion 15b when predicting by the counter 92 of frequency used, according to its count value, adding grade to memory a, b, c, output level information 90.The applying of this grade can such as be carried out for unit at the image (VOP) in certain moment with 1 frame or dynamic image target, also namely can carry out in each predicted area of image (macro block, arbitrary shape district, arbitrary shape block etc.) at thinner unit.

Therefore, can know which memory with what kind of frequency is used in prediction.In prediction, the frequent memory used is the memory that significance level is high when predicting, higher with reference to frequency, grade is also higher.

So, when the frequency information of the memory used during prediction being encoded in each predicted area of image unit, code assign ment short for the length of code is given the memory (memory that=grade is high) of frequent reference, can code efficiency be improved like this.

In addition, if the moving parameter detected in each predicted area of image unit also can distribute code length with the grade of the memory of reference accordingly, then can give the frequent moving parameter value occurred by code assign ment short for the length of code, effectively can carry out the coding of information of forecasting.These structures can realize like this, and namely the information of forecasting coding unit 91 in variable-length code multi-way multiplex portion 19 receives the grade of each memory from the counter 92 of memory updating portion 15b, according to its class information 90, change code length, carry out the coding of information of forecasting.

Example 12

Figure 24 represents the structure digital image after compressed encoding being launched the image decoding apparatus of regeneration in this example.In this figure, 21 is the bit streams compiling code, 119 is variable-length decoder unit, 12 is quantization DCT coefficient, 12a is quantization orthogonal transform coefficient, 12b is quantization step-length, 111 is structure decoder unit, 111a is inverse guantization (IQ) portion, 111b is inverse orthogonal transformation portion, 190 is decoding adders, 101 is decoding images, 191 is display control units, 8 is prediction mode, 1001 is memory b lastest imformations, 1002 is memory c lastest imformations, 4 is mobile vector (moving parameters), 800 is forecast memory indication informations, 195 is predicted area of image positions in picture, 105 is motion compensation portions, 116 is memory a, 117 is memory b, 118 is memory c, 115 is memory updating portions, 106 is predictive images.Predictive image generating unit 100a is formed by motion compensation portion 105 and memory updating portion 115.In addition, memory block 200a is formed by memory a, b, c.

In this example, assuming that memory a is the frame memory of the pictorial data of storage 1 frame, memory b is static sprite memory, and memory c is dynamic sprite memory.In addition, the bit stream 21 of code translator supposition input shown in Figure 22 of this example.In addition, although not shown in fig. 22, memory b lastest imformation 1001 and memory c lastest imformation 1002 are sent in bit stream.The renewal instruction comprehensively upgrading static sprite memory and the pictorial data comprehensively upgraded is comprised in memory b lastest imformation 1001.Equally, the renewal instruction comprising comprehensive Regeneration dynamics sprite memory in memory c lastest imformation 1002 and the pictorial data comprehensively upgraded.

The working condition of device as constructed as above is below described.First, in variable-length decoder unit 119, analyze bit stream 21, be divided into each coded data.Quantization orthogonal transform coefficient 12a is fed to inverse guantization (IQ) portion 111a, carries out inverse guantization (IQ) with quantization step-length 12b.Its result, in inverse orthogonal transformation portion 111b, carry out inverse orthogonal transformation, structure is decoded, and gives decoding adder 190.Orthogonal transform adopts DCT (discrete cosine transform) etc. identical with used in code device side.

Mobile vector 4 after carrying out decoding by variable-length decoder unit 119 according to bit stream, forecast memory indication information 800 and represent that the information of the position 195 of predicted area of image in picture is transfused in motion compensation portion 105.Motion compensation portion 105 according to these 3 kinds of information, from be stored in multiple memory a, b, c with reference to image take out desired by predictive image.The position 195 of predicted area of image in picture is not included in the information in bit stream, but can be calculated by counting macroblock number.About the processing method that predictive image generates, the part of the working condition in following explanation motion compensation portion 105 is described in detail.

Decoding adder 190 is according to the information of prediction mode 8, if the block of frame interior coding, then the output of inverse orthogonal transformation portion 111b is directly exported as decoding image 101, if the block of intermediate frame coding, just predictive image 106 is added in the output of inverse orthogonal transformation portion 111b, exports as decoding image 101.Decoding image 101 is fed to display control unit 191, and is exported to display unit, simultaneously in decoding process afterwards as with reference to visual use, so be written into memory a ~ c.According to prediction mode 8, control the write to memory by memory updating portion 115.

Secondly, the predictive image generating process method in the motion compensation portion 105 of predictive image generating unit 100a is described.In this example, the prediction mode of image is determined according to forecast memory indication information 800.In the code translator of this example during generation forecast image, according to mobile vector 4 and forecast memory indication information 800, by coordinate transform processing and the interpolation process of regulation, from reference image denoise predictive image.The method of coordinate transform processing is just allocated to the memory used in prediction in advance accordingly.Such as, the following deformation method example identical with the anamorphose mode illustrated in example 3 can be considered.

(1) situation (=forecast memory indication information 800 indicates the situation using memory a) of memory a is used in prediction

According to mobile vector, the coordinate of each pixel of predicted target area is moved in parallel, the pictorial data of correspondence position in memory a is taken out as predictive image.

(2) situation of memory b (instruction of=forecast memory indication information uses the situation of memory b) is used in prediction

According to mobile vector, ask affine transformation formula, make the coordinate displacement of each pixel of predicted target area according to this transform, the pictorial data of correspondence position in memory b is taken out as predictive image.

(3) situation of memory c (instruction of=forecast memory indication information uses the situation of memory c) is used in prediction

According to mobile vector, ask far and near method transform, make the coordinate displacement of each pixel of predicted target area according to this transform, the pictorial data of correspondence position in memory c is taken out as predictive image.

The internal structure in motion compensation portion 105 is shown in Figure 25.In the figure, 161 is switches, and 162 is memory a corresponding points determination portions, and 163 is memory b corresponding points determination portions, and 164 is memory c corresponding points determination portions, and 165 is that memory reads address generating unit, and 166 is switches, and 167 is interpolation handling parts.In addition, Figure 26 is the flow chart representing its working condition.

Below, the working condition in the motion compensation portion 105 of this example is described according to Figure 25 and Figure 26.

1) determination of corresponding points

First, according to forecast memory indication information 800, selected the corresponding points determination portion of corresponding memory by switch 161, mobile vector 4 is inputted selected corresponding points determination portion.Carry out the calculating of the predictive image position corresponding with each memory here.Below correspond to memory to be described.

1-1) forecast memory indication information 800 indicates the situation (step S100) using memory a

According to moving in parallel of being undertaken by mobile vector, calculate predictive image position (step S101).Specifically, according to mobile vector (a, b), determined the predictive image position (x ', y ') of the pixel of the position (x, y) of predicted area of image by following formula.

x’＝x+a

y’＝y+b

Determined predictive image position is exported to memory and read address generating unit 165.

1-2) forecast memory indication information 800 indicates the situation (step S103) using memory b

Affine transformation formula is determined according to mobile vector 4.Specifically, with the mobile vector on the summit of the rectangle region of the predicted area of image of encirclement, the affine parameter (a, b, c, θ) in following formula is determined.

x’＝a(cosθ)x+a(sinθ)y+b

y’＝a(-sinθ)x+a(cosθ)y+c

Accordingly, obtain the predictive image position (x ', y ') of the pixel of the position (x, y) of predicted area of image, export to memory and read address generating unit 165 (step S104).

1-3) forecast memory indication information 800 indicates the situation (step S106) using memory c

Far and near method transform is determined according to mobile vector.Specifically, with the mobile vector on the summit of the rectangle region of the predicted area of image of encirclement, the far and near method parameter (a, b, c, d, e, f) in following formula is determined.

x’＝(ax+by+c)/(gx+hy+1)

y’＝(dx+ey+f)/(gx+hy+1)

Accordingly, obtain the predictive image position (x ', y ') of the pixel of the position (x, y) of predicted area of image, export to memory and read address generating unit (step S107).

2) reading of predictive image generation data

According to the predictive image position exported from selected corresponding points determination portion (x ', y '), memory reads the storage address of the position of necessary pictorial data when address generating unit 165 predictive image generated in the reference image stored in specific memory generates, and reads predictive image and generates with data (walking S102, S105, S108).

3) generation of predictive image

Form if the pixel of integer pixel location in the pixel of predictive image, then predictive image generation data just directly become predictive image formation pixel.On the other hand, when the pixel of the pixel location of real number precision, by being carried out the interpolation process of predictive image generation data by interpolation handling part 167, generate interpolation pixel value (step S109, S110, S111).The generation of interpolation pixel value is carried out according to Figure 27.In figure 27, (ip, jp) represents integer pixel location, and (i ', j ') represents the pixel location of real number precision, and w represents weight.

4) renewal of memory (with reference to image)

Figure 28 shows the controlling extent in memory updating portion 115.In memory updating portion 115, according to prediction mode 8 (or forecast memory indication information 800), control the renewal of each memory by predictive image retrieval unit (such as macroblock unit).When predicting with memory a (step S112), according to the content (step S113) of decoding image 101 successively more new memory a and c.When predicting with memory b (step S114), because memory b is static sprite memory, so the reference image of memory b upgrades not according to predictive image retrieval unit., according to the content (step S115) of decoding image 101 successively more new memory a and c.On the other hand, after the renewal instruction receiving memory b lastest imformation 1001, the pictorial data comprised in reception memorizer b lastest imformation 1001, the comprehensively content (step S116) of more new memory b.In addition, when predicting with memory c (step S117), according to the content (step S118) of decoding image 101 successively more new memory a and c.In addition, after received renewal instruction by memory updating information, the pictorial data comprised in reception memorizer c lastest imformation 1002, the more content (step S119) of new memory c.

In this example, describe the situation of use 3 memories a, b, c, even if but use 2 memories also to have no relations.Such as, even if use memory a and b, also have no relations when namely using frame memory and static sprite memory.Even if or also have no relations when use memory a and c, i.e. use frame memory and dynamic sprite memory.

As mentioned above, if adopt the code translator of this example, then correspond to the movement of image, various types of moving parameter can be used, decoding is carried out to the encoded bit stream 21 effectively carrying out predicting, structure simultaneously owing at random can upgrade with the moment fixed at side mark of encoding with reference to the content of image is corresponding, so more can be adapted to the decoding process of image property.

In this example, even the bit stream that other coded system beyond employing orthogonal transform coding is encoded to predictive error signal, by changing the component of the predictive error signal decoding process beyond memory updating portion of motion compensation portion, same effect can be obtained.

In addition, this example can not only be applicable to fixed dimension block as unit carries out the code translator being unit with the frame of common vision signal of decoding process and the code translator that can be applicable to predicted target area is not limited to fixed dimension block and be unit with arbitrary shape image object (such as: Video Object Plane etc. disclosed in ISO/IEC JTC1/SC29/WG11/N1902).

The possibility of industrial utilization

As mentioned above, if employing the present invention, owing to having the memory block of multiple storage reference image, so can according to the character of dynamic image order, distribute stored in memory, in addition, due to the content of the memory block of more than 1 in multiple memory block can be upgraded at any time, so the time-independent image content such as background images does not carry out the renewal of content for a long time, thus the control of the area of image frequent memory updates content of localized variation etc. can be carried out, the prediction effectively of the history producing dynamic image order can be carried out.

In addition, owing to setting the deformation parameter value scope making each memory become effective district respectively to multiple memory, according to the deformation parameter value of predicted area of image, switch and use memory block in prediction, so can according to the local of dynamic image order/size of on a large scale movement, effectively predict.Tackle the moving parameter that each predicted area of image carries out encoding effectively to encode in the scope in the effective moving parameter value district with reference to memory block simultaneously.

In addition, make each memory become effective deformation method owing to setting respectively multiple memory block, according to the kind of the deformation parameter of predicted area of image, can switch and forecast memory, so can according to the local of dynamic image order/complexity of on a large scale movement, effectively predict.As one man suitably select deformation method with the mobile character of predicted area of image simultaneously, can effectively encode to moving parameter.

Claims (2)

1. predict a dynamic image prediction unit for dynamic image, it is characterized in that having:

Multiple memories of the view data of reference during Storage Estimation; And

Predicted picture generating unit, it has motion compensation portion and memory updating portion, the input of wherein said motion compensation portion represents the parameter of the movement of predicted image-region, according to this parameter, utilize the view data stored in described multiple memory, generation forecast image, described memory updating portion upgrades the view data stored at least one memory in described multiple memory at any time

Described dynamic image prediction unit is used for coding,

The information that described dynamic image prediction unit is relevant according to the usage frequency of described memory during prediction, carries out the code assign ment with reference to memory numbering.

2. predict a method for dynamic image, it is characterized in that comprising following operation:

By the operation of the view data of reference during prediction stored in multiple memory;

Input represents the operation of the parameter of the movement of predicted image-region;

According to described parameter, utilize the operation of the view data generation forecast image stored in described multiple memory; And

Upgrade the operation of the view data stored at least one memory in described multiple memory at any time,

The method of described prediction dynamic image is used for coding,

The information that the method for described prediction dynamic image is relevant according to the usage frequency of described memory during prediction, carries out the code assign ment with reference to memory numbering.