CN101310530A - Multimedia coding techniques for transitional effects - Google Patents
Multimedia coding techniques for transitional effects Download PDFInfo
- Publication number
- CN101310530A CN101310530A CNA2006800429097A CN200680042909A CN101310530A CN 101310530 A CN101310530 A CN 101310530A CN A2006800429097 A CNA2006800429097 A CN A2006800429097A CN 200680042909 A CN200680042909 A CN 200680042909A CN 101310530 A CN101310530 A CN 101310530A
- Authority
- CN
- China
- Prior art keywords
- transition effect
- effect
- information
- multimedia sequence
- sequence
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
This disclosure is directed to techniques for encoding and decoding transitional effects, i.e., visual video effects that are used to transition from a current scene of a multimedia sequence. According to the disclosed techniques, an encoding device detects a transitional effect associated with a multimedia sequence during the encoding of the multimedia sequence, and transmits information as part of an encoded multimedia sequence to identify the transitional effect associated with the encoded multimedia sequence to a decoder. The information may comprise metadata that can be used by the decoder to simulate or re-create the transitional effect. The decoder simulates a transitional effect in response to the information.
Description
Advocate the priority under the 35U.S.C. § 119
Present application for patent is advocated the priority of the 60/721st, No. 417 provisional application case of application on September 27th, 2005, and described provisional application case transfers this assignee and is incorporated herein clearly by reference at this.
Technical field
The present invention relates to the digital multimedia Code And Decode, and more particularly relate to the technology that is used for the transition effect that Code And Decode is associated with multimedia sequence.
Background technology
Set up multiple different video encoding standard and be used for the coded digital multimedia sequence.For example motion picture expert group (MPEG) has developed multiple standards, comprises MPEG-1, MPEG-2 and MPEG-4.Other example comprises H.263 standard and emerging ITU standard H.264 of Union of International Telecommunication (ITU), the also statement in MPEG-4 the 10th part (being entitled as " Advanced Audio Coding ") of described standard.These video encoding standards are generally by supporting the improved efficiency of transmission of multimedia sequence with the compress mode coded data.Compression has reduced the total amount of the data that effective transmission of being used for the multimedia frame need transmit.Video coding is used for the many situations on wired and the wireless medium, comprises video streaming, video camera, visual telephone (VT) and video broadcasting application.
In particular, MPEG-4, ITU H.263 and ITU H.264 the similitude (being called time or frame-to-frame correlation) between the continuous multimedia frame of standard support utilization the video coding technique of interframe compression is provided.Inter-frame compression techniques is converted to the movement representation form by the representation based on pixel with the multimedia frame and utilizes data redundancy on the frame.Usually will use the frame of interframe technology for encoding to be called P (" prediction ") frame or B (" two-way ") frame.Some frames that are commonly referred to I (" interior ") frame are to use space compression (it is for nonanticipating) to encode.In addition, some frames can comprise in-line coding and the combination of the block of coding each other.
Especially when using the interframe compression, transition effect can be challenged in video coding.Transition effect is meant the vision video effect that is used for from the current scene transition of multimedia sequence.Transition effect comprises fade effect, dissolve effect, sliding effect, the effect of breaking, rollover effect, and being used for from the visual effect of current scene transition of various other types.It is second scene and become second scene from first scene that transition effect is usually used in that (for example) take off or fade out by first scene is declined.Yet transition effect also can be used for other situation, for example by the most last scene that makes a sequence decline take off or fade out for the barnyard scape to represent the end of described multimedia sequence.
Summary of the invention
The present invention is directed to the technology that is used for transition effect is carried out Code And Decode.Described technology can be improved the data compression in the encoded multimedia sequence and/or improve the visual quality of transition effect.According to the technology that is disclosed, code device detects the transition effect that is associated with described multimedia sequence during the coding of multimedia sequence, and will as the part of encoded multimedia sequence in order to the message transmission of discerning described transition effect to decoder.
Described decoder receives the information of discerning the transition effect that is associated with described multimedia sequence.The information of discerning described transition effect can comprise can be by described decoder in order to simulation or rebulid the metadata of transition effect.Subsequently in response to described information the described encoded multimedia sequence of decoding to simulate described transition effect.Also describing can be through sending so that simulate some examples of the customizing messages of different transition effects.In some cases, the information of identification transition effect can only trigger the simulation of a certain type of transitional effect, and described transition effect needn't be identical with the transition effect in the original video sequence.In other cases, the information of identification transition effect can be discerned the effect to the particular type of decoder.
In certain embodiments, technology of the present invention can implement with often be called frame rate upwards like the technology type of conversion (FRUC) interpositioning so that allow the auxiliary transition effect simulation of the encoder at decoder place.In the case, for transition effect, can use the technology of similar FRUC substitute the frame be used for transition effect routine based on prediction or based on the coding in space, to improve the data compression in the encoded multimedia sequence and/or to improve the visual quality of transition effect.
In certain embodiments, the invention provides a kind of method, it comprises: receive the information as the part of encoded multimedia sequence, the transition effect that wherein said information Recognition is associated with described multimedia sequence; With the described encoded multimedia sequence of decoding, comprise in response to described information and simulate described transition effect.
In certain embodiments, the invention provides a kind of method, it comprises: detect the transition effect that is associated with described multimedia sequence during the coding of multimedia sequence; With produce as the part of encoded multimedia sequence in order to discern the information of described transition effect.
In certain embodiments, the invention provides a kind of equipment, described equipment comprises: receiver, and it is configured to receive the information as the part of encoded multimedia sequence, the transition effect that wherein said information Recognition is associated with described multimedia sequence; And decoder, it simulates described transition effect in response to described information.
In certain embodiments, the invention provides a kind of equipment, described equipment comprises: detector, and it detects the transition effect that is associated with described multimedia sequence during the coding of multimedia sequence; And encoder, its produce as the part of encoded multimedia sequence in order to discern the information of described transition effect.
In certain embodiments, the invention provides a kind of processor, it is configured to: receive the information as the part of encoded multimedia sequence, the transition effect that wherein said information Recognition is associated with described multimedia sequence; And the described encoded multimedia sequence of decoding comprises in response to described information and simulates described transition effect.
In certain embodiments, the invention provides a kind of processor, it is configured to: detect the transition effect that is associated with described multimedia sequence during the coding of multimedia sequence; And produce information as the described transition effect that is associated with described encoded multimedia sequence in order to identification of the part of encoded multimedia sequence to decoder.
In certain embodiments, the invention provides a kind of equipment, described equipment comprises: receiving system, and it is used to receive the information as the part of encoded multimedia sequence, the transition effect that wherein said information Recognition is associated with described multimedia sequence; And decoding device, it simulates the transition effect in the multimedia sequence in response to the information of the described transition effect of identification.
In certain embodiments, the invention provides a kind of equipment, described equipment comprises: checkout gear, and it is used for detecting the transition effect that is associated with described multimedia sequence during the coding of multimedia sequence; And code device, it produces the information of the transition effect that is associated with described multimedia sequence in order to identification.
Technology described herein may be implemented in hardware, software, firmware or its combination in any.If be implemented in the software, so described software can be carried out in the processor of digital signal processor (DSP) or other type.But the software initial storage of carrying out described technology and is written into and carries out in processor or other machine to allow as described herein transition effect to be carried out video coding or decoding in machine-readable medium (for example computer-readable media).
Therefore, the present invention also expects a kind of machine-readable medium that comprises instruction, described instruction is in case execution just impels a machine: receive the information as the part of encoded multimedia sequence, the transition effect that wherein said information Recognition is associated with described multimedia sequence; And the described encoded multimedia sequence of decoding comprises in response to described information and simulates described transition effect.
Equally, the present invention expects a kind of machine-readable medium that comprises instruction, and described instruction is in case carry out and just impel a machine: detect the transition effect that is associated with described multimedia sequence during the coding of multimedia sequence; And produce as the part of encoded multimedia sequence in order to discern the information of described transition effect.
State the details of one or more embodiment in the accompanying drawings and the description below.From describe content and accompanying drawing and accessory rights claim, will understand other features, objects and advantages.
Description of drawings
Fig. 1 is that explanation can be implemented the video coding of transition effect coding and decoding technology and the block diagram of decode system according to the present invention.
Fig. 2 is the flow chart of explanation according to transition effect coding techniques of the present invention.
Fig. 3 is the flow chart of explanation according to transition effect decoding technique of the present invention.
Fig. 4 is the concept map of the exemplary frame of the explanation different transition effects of can technology according to the present invention encoding to Figure 16.
Figure 17 is that explanation can be implemented the exemplary video coding of transition effect coding and decoding technology and another block diagram of decode system according to the present invention.
Embodiment
The present invention is directed to the technology that is used for transition effect is carried out Code And Decode.Transition effect is meant the vision video effect that is used for from the current scene transition of multimedia sequence.The example of transition effect comprises fade effect, smears and wipe effect, dissolve effect, sliding effect, the effect of breaking and rollover effect, only enumerates this several examples.It is second scene and become second scene from first scene that transition effect is usually used in that (for example) take off or fade out by first scene is declined.Yet transition effect also can be used for other situation, for example by the most last scene that makes sequence decline take off or fade out for the barnyard scape to represent the end of described multimedia sequence.
According to the technology that is disclosed, code device detects the transition effect that is associated with multimedia sequence during the multimedia sequence coding.This transition effect detects and can be used for detecting the reliable method of the transition effect of multimedia sequence based on conventional shot boundary (shot boundary) detection algorithm or any other.Perhaps, but the signal or the order of existence of indication transition effect in the code device detection resources bit stream.Thereby described code device produces information subsequently with the unconventional mode transition effect of encoding.
Decoder can use interpositioning (for example and routine be used between two encoded frames inserting one or more frames to be used for the upwards technology similar techniques of conversion (FRUC) of frame rate) simulation transition effect.Code device transmission as the part of encoded multimedia sequence in order to discern the transitional effect information of the transition effect in the described encoded multimedia sequence.In certain embodiments, can comprise can be by decoder in order to produce the metadata of transition effect for transitional effect information.In the case, but the transitional effect information aided decoder produces the transition effect be similar to the transition effect in the original video source bitstream.Perhaps, transitional effect information can only send the signal that the indication transition effect exists.In the case, decoder can produce transition effect independently in response to transitional effect information, and described transition effect needn't be identical with the original transition effect in the original video sequence.
Fig. 1 is that explanation can be implemented the video coding of transition effect coding and decoding technology and the block diagram of decode system 10 according to the present invention.As shown in the figure, system 10 comprises video encoder apparatus 12 and video decoder devices 14.Encoded multimedia sequence can be transferred to video decoder devices 14 in communication channel 15 from video encoder apparatus 12.For this reason, video encoder apparatus 12 and video decoder devices 14 comprise respectively reflector 23 with receiver 27 so that this communicates by letter, described communication can be wired or wireless communication.
Video encoder apparatus 12 (for example) can be formed for one or more video channels are broadcast to the part of the broadcasting network assembly of wireless subscriber devices.In the case, video encoder apparatus 12 can be with encoded transfer of data to some Video Decoders (that is many subscribers of broadcast service).Yet, for clarity, the single video decoder devices 14 of explanation in Fig. 1.Video encoder apparatus 12 and video decoder devices 14 can be embodied as one or more processors, digital signal processor, application-specific integrated circuit (ASIC) (ASIC), field programmable gate array (FPGA), discrete logic, software, hardware, firmware, or its combination in any.The illustrated assembly of device 12 and device 14 can be included in one or more encoders or the decoder, and wherein any one can be integrated into the part of encoder/decoder (CODEC).Yet reflector 23 can be formed on the assembly or chip different with encoder, decoder or CODEC with receiver 27.
As mentioned above, video encoder apparatus 12 can be formed for broadcasting the part of the broadcasting network assembly of one or more channels of video data.The example of this class component comprises wireless base station or any infrastructure node that is used to broadcast encoded video data.On the other hand, video decoder devices 14 can comprise the user's set that receives encoded video.For instance, video decoder devices 14 can be embodied as the part of Digital Television, radio communication device, portable digital-assistant (PDA), laptop computer or desktop computer, digital music and video-unit (for example device of selling with trade mark " iPod ") or the radio telephone radio telephone of honeycomb fashion, satellite or land (for example based on).Assembly in the illustrated system 10 is the assembly that is suitable for implementing technology described herein most, but encoder apparatus 12 and decoder device 14 can comprise many other assemblies when needed.In addition, technology of the present invention is not necessarily limited in the system of the system of being similar to 10 or the broadcast system and uses.Described technology can be applicable to transition effect wherein and is used in any video coding environment of multimedia sequence of to be encoded and/or decoding.
As shown in Figure 1, video encoder apparatus 12 (for example) receives multimedia sequence 5 from memory location.Described memory (not shown in Fig. 1 for brevity) can be the part of encoder apparatus 12, perhaps can be the external memory storage that multimedia sequence is provided to video encoder apparatus 12.Multimedia sequence 5 can comprise to be encoded and as broadcasting and live real-time video or the video and the tonic train of transmission, perhaps can comprise to be encoded and as broadcasting or by the video that writes down in advance and store or the video and the tonic train of required transmission.
Under any circumstance, video encoder apparatus 12 helps the coding of multimedia sequence 5, and implements technology of the present invention and improve this coding when meeting with transition effect in multimedia sequence 5.Video encoder apparatus 12 comprises transitional effect detector 20, and it carries out the detection of the transition effect that is associated with sequence 5.For instance, transitional effect detector 20 can be to multimedia sequence 5 samplings, seek one or more marks of transition effect, for example produce fade effect, smear and wipe being used for of effect, dissolve effect, sliding effect, the effect of breaking, rollover effect or various other types from a framing of the visual effect of current scene transition.As an example, transitional effect detector 20 can be implemented one or more so-called shot boundary detection algorithms, or any other is used to detect the reliable method of the transition effect that is associated with multimedia sequence.In this way, the existence of transition effect in the transitional effect detector 20 identification multimedia sequences 5.
Transition effect can be challenged in video coding.In particular, transition effect can not showed the highly effective relative scene motion based on the video coding of predicting of permission usually.In addition, transition effect is usually used in the scene change place, and is insensitive to the higher order signal noise in the described scene change observer's of place sensation.When using conventional encoder use to encode transition effect based on the technology of prediction with standard mode, compression level may be very limited, and video quality may be relatively poor.In addition, because observer's sensation usually not as to the higher order signal noise sensitivity in the entire motion part of multimedia sequence, from quality viewpoint may be unnecessary so the poor efficiency standard pattern of this type of transition effect is encoded to the higher order signal noise in the transition effect.In other words, when scene changed, the user can stand usually and can not feel that than high noise levels the quality of any subjectivity descends.
For these and other reason, technology of the present invention provides the different coding method with respect to the transition effect of other non-transition portion of multimedia sequence.Specifically, video encoder apparatus 12 comprises standard code module 16 and transition effect coding module 18, and it can be formed in an encoder or a plurality of encoder.The coding techniques based on prediction of standard code module 16 application standards, for example estimation and motion compensation is with the frame of encoded multimedia sequence 5.Standard code module 16 also can be used non-motion encoded technology, and the space that for example is used for some frames is estimated and intra-prediction.According to the technology based on prediction of standard, standard code module 16 also can comprise the various unit that are used for entropy coding, scanning, quantification, conversion and possible deblocking (deblock) filtering.These unit application are in normal (non-transition) frame, with the coding based on prediction of the standard that promotes this type of frame.Yet for any frame that is identified as transition effect, transition effect coding module 18 is carried out different non-standard coding techniquess.
In particular, for the frame that comprises transition effect, transitional effect encoder 18 produces can be by the information of video decoder devices 14 in order to the simulation transition effect.Decoder device 14 is by the frame of reconstruction transition effect, but not this type of frame of decoding individually, and use described information to simulate described effect.Described information can comprise the metadata that can be used in the mode that is similar to the FRUC technology by decoder device 14.For instance, metadata can be discerned the effect of a type, so that can simulate the effect of described type at video decoder devices 14 places at those particular frames that comprise described transition effect.Under better simply situation, metadata can only be discerned the existence of transition effect, and need not consider the type of effect, and in the case, video decoder devices 14 can be simulated not necessarily identical with original effect transition effect.Yet than under the complicated situation, metadata can comprise in order to the information in recognition effect cycle, possible one or more geometric maps that are associated with frame transition effect, or help to simulate at video decoder devices 14 places other data of described effect.Yet in most of the cases, can or produce in decoder-side storage and be used to simulate the geometric maps of transition effect, and based on the indication of type of effect and cycle of being associated with effect are used described geometric maps.According to the technology that is disclosed, can avoid needing to encode, send and decoding being used for the complete predictive frame of transition effect, thus the processing overhead at saving encoder place, and the bandwidth on the saving radio communication channel.
Transition effect decoder module 24 is handled the decoding of transition effect based on the information that sends from video encoder apparatus 12.Use the standard technique required information much less of this type of effect of encoding through sending the comparable script of information with the decoding transition effect.In fact, video encoder apparatus 12 produces information and the possible out of Memory about effect, for example cycle that is associated with effect, type of effect and the beginning that is associated with effect and the end scene of identification transition effect.Decoding device 14 receives this information via receiver 27, and transition effect decoder module 24 is simulated described effect in video decoder devices 14 in response to described information.For instance, given beginning and end scene, to the indication of type of effect and the cycle of effect, can rebulid the simulation of described effect by transition effect decoder module 24, and required coded message more widely when not needing originally to use standard technique to encode this type of effect.
Fig. 2 is the flow chart of explanation according to coding techniques of the present invention.As shown in Figure 2, video encoder apparatus 12 receives multimedia sequence 5 (31) to be encoded.Transitional effect detector 20 (for example) detects any of multimedia sequence 5 and whether comprises transition effect (32) to certain portions by implementing shot boundary detection algorithms or any reliable method that other is used to detect the transition effect that is associated with multimedia sequence.If transitional effect detector 20 is unidentified to any transition effect (32 negative branch), call standard code module 16 so with coding whole multimedia sequence 5.In the case, the motion encoded and non-space encoding based on prediction of 16 pairs of multimedia sequence 5 operative norms of standard code module based on prediction.
Yet,, call transition effect coding module 18 so to be used for the transition effect part of multimedia sequence 5 if transitional effect detector 20 recognizes the transition effect that is associated with multimedia sequence 5.In the case, the transitional effect information (34) that transition effect coding module 18 produces at transition effect, it can be used with the simulation transition effect by video decoder devices 14.This information can be to help the information of interpolation or to the existence of transition effect, position and indication regularly, perhaps may only be the flag that indication need produce transition effect at the decoder place.Under any circumstance, standard code module 16 is still to the non-space encoding (35) based on prediction based on the motion encoded and standard of prediction of the non-transition portion operative norm of multimedia sequence 5.
Fig. 3 is the flow chart of explanation according to decoding technique of the present invention.As shown in Figure 3, the receiver 27 (for example) of video decoder devices 14 is via the reflector 23 reception encoded multimedia sequences (51) of communication channel 15 from code device 12.If the multimedia sequence that receives does not comprise the transitional effect information of the existence of discerning transition effect, call standard decoder module 22 so to be used for the decoding of whole multimedia sequence.In the case, the motion decoding and the non-space decoding based on prediction of standard decoder module 22 operative norms based on prediction.
Yet,, call transition effect decoder module 24 is received multimedia sequence to be used for transition effect part so if the multimedia sequence that receives comprises the transitional effect information of the existence of discerning transition effect.In the case, transition effect decoder module 24 is simulated transition effect (53) in response to transitional effect information.Equally, this simulation of being undertaken by transition effect decoder module 24 can be similar to interpositioning or the auxiliary FRUC technology of encoder that is used to support the FRUC technology, wherein produces frame to simulate required effect.For the non-transition portion of reception multimedia sequence, standard decoder module 22 still operative norm based on prediction and non-based on the decoding of predicting (55).
Fig. 4 is that explanation can be according to the concept map of other frame one by one in indivedual transition effects of technology for encoding of the present invention to Figure 16.Fig. 4 will be used to explain to different exemplary situation illustrated in fig. 16 and can be used for encoding transition effect and allow the dissimilar transitional effect information or the metadata of the indivedual effects of good simulation at the decoder place.Once more, under some simple scenarios, metadata can only be discerned the existence of transition effect, and need not consider the type of effect, and in the case, video decoder devices 14 can be simulated needn't the transition effect identical with original effect.Yet than under the complicated situation, metadata can comprise in order to the beginning of recognition effect type, effect cycle, effect and the information of end frame, and other data that may aid in video decoder devices 14 place's simulate effects.In these cases, information can be used to assist the simulation of transition effect by decoder.
Fig. 4 is the concept map of a frame 62 of explanation fade effect (taking off for intersecting to decline in the case).In the case, scenario A declines to take off in cycle P and is scenario B.Fade effect can comprise " (fade-in) fades in ", and scenario B is faded in from the barnyard scape in the case; " (fade-out) fades out ", scenario A is faded out and is the barnyard scape in the case.Fig. 4 illustrates " intersection declines and takes off ", and wherein scenario A declines to take off and is scenario B.Technical, these fade in, fade out and intersect when declining situation about taking off in processing, may not there are differences.Can be considered as taking off identically with intersecting to decline with fading out with fading in, but only scenario A or B are considered as being used to the blank scene of fading in or fading out.
Under any circumstance, for the fade effect that illustrated intersection in the frame 62 of for example Fig. 4 declines and takes off, the weighted bi-directional motion compensation can be used with zero motion vector.In the case, use information with the coding transition effect effect can be identified as fade effect by decoding device 14 and define cycle of described effect.A copy of scenario A and a copy of scenario B also can be included in the transitional effect information.Yet, can be omitted at least some information of the intervention frame of expanding between scenario A and the B.
The given relatively limited information that receives from video encoder apparatus 12, video decoder devices 14 can be simulated fade effect.Specifically, transition effect decoder module 24 can based on the copy of the copy of scenario A, scenario B with take off the framing that the related cycle of transitional face rebuilds transition from scenario A to declining of scene B.In the case, transition effect decoder module 24 can use the motion vector of null value to carry out the weighted bi-directional reconstruction of multimedia frame.The weighting that is used for scenario A and scenario B can relative to each other change producing one group of transition frames in the cycle, and can be stored in during decode procedure in one or more look-up tables (LUT) by 24 accesses of transition effect decoder module.The cycle of fade effect can be defined some indexes, and described index can be applicable to LUT is used for each successive frame of effect with selection weighting.Needing to send with the information that promotes this reconstruction is the copy of indication, scenario A and the scenario B of fade effect type, and the cycle yet importantly.The reconstruction of frame can be carried out separately by transition effect decoder module 24 in the described cycle.In this way, transmitting the required information of transition effect in video coding mechanism can significantly reduce.
Fig. 5 is the concept map of a frame 64 of explanation fade effect (being dissolve effect in the case).In dissolve effect, scenario A is faded out in cycle P and is scenario B.Dissolve effect also can comprise " being fade-in (dissolve-in) ", and scenario B is fade-in from the barnyard scape in the case; Or " gradually going out (dissolve-out) ", scenario A gradually goes out to be the barnyard scape in the case.Fig. 5 illustrates " intersection is faded out ", and wherein scenario A is faded out and is scenario B.Can fade out identically with gradually going out to be considered as with intersecting with being fade-in, but only scenario A or B are considered as being used to the blank scene that is fade-in or gradually goes out.
Under any circumstance, for illustrated dissolve effect in the frame 64 of for example Fig. 5, a pair of selection is possible.In one case, define geometric maps in advance to define bidirectional-movement at transition effect decoder module 24 places with zero motion vector.For each successive frame of transition effect, described mapping defines which pixel is taken from scenario A and which pixel is taken from scenario B.The number of employed mapping can be defined by the cycle that is associated with transition effect in fading out.The predefined mapping that is used for transition effect can be stored in can be by one or more look-up tables (LUT) of transition effect decoder module 24 accesses.The effect cycle can be defined index, and described index can be applicable to LUT is used for each successive frame of effect with selection predefined mapping.
Yet in another case, geometric maps can random fashion produce at video decoder devices 14 places.Be similar to other dissolve example, in the case, transitional effect information can be discerned scenario A and B, dissolve effect and cycle.Transition effect decoder module 24 can produce one group based on the random number generation subsequently and fade out mapping.This situation causes dissolve effect, but can select the specific pixel of fading out at video decoder devices 14 places at random.Therefore, any dissolve effect subsequently will differ from one another with random fashion.
Fig. 6 is respectively the concept map of explanation box effect (box effect), sliding effect and J curve effectJ to Fig. 8.In the box effect shown in the frame 65 of Fig. 6, scenario B 68 is presented on from scenario A 66 amplifies the frame.In the sliding effect shown in the frame 75 of Fig. 7, scenario B 78 presents with sliding type 76 from scenario A.In the J curve effectJ shown in the frame 85 of Fig. 7, scenario B 88 is presented in the sets of curves zone that scene is amplified from scenario A 86.Be similar to the effect of Fig. 4 and Fig. 5, Fig. 6 also can use with the blank scene that is used for scenario A or is used for scenario B to the effect of Fig. 8.Under any circumstance, the information of this effect that is used to encode can be identified as box effect with the transition effect of described type, and defines the cycle.The copy of scenario A and scenario B also is used for producing simulation.
Specifically, transition effect decoder module 24 can come the transition effect of simulation drawing 6 to Fig. 8 according to each predefined mapping of the transition effect that is used for these types.Given scenario A, scenario B, transition effect type and the cycle that is associated with effect, transition effect decoder module 24 can select to be stored in one group of predefined mapping among one or more LUT.Selected mapping can be defined the geometric figure of each successive frame on the transitional sequence so that the simulation transition effect.In other words, transition effect decoder module 24 can use the mapping that is used for each frame to carry out the bi-directional reconstruction of each successive frame, and described mapping defines which pixel (or block of pixels) is taken from scenario A and which pixel (or block of pixels) is taken from scenario B.For each individual pixel or block, the motion vector that is used for each pixel (or block of pixels) will be zero.LUT can be through defining being used for the transition effect of each type in advance, and can select from LUT based on the transient period, if for example the cycle shorter, skip one or more mappings among the LUT so.
In other example, any one the geometric maps that is used for effect described in the present invention can produce and send to decoding device 14 at code device 12 places.In other words, be used for the geometric maps of indivedual effects by generation, or, can simulate described effect at decoding device 14 places by being applied in the generation of code device 12 places and sending to the mapping of decoding device 14 as the part of transitional effect information.Produce the complexity that geometric maps can reduce decoding device 14 by code device 12, decoding device 14 has the ability of less complicated calculations usually than code device 12, especially true in broadcast scenarios.On the other hand, produce geometric maps by decoding device 14 and realize distinct advantage, because this has reduced the amount of information of the required transmission of efficient coding transition effect.In the case, can allow between code device 12 and the decoding device 14 minimizing of the information of transmission at decoder place simulate effect, with the efficient coding transition effect, this especially may be exceedingly useful with wireless mode transmission information the time.
Fig. 9 is another concept map that (door opening) effect of opening the door is described.In the frame 95 of Fig. 9, present scenario B 98 when removing scenario A 96 opening the door.Be similar to other effect, the open door effect of Fig. 9 also can be used with the blank scene that is used for scenario A or is used for scenario B.Under any circumstance, the information of this effect that is used to encode can be identified as open door effect with the transition effect of described type, and defines the cycle.The copy of scenario A and scenario B also is used for producing simulation.
In the case, the frame of transitional sequence is segmented into forward prediction part and back forecast part.Described forward prediction part is decoded in the mode that is different from described back forecast part.One group of geometric maps can be defined in the transient period particular through the scene of forward prediction and back forecast.The motion vector that is associated with the scenario B that presents is zero, and the motion vector that is associated with opening the door of scenario A can and be calibrated to produce this effect through smoothing.Therefore, scenario A 96 opens the door through calibration and smoothing so that complete scenario A is included in and opens the door, also through regulating to produce open door effect.Be similar to other example, the geometric maps that calibration and smoothing factor and any and described effect are associated can be stored in the Video Decoder 12 (for example) in advance as one or more LUT in the transition effect decoder module 24.
Therefore, the smoothing of the motion vector that is associated with scenario A and calibration can be carried out by transition effect decoding unit 24 at video decoder devices 14 places, thereby have eliminated the needs that send complex motion information in encoded data.The transition effect coding module 18 of code device 12 is only discerned open door effect and cycle, and sends the copy of scenario A and scenario B.The transition effect decoder module 24 of decoding device 14 can use the zone that presents of the scenario B 98 with zero motion vector to simulate this effect with the differential decoding that opens the door of the scenario A 96 of the smoothing of using motion vector and calibration subsequently.Scenario A and B can be through space compression to be used to increase efficient.
The different effect of breaking that Figure 10 also can technology according to the present invention simulates to Figure 12 explanation.In the frame 105 of Figure 10, scenario A 106 is split into scenario B 108.In the frame 115 of Figure 11, scenario A 116 break for some exposing barnyard scape 119, and scenario B 118 occurs with failure mode, finally rebuilds the B scene.In the frame 125 of Figure 12, scenario A 126 break for some exposing barnyard scape 129, and scenario B 128 is rendered as and assembles some that form the B scene subsequently.
Figure 10 also can be simulated based on the transitional effect information of the content of recognition effect type, cycle and beginning scenario A and end scene B by decoder device 14 to the example of Figure 12.In the case, frame is segmented into forward direction and back forecast part, in part, carries out calibration and rotation in forward direction and back to realize breaking individually transition effect.
Figure 13 shows the frame 135 of exemplary rollover effect.In the case, scenario A 136 its size when rotated reduces, thereby final upset is scenario B (not shown), and scenario B is amplified with the successive frame of transition, till it comprises full scene.Background 139 is shown as empty in this example.Be similar to the example of Figure 10 to Figure 12, in order to simulate the effect of Figure 13, frame is segmented into forward direction and back forecast part.Yet in the case, in part, carry out calibration and rotation in forward direction and back to realize breaking individually transition effect.Described effect can be simulated based on the transitional effect information of the content of recognition effect type, cycle and beginning scenario A and end scene B by decoder device 14.The calibration of effect, rotation and geometry can be stored among (for example) one or more LUT in advance, and can select to be used for each continuous scene of described effect based on the cycle of discerning at described effect.
Figure 14 illustrates the frame 145 of movie film effect, and wherein scenario A 146 shifts out when scenario B 148 moves into.Motion is transition.In the case, motion vector defines in the homogeneous mode, and different prediction direction is used in the both sides of boundary line.Be similar to other transition effect, can simulate this effect based on the information of the content that defines transition effect type, cycle and scenario A and B at video decoder devices 14 places.Therefore, can avoid Code And Decode, and transitional effect decoder module 24 can be simulated this effect based on the information of lacking than needed information originally based on normative forecast at each frame of the transition portion of sequence.
Figure 15 explanation comprises the frame 155 of the combination of two kinds of transition effects.In the case, described effect is the combination of J curve effectJ shown in Figure 8 and cross-fade effect shown in Figure 4.The effect of supposing Figure 15 is the combination of these two kinds of different-effects, and transitional effect decoder module 24 can be simulated this effect by the technology that combination is used for J curve effectJ and cross-fade effect.In the case, carry out the weighted bi-directional reconstruction of the use motion vectors of zero values of the multimedia frame that is associated with transition effect, and define prediction weights according to the predefined mapping of J curve effectJ by transitional effect decoder module 24.
Some complex effects, illustrated ripple effect in the frame 165 of Figure 16 for example may be too complicated and can not be identified as the combination of simple effects.In the case, can introduce the physics warpage, intersect decline take off with other random motion to simulate described effect at video decoder devices 14 places.At random and/or nonlinear distortion also can be used for simulating this type of complex transitional effects.Under these complex situations, be alternative in each details of each block of transmission, can send statistics, for example fade out the ratio of bright pixels in the ratio of the block that is updated to new scene and the white noise with intersection.Given this information, but the effect of the similar ripple of Simulation of Complex.
Yet especially when content can be revised, may need in some cases, with better simply transition effect (for example intersect to decline and take off) replace complex effect (for example ripple effect).In the case, encoder apparatus 12 can only be discerned the existence of transition, and decoder device 14 can simulate needn't the transition effect identical with the transition effect in the original series.In the case, the observer will observe transition, and can significantly reduce complexity by the Code And Decode of avoiding the required high cost of transfer complex transition effect.
When at the transition effect Simulation execution weighted motion compensated at decoder place, can in the cycle of transition effect, on frame, carry out smoothing to weighting.When the motion vector that is used for given effect is carried out smoothing, the adjacent part that this smoothing is not necessarily limited to be close to, but can comprise just whole section or the smoothing of part of smoothedization.Smoothing also is not limited to transient motion, and can comprise (for example) calibration and rotation.For given transition effect, smoothing, weighting, filtering etc. all can be defined in advance at the decoder place.
Figure 17 is that explanation can be implemented the exemplary video coding of transition effect coding and decoding technology and another block diagram of decode system according to the present invention.As shown in the figure, system 170 comprises code device 172 and decoding device 174.Code device 172 can comprise encoder, and decoding device can comprise decoder.Code device 172 (for example) receives multimedia sequence 171 from memory location.Described memory (not shown in Figure 17 for brevity) can be the part of code device 172, perhaps can be the external memory storage that multimedia sequence is provided to code device 172.
Decoding device 174 comprises standard decoding device 182 and transition effect decoding device 184, and described device can be described as first and second decoding devices.Standard decoding device 182 and transition effect code device 184 can comprise independent decoder or may be implemented in the common decoder.Standard decoding device 182 operative norm decoding techniques, for example inner decoding of motion compensation decoding and any intra-coded frame.Yet transition effect decoding device 184 is carried out non-standard technology, simulates any transition effect based on the metadata of transmitting at this type of effect described herein in essence.
Having described can be through sending the some examples with the information that helps to simulate different transition effects.Generally, technology of the present invention can be implemented and interpositioning like the technology type that is used for conversion (FRUC) technology on some frame rate, simulates so that allow to carry out the auxiliary transition effect of encoder at the decoder place.In the case,, can use the technology of similar FRUC to substitute conventional coding, so that improve the data compression in the encoded multimedia sequence and/or improve the visual quality of transition effect based on prediction for transition effect.To begin and end scene (with may be in the middle of some scenes) send to decoder, and decoder inserts other frame to produce transition effect.
Under any circumstance, technology described herein may be implemented in hardware, software, firmware or its arbitrary combination.If be implemented in the software, so described technology can partly realize by the computer-readable media that comprises the program code that contains instruction (or other machine-readable medium), and described program code is carried out one or more in the above-described technology when being performed.In the case, described computer-readable media can comprise for example Synchronous Dynamic Random Access Memory random-access memory (ram)s such as (SDRAM), read-only memory (ROM), nonvolatile RAM (NVRAM), Electrically Erasable Read Only Memory (EEPROM), flash memory, magnetic or optical data storage media etc.
Described instruction can be carried out by one or more processors or other machine, and described other machine for example is the integrated or discrete logic of one or more digital signal processors (DSP), general purpose microprocessor, application-specific integrated circuit (ASIC) (ASIC), field programmable logic array (FPGA) or other equivalence.The machine of carrying out described instruction can comprise an equipment usually.In certain embodiments, functional being provided in described herein is configured for use in the dedicated software modules or hardware cell of Code And Decode, or is incorporated in the video encoder-decoder (CODEC) of combination.
Various embodiment have been described.Yet, under the situation of the scope that does not depart from appended claims, can make various modifications to described technology.For instance, affine motion can be used for transition effect according to some types and defines mapping and pixel value in the simulation.Affine motion allows six-freedom degree, and codified translation, compression or amplification.Accelerated motion also can be used for defining simulation.In addition, also can be but not the look-up table in encoder or the decoder (LUT) uses technology of the present invention by applicable equations.These and other embodiment within the scope of the appended claims.
Claims (41)
1. method, it comprises:
Reception is as the information of the part of encoded multimedia sequence, the transition effect that wherein said information Recognition is associated with described multimedia sequence; And
The described encoded multimedia sequence of decoding comprises in response to described information and simulates described transition effect.
2. method according to claim 1, the transition effect of wherein said information Recognition one type, described method further comprises based on the transition effect of described type simulates described transition effect.
3. method according to claim 1, the transition effect of wherein said information Recognition one type and the cycle that is associated with described transition effect, and wherein simulate the transition effect that described transition effect is included in the described type of simulation during the described cycle.
4. method according to claim 1, the transition effect of wherein said information Recognition one type, the cycle that is associated with described transition effect and the beginning and the end scene that are associated with described transition effect, and wherein simulate described transition effect and comprise and produce one or more geometric maps.
5. method according to claim 4 wherein produces described one or more geometric maps from one or more look-up tables.
6. method according to claim 1, wherein said information is identified as fade effect with described transition effect, and wherein simulates described transition effect and comprise the weighted bi-directional reconstruction that the motion vector that uses null value is carried out the multimedia frame that is associated with described fade effect.
7. method according to claim 1, wherein said information is identified as dissolve effect with described transition effect, and wherein simulates described transition effect and comprise and produce described dissolve effect at random.
8. method according to claim 1, wherein said information is identified as second scene wherein with described transition effect and is presented in effect on first scene, and simulates described transition effect and comprise the motion vector that uses null value and be presented in the bi-directional reconstruction that the multimedia frame that is associated with described transition effect is carried out in predefined mapping on described first scene according to wherein said second scene.
9. method according to claim 1, wherein simulate described transition effect and comprise at least one frame in the transitional sequence is segmented into forward prediction part and back forecast part, and by different way described forward prediction is partly partly decoded with described back forecast.
10. method according to claim 1 is wherein simulated described transition effect and is comprised the weighted bi-directional reconstruction of carrying out the multimedia frame that is associated with described transition effect according to the predefined mapping of described transition effect.
11. a method, it comprises:
During the coding of multimedia sequence, detect the transition effect that is associated with described multimedia sequence; And the information as the part of encoded multimedia sequence of producing is to discern described transition effect.
12. method according to claim 11, wherein said information comprises the identification of the transition effect of a type.
13. method according to claim 11, wherein said information comprise the transition effect of a type and the identification in cycle of being associated with described transition effect.
14. method according to claim 11, wherein said information comprise the transition effect of a type and the identification of one or more geometric maps of being associated with described transition effect.
15. method according to claim 11, it further comprises described information is transferred to decoder as the part of described encoded multimedia sequence.
16. method according to claim 11, wherein said information comprise the transition effect of a type, the cycle that is associated with described transition effect and the beginning that is associated with described transition effect and the identification of end scene.
17. an equipment, it comprises:
Receiver, it receives the information as the part of encoded multimedia sequence, the transition effect that wherein said information Recognition is associated with described multimedia sequence; And
Decoder, it simulates described transition effect in response to described information.
18. equipment according to claim 17, wherein said decoder comprise the standard decoder module of the non-transition portion of the transition effect decoder module of simulating described transition effect and the described encoded multimedia sequence of decoding.
19. equipment according to claim 17, the transition effect of wherein said information Recognition one type, the cycle that is associated with described transition effect and the beginning and the end scene that are associated with described transition effect, and wherein said decoder is simulated the transition effect of described type through discerning during the described cycle.
20. equipment according to claim 17, wherein said decoder produces one or more geometric maps that are associated with described transition effect, and uses described one or more geometric maps to simulate described transition effect.
21. equipment according to claim 20, wherein said decoder produce described one or more geometric maps from one or more look-up tables.
22. equipment according to claim 17, wherein said decoder is segmented into forward prediction part and back forecast part by the frame with transitional sequence and described forward prediction part is partly decoded with described back forecast simulate described transition effect by different way.
23. an equipment, it comprises:
Detector, it detects the transition effect that is associated with described multimedia sequence during the coding of multimedia sequence;
And
Encoder, it produces information as the part of encoded video sequence to discern described transition effect.
24. equipment according to claim 23, the transition effect of wherein said information Recognition one type, the cycle that is associated with described transition effect and the beginning and the end scene that are associated with described transition effect.
25. equipment according to claim 23, wherein said encoder comprises: first coding module, and it produces in order to discern the described information of described transition effect; With second coding module, it uses the encode non-transition portion of described multimedia sequence of predictive coding.
26. a machine-readable medium that comprises instruction, described instruction is in case execution just impels a machine:
Reception is as the information of the part of encoded multimedia sequence, the transition effect that wherein said information Recognition is associated with described multimedia sequence; And
The described encoded multimedia sequence of decoding comprises in response to described information and simulates described transition effect.
27. machine-readable medium according to claim 26, the transition effect of wherein said information Recognition one type, the cycle that is associated with described transition effect and the beginning and the end scene that are associated with described transition effect, and wherein simulate described transition effect and comprise and produce one or more geometric maps.
28. machine-readable medium according to claim 26, wherein said instruction is segmented into forward prediction part and back forecast part by the frame with transitional sequence and described forward prediction part is partly decoded with described back forecast simulate described transition effect by different way.
29. a machine-readable medium that comprises instruction, described instruction is in case execution just impels a machine:
During the coding of multimedia sequence, detect the transition effect that is associated with described multimedia sequence; And
Generation as the information of the part of encoded multimedia sequence to discern described transition effect.
30. a processor, it is configured to:
Reception is as the information of the part of encoded multimedia sequence, the transition effect that wherein said information Recognition is associated with described multimedia sequence; And
The described encoded multimedia sequence of decoding comprises in response to described information and simulates described transition effect.
31. a processor, it is configured to:
During the coding of multimedia sequence, detect the transition effect that is associated with described multimedia sequence; And
Produce information to decoder as the described transition effect that is associated with described encoded multimedia sequence in order to identification of the part of encoded multimedia sequence.
32. processor according to claim 31, wherein said information comprise the transition effect of a type, the cycle that is associated with described transition effect and the beginning that is associated with described transition effect and the identification of end scene.
33. processor according to claim 31, wherein said information comprise by decoder in order to simulate one or more geometric maps of described transition effect.
34. an equipment, it comprises:
Receiving system, it is used to receive the information as the part of encoded multimedia sequence, the transition effect that wherein said information Recognition is associated with described multimedia sequence; And
Decoding device, it simulates the transition effect in the multimedia sequence in response to the described information of the described transition effect of identification.
35. equipment according to claim 34, wherein said decoding device comprises the transition effect decoding device, and described equipment further comprises the standard decoding device, the decode non-transition portion of described multimedia sequence of described standard decoding device.
36. equipment according to claim 34, the transition effect of wherein said information Recognition one type, the cycle that is associated with described transition effect and the beginning and the end scene that are associated with described transition effect, and wherein said decoding device uses described information to simulate the transition effect of described type through discerning in the described cycle.
37. equipment according to claim 34, wherein said decoding device produces one or more geometric maps that are associated with described transition effect, and uses described one or more geometric maps to simulate described transition effect.
38. equipment according to claim 34, wherein said decoding device is segmented into forward prediction part and back forecast part by the frame with transitional sequence and described forward prediction part is partly decoded with described back forecast simulate described transition effect by different way.
39. an equipment, it comprises:
The transition effect checkout gear, it is used for detecting the transition effect that is associated with described multimedia sequence during the coding of multimedia sequence; And
Code device, it produces the information of the described transition effect that is associated with described multimedia sequence in order to identification.
40. according to the described equipment of claim 39, the transition effect of wherein said information Recognition one type, the cycle that is associated with described transition effect and the beginning and the end scene that are associated with described transition effect.
41. according to the described equipment of claim 39, wherein said code device comprises the transition effect code device, described equipment further comprises the standard code device, and described standard code device uses the encode non-transition portion of described multimedia sequence of predictive coding.
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US72141705P | 2005-09-27 | 2005-09-27 | |
| US60/721,417 | 2005-09-27 | ||
| US11/501,969 | 2006-08-09 |
Related Child Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2013100630092A Division CN103096091A (en) | 2005-09-27 | 2006-09-27 | Multimedia coding techniques for transitional effects |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN101310530A true CN101310530A (en) | 2008-11-19 |
Family
ID=40125840
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNA2006800429097A Pending CN101310530A (en) | 2005-09-27 | 2006-09-27 | Multimedia coding techniques for transitional effects |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101310530A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109792526A (en) * | 2016-09-30 | 2019-05-21 | 高通股份有限公司 | The improvement of decoding mode is converted in frame per second |
| CN111182308A (en) * | 2018-11-09 | 2020-05-19 | 腾讯美国有限责任公司 | Video decoding method, video decoding device, computer equipment and storage medium |
-
2006
- 2006-09-27 CN CNA2006800429097A patent/CN101310530A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109792526A (en) * | 2016-09-30 | 2019-05-21 | 高通股份有限公司 | The improvement of decoding mode is converted in frame per second |
| CN109792526B (en) * | 2016-09-30 | 2022-05-24 | 高通股份有限公司 | Improvements in frame rate up-conversion decoding schemes |
| CN111182308A (en) * | 2018-11-09 | 2020-05-19 | 腾讯美国有限责任公司 | Video decoding method, video decoding device, computer equipment and storage medium |
| CN111182308B (en) * | 2018-11-09 | 2024-03-01 | 腾讯美国有限责任公司 | Video decoding method, device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103096091A (en) | Multimedia coding techniques for transitional effects | |
| US10306236B2 (en) | Image coding device and image decoding device | |
| CN101415122B (en) | Forecasting encoding/decoding method and apparatus between frames | |
| US9426480B2 (en) | Image signal decoding apparatus and image signal decoding method | |
| CN102037730B (en) | Content adaptive video encoder and coding method | |
| WO2013099285A1 (en) | Video encoding device, video encoding method and video encoding program, and video decoding device, video decoding method and video decoding program | |
| CN104350750A (en) | Constraints and unit types to simplify video random access | |
| WO2013099283A1 (en) | Video encoding device, video encoding method and video encoding program, and video decoding device, video decoding method and video decoding program | |
| CN104969557A (en) | Explicit way for signaling a collocated reference picture for video coding | |
| CN100539670C (en) | The editor of the A/V sequence of having encoded | |
| US11445212B2 (en) | Moving picture decoding device, moving picture decoding method, and moving picture decoding program | |
| US6473465B1 (en) | Method and apparatus for video coding at high efficiency | |
| CN101310530A (en) | Multimedia coding techniques for transitional effects | |
| CN101986713A (en) | View synthesis-based multi-viewpoint error-resilient encoding frame | |
| JPH03139083A (en) | Dynamic image encoding device | |
| JP2013153434A (en) | Video decoding device, video decoding method, and video decoding program | |
| US11812052B2 (en) | Moving picture decoding device, moving picture decoding method, and moving picture decoding program | |
| JPH1023415A (en) | Method and device for encoding and decoding picture | |
| AU2015202063B2 (en) | Image signal decoding device, image signal decoding method, image signal encoding device, image signal encoding method, and program | |
| JP2003274412A (en) | Image encoding method and image decoding method | |
| CN101090500A (en) | Code-decode method and device for video fast forward |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20081119 |