Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
  • H04N19/31—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability in the temporal domain
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T9/00—Image coding
  • G06T9/001—Model-based coding, e.g. wire frame
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T9/00—Image coding
  • G06T9/004—Predictors, e.g. intraframe, interframe coding
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/17—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object
  • H04N19/172—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being an image region, e.g. an object the region being a picture, frame or field
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
  • H04N19/169—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding
  • H04N19/177—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the coding unit, i.e. the structural portion or semantic portion of the video signal being the object or the subject of the adaptive coding the unit being a group of pictures [GOP]
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
  • H04N19/51—Motion estimation or motion compensation
  • H04N19/537—Motion estimation other than block-based
  • H04N19/54—Motion estimation other than block-based using feature points or meshes
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding

Definitions

• the present invention relates to an apparatus and a method for coding a video signal, and particularly, but not exclusively, to a method and an apparatus for implementing a representation of displacement information (i.e., a model) between video frames.
• a representation of displacement information i.e., a model
• frame refers to frames of a video sequence, as well as views in a multi-view setting.
• Embodiments of the invention are not concerned with the generation of such models, but how they can be used to "infer" frames in between already decoded frames.
• motion compensated temporal lifting transforms [4] also known as motion compensated temporal filtering, or just MCTF
• MCTF motion compensated temporal filtering
• the present invention provides a method of representing displacement information between the frames of a video and/or multiview sequence, comprising the steps of assigning a plurality of the frames to a Group of Pictures (GOP) , providing a base displacement model for each GOP, the base displacement model describing a displacement field that carries each location in a designated base frame of the GOP to a corresponding location in each other the frame of the GOP, and inferring other displacement relationships between the frames of the GOP from the base displacement model.
• GOP Group of Pictures
• a video signal may be a multi-view video signal.
• a GOP may consist of frames from multiple views at the same time instance and/or frames from a view taken at different time instances.
• a video signal may be a single dimensional video sequence.
• a GOP need not be one-dimensional, as is traditionally the case for single view video compression, or in the case of a multi-view arrangement where all views are arranged in a ID array.
• 2D groups of pictures are the most appropriate construct for multi-view imagery associated with a 2D array of cameras, while 3D GOPs are the most appropriate construct when the cameras in such an array each capture a video sequence.
• the term "displacement” covers a number of parameters associated with the images, including motion, depth and disparity (particularly for multi-view imagery and video), location information, and other parameters .
• this represents a new way to describe, compress and infer displacements, in which all displacement information for a group of pictures (GOP) is derived from a base model, whose displacement representations are anchored at the GOP' s base frame.
• GOP group of pictures
• one piecewise smooth 2D displacement field is encoded for each frame, but all of the displacement fields associated with a GOP are anchored at its base frame. Collectively, we identify these displacement fields as the base model.
• One advantage of anchoring all descriptions for the GOP at the same frame is that it facilitates various compact descriptions of the multitude of displacement fields. By anchoring all displacements at the base frame, a single description of boundary discontinuities can be applied to all displacement fields, these boundary discontinuities being critical to the description of piecewise continuous models in general.
• energy compacting transforms are readily applied directly to the collection of displacement fields.
• parametric models may be employed to express the base model using a reduced set of displacement parameters.
• parametric representations can be based on physical attributes such as velocity and acceleration.
• the apparent displacement between frames of the GOP may be related to geometric properties, notably scene depth, so that depth or reciprocal depth provides the natural basis for parametric descriptions.
• the base-anchored framework can support high quality temporal motion inference, which is computationally efficient and requires as few as half the coded motion fields used in conventional codecs, where the common tool of bi-directional prediction assigns two motion fields to each target frame .
• the base-anchored approach advantageously provides more geometrically consistent and meaningful displacement information.
• the availability of geometrically consistent displacement information improves visual perception, and facilitates the efficient deployment of highly scalable video and multi-view compression systems based on displacement-compensated lifting, where the feedback state machine used in traditional codecs is replaced by purely feed-forward transforms .
• the present invention provides a method for coding displacement fields within a video sequence, wherein the video frames are assigned to Groups of Pictures known as GOPs, a base displacement model is coded for each GOP, describing the displacement that carries each location in a designated base frame of the GOP to a corresponding location in each other frame of the GOP, and other displacement relationships between the frames of the GOP are inferred from the base displacement model, in accordance with the method of the first aspect of the invention.
• GOPs Groups of Pictures
• the present invention provides a method for displacement compensated prediction of certain image frames from other frames, wherein the frames are assigned to groups of pictures (GOPs), a base displacement model is provided for each GOP, describing the displacement that carries each location in a designated base frame of the GOP to a corresponding location in each other frame of the GOP, this base displacement model being used to infer displacement relationships between the frames of the GOP, and the inferred displacement field at a prediction target frame being used to predict that frame from one or more other frames in the GOP.
• GOPs groups of pictures
• the present invention provides a coding apparatus arranged to implement a method for representing displacement information in accordance with the first aspect of the invention.
• the present invention provides a coding apparatus arranged to implement a method for coding displacement fields in accordance with the second aspect of the invention.
• the present invention provides a coding apparatus arranged to implement a method for displacement compensated prediction in accordance with the third aspect of the invention.
• the present invention provides a decoding apparatus arranged to decode a signal coded by an apparatus in accordance with the fourth aspect of the invention or the fifth aspect of the invention or the sixth aspect of the invention.
• the present invention provides a computer program, comprising instructions for controlling a computer to implement a method in accordance with the first aspect, second aspect or third aspect of the invention.
• the present invention provides a non-volatile computer readable medium, providing a computer program, in accordance with the eight aspect of the invention.
• the present invention provides a data signal, comprising a computer program in accordance with the eight aspect of the invention.
• Figure 1 is an illustration of base-anchored displacement in the case of ID Group of Pictures (GOP) , in accordance with an embodiment.
• Figure 2 is an illustration of a number of representative frames of an image/ video sequence, illustrating principles of the base anchoring, in accordance with an embodiment
• Figure 3 is an illustration of a displacement backfilling strategy, in accordance with an embodiment
• Figure 4 is an illustration of double mapping resolving procedure in accordance with an embodiment of the invention, in accordance with an embodiment
• Figure 5 is an illustration of the extension of the base anchoring to higher-dimensional GOPs, in accordance with an embodiment
• FIG. 6 is an overview of an encoder that employs the base model and inference scheme in accordance with an embodiment of the invention.
• Figure 7 is an overview of a decoder that employs the base model and inference scheme in accordance with an embodiment of the invention. Detailed description of embodiments
• FIG. 4.5 illustrates some of the key ideas behind the base-anchored framework.
• displacement information for a GOP is described within its base frame, denoted here as f Q .
• the displacement field is piecewise smooth, being expected to exhibit discontinuities around object boundaries.
• One way to describe such displacement fields is through a triangular mesh that is allowed to tear at certain locations - its breaks.
• Methods for representing and encoding such displacement models exist. For example, in the case of single-view video compression, [19] generalizes the mesh to a wavelet-based motion model based on affine interpolation, which is coupled with an efficient and highly scalable method for coding "arc breakpoints" [20] , which adapt the wavelet basis functions in the vicinity of displacement discontinuities.
• Figure 2 shows only a coarse regular triangular mesh, with 3 illustrative breakpoints.
• nodes of the mesh carry one displacement vector for each of the N-l non-base frames in the iV-frame GOP, denoted as u 0 ⁇ j,jE ⁇ l,N ⁇ .
• the node displacement vectors serve to continuously warp the mesh from the base frame to each other frame in the GOP.
• mesh nodes outside the original frame can be assigned the displacement of their adjacent node(s) in the frame.
• these nodes have to be mapped according to their displacement vector, effectively extrapolating the displacement at frame boundaries rather than creating a linear ramp as achieved by assigning a 0 displacement vector.
• these elementary extension methods are sufficient, more physically meaningful extension mechanisms will be apparent to those skilled in the art.
• information from base meshes from adjacent GOPs can be used in such regions; one way to achieve this is explained in Section 4.1.2, which describes a general method for "augmenting" the base mesh in disoccluded regions .
• the base model incorporates scene depth (or reciprocal depth) information
• the ambiguity created by double mappings can be resolved immediately by identifying the visible location r k l as the one with smallest depth.
• explicit depth information is either not available or not sufficient to describe the displacement relationships between the base frame and all other frames of the GOP
• more advanced techniques can be used to resolve double mappings - see Section 4.2 for a description of how discontinuities in the displacement field can be used to identify local foreground objects.
• each breakpoint effectively introduces two new mesh nodes, illustrated by green and orange dots in the figure, whose locations coincide with the break in the base frame and whose displacement vectors are obtained by replicating or extrapolating the displacement vector from each end of the arc that is broken. Mapping these new break-induced nodes into each non-base frame, using their respective displacement vectors, can open up "holes" in the mesh, corresponding to regions in the non-base frame that are not visible from the base frame. These so-called disoccluded regions are illustrated by pink shading in the figure.
• Break-induced discontinuities in the displacement field also provide a rich source of double mappings (not shown in the figure) , corresponding to areas of occlusion; as one side of a foreground object disoccludes background content, the other side typically produces double mappings .
• an embodiment employs a novel backfilling methodology.
• Triangles formed by breakpoint-induced nodes in the base frame are necessarily stretching (their area increases significantly when they are mapped) . Roughly half of these stretching triangles, which are expected to form around discontinuities (i.e., object boundaries) in the displacement field, are "disoccluding"; the remaining stretching triangles are "folding", indicating regions of double mappings. Disoccluding triangles are identified as triangles with a positive determinant, whereas folding triangles are characterized by a negative determinant.
• Folding triangles map to regions where at least two other triangles (one of the local foreground and one of the local background object) will map to, and hence are discarded.
• Disoccluding triangles can map to regions where no other triangle maps to; these triangles need to be handled separately, which is described in the following. Holes created by disocclusion are first filled by adding so-called "break-induced mesh elements" that link the breakpoint-induced nodes in the base frame. These mesh elements have zero size in the base frame, but expand to fill disoccluded regions in the non-base frames, as illustrated by the dashed red lines in Figure 2.
• disoccluded regions in a non-base rame always involve substantial expansion, where the displacement found within a small region in the base rame expands that region to a much larger one in the non-base frame.
• break-induced nodes avoids any ambiguity in the identification of disocclusions , because the break-induced nodes from each side of a discontinuity in the displacement field are co- located in the base frame, leading to mesh elements with zero area that exhibit infinite expansion ratios wherever disocclusions arise in the non-base frame.
• ⁇ elements in the mesh is sufficient to cover all regions of disocclusion in every non-base frame. This means that each frame in a GOP is certain to be covered by mesh elements from the base frame that are mapped in accordance with the associated displacements.
• ⁇ elements do ensure that a reverse displacement field exists everywhere, pointing back to the base frame from any non-base frame, they do not lead to physically meaningful reverse displacement values. This is because half the break-induced nodes (e.g., full black circles (dots)) associated with an ⁇ element move with the background, while the other half (e.g., black dots in circles) move with the foreground. Displacement within a disoccluded region, however, should be associated entirely with the (local) background.
• the backfilling scheme starts by assigning new displacements to disoccluded regions in the last frame of the GOP, which can be identified as the "back-filled" frame for the purpose of this description.
• each disoccluded region in the back-filled frame starts out being covered by ⁇ elements that have zero size in the base frame.
• the first more generic method, extrapolates the local background information.
• the second method called basemesh augmentation, leverages displacement information that is provided at the back-filled frame by other means, to augment the current base mesh; this method is of particular interest when the back-filled frame coincides with the baseframe of another GOP.
• each mapped ⁇ element that is visible in the back-filled frame is first replicated to produce what will become a back-fill element. Consequently, the mesh nodes that delineate each back-fill element include at least two break-induced nodes, being co-located in the base frame. For each pair of break-induced nodes, one belongs to the foreground side of the break and one belongs to the background side of the break. Distinguishing between these is very important to the back-filling procedure.
• Each break-induced node that is identified as belonging to the foreground is also replicated, and the replica is associated with the relevant back-fill element (s) in place of the original break-induced node that is associated only with ⁇ element (s) .
• the replicated nodes are identified as back-fill nodes, and are illustrated in Figure 3b as purple dots; these can also be viewed as "free nodes", since the displacement vector initially assigned to these nodes from the base model can be freely changed to improve consistency with the uncovered background, whose displacement vectors should be modeled by the back-fill elements .
• the mapped locations of back-fill nodes in the back-filled frame must agree with the mapped locations of the break- induced nodes from which they were spawned, but the backfilling strategy assigns new displacement vectors to these nodes, effectively changing their locations within all other frames, including the base frame.
• the ⁇ elements that span disocclusions in the back-filled frame are remapped into back-fill elements (triangles g-h in Figure 3b) , whose appearance is identical to the corresponding ⁇ elements in the back-filled frame, but not in the other frames .
• Displacement vectors are assigned to back-fill nodes based on an extrapolation procedure that averages the displacement vectors found on the other node(s) of the back-fill mesh elements - these are the non-free nodes from each pair of break-induced nodes that defined the original ⁇ elements.
• the displacement vectors for the free nodes are found by assigning a weighted average of all the "fixed" nodes (i.e., the ones carrying local background displacement information) in the grid of the base mesh, obtained via a splatting procedure. This creates a lookup table of displacement values that can efficiently be implemented in a computer graphics card.
• the original 2D base mesh is converted to a layered mesh through inter-frame reasoning alone, without the need for any additional coded displacement information or other side information.
• This section describes a way of augmenting the current base mesh with information from another mesh - this can be either a base mesh of another GOP, or another mesh that has been coded; we refer to this mesh as the augmenting mesh.
• all triangles of the other base frame that cover regions mapped by infinity triangles from the current base mesh i.e., the set of all disoccluded regions
• the main appeal of base mesh augmentation is that it is able to handle discoccluded regions where new objects are appearing.
• the main issue that arises with base mesh augmentation is that since the augmenting mesh elements are only applied in regions where the current base mesh has no valid values, inconsistencies can be created at the (hard) transition boundary from the current base mesh to the backfilled augmenting mesh elements .
• the augmenting mesh elements can potentially be large enough to span unrelated disocclusion regions, leading to ambiguities in the back-filling procedure.
• the back-fill mesh elements are all collected within the base frame as part of an augmented base model, where they can be interpreted as an inferred local background layer, which guarantees consistent displacement assignments in disoccluded regions across multiple interpolated frames .
• back-filling because the assignment of displacement vectors to backfill nodes, which leads to new underlying mesh elements in the base frame, is driven initially from the last frame of the GOP - the back-filled frame. This is the frame that is furthest from the base frame, where regions of disocclusion are likely to be largest. While not strictly necessary, it is desirable to arrange for the inter-frame transform to provide intra-coded texture information at the base frames .
• the remapped mesh elements produced by backfilling correspond to content that can be predicted from the last frame of the GOP (the next base frame) but not from the GOP' s own base frame.
• the new back-fill elements are defined by mesh nodes that arise from pairs of original break-induced nodes, whose foreground/background assignment is either already known (from preceding back-filling steps) or needs to be newly determined, as explained above. Once determined, the background node within each break-induced pair is replicated to form a new back-fill node that is associated with the relevant back-fill element. These new back-fill nodes are free to be assigned new displacement vectors, using the same extrapolation procedure described in Section 4.1.1, which results in the back-fill mesh elements constituting a new local background layer within the base frame.
• the base model is progressively augmented with backfill nodes and back-fill mesh elements, so that the base model eventually describes the relationship between all frames of the GOP in a complete and geometrically consistent manner.
• back-fill elements means that double mappings become increasingly likely, as mesh elements from the augmented base model are mapped to new intermediate frames. All such double mappings can be resolved using the methods briefly introduced already. It is helpful, however, to assign a layer-ID to each back-fill element, which identifies the back-filled frame in which the backfill element was discovered.
• Original elements of the base mesh are assigned layer-ID 0.
• Elements introduced when back-filling the last frame of the GOP e.g., f N
• Elements introduced when backfilling the first intermediate frame e.g., /jv/2) are assigned layer-ID 2, and so forth.
• the ⁇ elements are conceptually assigned a layer-id of ⁇ . This way, when a frame location is mapped by different mesh elements, the double mapping can be resolved in favour of the element with smaller ID.
• each node in the (augmented) base model is assigned a unique ID and each mesh element is also assigned a unique ID.
• mapping base model elements to another frame that frame is assigned an ID-map, with one ID for every pixel location.
• the ID-map is populated progressively as each mesh element is mapped, transferring the element's ID to all empty locations in the ID-map that are covered by the mapped element. Double mappings are discovered immediately when a mesh element covers a location in the mapped frame's ID-map that is not empty.
• the existing ID is used to immediately discover the mesh element that maps to the same location as the one currently being considered, and the double mapping resolution techniques are applied, as described already.
• any locations in the ID-map that identify ⁇ elements are the ones for which back-filling is required.
• Simple pixel and reference counting techniques can be used to identify all ⁇ elements that remain visible, each of which is replicated to produce a back-fill element and back-fill nodes that are remapped.
• an embodiment employs a recursive back-filling strategy.
• the first back-filled frame associated with the GOP based at frame f 0 is f N .
• the next back-filled frame is ⁇ N/2 - After this, frames f N / and fsN/2 are back-filled. The process continues in this way, following a breadth-first dyadic tree scan of the GOP.
• the foreground displacement model should be the one that maps discontinuities in the base mesh for frame f 0 to discontinuities in the next base mesh, associated with frame f N
• the last frame of each GOP is also the first frame of the next GOP, so that the next GOP' s base displacement model M Nr that is anchored in frame f Nr can be compared with the displacement found in the base displacement model M 0 of the current GOP.
• a discontinuity (or break) in the base displacement model M Q can be mapped to frame f N using the displacement vector found on either side of the break; the displacement vector which maps the discontinuity to a region of similar divergence or convergence in model M Nr is the one that is more likely to correspond to the foreground displacement vector .
• each pair of co-located break-induced nodes in the base mesh maps to a line segment in the back-filled frame that spans the disoccluded region. This phenomenon corresponds to divergence in the base model M 0 . From each such pair of break-induced nodes, the free node is identified as the one whose location in frame f N exhibits a divergence value in the next base model M N that is most similar to the divergence in M 0 .
• the mesh nodes or regions that belong to the foreground are determined as follows (see Figure 4).
• the "origin" of any detected double mapping s k in non-base frame f k is found by searching the line segment connecting the corresponding source locations r and r ⁇ in the base-frame, looking for the location at which the displacement field folds.
• This line segment was refer to this line segment as the "fold search path.” Folding is associated with convergence in the base displacement field, so the fold location is identified as the point along the search path at which the displacement convergence value (negative divergence) is largest. This location usually corresponds to a break in the base displacement field.
• the fold location is mapped to frame f Nr using the displacement vectors on each side of the fold (e.g., at a distance of 1 pixel in each direction along the fold search path) , and the divergence in the next base displacement model M N is compared with that in the base displacement model M 0r to discover which displacement vector belongs to the foreground.
• the foreground side of the fold is the one whose displacement vector carries it to a location of similar convergence (negative divergence) in the next base-frame.
• Figure 4 is an illustration of a double mapping resolving procedure that uses divergence of the displacement field to identify the foreground object.
• T 0 ⁇ t defines the affine mapping from frame f 0
• i"o and i"g map to the same location m in f t .
• each location m in the target frame can be computed
• each location in the target frame is predicted as a weighted combination of all reference frames where the location is visible, weighted by the distance of the target frame to the respective reference frame.
• d(v) is a distance measure. That is, each location in the target frame is predicted as a weighted combination of all reference frames where the location is visible, weighted by the distance of the target frame to the respective reference frame.
• a number of methods can applied. In the formulation above, we resort to simple weighted prediction for locations that are deemed not visible. Another, preferred way is to employ inpainting strategies to fill in all locations that are not visible in any reference frame, which generally lead to more plausible interpolations in regions that are not visible in any of the reference frames.
• Figure 5 is an example tiling of a higher-dimensional group of pictures (GOP) .
• Each GOP has one base frame in its upper left corner. Adjacent GOPs overlap by one frame, in each direction and the frames that would be base frames for additional overlapping GOPs that do not exist are shown in light grey text - these are so-called "disenfranchised base frames.
• Figure 5 shows a GOP tiling scheme that represents the most natural extension of the ID GOP structure presented before.
• the base frame for a GOP is in its upper left corner, and adjacent GOPs overlap by one frame, horizontally and vertically.
• the displacement-compensated inter-frame transform involves prediction only between frames that are found in the same GOP. This is why we require the GOPs to overlap. This way, common frames found at the intersection between adjacent GOPs can be used to predict frames found within each of those GOPs.
• frame 2 M,O has no GOP of its own, unless the tiling continues to a third row of GOPs .
• These disenfranchised base frames need carry no coded displacement information. However, it could be beneficial to selectively encode displacement information within a disenfranchised base so that it can be used to improve the quality of back-filled geometry for the GOPs that include it .
• the back-filling algorithm is identical for ID and 2D GOPs, but in the 2D case there is no obvious order in which frames should be visited to generate back-fill mesh elements.
• the back-filling order determines the order in which the base model for a GOP is augmented, which ultimately affects the inferred displacement values that are generated for any given frame of the GOP. It is important that the back-filling order is well-defined, since the displacement-compensated inter-frame transform generally depends upon it.
• the motion-compensated prediction of a target frame f t can exhibit visible artefacts around object boundaries, where the displacement field is likely to be discontinuous.
• the target frame f t is predicted from N reference frames fry E ⁇ 1, ...,N ⁇ ; we note here that depending on the transform structure, the base frame itself might be a target frame.
• the upsampled frame interpolated using the occlusion-aware frame interpolation method of can have problems; the sudden transition from uni- to multi-directional prediction can lead to artificial boundaries in places where the illumination changes between the two reference frames.
• the method we propose therefore consists of limiting the frequency content at each location of f t to the one of the motion- compensated reference frames.
• this is achieved in the wavelet domain.
• f t to denote the 2D-wavelet decomposition (interleaved) of frame / " ;
• i[k] to access a specific wavelet coefficient k, where k collects information about level, subband, and spatial position in the transform.
• r[k] max(/ ri [k] / ri ⁇ t [fc], ...,/ r Jfc] rw ⁇ t M) ⁇ That is,T[fc] represents the largest (visible) wavelet coefficient of the wavelet decomposition of the
• synthesizing optical blur uses the divergence of the mapped (and inverted) displacement field M t ⁇ b as an indication of displacing object boundaries.
• a low-pass filter is applied to all pixels where the absolute value of the divergence of the displacement field is larger than a certain threshold ⁇ ; the displacement compensated non- base frame with optical blur synthesis, denoted as f ⁇ ° is then obtained as :
• h[m] is the kernel of a two-dimensional low-pass filter.
• Figure 6 shows an overview of an encoder that employs the proposed base model.
• Input to the encoder scheme is either a video sequence, multi view imagery, or a multi view video sequence.
• the base model is
• FIG. 7 shows an overview of a decoder that employs the proposed base model. Firstly, the sub bands and base model are decoded 110. Then, the sub bands are subjected to an inverse spatial transform 111. Lastly, the decoded sequence is obtained by reversing the interframe transform 112.
• Encoders and de-coders implementing the methods, processes described above may be implemented using hardware, software, a combination of hardware, software and
• firmware may be provided on computer readable mediums, or transmitted as a data signal, or in other way.
• An element of the base-anchored approach in accordance with embodiments described above is the displacement backfilling procedure, whereby local background
• displacement layers are added to the base model whenever disocclusion holes are observed during displacement inference. These "background layers” guarantee the assignment of geometrically consistent displacement information in regions of disocclusion, which is highly important for visual perception.
• Another element is a robust method of identifying local foreground/background relationships around displacing objects in cases where such information cannot be deduced from the displacement information, as is for example the case when the
• the base anchoring approach facilitates the deployment of compression systems that are highly scalable across space (i.e., multi-view) and/or time (i.e., video), enabling a seamless upsampling of arbitrary frame-rates across both dimensions .
• a compelling feature of the base-anchored approach is that not all the frames that are used to estimate the base displacement model must be coded. That is, one might use all frames that were recorded to estimate a high-quality displacement model; however, only a fraction of these frames is coded, and all "in-between" frames are purely interpolated using the described geometrically consistent frame interpolation procedure. This is in contrast to existing video

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)
• Image Analysis (AREA)

Priority Applications (4)

Applications Claiming Priority (4)

Publications (1)

Family

ID=61689250

Family Applications (1)

Country Status (5)

Cited By (3)

Families Citing this family (10)

Citations (4)

Family Cites Families (6)

• 2017
  • 2017-09-21 EP EP17851993.0A patent/EP3516872A4/en active Pending
  • 2017-09-21 US US16/335,297 patent/US11122281B2/en active Active
  • 2017-09-21 AU AU2017331736A patent/AU2017331736B2/en active Active
  • 2017-09-21 WO PCT/AU2017/051030 patent/WO2018053591A1/en not_active Ceased
  • 2017-09-21 JP JP2019536625A patent/JP7279939B2/ja active Active

Patent Citations (4)

Non-Patent Citations (31)

Also Published As

Similar Documents

Legal Events