Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N7/00—Television systems
  • H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N7/00—Television systems
  • H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
  • H04N7/0117—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving conversion of the spatial resolution of the incoming video signal
  • H04N7/012—Conversion between an interlaced and a progressive signal
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T3/00—Geometric image transformations in the plane of the image
  • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
  • G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/59—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N7/00—Television systems
  • H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
  • H04N7/0135—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving interpolation processes
  • H04N7/014—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving interpolation processes involving the use of motion vectors
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/14—Picture signal circuitry for video frequency region
  • H04N5/144—Movement detection
  • H04N5/145—Movement estimation

Definitions

• the present invention relates generally to video processing. More particularly, the present invention relates to video spatial-up conversion using motion compensation in video processing.
• Video spatial up-conversion is also known as video resolution enhancement.
• V-SUC is used to enhance the spatial resolution of an arbitrary video sequence through both horizontal and vertical spatial interpolation.
• Video spatial up- conversion is one aspect of video format conversion (VFC), in which video signals are converted from one format to another.
• VFC video format conversion
• Two typical aspects of VFC are video deinterlacing, also known as video scan rate up-conversion and video picture rate up- conversion.
• Deinterlacing involves enhancing the spatial resolution of a video signal through interpolation in the vertical direction.
• Video picture rate up-conversion enhances the picture rate (also known as frame rate) of a video signal through temporal interpolation.
• Video spatial up-conversion is required for TV-out of mobile phone captured videos.
• Typical spatial resolutions of NTSC TV are 640x480 or 800x576.
• videos captured by conventional mobile telephones have a spatial resolution typically as SIF (320x240), CIF (352x288), or QCIF (176x144). Therefore, the spatial resolution needs to be enhanced before mobile telephone-captured videos are displayed in a regular TV device.
• Another example of video spatial up-conversion involves the display of standard definition TV (SDTV) signals in a high definition TV (HDTV) device.
• SDTV standard definition TV
• HDTV high definition TV
• Video spatial up-conversion mainly needs to fulfill two tasks in the process of spatial resolution enhancement: anti-aliasing and high spatial frequency generation to overcome the over-smoothness artifact.
• a digital video signal is obtained through three-dimensional (3D) sampling of the original continuous video signal.
• Ax , Ay , and T can denote the sampling distances in the horizontal direction, the vertical direction, and the temporal direction, respectively, which specify a 3D sampling grid.
• the replication centered at the coordinates ( ⁇ ,O, ⁇ ) is referred to as the baseband spectrum If the original continuous signal is band-limited and the maximum frequencies in the respective directions, denoted as /J 3x , / m v ax , and /L x .
• an ideal interpolation filter When a digital video signal is upsampled, an ideal interpolation filter should all-pass the baseband spectrum, without aliasing, while suppressing the aliasing portion as much as possible. As shown in Figure l(b), if a vertical motion is present, an ideal low pass filter for interpolation should be motion-compensated to effectively extract the baseband spectrum without aliasing,
• FIR filtering Spatial interpolation using finite impulse response (FIR) filtering is the most commonly used technique, where image independent FIR filters are applied in both the ho ⁇ zontal direction and vertical direction of a still image.
• Vanous interpolation FIR filters have been designed, with typical examples as bilinear filter, bicubic filter, bicubic spline filter, Gaussian filter, and Lanczos filter. These FIR filters are differentiated from each other mamly by different passband and stopband frequencies, as well as the length of the filter kernels.
• the design of these FIR filters mainly aims to all-pass the baseband spectrum containing no alias, suppress the aliasing spectrum component, and boost high frequencies to preserve image details such as edges.
• proper anti-aliasing is usually applied p ⁇ or to ho ⁇ zontal sampling but not in vertical sampling, it is suggested that different filters be used for horizontal interpolation and for vertical interpolation
• Image content-dependent filters have also been developed for image spatial interpolation On such filter is referred to as the Wiener filter, which is a linear filter with a target at the least mean square error (MSE).
• MSE least mean square error
• the coefficients of these types of filters are derived from the local image content, thus adapting to the local image characteristics.
• Other image spatial interpolation techniques are also conventionally known. These techniques include New Edge-Directed Interpolation (NEDI), which uses the geometrical duality across different resolutions of the image content, and Adaptive Quadratic (A Qua) image interpolation, which is based upon the optimal recovery theory and can be used to permit the interpolation of images by arbitrary factors. It has been shown that longer FIR filter kernels or image dependent filters are often preferred.
• video spatial up- conversion aims to enhance the spatial resolution of every picture in the video sequence.
• An effective video spatial up-conversion technique is only permitted to use a limited number of adjacent frames to enhance the resolution of current frame and the computational complexity should be kept reasonably low. Therefore, the concept of motion compensated video spatial up-conversion has not been extensively examined.
• the present invention involves the designing of effective motion compensated video up-conversion techniques by taking advantage of the connection between video spatial up-convcrsion and video deinterlacing. Specifically, the present invention involves the idea that interpolation in the two spatial directions for video spatial up-conversion be treated differently, and motion compensated techniques be used for the interpolation in the vertical direction. [0014]
• the present invention addresses the two primary tasks involved in the process of spatial resolution enhancement for video spatial up-conversion. In particular, the present invention addresses both anti-aliasing and high spatial frequency generation, which serves to overcome the over-smoothness artifact that would otherwise exist using conventional approaches. With the present invention, video resolution is enhanced by a scaling parameter of 2 in both the horizontal and the vertical directions.
• Figure l(a) is a representation of the f y - f t frequency space for progressively scanned videos with vertical motion
• Figure l(b) is a representation of the f v - f t frequency space for progressively scanned videos without vertical motion
• Figure 2 is a representation of a three-dimensional sampling grid for video deinterlacing
• Figure 3 is a representation a three-dimensional sampling grid for video spatial up-conversion
• Figure 4(a) is a representation showing an example of vertical interpolation using motion compensated samples with video deinterlacing
• Figure 4(b) is a representation showing an example of vertical interpolation using motion compensated samples with video spatial up-conversion
• Figure 5 is a representation of four types of samples in video spatial up- conversion;
• Figure 6 is a representation of motion compensated interpolation using the
• GST Generalized Sampling Theorem
• Figure 7 is a perspective view of an electronic device that can incorporate the principles of the present invention.
• Figure 8 is a schematic representation of the circuitry of the electronic device of Figure 7.
• a unique and significant characteristic of video signals is motion. Considering the correlation along the motion trajectory in the task of video spatial resolution enhancement is beneficial. Motion compensated video spatial up- conversion techniques, however, is even more advantageous than the consideration of such a correlation without using temporal correlations in constructing a spatial resolution enhanced video with a superior quality. This fact is supported by the advantages of motion compensation in video deinterlacing and the close connection between video spatial up-conversion and video deinterlacing. [0026] The present invention involves the concept of designing effective motion compensated video up-conversion techniques by taking advantage of the connection between video spatial up-conversion and video deinterlacing.
• the present invention involves the idea that interpolation in the two spatial directions for video spatial up-conversion be treated differently, and motion compensated techniques be used for the interpolation in the vertical direction.
• Video spatial up- conversion is required for TV-out of mobile visual content.
• motion compensated video spatial up-conversion accurate motion vectors are required. It should be noted that the motion for video predictive coding is different from the motion for video format conversion (VFC). In video predictive coding, the motion vectors of one block do not have to be correlated to that of adjacent blocks. For video format conversion, on the other hand, the true motion is supposed to be identified, where the motion vectors of adjacent blocks that belong to one object should be correlated to each other. Such motions can be obtained for video spatial up-conversion in a similar manner as the motion estimation operation implemented for video deinterlacing or video frame rate up-conversion. Motion compensated video spatial up-conversion techniques require more computational resources than non-motion compensated techniques due to the requirement of motion estimation. However, motion estimators are conventionally known and can be used for video deinterlacing. Therefore, the additional cost for motion estimation in video spatial up-conversion is limited.
• video deinterlacing is used to enhance the video vertical resolution, as shown in Figure 2.
• video spatial up- conversion is used to enhance the video resolution both horizontally and vertically. This is represented in Figure 3,
• the interpolation of vertical samples for video spatial up-conversion in Figure 3 can be realized in a similar manner, except that the "original" vertical sampling lines in adjacent frames are located in the same positions for video spatial up-conversion, instead of being interlaced for the scenario of video deinter ⁇ acing. Therefore, a close connection between video spatial up-conversion and video deinterlacing is established from the 3D sampling grid perspective.
• Motion compensated deinterlacing techniques take the motion compensated samples from the previous frame (or from both the previous and the successive frames) as a candidate for the interpolated samples of the current frame.
• a motion compensated video deinterlacing technique of the present invention should be able to be modified and extended for the use of video spatial up- conversion.
• methods for motion compensated video spatial up-conversion can be implemented through two steps.
• the first step involves interpolating the horizontal samples using a wide variety of spatial interpolation techniques.
• the second step involves interpolating the vertical sampling through the use of a motion compensated deinteriacing-like technique.
• the use of different methods for spatial interpolation in the horizontal direction and in the vertical direction is permissible because horizontal sampling is implemented after the image acquisition procedure, while vertical sampling is realized as a part of the image acquisition process by the cameras.
• the success of motion compensation in video deinterlacing, and the close connection between the two aspects of video format conversion (VFC), imply the success of motion compensation in video spatial up- conversion.
• two video deinterlacing methods are selected. These methods have demonstrated particularly strong deinterlaced video quality. These methods are used to develop two motion compensated video spatial up-conversion techniques. [0034] Algorithm I: Adaptivelv Recursive Motion Compensated Video Spatial Up- Conversion.
• Adaptive Recursive Motion Compensated (ARMC) video spatial up- conversion technique can be implemented:
• F(x,n) denotes the original sample
• F im (x,n) denotes the initially interpolated sample
• ( ⁇ ) denotes the transpose of a vector/matrix.
• Any spatial interpolation technique can be used for generating the initially interpolated samples F mt (x, n) at locations B, C, and D.
• Different FIR filters can be selected for the interpolation of horizontal samples (B) and vertical samples (C and
• a h (ic, n) is determined by the reliability of the motion vector for the original sample A:
• a B (x, n) is selected in a way such that the non-stationary pixels along the motion trajectory for sample B is the same as that of its horizontally neighboring pixels after video spatial up-conversion:
• Equation (3) ⁇ is a small constant preventing division by zero
• a c (x, n) is selected in a way such that the non-stationary pixels along the motion trajectory for sample C is the same as that of its vertically neighboring pixels after video spatial up-conversion:
• Equation (4) ⁇ is a small constant preventing division by zero
• ⁇ ci F ( X + fiy » «)- ⁇ SUC I* + Ay ⁇ d ⁇ x, «), /I - 1 J .
• ⁇ D (x, n) is selected in a way such that the non-stationary pixels along the motion trajectory for sample D is the same as that of its four diagonally neighboring pixels after video spatial up-conversion:
• Equation (5) ⁇ is a small constant preventing division by zero, and ⁇ D , ⁇ ⁇ y - d(x,n),n - l ⁇ ,
• ⁇ oi fFfc - M x + My, »)- ⁇ suc ⁇ - Mx + My - d(x, »), » - 1
• , ⁇ m F Mc ⁇ + ⁇ x + ⁇ y -d(x,n),n -l] .
• Algorithm H Adaptively Recursive Video Spatial Up-Conversion Using a Generalized Sampling Theorem (GST).
• “disjointness” refers to a shift in the time/spatial domain or equivalently, a phase difference in the frequency domain.
• GST has been successfully used in video deinterlacing.
• Extended from the known deinterlacing algorithm Adaptive Recursive GST the following Adaptive Recursive Motion Compensated scheme using GST (ARMC-GST) can be used for video spatial up-conversion.
• ARMC-GST is a two-step algorithm. The first step involves horizontal interpolation. An Optimized FIR filter is designed for the interpolation in the horizontal direction, which is a ID interpolation problem that conventionally understood. Through horizontal interpolation, samples at B positions are obtained. The second step involves vertical interpolation. Vertical interpolation is implemented as follows to obtain samples at C and D positions:
• Equation (6) a c (x, n) is obtained by Equation (4), a O (x,n) is obtained by Equation (5), F hit (x,n) is obtained by any spatial interpolation technique, and
• Equation (7) /z, (k, d y (x, nyj and h 2 (m, d (x, n)j , k,m e Z , are two FIR filters in the vertical direction as a function of the vertical component of the motion vector.
• the FIR filters can be designed in exactly the same way as their design in video deinterlacing using GST.
• it is known that, in recovering a continuous signal from its two sets of disjoint samples, if the condition / max - 2// /2 // is satisfied, aliasing for each set of samples is caused only by the interference of the two adjacent spectral replications.
• FIGS 7 and 8 show one representative electronic device 12 upon which the present invention may be implemented.
• the electronic device 12 shown in Figures 7 and 8 comprises a mobile telephone.
• the present invention is not limited to any type of electronic device and could be incorporated into devices such as personal digital assistants, personal computers, integrated messaging devices, and a wide variety of other devices. It should be understood that the present invention could be incorporated on a wide variety of electronic device 12.
• the electronic device 12 of Figures 7 and 8 includes a housing 30, a display 32 in the form of a liquid crystal display, a keypad 34, a microphone 36, an ear-piece 38, a battery 40, an infrared port 42, an antenna 44, a smart card 46 in the form of a universal integrated circuit card (UICC) according to one embodiment of the invention, a card reader 48, radio interface circuitry 52, codec circuitry 54, a controller 56 and a memory 58.
• the controller 56 can be the same unit or a different unit than the camera processor 116.
• the memory 58 may or may not be the same component as the primary memory unit 114 in various embodiments of the present invention.
• the present invention can be implemented as a part of a TV-out system for mobile terminals. Such a system can permit a user to display videos captured by a handset in a separate TV device.
• program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types.
• Computer-executable instructions, associated data structures, and program modules represent examples of program code for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Computer Graphics (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Television Systems (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)
• Television Signal Processing For Recording (AREA)

Priority Applications (2)

Applications Claiming Priority (2)

Publications (2)

Family

ID=38283244

Family Applications (1)

Country Status (6)

Families Citing this family (15)

Family Cites Families (17)

• 2006
  • 2006-04-25 US US11/411,700 patent/US7701509B2/en active Active
• 2007
  • 2007-04-24 WO PCT/IB2007/051516 patent/WO2007122589A2/en not_active Ceased
  • 2007-04-24 KR KR1020087027242A patent/KR20090035473A/ko not_active Ceased
  • 2007-04-24 CN CN200780014851XA patent/CN101433093B/zh active Active
  • 2007-04-24 JP JP2009507228A patent/JP5414519B2/ja not_active Expired - Fee Related
  • 2007-04-25 EP EP07106950A patent/EP1850590A3/en not_active Withdrawn
• 2013
  • 2013-02-12 JP JP2013024748A patent/JP5667650B2/ja not_active Expired - Fee Related
• 2014
  • 2014-04-23 JP JP2014089031A patent/JP2014161093A/ja active Pending

Also Published As

Similar Documents

Legal Events