EP1161833A1

EP1161833A1 - Conversion of video data

Info

Publication number: EP1161833A1
Application number: EP00902407A
Authority: EP
Inventors: Ronald D. Smith
Original assignee: Intel Corp
Current assignee: Intel Corp
Priority date: 1999-03-15
Filing date: 2000-01-12
Publication date: 2001-12-12
Also published as: KR20010102478A; JP2002539725A; WO2000056069A1; AU2412900A

Abstract

A method and apparatus of modifying video data according to a first format (e.g., interlaced scanned format) in a system includes retrieving one or more indications of movement associated with the video data (502). The video data is converted from the first format to a second, different format (e.g., progressive scanned format) (510) based on the one or more indications of movement. The video data may be digitally encoded according to a Motion Picture Experts Group (MPEG) standard, such as MPEG-2.

Description

Conversion Of Video Data Background

The invention relates to converting video data from a first format to a different format.

Many current television and video standards utilize interlaced scanning of an image frame. Such standards include the National Television System Committee (NTSC) standard of the Electronics Industries Association, the Phase Alternation by Line (PAL) standard, the Sequential Couleur Avec Memoire or Sequential Color With Memory (SECAM) standard, the Digital Video Disc (DND) standard, as well as some of the high definition television (HDTV) standards. A description of ΝTSC, PAL, and SECAM may be found in Keith Jack, "Video Demystified: A Handbook for the Digital Engineer," HighText Publications (2d ed. 1996). A description of DVD may be found at the Internet site

{http://www.dvdresource.com/dvdfaq}, and a description of an HDTV standard may be found in the ATSC Digital Television Standard, dated September 16, 1995, and provided by the Advanced Television Systems Committee (ATSC). Interlaced scanning was adopted for television to reduce flicker and other artifacts created by video equipment. For example, an ΝTSC signal produced by an electron beam may scan the picture tube of a television at 525 lines of picture data 30 times per second. Interlaced scanning involves dividing the 525 lines of the television frame into two fields, each made up of 262.5 lines of picture data. One field is made up of odd lines, while the second field is made up of even lines. Lines in the first field are first scanned, followed by lines in the second field. The net effect is that each scan involves scanning every other line in the picture frame. The other standards may have differing numbers of lines but the interlaced scanning technique is similarly applied.

To overcome some limitations of interlaced scanning, a progressive scanning technique (in which every line is scanned in sequence) has been used. One type of display that has used progressive scanning includes monitors of computer systems. Progressive scanning generally provides a sharper image at the cost of twice the transmission bandwidth. Currently, some video display devices such as cathode ray tube and vidicon tube systems directly support interlaced scanning. However, other types of devices such as liquid crystal displays (LCDs) and charge coupled device (CCD) displays may not be optimized to support interlaced scanning. If interlaced scanned video data is applied for display in systems that are not optimized to receive interlaced scanned data, then artifacts may be created. Such artifacts may include a motion tilt artifact, a comb-like artifact, and other artifacts.

To remove such artifacts, some techniques have been provided to convert video data from an interlaced scanned format to a progressive scanned format. Conventional techniques in use are variations of the adaptive filtering technique, which typically performs

2! ₂-dimensional (2 2-D) processing (2-D for processing a horizontal and vertical array of video data and '/i-D for processing in the temporal dimension to compare the current frame to a previous frame). In the adaptive filtering technique, the missing lines (odd lines or even lines) of information in an interlaced frame may be generated and filled into the frame to create a progressive scanned frame.

The adaptive filtering technique typically compares the current interlaced frame (frame 2) to a prior frame (frame 1) and assumes that there is some degree of vertical correlation between the frames. The amount of motion between the frames may be determined based on some predefined threshold. If no motion or low motion is indicated, then data of frame 1 may be used to fill in the blank lines of frame 2. However, if high motion is detected, then some algorithm may be used to fill in the blank lines of frame 2, including line replication or interpolation. Since the target image is rarely still in video, the success of the conversion may vary widely with the source material. In addition, the changing back and forth from copying the previous frame to interpolating the current one provides a source of noise to the video image.

To improve accuracy, some high-end consumer devices have begun using full 3-D algorithms for interlaced- to-progressive scanning conversion. With the 3-D schemes, the current frame is compared to the previous frame as well as to the next frame to both better detect motion and interpolate video data. Using the 3-D algorithm improves the results, but the improvement comes at the expense of significantly more memory to store the extra frames of video, significantly more processing power to correctly detect motion. Further, audio portions are delayed so that the video and audio components are properly synchronized.

Regardless of whether a conventional 2Ϊ -.-D or 3-D scheme is used, the decision on when to switch between frame copying and interpolation may be difficult as accurate motion detection (which involves determining the existence of motion as well as the direction of motion) may be difficult to achieve. Thus, a need exists for an improved method and apparatus for converting video data produced by interlaced scanning to progressive scanned video data.

Summary In general, according to one embodiment, a method of modifying video data according to a first format in a system includes retrieving one or more indications of movement encoded with the video data. The video data is converted from the first format to a second, different format based on the one or more indications of movement.

Other features and embodiments will become apparent from the following description and from the claims.

Brief Description Of The Drawings

Fig. 1 illustrates an embodiment of a video system. Figs. 2-5 illustrate video pictures processed by the video system of Fig. 1. Fig. 6 is a flow diagram of a process performed by the video system of Fig. 1.

Detailed Description

In the following description, numerous details are set forth to provide an understanding of the present invention. However, it is to be understood by those skilled in the art that the present invention may be practiced without these details and that numerous variations or modifications from the described embodiments may be possible.

Referring to Fig. 1, an example video system 10 according to one embodiment includes a graphics controller 12 and a display monitor 34 coupled to the graphics controller 12. The graphics controller 12 may be coupled to a system bus 36 in the system 10. In one embodiment, the system 10 may be a computer or another type of system such as a set-top box, a hand-held computing device, an appliance, a game system, a display system such as a projection display system, or any other system that includes video capabilities. In other embodiments, the system 10 may include different components and architectures.

According to one embodiment, the graphics controller 12 is adapted to receive video data from one or more sources. For example, the graphics controller 12 may receive video signals over the system bus 36 or from another source that may provide analog or digital video signals through port 31. Such video signals may include interlaced scanned video data. According to some embodiments, digitally encoded source video material containing interlaced scanned data may include indications of movement associated with video frames or video frame portions that may be created at the source of the video material. The indications of movement are digitally encoded into the source material to indicate how an image has moved with respect to a prior image. Conversion of the interlaced scanned video data to progressive scanned video data is performed based on the indications of movement, as further described below.

Digitally encoded standards for storing digital audio and video signals include those from the Moving Picture Experts Group (MPEG), including the MPEG-2 standard. The MPEG-2 standard is described in ISO/IEC 13818-1 (MPEG-2 Systems), ISO/IEC 13818-2 (MPEG-2 Video), and ISO/IEC 13818-3 (MPEG-2 Audio), dated in 1994 and provided by the International Organization For Standardization (ISO) and the International Electrotechnical Commission (IEC). MPEG-2 provides a generic coding technique for moving pictures and associated sound of various applications, including digital storage media, television transmissions, and data communications. MPEG-2 also provides for representations of both progressive and interlaced scanned video sources. Video and audio data coded according to MPEG-2 may be manipulated as digital data, stored on various storage media, transmitted and received over networks, and/or distributed on transmission channels. Although reference is made to MPEG-2 video frames in this description, the invention is not to be limited in this respect as other types of digitally encoded video standards may be used in further embodiments.

As illustrated in Fig. 2, a picture 100 represented according to MPEG-2 may be divided into macroblocks 102. In the illustrated embodiment, each picture frame 100 includes an array of macroblocks 102 according to the MPEG-2 standard. According to the MPEG-2 standard, each frame of interlaced scanned video data may include two fields, with one field containing odd lines and the other field containing even lines. In this description, the term "picture" may refer to either a frame or a field.

MPEG-2 video data includes motion compensation at the macroblock level to improve compression of the video data by removing temporal redundancies between successive pictures (frames or fields). Motion compensation depends on the fact that within a short sequence of the same general image, most objects remain in the same location while others move only a short distance. The motion may be described as a two-dimensional motion vector that specifies where to retrieve a macroblock from a previously decoded picture to predict the pixel values of the current macroblock. After a macroblock has been compressed using motion compensation, it contains both the spatial difference (motion vectors) and content difference (error terms) between the reference macroblock and the current macroblock being coded.

As successively more pictures are added, the motion vectors of the latest picture may be encoded differentially with respect to the last encoded motion vector using variable length codes. In deriving the motion vectors, "true" or reliable motion detection is performed at the data source to compare current pictures with previous pictures to determine the amount of motion compensation needed. Conventionally, relatively sophisticated equipment may be used at the data source to encode video data according to the MPEG-2 standard. The data source may include service providers such as studios or other producers that create material including DVD (digital video disc) movies, digital cable programming, and programming for terrestrial transmission of HDTV (high definition television). Such sophisticated equipment may provide higher quality video with the relatively high cost effectively spread over viewers and consumers of the produced video data. Consequently, more accurate motion detection may be incorporated into the source video data (such as MPEG-2 data) than may be expected from consumer devices at the receiving end.

Further, the MPEG-2 standard allows for significant compression by interpolating many intermediate pictures from both well into the past and well into the future. This introduces the requirement at the source end for relatively large frame memory storage and processing power to determine optimal coding for maximum compression at high quality. In further embodiments, other types of standards for storing video data that include indications of motion (e.g., vectors, coded data, and so forth) between or among pictures may be implemented.

To receive video data, the graphics controller 12 may include a video decompression and decoder circuit 32 that is adapted to receive video signals from the system bus 36 or from the port 31. Data from the decoder circuit 32 may be provided to a conversion block 26 that is adapted to perform interlaced-to-progressive scan conversion according to some embodiments of the invention. If the video data is from a source that provides digitally encoded indications of motion, then the conversion block 26 can perform the interlaced-to- progressive scan conversion based on the indications of motion. The data provided from the decoder circuit 32 to the conversion block 26 indicates video data along with associated indications of motion. The decoder circuit 32 may be different from conventional decoders which may remove the indications of motion after the digitally encoded video data has been decoded. The output of the conversion block 26 is provided to a controller 20 that is coupled to video memory 22, e.g., dynamic random access memory (DRAM), synchronous DRAM (SDRAM), video random access memory (VRAM), static random access memory (SRAM), and so forth. The output of the controller 20 is provided to an interface block 28, which may include a random access memory digital-to-analog converter (RAMDAC), that feeds video signals (e.g., analog RGB signals) to the display monitor 34.

In some embodiments, the controller 20 may be run under control of a graphics routine 24, which may be a software or firmware layer. In further embodiments, the interlaced-to-progressive scanned conversion may be performed by the graphics routine 24 instead of the conversion block 26, or the conversion may be performed by a combination of the graphics routine 24 and conversion block 26. During the conversion process, the received video data and the processed video data may be stored in some storage device, such as the video memory 22 or other suitable storage medium or media in the system 10.

In the example system 10, the system bus 36 may be coupled to a network interface controller 50 that is coupled to a network communications channel 52. In addition, the system bus 36 may also be coupled to a storage controller 44 that may be coupled to a hard disk drive 48 or to a compact disc (CD) or digital video disc (DVD) drive 46. The system bus in one embodiment may include a Peripheral Component Interconnect (PCI) bus, as described in the PCI Local Bus Specification, Production Version, Revision 2.1, dated June 1995. The system bus 36 may also be coupled to a bridge controller 40 that is coupled to a central processing unit (CPU) 38, which may be a processor such as a microprocessor, a microcontroller, an application-specific integrated circuit (ASIC), a programmable gate array (PGA), and the like. The bridge controller 40 may also include a memory controller (not shown) coupled to a system memory 42. In another embodiment, instead of the graphics controller 12 being coupled to the system bus 36 directly, it may be coupled to a graphics port in the bridge controller 40 over a link 54. In this other embodiment, the link 54 between the bridge controller 40 and the graphics controller 12 may be an Accelerated Graphics Port (AGP) link, as described in the Accelerated Graphics Port Interface Specification, Revision 2.0, dated May 1998.

The system 10 may also include a modem 56 that may be coupled to the system bus 36 or to a secondary or expansion bus (not shown) in the system 10. The modem 56 may be coupled to a communications link 58, such as a telephone line, a cable link, or other types of links. The modem 56 may be any type of transceiver, including a transceiver that acts as an interface between analog and digital channels as well as a transceiver that acts as an interface between digital and digital channels.

As illustrated, several possible sources of video data are available. The video data may be provided over the network 52 through the network interface controller 50, over the communications link 58 through the modem 56, through the port 31, on a CD, DVD or other storage medium that may be coupled to the storage controller 44, or over other paths. The configuration and architecture of the system 10 as shown in Fig. 1 are for illustrative purposes only and are not to be considered limiting for purposes of the invention — other types of systems are contemplated.

Digitally encoded video data provided to the graphics controller 12, either through the port 31 or over the system bus 36, containing interlaced scanned data may include indications of motion (such as the motion vectors found in MPEG-2 video data). Using indications of motion, interlaced scanned data may be converted by the interlaced-to-progressive scan conversion block 26 (or by some other component such as the graphics routine 24 or by some combination of components) to progressive scanned data. The progressive scanned data in turn may be provided to the controller 20, which further processes the data for presentation on the display monitor 34.

Fig. 3 illustrates the basic interlaced-to-progressive scanned problem. Block 310 represents a progressive scanned frame, while blocks 312, 314 and 316 represent interlaced scanned fields. As illustrated, fields 312 and 316 include odd lines while field 314 includes even lines. The fields 312, 314, and 316 represent successively received fields. Two interlaced scanned fields make up one interlaced scanned frame. To convert to progressive scanned format, the information from two interlaced scanned fields are used to derive the progressive scanned frame, as further described below.

Referring to Fig. 6, a process according to one embodiment performed by the graphics controller 12 (by the conversion block 26 or the graphics routine 24 or both) is illustrated. The graphics controller 12 first checks (at 502) to determine if video data according to the interlaced scanned format has been received. If not, then interlaced-to-progressive conversion is bypassed. If interlaced data has been received, then the graphics controller 12 determines (at 504) motion of a portion of the image picture. In one embodiment in which MPEG-2 pictures are used, a picture portion includes a macroblock 102 (Fig. 2). Motion of macroblocks may be conveniently detected by retrieving motion vectors associated with each macroblock. With other standards, other types of motion indications may be retrieved to determine motion of the picture or picture portion.

As illustrated in Fig. 4, three interlaced scanned fields 312, 314, and 316, are illustrated in which the field 312 is the first field and fields 314 and 316 are successively received fields. If no motion is detected (as determined at 506), then the lines of the previous field (e.g., 312) are copied (at 508) along with the lines of the current field (e.g., 314) to form a progressive scanned frame 320. Similarly, to create a progressive scanned frame 322 when no motion is detected, the lines of the field 314 are copied along with the lines of the field 316 into the frame 322. Effectively, in the case of no motion, the previous interlaced scanned field is combined with the current interlaced scanned field to form a progressive scanned frame.

The conversion of the first received picture may be handled differently. For example, if the interlaced scanned field 312 in Fig. 4 is the first field received, then a previous field does not exist. To create a progressive scanned frame 324, the odd lines of the field 312 are copied, with the even lines duplicated or interpolated from the odd lines to fill up the progressive scanned frame 324.

The conversion is performed differently if motion exists. Referring again to Fig. 6, if some motion is detected, then the current picture portion is processed (at 510) using the motion vector or other indication of movement. As shown in Fig. 5, a field 316A includes a motion vector 306 to indicate movement of a macroblock 304. The result of the move indicated by the motion vector 306 of the macroblock 304 is a block 304A. To create a progressive scanned frame 330, the lines from fields 314 and 316A are combined and processed. Data in the macroblock 304 is modified according to the motion vector 306 and associated difference vector indicating how data is to be modified. The unmoved portions of the frame 330 are copied directly from the fields 314 and 316A. Odd lines 340 in the block 304A are the odd lines in the macroblock 304 moved according to the motion vector 306. The odd lines 344 in the macroblock 304 have been "uncovered" due to motion of the macroblock 304. The values of the lines 344, however, can be derived using the difference vector associated with the macroblock 304. The difference vector indicates how pixel values in the macroblock 304 change after the move indicated by the motion vector 306. From the difference vector, the uncovered pixels remaining in the macroblock 304 can be calculated.

The even lines 342 in the block 304 A may be interpolated from the odd lines 340 above and below the even lines. In the macroblock 304, the uncovered even lines 346 are interpolated from the adjacent odd lines 344.

Next, the process determines (at 512) if all frame portions of the current frame have been processed. If not, then the next frame portion is processed (at 516). If all frame portions have been processed, then the progressive scanned frame is sent (at 514) to the controller 20 for further processing. Thus, a system according to some embodiments performs conversion of interlaced scanned video data to progressive scanned video data using indications of movement that have been encoded with the video data at the data source. For example, with digitally encoded video data such as MPEG-2 data, motion vectors are encoded with video data to reduce redundancies. Such indications of movement are provided either at the source or by some intermediary where relatively sophisticated equipment may be available to accurately detect motion of successive frames. Using the indications of movement, video data from a previous frame may be copied into missing lines of a current frame to generate a progressive scanned video frame. Inteφolation or other calculation of unknown data may thus be limited to instances of new data that may be newly presented in the current frame. By using indications of motion that have already been incorporated into video data, some embodiments of the invention allow the conversion of video data to be performed by relatively simple components, which may include hardware, software, or a combination of both. In addition, because the indications of motion associated with the video data are more accurate than would be achievable at the receiving end with less sophisticated equipment, higher quality conversion results may be attained.

Various software or firmware layers including applications, routines, or microcode may be stored or otherwise tangibly embodied in one or more machine-readable storage media in the system 10. Storage media suitable for tangibly embodying software or firmware instructions may include different forms of memory including semiconductor memory devices such as dynamic or static random access memories, erasable and programmable readonly memories (EPROMs), electrically erasable and programmable read-only memories (EEPROMs), and flash memories; magnetic disks such as fixed, floppy and removable disks; other magnetic media including tape; and optical media such as CD or DVD disks. The instructions stored in the one or more storage media when executed cause the system 10 to perform programmed acts.

The software or firmware can be loaded into the system 10 in one of many different ways. For example, instructions or other code segments stored on one or more storage media or transported through a network interface card, modem, or other interface mechanism may be loaded into the system 10 and executed to perform programmed acts. In the loading or transport process, data signals that are embodied as carrier waves (transmitted over telephone lines, network lines, wireless links, cables and the like) may communicate the instructions or code segments to the system 10.

While the invention has been disclosed with respect to a limited number of embodiments, those skilled in the art will appreciate numerous modifications and variations therefrom. It is intended that the appended claims cover all such modifications and variations as fall within the true spirit and scope of the invention.

Claims

What is claimed is: 1. A system comprising: a storage device containing successive pictures of video data according to a first format, the successive pictures containing one or more indications of movement between successive pictures; and a controller adapted to convert the video data from the first format to a second, different format by combining portions of the successive pictures of video data using the one or more indications of movement.

2. The system of claim 1, wherein the successive pictures include digitally encoded video data and motion vectors indicating movement of portions of the video data.

3. The system of claim 2, wherein the digitally encoded video data is defined by a standard of the Motion Picture Experts Group.

4. The system of claim 1, wherein the first format includes an interlaced scanned format.

5. The system of claim 1, wherein the second format includes a progressive scanned format.

6. A system comprising: a storage device containing video data encoded with one or more indications of movement, the video data being in a first format; and a controller adapted to perform conversion of the video data from the first format to a second, different format based on the one or more indications of movement.

7. The system of claim 6, wherein the first format includes an interlaced scanned format.

8. The system of claim 6, wherein the second format includes a progressive scanned format.

9. The system of claim 6, wherein the video data includes digitally encoded video data.

10. The system of claim 9, wherein the video data has a format according to a Motion Picture Experts Group standard.

11. The system of claim 6, further comprising: a receiver adapted to receive video data according to the first format; and a monitor adapted to display video data according to the second format.

12. The system of claim 6, wherein the video data according to the first format includes a first picture and a second picture received prior to the first picture, the controller adapted to combine the first picture and the second picture to form video data according to the second format.

13 The system of claim 12, wherein the controller is adapted to combine the first and second pictures based on the one or more indications of movement.

14. A method of modifying video data according to a first format in a system, comprising: retrieving one or more indications of movement encoded with the video data; and converting the video data from the first format to a second, different format based on the one or more indications of movement.

15. The method of claim 14, further comprising: storing the video data according to the first format as a first and a second picture, the second picture being received prior to the first picture; and copying at least a portion of the second picture into the first picture based on the one or more indications of movement to form the video data according to the second format.

16. The method of claim 14, wherein the converting includes converting the video data from an interlaced scanned format to a progressive scanned format.

17. The method of claim 16, wherein the storing includes storing the first picture including odd lines and the second picture including even lines.

18. A method of converting a video picture including interlaced scanned data to a video picture including progressive scanned data, comprising: receiving successive video pictures including interlaced scanned data, at least one video picture including one or more indications of movement; and combining the video picture including one or more indications of movement with a previously received video picture based on the one or more indications of movement to form the video picture including progressive scanned data.

19. An article including one or more machine-readable storage media containing instructions for modifying video data in a system, the instructions when executed causing the system to: retrieve one or more indications of movement from video data in a first format; and convert the video data based on the indications of movement from the first format to a second, different format.

20. The article of claim 19, wherein the one or more storage media contain instructions that when executed cause the system to further: capture at least two successive pictures of the video data in the first format; and combine portions of the two successive pictures based on the indications of movement to form video data in the second format.