WO2007018535A1

WO2007018535A1 - System and method for displaying widescreen format images using a standard format display

Info

Publication number: WO2007018535A1
Application number: PCT/US2005/027725
Authority: WO
Inventors: David Jon Nicoson; Scott Allan Kendall; Ronald Lynn Blair
Original assignee: Thomson Licensing SAS
Current assignee: Thomson Licensing SAS
Priority date: 2005-08-04
Filing date: 2005-08-04
Publication date: 2007-02-15
Anticipated expiration: 2008-02-04

Abstract

A method and system for displaying widescreen video images. A stream of multimedia content (435) is received. A difference Ave in amplitudes of right R(t) and left L(t) channel audio signals contained in a portion of the stream of multimedia content is evaluated. A display sector identifier S(t) is determined correlating to the difference in amplitudes: The current display sector is also determined. A pan direction for panning within a presentation is determined from the current display sector and the display sector identifier.

Description

SYSTEM AND METHOD FOR DISPLAYING WIDESCREEN FORMAT IMAGES USING A STANDARD FORMAT DISPLAY

Field of the Invention [0001] The present invention generally relates to video display systems, and more particularly to display systems for displaying widescreen video using standard video display systems.

Background of the Invention [0002] Widescreen video typically has a 16x9 aspect ratio. Standard television signals, such as National Television System Committee (NTSC) and phase alternation by line (PAL) video signals, are typically presented with a 4x3 aspect ratio. Thus, widescreen video images are modified for presentation on standard television systems. Modification of the images sometimes results in a presentation that lacks portions of the images intended for display. Therefore, the artistic intent of the creator of the video presentation is not always adequately achieved when widescreen video images are modified for standard television systems. [0003] Therefore, systems and methods for displaying images while preserving, as much as possible, the intended artistic content of the presentation are needed.

Summary of the Invention

[0004] The present invention relates to a system and method for displaying widescreen video images on a standard television system. According to an embodiment of the invention, a stream of multimedia content is received. A difference between right and left channel audio signals is determined. A display sector identifier is determined correlating to the difference in amplitudes. The current display sector also is determined. A pan direction for panning within a presentation is determined from the current display sector and the display sector identifier.

Brief Description of the Drawings

[0005] Embodiments of the present invention will be described below in more detail, with reference to the accompanying drawings, in which: [0006] Fig. 1 is a pictorial diagram representing a video frame useful for understanding the present invention.

[0007] Fig. 2 is a pictorial diagram representing a video frame useful for understanding the present invention. [0008] Fig. 3 is a flowchart illustrating a pan and scan method according to an embodiment of the invention.

[0009] Fig. 4 is a block diagram of a pan and scan circuit according to an embodiment of the present invention.

[0010] Fig. 5 illustrates operations of a pan processor of Fig. 2 according to an embodiment of the invention.

[0011] Fig. 6 illustrates operations of a pan transition dampener of Fig. 2 according to an embodiment of the invention.

Detailed Description [0012] The present invention relates to a method and a system for displaying widescreen format multimedia content on standard display systems. The invention is useful, for example, for displaying with a second aspect ratio, content recorded with a first aspect ratio wider than the second aspect ratio. In an embodiment of the invention, the method and system is used to display multimedia content recorded with a 16x9 aspect ratio on a television designed to display content having a 4x3 aspect ratio.

[0013] Fig. 1 illustrates an example widescreen formatted frame of a video image 100 as it would appear on a widescreen display. Video image 100 comprises a first image portion 110 and a second image portion 121. First image portion 110 corresponds to a standard 4x3 aspect ratio display. This portion is referred to hereinafter as a "standard image portion". A second image portion 121 remains outside image portion of frame 100. Second image portion 121 represents an image portion of the video frame 100 not displayable on a standard 4X3 display system at the time of display of standard image portion 110. In the example of Figure 1 , standard image portion 110 is depicted as comprising a leftmost portion of the widescreen formatted image 100. However, this is by way of illustration only. According to embodiments of the invention, standard image portion 110 is selectable from any portion of the widescreen image 100. This is in contrast to conventional systems that rely on the same location for the standard image portion to be displayed from frame to frame.

[0014] By way of illustration of one embodiment of the invention, a frame of a widescreen image 100 is illustrated in Fig. 1. The widescreen image of frame 100 comprises an image including a first person 120 and a second person 230. In this example, person 120 would appear to a viewer of the video frame on Standard display system because person 120 is included in the standard image portion 110 of frame 100. However, person 230 would not appear during presentation of this frame because person 230 is included in an image portion outside standard image portion 110 of frame 100. In conventional systems, person 230 would appear in the displayed image during presentation of video images only if person 230 changed position during recording of the video so as to move into first image portion 110. In that case person 230 would be displayed during subsequent frames. [0015] The present invention selects standard image portions of widescreen image frames for display based upon the location, within widescreen image 100, of a center of interest. A center of interest is determined based upon audio information included in multimedia video data-stream. According to embodiments of the invention, when person first standard image portion corresponding to a first displayed frame differs from a second standard image portion corresponding to a second displayed frame.

[0016] To illustrate an embodiment of the invention, assume person 120 is speaking. According to an embodiment of the invention speech determines a center of interest. A "center of interest" refers to a locus within the widescreen video image 100. The locus is used to define a standard image portion 110 of the widescreen video image 100. For example, a standard image portion is determined by defining a region about the locus that corresponds to an image portion suitable for display on a standard television display with a 4X3 aspect ratio. The standard image portion including the locus provides a video image likely to be of interest to a person viewing the video frame 100 on a standard display system. In this case, when person 120 speaks, a first center of interest has a locus at the speaker. The displayed standard image portion including the speaker, person 120. [0017] Figure 2 illustrates a second person 230 in a second portion 240 of the same frame 100. The second portion 240 is represented in Fig. 2 by a rightmost portion of the video frame 100 and is represented having a 4x3 aspect ratio. In one embodiment of the invention, the second portion 240 generally includes the entire height of the video frame 100, but has a width limited to the rightmost 75% of the video frame 100. When the second person 230 begins speaking, a second standard image portion 240 is defined by a second locus defining a second center of interest. A system of the invention pans the video presentation from the first locus and its corresponding region of interest, to the second locus and its corresponding region of interest. The system accomplishes this by selecting a standard image portion from the video frame 100 based on the first locus, and displaying the selected portion, the displayed portion including person 120 when person 120 is speaking. When person 230 speaks, a second locus defining a standard image portion including person 230 is determined. The distance between the first locus and the second locus is referred to herein as a pan offset.

[0018] According to an embodiment of the invention, a locus defining a region of interest is determined by comparing audio signals during presentation of the multimedia content. In another embodiment, the position of closed captioning information determines a locus. [0019] The pan offset therefore determines an amount of pan required to pan from a currently displayed standard image portion to a subsequent displayed standard image portion, when the first standard image portion has a locus different from the second standard image portion. As defined herein, a pan is a movement of a viewing window as defined by a current standard image portion, to display areas in a video presentation which lie outside the currently selected standard image portion. The invention is applied so that the video presentation is displayed as if the camera pans from a first locus defining a first region of interest to a second locus defining a second region of interest shown in Fig. 2. [0020] Fig. 3 shows a flow chart 300 according to an embodiment of the invention. Beginning at step 310, a stream of multimedia content is received by a display system. The stream of multimedia content is a bitstream of digital data containing both video and audio data, typically in a compressed format, such as MPEG. The audio data contains amplitude and frequency information for two or more audio channels corresponding to left and right audio transducers. The video data provides video images in widescreen format. According to one embodiment of the invention, the amplitude information from at least two channels is evaluated to determine a locus within the video image, and thereby defines a standard image portion of video image for display on a standard display system. When the amplitude information associated with the received video image indicates a change in locus within the widescreen format video image, the system determines a pan offset applicable to define a new standard image portion for display to a viewer. For example, left and right audio channels from the portion of the stream are evaluated, as shown in step 320. A difference between the average amplitude of the audio signals on the right and left channels is used to determined a locus, as shown in step 330. [0021] Continuing at step 340, a display sector identifier representing a standard image portion for display is determined from the measured difference in the average amplitudes. The display sector identifier correlates to the standard image portion of multimedia content likely to be most interesting. In one embodiment of the invention, display sectors are defined as follows: left sector, middle sector and right sector. For this example, the left sector is the leftmost portion of the video frame and the right sector is the rightmost portion of the video frame. The middle sector is a center portion of the video frame that generally includes the entire height of the video frame, but has a limited width. For instance, in the case that the received stream of multimedia has a 16x9 aspect ratio and the display aspect ratio is 4x3, the middle sector includes the centermost 75% of the video frame width and a center of the middle sector correlates to a center of the video frame.

[0022] In an embodiment in which the stream of multimedia is an MPEG-2 stream, the aspect ratio of the received multimedia is defined in the pel_aspect_ratio field. As noted, the aspect ratio of typical widescreen formatted multimedia content is 16x9. However, there are numerous other aspect ratios with which the present invention can be used and the present invention is not limited in this regard. [0023] Referring to step 350, a currently displayed sector is determined. A current display sector is one of the left, middle, or right sector. The current display sector corresponds to a standard image portion currently being displayed on a standard format display system based on a first locus. Proceeding to step 360, when the locus changes a pan direction and a pan offset for panning from the current display sector to the display sector including a second locus is determined. Referring to step 370, a pan transition time also is determined. The pan transition time is the time for the pan to move from a display sector including a first locus to a display sector including a second locus. In one embodiment the pan transition time is selected to be proportional to the amount of pan. In one embodiment the pan transition time is directly proportional to the amount of pan. In another embodiment, the pan transition time is selected based upon an average frequency of pans. The average frequency of pans is the number of pans in a specific time period divided by that time period. In yet another embodiment, the pan transition time can be a fixed time value, for instance 500 ms. The amount of pan to be implemented, the direction of pan and the pan transition time are pan parameters. [0024] Finally, the pan is applied within the presentation as shown in step 380 in the determined direction. The pan continues for the determined transition time to pan the determined amount of pan. In one embodiment of the invention, Jitter prevention is incorporated to insure a jitter-free pan. Pan dampening also is applied at the beginning and end portions of the pan to simulate acceleration and deceleration characteristics of a camera pan and overdamps the pan to prevent overshoot at a completion of a pan.

[0025] Fig. 4 is a block diagram of a pan circuit 400 according to an embodiment of the invention. A pan circuit 400 includes a first moving picture experts group (MPEG) decoder 405, a pan processor 410, a pan transition dampener 415, a pan controller 420, a buffer 425 and a second MPEG decoder 430. The pan circuit 400 is operatively connected to a display 490. In the case that the display is analog, the display includes a D/A converter to convert digital multimedia content to an analog format for presentation on the display. In an alternate arrangement, a separate D/A converter (not shown) is provided as a component of the pan circuit 400 or as a stand alone unit.

[0026] In one embodiment, functions of the pan circuit 400 are implemented as discrete components within the pan circuit 400. Other embodiments of the invention include various combinations of hardware and software, for example an application specific integrated circuit (ASIC) or processor and a computer program which implement the methods described herein. A computer program in the present context means any expression, in any language, code or notation, of a set of instructions intended to cause a system having an information processing capability to perform a particular function either directly or after either or both of the following: a) conversion to another language, code or notation; b) reproduction in a different material form. [0027] In one arrangement, the first MPEG decoder 405 decodes an audio portion of a stream of multimedia content 435. In an embodiment of the invention, audio information is digitally encoded in an MPEG format. Decoder 405 provides right and left channel audio information 440. Accordingly, in this arrangement the first MPEG decoder 405 optionally includes video processing capability. [0028] The pan processor 410 and pan transition dampener 415 determine pan parameters and provide these parameters to the pan controller 420 with a control signal 445. Pan parameters include, but are not limited to: the amount of pan to be implemented by the pan controller 420, the direction of pan and the pan transition time. As noted, in addition to determining the pan parameters, the pan processor 410 also prevents jittering of pan movement. Further, the pan transition dampener 415 provides dampening for the pan movement. In one arrangement, the pan processor 410 and the pan transition dampener 415 are implemented as a single digital signal processor (DSP) which processes firmware or software commands. Operation of the pan processor 410 and the pan transition dampener 415 are discussed in greater detail with reference to Figs. 5 and 6, respectively. [0029] The buffer 425 also receives the stream of multimedia content 435. The buffer 425 holds in the buffer the stream of multimedia content 435 for an amount of time approximately equal to the time (W_f) given by the equation: tbuff ⁼ tave/2 ⁺ Wans ⁺ tdec where t_aVe is a length of time over which the pan processor 410 averages timet_trans is a transition time that it takes for the pan to move from one display sector to another display sector; and t_dec is a difference in the time that it takes the second MPEG decoder 430 to decode an MPEG frame in comparison to a combined processing time for the first MPEG decoder 405 to decode the MPEG frame, the pan processor 410 to process the pan, and the pan transition dampener 415 to dampen the pan. Accordingly, a portion of decoded multimedia content 450 is not propagated to the pan controller 420 before the pan controller 420 has received the pan parameters applicable to that portion of the decoded multimedia content 450. Thus, the pan controller 420 can apply the pan parameters to the decoded multimedia content 450 when it receives the decoded multimedia content 450. [0030] The pan controller 420 is a pan controller that receives a pan control signal as an input to control how pan is to be applied to multimedia content. Referring to Fig. 5, a flow diagram of the operation of a pan processor according to one embodiment of the invention is shown. The pan processor determines the pan parameters to be applied by the pan controller and prevents pan jitter from occurring by insuring that a first pan operation is complete before a second pan operation is initiated. At step 505, the pan processor determines an amplitude difference A(t) between the amplitude of the right and left audio signals. For instance, in one arrangement the audio amplitude data on each channel is converted to the decibel (dB) scale, for example, based on a pre-defined reference amplitude level. The audio signal amplitude on each channel then is scaled from a dB value to a value between 0 and 1. A(t) is determined by subtracting the scaled value of the left audio channel L(t) from the scaled value of the right audio channel R(t).

[0031] At step 510 the pan processor averages A(t) over the period of time t_ave to generate an average amplitude difference (A_ave). The amount of time over which amplitude is averaged to obtain t_ave depends on the maximum acceptable frequency of pans within the multimedia presentation. For instance, if the maximum acceptable frequency of pans is once every 500 ms, then t_ave should be 500 ms.

[0032] Proceeding to step 515, a display sector identifier S(t) then is determined based upon A_ave- S(t) correlates to a display sector on a display. For instance, in one arrangement a display comprises three display sectors: a first display sector corresponds to a left portion of the display, a second display sector corresponds to a middle portion of the display, etc, and a right display sector. S(t) is assigned a value of 1 (first display sector) if -1≤Aave<-0.5. S(t) is assigned a value of 2 (second display sector) if -0.5<A_aVe≤-0.5. S(t) is assigned a value of 3 (third display sector) if 0.5<

[0033] Continuing to step 520, S(t) determines whether to pan from a first display sector to a second display sector and through how many sectors to pan. If the second display sector identifier S(t) is not equal to the current display sector identifier S(t-tt_ran_s), and the previous pan transition ("Trans") is complete, then a pan value C(t), is determined by C(t) = S(t) - S(t-t_trans)- C(t) represents a number of display sectors through which to pan. Thus, if S(t) = 1 and S(t-t_tran_s) = 3, then the pan value is computed as C(t) = S(t) - S(t-t_trans) = 1 - 3 = -2. Thus, a pan is from right to left through two display sectors. Further, according to an embodiment of the invention, a new pan value C(t) is processed after the Trans is set to "Complete." [0034] Referring to Fig. 6, a flow diagram of the operation of the pan transition dampener is shown. At step 605, the pan value C(t) is evaluated. If C(t) ≠ 0, which means that a pan is initiated, the current position of the display P(t) as it moves from a first display sector to a next display sector is given by the equation:

P(t) = P(Hrans) + (C(t)COS((t/t,_rans)Tr))N where P(t) is the position of the display, P(t- tt_rans) is the position of the display at a pan sector prior to initiation of the pan, and N is a normalizing factor that normalizes C(t) to the actual distance panned between display sectors. Implementation of the equation dampens the pan and simulates acceleration and deceleration characteristics of a camera pan. During the pan transition, the value "Trans" is maintained as "Not Complete." For example, "Trans" is set to a value of 1. When the pan transition is completed, "Trans" is set to "Complete." For example, "Trans" is set to a value of 0. At this point C(t) = 0 and thus P(t) = P(t-t_trans), which indicates that no position change takes place.

[0035] While the foregoing is directed to the preferred embodiment of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.

Claims

1. A method for displaying video images comprising: receiving a stream of multimedia content comprising video images characterized by a widescreen format and at least first and second channel audio signals; determining a difference between said first and said second channel audio signals; selecting and displaying respective standard image portions corresponding to said video images on a standard format display system based on the difference.

2. The method according to claim 1 including steps of determining a pan direction and a pan offset for panning within said video images.

3. The method according to claim 2 further comprising: defining a pan transition time; and panning the pan offset in an amount of time approximately equal to the defined pan transition time.

4. The method according to claim 3, said panning step further comprising applying pan dampening during a portion of the pan.

5. The method according to claim 3, said panning step further comprising applying pan dampening during an end portion of the pan.

6. The method according to claim 1 further comprising: performing a first pan; determining when the first pan is complete; and performing a second pan after the first pan is complete.

7. A system for displaying widescreen video images comprising: at least one MPEG decoder for receiving a stream of multimedia content; a pan processor for determining a display sector identifier correlating to a difference in amplitude between left channel and right channel audio signals contained in a portion of the stream of multimedia content and, from the display sector identifier and a current display sector, determining a pan direction for panning within a presentation of the portion of the stream of multimedia content.

8. The system of claim 7 wherein said pan processor further determines a current display sector within the presentation and, from the determined display sector and the determined display sector identifier, determines a pan offset for panning within the presentation.

9. The system of claim 7 wherein said pan processor further initiates a first pan, determines when the first pan is complete, and initiates a second pan after the first pan is complete.

10. The system of claim 7 further comprising a pan dampener which receives data correlating to the pan from the pan processor and generates display position parameters for implementing the pan, the display position parameters defining a pan that is damped.

11. The system of claim 10 wherein the pan is damped during a portion of the pan.

12. The system of claim 10 wherein the pan is damped during an end portion of the pan.