WO2012044272A1

WO2012044272A1 - Automatically switching between three dimensional and two dimensional contents for display

Info

Publication number: WO2012044272A1
Application number: PCT/US2010/002629
Authority: WO
Inventors: Tao Zhang; Shan He; Dekun Zou; Jill Macdonald Boyce
Original assignee: Thomson Licensing
Priority date: 2010-09-29
Filing date: 2010-09-29
Publication date: 2012-04-05

Abstract

A method and system for automatic switching between three dimensional (3D) and two-dimensional (2D) contents on a 3D display system includes determining if a received video frame content is in a 3D format or 2D format. If the received video frame is in a 2D format, then the brightness of the received video frame is adjusted and the brightness adjusted video frame is converted into a 3D format for display. If the received video frame content is in 3D format, then the received video frame is displayed after converting the received 3D video frame to a 3D format compatible with the display.

Description

AUTOMATICALLY SWITCHING BETWEEN THREE DIMENSIONAL AND TWO DIMENSIONAL CONTENTS

FOR DISPLAY

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This invention is related to Application Number PCT/US2009/006454, entitled "Method For Distinguishing A 3D Image From A 2D Image And For Identifying The Presence Of A 3D Image Format By Feature Correspondence Determination", and Application Number PCT/US2009/006469, entitled "Method For Distinguishing A 3D Image From A 2D Image And For Identifying The Presence Of A 3D Image Format By Image Difference Determination", both filed 9 December 2009, wherein the entire disclosure of both commonly assigned applications are included by reference in their entirety for all purposes.

FIELD

[0002] The present invention relates to the display of 3D or 2D images, specifically, the invention related to the automatic and user-transparent display of images that transition from 3D to 2D or the reverse.

BACKGROUND

[0003] Recently, some television (TV) channels, such as the commonly known sports channel ESPN, have announced three-dimensional (3D) TV programming. It is predictable that more and more TV channels will provide 3D programs in the near future.

[0004] Currently, 3D contents are scarce and cannot fill in all the time slots for a 3D TV channel's programming needs. Therefore, it is very likely that a significant amount of two- dimensional (2D) content may transmitted (or aired) on a TV channel along with 3D content. Currently, many TV commercials that are aired may only have a 2D version of the commercial. This is similar to the situation where many standard-definition (SD) commercials are still been aired on high-definition (HD) TV channels. As a result, 2D and 3D contents will be displayed sequentially and the switching between these two types of contents can be quite frequent. For example, a program on a TV channel may have 2D commercials inserted into a 3D movie at multiple locations.

[0005] At least one optical disc standard association, the Blu-ray Disc Association, has recently announced that 1080p frame sequential format will be its official output format. The 1080p frame sequential format includes left and right views that are displayed alternatively in full frames. However, it is likely that broadcast industry will stick to one or more of the existing 3D formats such as the side-by-side format, the over-under format and/or the checkerboard pattern so that both left and right view images may be stored in a single HD frame. When the contents in 3D coexist with 2D contents in a channel, it is very possible that the 3D contents are in certain 3D formats in which left and right views can be encoded into a single frame, while the 2D contents are in regular 2D format.

[0006] Different 3D display systems present 3D in different native 3D display formats, which is the format that the display shows an image in 3D without any further processing of the images in its 3D mode. Such native formats include the horizontal interlaced format and the checkerboard pattern, among others. It is commonly accepted that users of 3D systems have to change settings on a TV or Set top box (STB) or to put on or take off 3D glasses in order to properly view a 3D or 2D program respectively. No widely accepted method is in place for a TV display to switch between 3D and 2D contents on different TVs, such as on a 3D TV or non-3D TV. It is noted that switching and conversion may be done at the end user side such as on STB or TVs, but it is very likely that they will happen just before broadcasting.

SUMMARY

[0007] The current invention addresses the above-mentioned problems by introducing a technology that allows TV displays to automatically (i.e. user-transparently) switch between 2D and 3D contents on a 3D channel when not all contents are delivered in 3D format. This technology includes 2D content detection in a 3D channel, 3D format detection, format conversion between 2D and 3D, and display of the incoming video frames in a native display system format.

[0008] A method and system for automatic switching between three dimensional (3D) and two-dimensional (2D) contents on a 3D display system includes determining if a received video frame content is in a 3D format or 2D format. If the received video frame is in a 2D format, then the brightness of the received video frame is adjusted and the brightness adjusted video, frame is converted into a 3D format for display. If the received video frame content is in 3D format, then the received video frame is displayed after converting the received 3D video frame to a 3D format compatible with the display.

[0009] Additional features and advantages of the invention will be made apparent from the following detailed description of illustrative embodiments which proceeds with reference to the accompanying figures. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 illustrates an example method to make automatic and user-transparent adjustments for a 3D image system to display an image that transitions between 3D and 2D or the reverse according to the invention;

Figure 2 illustrates an example method to make automatic and user-transparent adjustments for a 2D image system to display an image that transitions between 3D and 2D or the reverse according to the invention; and

Figure 3 illustrates an example apparatus serving as an environment for operation of the invention.

DETAILED DISCUSSION OF THE EMBODIMENTS

[0011] In one embodiment, an automatic detection, switching, and display using a 3D display system is described where the broadcasted content changes from 3D content to 2D content or the reverse. In another embodiment, an automatic detection, switching, and display using a 2D display system is described where the broadcasted content changes from 2D content to 3D content or the reverse. The present invention has the advantage that no active user intervention is needed to properly display a 2D image on a 3D display when the incoming video frames change from 3D to 2D or the reverse. That is, using the present invention, the user need not remove his 3D glasses during 3D to 2D program transitions or actively make a mode adjustment on the 3D display system. The same is true for 2D to 3D program transitions.

[0012] Figure 1 describes an example adjustment algorithm for 3D TV operation under 3D mode. The Figure 1 method 100 example adjusts one received frame for display on a 3D display system. The method 100 starts at step 105 where a frame of incoming video is received. At step 110, the frame is analyzed to determine if the frame is a 3D frame or a 2D frame.

[0013] One technique for performing this detection is disclosed in International Application Number PCT/US 2009/006454, filed December 9, 2009, entitled "Method For Distinguishing A 3D Image From A 2D Image And For Identifying The Presence Of A 3D Image Format By Feature Correspondence Determination", the entire disclosure of which is included by reference in its entirety for all purposes. One technique to determine a 3D frame format is by identifying the presence of a three-dimensional (3D) image format for a received image through the use of feature matching and correspondence. The received image is sampled using a candidate 3D format to generate two sub-images from the received image. Initially, these sub-images are compared to determine whether these sub-images are similar with respect to structure. If the sub-images are not similar, a new 3D format is selected and the method is repeated. If the sub-images are found to be similar, features are detected in the two sub-images and a correspondence is formed between features that match in the two sub- images. Positional differences are computed between corresponding features. The amount and uniformity of the positional differences are then used to determine whether the format is 2D or 3D and, if 3D, which of the 3D formats was used for the received image. When the format of the received image is determined, that format can be used to process and display the received image.

[0014] Another technique for performing this detection is disclosed in International Application Number PCT/US 2009/006469, filed December 9, 2009, entitled "Method For Distinguishing A 3D Image From A 2D Image And For Identifying The Presence Of A 3D Image Format By Image Difference Determination", the entire disclosure of which is included by reference in its entirety for all purposes. One technique to determine a 3D frame format is by identifying the presence of a three-dimensional (3D) image format for a received image through the use of image difference determination. In one embodiment, the received image is sampled using a candidate 3D format to generate two sub-images from the received image. When the candidate 3D format is a non-blended 3D format, these sub-images are compared to determine whether these sub-images are similar with respect to structure. If the sub-images are not similar, a new 3D format is selected and the method is repeated. If the sub-images are found to be similar or if the candidate 3D format is a blended 3D format, an image difference is computed between the two sub-images to form an edge map. Thicknesses are computed for the edges in the edge map. The thickness and uniformity distribution of the edges are then used to determine whether the format is 2D or 3D and, if 3D, which of the 3D formats was used for the received image. When the format of the received image is determined, that format can be used to process and display the received image.

[0015] If the frame is determined to be a 2D frame, then step 130 is performed. At step 130, brightness compensation is performed on the 2D frame content. Brightness compensation is necessary if a blending-type of 3D frame is the native frame format for the 3d display. Blending formats include formats such as an interlaced format and checkerboard format. A blending format is considered in contrast to a non-blended format, wherein, for example, a left portion of a frame is positioned next to a right portion of a frame within a single frame. In a blended format, pixels from one view are always surrounded by pixels from the other view. Step 130 is performed to adjust the brightness of the frame so that the 2D frame, when presented on a 3D display, has the same brightness as other 3D frames. [0016] After brightness adjustment of the 2D frame at step 130, the 2D frame is converted to the native 3D frame format of the 3D display system at step 135. Format conversion from 2D to 3D may be done in different ways. One simple conversion method involves conversion from a 2D to a 3D horizontal interlaced 3D format. In this example, the even lines of the 2D frame may be placed as the left view of the 3D frame and the odd lines of the 2D frame may be put as the right view of the 3D frame. Similarly, conversion to other 3D formats may be done by considering the characteristics of that 3D format and accommodating those characteristics in the 2D to 3D format conversion. The 2D to 3D format conversion allows the incoming 2D frame to be displayed using the native 3D display system equipment.

[0017] After conversion of the 2D frame to a 3D native format of the particular display system used, then method 300 moves to step 125 where the converted video frame is displayed on the 3D system equipment.

[0018] Returning to step 110, if the result of the determination is that the incoming frame content is a 3D frame, the process 100 moves to step 115 where it is determined if the 3D frame content is the specific native format of the 3D display system equipment being used. If the frame format is the native format, then the 3D frame is displayed at step 125. If the frame format at step 1 15 is some other, non-native format, then step 120 is conducted to convert the non-native 3D frame to a native 3D frame. After the conversion at step 120, the 3D frame can be displayed at step 125. This is generally accomplished by moving the 3D frame into a frame buffer or the like in the 3D display equipment to present the frame to the 3D display device.

[0019] From step 125, the method 100 can be repeated using the next frame or may end if more frames are not available. In another embodiment, a batch mode may be implemented where a number n of frames could be read and then processed according to the method 100 of Figure 1.

[0020] Figure 2 depicts a method 200 for adjustment of one frame for a 2D display system that is receiving a 3D broadcast. This situation could occur in a 3D TV that is operated in a 2D mode. The method 200 starts at step 205 by receiving a frame of incoming video information. The frame content is tested at step 210 to determine if the received frame was received as a 2D frame. In one embodiment, the determination at step 210 can be made by employing the technique of International Application Number PCT US 2009/006454 or PCT US2009/06469 as mentioned above.

[0021] If the frame is determined to be a 2D frame at step 210, then step 220 is performed where the 2D frame is displayed in a native 2D format. If the frame is determined to be a 3D frame at step 210, then step 215 converts the 3D frame to a 2D format compatible with the TV display system being used. Format conversion from 3D to 2D may be done in different ways. In one embodiment of a 3D to 2D format conversion, a side-by-side 3D format is converted to a 2D format by selecting either the left view or the right view of a side-by-side 3D image, scaling the resolution or other factors as needed of selected image, then using the scaled image as the 2D image.

[0022] Although the methods of Figures 1 and 2 describe an algorithm on a frame by frame base, it is also reasonable to apply the either or both methods 100 and 200 only when program switching occurs. This is possible since usually all the frames of one program will have the same format. Once a decision has been made at the beginning of a program, all the frames of the same program will use the same settings. The information of program switching is available through metadata in the TV stream.

[0023] Figure 3 depicts a device 300 useful to receive input video information and perform automatic switching between 3D and 2D contents according to aspects of the present invention. Device 300 may be part of a TV, a STB, or other equipment in a 3D display system.

[0024] Input video 305 is received by the receiver function 315 of device 300 and provides video frame information that may be stored in memory 320 or in a frame buffer. CPU/Arithmetic unit 325 may serve as a controller and/or processing unit that performs the automatic or transparent switching methods described herein, such as described with respect to Figures 1 and 2. The user remote control 310 is useful as an interface to allow the user to select different program that may be in 3D, 2D or a combination of 3D and 2D, where the program format may change over time. Memory 320 may be used as a store for instructions which can execute the methods described herein or may be used for the conversion of video frame data. Alternately or in conjunction, the CPU 325 uses output driver 330, such as a frame buffer, to drive a display device 350 as needed for the automatic (i.e. user-transparent) switching process from 3D to 2D or the reverse.

[0025] The implementations described herein may be implemented in, for example, a method or process, an apparatus, or a combination of hardware and software. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed may also be implemented in other forms (for example, a hardware apparatus, hardware and software apparatus, or a computer-readable media). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, an apparatus such as, for example, a processor, which refers to any processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processing devices also include communication devices, such as, for example, computers, cell phones, portable/personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end-users.

[0026] Additionally, the methods may be implemented by instructions being performed by a processor, and such instructions may be stored on a processor or computer-readable media such as, for example, an integrated circuit, a software carrier or other storage device such as, for example, a hard disk, a compact diskette, a random access memory ("RAM"), a read-only memory ("ROM") or any other magnetic, optical, or solid state media. The instructions may form an application program tangibly embodied on a computer-readable medium such as any of the media listed above. As should be clear, a processor may include, as part of the processor unit, a computer-readable media having, for example, instructions for carrying out a process. The instructions, corresponding to the method of the present invention, when executed, can transform a general purpose computer into a specific machine that performs the methods of the present invention.

Claims

CLAIMS:

I . A method performed by an apparatus for automatic switching between three dimensional (3D) and two-dimensional (2D) contents on a 3D display system, the method comprising:

determining if a received video frame content is in a 3D format or 2D format;

if the received video frame is in a 2D format:

converting the received video frame into a 3D format; and

displaying the converted video frame on the 3D display system; if the received video frame content is in 3D format:

displaying the 3D frame on the 3D display system.

2. The method of claim 1, wherein if the received video frame is in a 2D format, then the received video frame is adjusted in brightness before the step of converting the received video frame into a 3D format.

3. The method of claim 1, wherein the step of converting the brightness adjusted video frame into a 3D format comprises converting into a native 3D format compatible with the 3D display system.

4. The method of claim 1, wherein the step of displaying the 3D frame on the 3D display system comprises determining if the 3D frame is in a native 3D format compatible with the 3D system, and if not, then converting the 3D frame into a native 3D format compatible with the 3D display system before displaying the 3D frame on the 3D display system.

5. The method of claim 1, wherein the step of determining if a received video frame content is in a 3D format or 2D format comprises comparing features in sub-images to determine if the received video frame is a 3D format.

6. An apparatus for switching between three dimensional (3D) and two-dimensional (2D) contents on a 3D display system, the apparatus comprising:

a receiver to receive a frame of video data;

a processor to determine if the frame of video data is in 3D format or 2D format, and to automatically convert a 2D format into a 3D format if the received frame is in a 2D format; and

a display device to display the 3D formatted video frame.

7. The apparatus of claim 6, wherein the processor, upon determining that the received video frame is a 2D video frame, acts to adjust the brightness of the received video frame and convert the brightness adjusted video frame into a 3D format for display using the 3D display system.

8. The apparatus of claim 6, wherein the processor, upon determining that the received video frame is a 3D video frame, acts to convert the received 3D video frame to a 3D format that is compatible with the 3D display system.

9. The apparatus of claim 6, wherein the processor determines if received video frame content is in a 3D format or 2D format by comparing features in sub-images of the received video frame.