WO2010050691A2 - Methods and apparatuses for processing and displaying image - Google Patents

Methods and apparatuses for processing and displaying image Download PDF

Info

Publication number
WO2010050691A2
WO2010050691A2 PCT/KR2009/006067 KR2009006067W WO2010050691A2 WO 2010050691 A2 WO2010050691 A2 WO 2010050691A2 KR 2009006067 W KR2009006067 W KR 2009006067W WO 2010050691 A2 WO2010050691 A2 WO 2010050691A2
Authority
WO
WIPO (PCT)
Prior art keywords
metadata
images
disk
generating
depth map
Prior art date
Application number
PCT/KR2009/006067
Other languages
English (en)
French (fr)
Other versions
WO2010050691A3 (en
Inventor
Kil-Soo Jung
Sung-Wook Park
Hyun-Kwon Chung
Dae-Jong Lee
Original Assignee
Samsung Electronics Co,. Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from KR1020080105485A external-priority patent/KR20100002048A/ko
Application filed by Samsung Electronics Co,. Ltd. filed Critical Samsung Electronics Co,. Ltd.
Priority to EP09823771A priority Critical patent/EP2319247A4/de
Publication of WO2010050691A2 publication Critical patent/WO2010050691A2/en
Publication of WO2010050691A3 publication Critical patent/WO2010050691A3/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/172—Processing image signals image signals comprising non-image signal components, e.g. headers or format information
    • H04N13/178—Metadata, e.g. disparity information
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/161—Encoding, multiplexing or demultiplexing different image signal components
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/189—Recording image signals; Reproducing recorded image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/356—Image reproducers having separate monoscopic and stereoscopic modes
    • H04N13/359—Switching between monoscopic and stereoscopic modes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/261—Image signal generators with monoscopic-to-stereoscopic image conversion
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/356—Image reproducers having separate monoscopic and stereoscopic modes

Definitions

  • aspects of the present invention relate to methods and apparatuses for processing and displaying images, and more particularly, to methods and apparatuses for processing and displaying images for determining whether three-dimensional (3D) image conversion is possible for each of a plurality of titles recorded in a disk, by using metadata for disk management and controlling an image displaying apparatus according to the result of the determination.
  • 3D three-dimensional
  • 3D image technology is a technology for displaying more realistic images by adding depth information to two-dimensional (2D) images.
  • 3D image technologies There are two types of 3D image technologies; technology for generating video data with 3D images and technology for generating 3D images from video data generated with 2D images. These two technologies are being researched and developed together.
  • aspects of the present invention provide a method for determining whether images in titles recorded in a disk can be converted to 3D images for each of a plurality of titles by using metadata for disk management, generating graphic information indicating whether images displayed on a screen are 2D images or 3D images, and automatically changing an image displaying mode of an image displaying device.
  • FIG. 1 is a diagram illustrating metadata regarding video data, according to an embodiment of the present invention
  • FIG. 2 is a diagram showing metadata for disk management
  • FIGS. 3A and 3B are diagrams for describing depth information included in the metadata for generating a depth map, that is, the metadata shown in FIG. 1, wherein FIG. 3A is a diagram for describing a depth provided to an image, and FIG. 3B is a diagram for describing a depth provided to an image when the image is viewed from a side;
  • FIG. 4 is a diagram of an image processing system for describing methods of processing and displaying images, according to an embodiment of the present invention
  • FIG. 5 is a block diagram of the image processing device of FIG. 4;
  • FIG. 6 is a block diagram of the 3D image converting unit of FIG. 5 in closer detail
  • FIG. 7 is a module view of a server transmitting metadata for disk management, according to an embodiment of the present invention.
  • FIGs. 8A and 8B are diagrams showing an image, on which graphic information is overlaid, output by an image processing device.
  • FIG. 9 is a flowchart for describing a method of displaying images, according to an embodiment of the present invention.
  • aspects of the present invention provide a method of processing images, the method including determining whether metadata for generating a depth map corresponding to a predetermined title exists for each of a plurality of titles recorded in a disk by using metadata for disk management, and converting two-dimensional (2D) images of the predetermined title to three-dimensional (3D) images by using the metadata for generating the depth map if metadata for generating the depth map corresponding to the predetermined title exists.
  • the conversion of the 2D images to the 3D images includes extracting depth information regarding a frame included in the predetermined title from the metadata for generating a depth map, generating a depth map by using the depth information, and generating a left image and a right image corresponding to the frame by using the depth map and the frame.
  • the metadata for disk management may include information regarding a location of the metadata for generating a depth map corresponding to the predetermined title.
  • the conversion of the 2D images to the 3D images may include receiving information, from an image displaying device, indicating whether an image displaying device can only display 2D images or can display both 2D images and 3D images, and converting 2D images in the predetermined title to 3D images if the image displaying device can display both 2D images and 3D images.
  • the method may further include extracting a disk identifier from the disk, transmitting the disk identifier to a server via a communication network, and downloading metadata for disk management corresponding to the disk identifier from a server.
  • the method may further include generating graphic information indicating that the 2D images in the predetermined title are converted to the 3D images and overlaying the graphic information on the 3D images. Furthermore, the method may, if an image displaying mode of the image displaying device is a 2D image displaying mode, further include generating a mode switching control signal for changing the image displaying mode of the image displaying device to a 3D image displaying mode and transmitting the mode switching control signal to the image displaying device. Furthermore, the method may, if metadata for generating a depth map corresponding to the predetermined title does not exist, further include generating graphic information indicating that images in the predetermined title are 2D images and overlaying the graphic information on the 2D images.
  • the method may, if an image displaying mode of the image displaying device is a 3D image displaying mode, further include generating a mode switching control signal for changing the image displaying mode of the image displaying device to a 2D image displaying mode and transmitting the mode switching control signal to the image displaying device.
  • the conversion of the 2D images to the 3D images may include extracting shot information from the metadata for generating a depth map, wherein the shot information is for classifying video frames included in the predetermined title into shots and converting frames classified as a predetermined shot from among video frames included in the predetermined title to 3D images by using the shot information, wherein the shot information is information for classifying a series of frames from which the composition of a current frame can be predicted based on the composition of a previous frame in a same group.
  • the metadata for generating a depth map may include shot type information indicating whether frames in a shot are to be displayed as 2D images or 3D images for each shot, and the conversion of the frames in the predetermined shot to the 3D images may include converting frames in the predetermined shot to the 3D images by using shot type information corresponding to the predetermined shot.
  • aspects of the present invention provide a method of transmitting metadata for disk management, performed by a server communicating with an image processing device via a communication network, the method including receiving a request for metadata for disk management corresponding to a predetermined disk from the image processing device, searching for the metadata for disk management corresponding to the predetermined disk by using a disk identifier of the disk, and transmitting the metadata for disk management corresponding to the predetermined disk to the image processing device, wherein the metadata for disk management corresponding to the disk indicates whether metadata for generating a depth map corresponding to each of a plurality of titles recorded in the disk exists and, if the metadata for generating a depth map exists, indicates a location of the metadata for generating a depth map corresponding to the titles.
  • aspects of the present invention provide an image processing device including a disk management metadata processing unit for determining whether metadata for generating a depth map corresponding to a predetermined title exists for a plurality of titles in the disk by using metadata for disk management, a unit for decoding metadata for generating a depth map, and a three dimensional (3D) image converting unit for converting two dimensional (2D) images in the predetermined title to 3D images by using the metadata for generating a depth map corresponding to the predetermined title if the metadata for generating a depth map corresponding to the predetermined title exists.
  • a disk management metadata processing unit for determining whether metadata for generating a depth map corresponding to a predetermined title exists for a plurality of titles in the disk by using metadata for disk management
  • a unit for decoding metadata for generating a depth map and a three dimensional (3D) image converting unit for converting two dimensional (2D) images in the predetermined title to 3D images by using the metadata for generating a depth map corresponding to the predetermined title
  • aspects of the present invention provide a server communicating with an image processing device via a communication network, the server including a transmitting/receiving unit receiving a request for metadata for disk management corresponding to a predetermined disk from the image processing device and transmitting the metadata for disk management corresponding to the predetermined disk to the image processing device in response to the request, a disk management metadata storage unit storing metadata for disk management corresponding to a disk, and a disk management metadata searching unit searching for the metadata for disk management corresponding to the predetermined disk by using a disk identifier of the disk, wherein the disk management metadata corresponding to the disk indicates whether metadata for generating a depth map corresponding to each of a plurality of titles recorded in the disk exists and, if metadata for generating a depth map exists, indicates a location of the metadata for generating a depth map corresponding to the titles.
  • aspects of the present invention provide a computer readable recording medium having recorded thereon a method of processing images, the method including determining whether metadata for generating a depth map corresponding to a predetermined title exists for a plurality of titles recorded in a disk by using metadata for disk management, and if metadata for generating a depth map corresponding to the predetermined title exists, converting two-dimensional (2D) images in the predetermined title to three-dimensional (3D) images by using the metadata for generating a depth map.
  • aspects of the present invention provide a method by which a server, which communicates with an image processing device via a communication network, transmits metadata for disk management, the method including receiving a request for metadata for disk management corresponding to a predetermined disk from the image processing device, searching for the metadata for disk management corresponding to the predetermined disk by using a disk identifier of the disk, and transmitting the metadata for disk management corresponding to the predetermined disk to the image processing device, wherein the metadata for disk management corresponding to the disk indicates whether metadata for generating a depth map corresponding to each of a plurality of titles recorded in the disk exists and, if metadata for generating a depth map exists, indicates a location of the metadata for generating a depth map corresponding to the titles.
  • FIG. 1 is a diagram illustrating metadata regarding video data, according to an embodiment of the present invention.
  • Metadata includes metadata for disk management and metadata for generating a depth map.
  • a plurality of titles may be recorded in a single disk.
  • a first playback title, a top menu title, and n movie titles may be recorded in a single disk.
  • the first playback title is the title read initially when the disk is loaded into an image processing apparatus (not shown), and metadata for generating a depth map is not necessary therein.
  • the top menu title is a title for providing menus corresponding to movie titles, and metadata for generating a depth map may not be necessary therein, too.
  • metadata for generating a depth map corresponding to the n movie titles may be required to playback the movie title in 3D images.
  • metadata for generating a single depth map is generated with respect to a single title.
  • metadata for generating a depth map corresponding to a single title there may be either metadata for generating a depth map corresponding to a single title or a plurality of pieces of metadata for respectively generating depth maps corresponding to the plurality of titles. Therefore, in the example above, the number of pieces of metadata for generating depth maps, with respect to the n titles, may be n.
  • Metadata for disk management is data for managing metadata for generating depth maps corresponding to a plurality of titles recorded in a disk, and includes disk identification information and information for managing metadata for generating depth maps corresponding to a plurality of titles recorded in a disk. Descriptions on metadata for disk management will be provided below with reference to FIG. 2.
  • Metadata for generating a depth map includes information for converting video data frames to 3D images.
  • Video data is formed of a series of frames, and thus metadata for generating a depth map includes information regarding the frames.
  • the information regarding frames includes information for classifying frames according to a predetermined standard.
  • frames of video data may be classified into a plurality of units.
  • metadata for generating a depth map includes information for classifying frames of video data into predetermined units.
  • Metadata for generating a depth map includes shot information for classifying frames of video data into shots.
  • shot information may include information regarding times at which a shot begins and ends.
  • a time at which a shot begins is a time point at which the first frame from among frames classified as a predetermined shot is displayed, and a time at which a shot ends is a time point at which the last frame from among the frames is displayed.
  • Metadata for generating a depth map may include shot type information corresponding to frames classified into one shot.
  • Shot type information is information instructing frames classified as one shot to be displayed as 2D images or 3D images.
  • Metadata for generating a depth map may further include information required for converting the frames to 3D images.
  • parallax can be positive parallax, zero parallax, or negative parallax.
  • Positive parallax refers to the case where the image is formed inside the screen and the parallax is equal to or smaller than the distance between the eyes.
  • Zero parallax refers to the case in which an image is projected flat on a screen two-dimensionally. In the case of zero parallax, an image is displayed flat on a screen, and thus a user cannot perceive the 3D effect.
  • Negative parallax refers to the case in which an image is projected in front of a screen. Negative parallax occurs when lines of sight of both eyes intersect each other and gives a 3D effect as if an object protrudes from the screen.
  • metadata for generating a depth map includes depth information, which is information for providing depth to frames.
  • the depth information is information to convert 2D images to 3D images by providing depth to frames, and includes depth information for background and depth information for objects.
  • Images of a frame include a background image and an image of objects not including the background.
  • Depth information for a background is information for providing depth to the background image.
  • Providing depth to the background image means providing depth to composition such as location or structure of the background.
  • Frames may have various compositions, and thus depth information for the background, which is included in metadata for generating a depth map, includes information regarding type of composition to indicate composition of the background frames. Furthermore, depth information for the background may include coordinate values of backgrounds, depth values corresponding to the coordinate values, and panel position values.
  • the coordinate values of the backgrounds refer to coordinate values of the backgrounds in frames of 2D images.
  • the depth values refer to the degree of depth to be provided to images, and metadata for generating a depth map includes depth values to be provided to each of a plurality of coordinates in frames of 2D images.
  • the panel position values refer to a position in a screen at which images are formed.
  • Depth information for objects is information used to generate a depth map regarding objects, such as people or buildings, other than the backgrounds.
  • Depth information for objects includes information regarding times of displaying the objects and object region information.
  • the times of displaying objects refer to time points of displaying frames in which the objects appear.
  • the object region information is information for indicating regions occupied by the objects, and may include information regarding coordinates to indicate the regions occupied by the objects, wherein the information regarding coordinates includes information regarding coordinates at which the objects and backgrounds meet. If required, a mask in which regions of the objects are indicated may be used as the object region information. Descriptions on depth information for background and depth information for the objects will be provided below in closer detail with reference to FIGS. 3 through 6.
  • metadata regarding video data includes metadata for disk management and metadata for generating a depth map.
  • FIG. 2 is a diagram showing metadata for disk management.
  • the metadata for disk management includes disk identification information.
  • the disk identification information is information for indicating with which disk the metadata for disk management is associated.
  • the metadata for disk management includes information indicating whether metadata for generating a depth map exists for each title and information indicating locations of metadata for generating a depth map corresponding to titles having metadata for generating a depth map.
  • the metadata for disk management includes information indicating whether metadata for generating a depth map exists for each of the five titles. Referring to FIG. 2, it is clear that metadata for generating a depth map does not exist for the first, second, and fifth titles, whereas metadata for generating a depth map exist for the third and fourth titles.
  • the metadata for disk management further includes information indicating locations of metadata for generating a depth map, that is, the metadata to be applied to the third and fourth titles.
  • the metadata for disk management may further include information indicating which metadata for generating a depth map, the metadata applicable to a predetermined disk, is to be applied to predetermined titles.
  • FIGS. 3A and 3B are diagrams for describing depth information included in the metadata for generating a depth map, that is, the metadata shown in FIG. 1.
  • FIG. 3A is a diagram for describing a depth provided to an image
  • FIG. 3B is a diagram for describing a depth provided to an image when the image is viewed from a side.
  • a depth is provided to a 2D flat frame by using depth information.
  • the X-axis direction which is a direction parallel to a line of sight of a user, indicates the degree of depth of a frame.
  • a depth value refers to the degree of depth of an image, and the depth value in the present invention may be one of 256 values, that is, from 0 to 255. As the depth value approaches zero, the depth of an image increases, and thus the image appears farther from a viewer. In contrast, as the depth value approaches 255, the image appears closer to a viewer.
  • the panel position refers to a location of a screen at which images are formed
  • the panel position value refers to a depth value of an image when parallax is zero, that is, when the image is formed on the screen.
  • the panel position value may be one depth value from among 0 to 255.
  • the panel position value is 255, all images in a frame have either the same depth value as the screen or a depth value less than that of the screen.
  • the images are formed away from a viewer, that is, inside the screen. In other words, the images in the frame have zero or positive parallax.
  • An object is either a person or a building standing parallel to the surface of the screen.
  • a depth value of an object is the same as that of a portion of the background at which the object and the background contact each other.
  • the depth value of an object may be the same as a panel position value. Depth values of objects are constant in a direction parallel to the surface of the screen.
  • FIG. 4 is a diagram of an image processing system for describing methods of processing and displaying images, according to an embodiment of the present invention.
  • the image processing system includes a server 100, an image processing device 200, and an image displaying device 300.
  • the image processing device 200 is a device for decoding video data, generating 2D video images, and either converting the 2D video images to 3D images by using metadata for disk management and transmitting the 3D images to the image displaying device 300 or transmitting the 2D video images to the image displaying device 300 without conversion.
  • the image processing device 200 may be a DVD player, a set-top box, or other similar devices.
  • the image displaying device 300 is a device for displaying images transmitted from the image processing device 200 on a screen, and may be a monitor, a TV, or other similar devices.
  • the image processing device 200 and the image displaying device 300 are shown as individual devices in FIG. 4.
  • the image processing device 200 may include a display unit, which is a unit for performing functions of the image displaying device 300, so that the image processing device 200 may also include the image displaying device 300 to display images on a screen.
  • the image processing device 200 is connected to the server 100 via a communication network.
  • the communication network may be a wired communication network and/or a wireless communication network.
  • the server 100 may be operated by a content provider such as a broadcast station or a general content generating company.
  • the server 100 stores content such as audio data, video data, text data, and metadata regarding the audio data, the video data, and the text data.
  • the image processing device 200 When a user turns on the image processing device 200 by using a user interface such as a remote control device (not shown), the image processing device 200 receives information, which indicates whether the image displaying device 300 can display 2D images only or the image displaying device 300 can also display 3D images. In the case where the image displaying device 300 can display 3D images, the image processing device 200 extracts disk identification information from a loaded disk, transmits the disk identification information to the server 100, and requests metadata corresponding to the disk identification information.
  • a user interface such as a remote control device
  • the server 100 determines whether metadata corresponding to a predetermined disk is stored in the server 100 by using disk identification information transmitted from a user, and, if the metadata is stored in the server 100, transmits the metadata to the image processing device 200.
  • the image processing device 200 may identify metadata downloaded from the server 100 corresponding to disk identification information and store the metadata in a predetermined location within the image processing device 200.
  • Metadata may be recorded in a disk in which video data is stored.
  • the metadata may be recorded in one or more of a lead-in area, a user data area, and a lead-out area of a disk.
  • metadata for disk management and metadata for generating a depth map may be stored separately.
  • metadata for disk management may be stored in the server 100 and metadata for generating a depth map may be stored in a disk, however, aspects of the present invention are not limited thereto.
  • the image processing device 200 generates 2D images by decoding titles recorded in a disk.
  • the image processing device 200 uses metadata for disk management either downloaded from the server 100 or extracted from a disk to determine whether metadata for generating depth maps for each of a plurality of titles recorded in a loaded disk exists.
  • the image processing device 200 searches for metadata for generating a depth map corresponding to the predetermined title by using information regarding a location of the metadata for generating a depth map.
  • the image processing device 200 converts 2D images of a predetermined title to 3D images by using metadata for generating a depth map corresponding to the predetermined title.
  • the image processing device 200 extracts depth information for the background and depth information for the objects corresponding to frames included in a predetermined title from metadata for generating a depth map, and generates depth maps for each of the background and the objects by using the extracted depth information.
  • the image processing device 200 generates a complete depth map by combining the depth map for the background and the depth maps for the objects, and generates left images and right images, that is, 3D images corresponding to 2D images by using the complete depth map and the 2D images.
  • the image processing device 200 may generate graphic information to indicate that the images transmitted to the image displaying device 300 are 3D images.
  • the graphic information may be emoticons, texts, images, etc.
  • the image processing device 200 overlays generated graphic information on the 3D images and transmits the 3D images to the image displaying device 300.
  • the image displaying device 300 displays video images, on which graphic information is overlaid, as 3D images.
  • a user can recognize that images currently displayed by the image displaying device 300 are 3D images by using graphic information overlaid thereon.
  • the image displaying mode of the image displaying device 300 is a 2D image displaying mode
  • a user may change the image displaying mode of the image displaying device 300 by using a remote control device (not shown), or the like, to display 3D images.
  • the image processing device 200 may automatically change the image displaying mode of the image displaying device 300.
  • the image processing device 200 determines whether the image displaying mode of the image displaying device 300 is set to the 2D image displaying mode or the 3D image displaying mode either by receiving information regarding the image displaying mode of the image displaying device 300 or by using information indicating whether images transmitted to the image displaying device 300 prior to a current image are 2D images or 3D images.
  • the image processing device 200 may generate a control signal switching the image displaying device 300 to the 3D image displaying mode and transmit the control signal to the image displaying device 300.
  • the image displaying device 300 can display 3D images.
  • the image processing device 200 determines that metadata for generating a depth map corresponding to a predetermined title does not exist, decoded 2D images are transmitted to the image displaying device 300 without conversion. Before transmitting the 2D images to the image displaying device 300, the image processing device 200 generates graphic information to indicate that the images transmitted to the image displaying device 300 are 2D images, overlays the generated graphic information on the 2D images, and transmits the 2D images to the image displaying device 300.
  • the image displaying device 300 displays video images on which graphic information indicating that the video images are 2D images are overlaid. A user may recognize that the images currently displayed by the image displaying device 300 are 2D images based on the graphic information overlaid on the video images. In the case where the image displaying mode of the image displaying device 300 is the 3D image displaying mode, a user may switch the image displaying device 300 to display 2D images by using a remote control device (not shown), or the like. Furthermore, as described above, the image processing device 200 may automatically recognize and change the image displaying mode of the image displaying device 300.
  • the image processing device 200 may generate a control signal instructing the image displaying device 300 to switch to the 2D image displaying mode and transmit the control signal to the image displaying device 300 such that the image displaying device 300 can display 2D images.
  • the image displaying device 300 sequentially displays a left-eye image and a right-eye image on a screen.
  • a user perceives that images are continuously and seamlessly displayed when images are displayed at least at a frame rate of 60Hz per eye.
  • the image displaying device 300 alternately displays left-eye images and right-eye images in frames every 1/120th second.
  • HDMI is a non-compressive digital video/audio interface standard, and provides an interface between devices supporting HDMI.
  • HDMI includes three communication channels; a transition minimized differential signaling (TMDS) channel, a display data channel (DDC), and a consumer electronics control (CEC) channel.
  • TMDS transition minimized differential signaling
  • DDC display data channel
  • CEC consumer electronics control
  • Transition minimized differential signaling (TMDS) on HDMI carries video, audio, and auxiliary data via one of three modes called the video data period, the data island period, and the control period.
  • the video data period the pixels of an active video line are transmitted.
  • the data island period (which occurs during the horizontal and vertical blanking intervals)
  • audio and auxiliary data are transmitted within a series of packets.
  • the control period occurs between video and data island periods.
  • the image processing device 200 may transmit a mode switching control signal to the image displaying device 300 to instruct the image displaying device 300 to be switched to either the 3D image displaying mode or the 2D image displaying mode.
  • a CEC line transmits control data transmitted for controlling devices using HDMI.
  • Control data transmitted via the CEC line may include information indicating that transmitted data is control data regarding mode switching, information instructing the image displaying device 300 to be switched to the 3D image displaying mode or the 2D image displaying mode, and an address of the image displaying device 300 which will receive the control data.
  • the image processing device 200 may generate graphic information indicating whether images to be displayed are 2D images or 3D images by using metadata for disk management, overlay the graphic information on the images, generate a control signal for switching the image displaying mode of the image displaying device 300 according to the current image displaying mode of the image displaying device 300, and transmit the control signal to the image displaying device 300.
  • the image displaying mode of the image displaying device 300 may be automatically switched even if a user does not manually switch the image displaying mode of the image displaying device 300.
  • FIG. 5 is a block diagram of the image processing device 200 of FIG. 4, according to an embodiment of the present invention.
  • the image processing device 200 includes a video data decoding unit 210, a disk management metadata processing unit 220, a unit 230 for decoding metadata for generating a depth map, a 3D image converting unit 240, a video image buffer 250, a graphic information processing unit 260, and a graphic information buffer 270, and a blender 280.
  • the image processing device 200 may further include a communication unit for exchanging data with the external server 100 via a communication network, and include a local storage unit for storing data downloaded via the communication unit.
  • the image processing device 200 may include a system time clock (STC) counter. The image processing device 200 decodes and outputs data according to the STC counter.
  • STC system time clock
  • the video data decoding unit 210 reads either video data from a disk or video data downloaded and stored in the local storage unit and decodes the video data.
  • the metadata for disk management processing unit 220 reads metadata from the disk.
  • the disk management processing metadata unit 220 extracts disk identification information from the disk and transmits the disk identification information to the server 100 via the communication unit (not shown).
  • the image processing device 200 may receive metadata regarding a predetermined disk and store the metadata in the local storage unit (not shown) in order using disk identification information.
  • the disk management metadata processing unit 220 determines whether metadata for generating a depth map corresponding to video data of a predetermined title is included for each title in the disk by using metadata for disk management.
  • the 3D image converting unit 240 transmits 2D images decoded by the video data decoding unit 210 to the video image buffer 250 without conversion.
  • the disk management metadata processing unit 220 controls the graphic information processing unit 260 to generate graphic information indicating that the images are 2D images.
  • the graphic information processing unit 260 transmits the generated graphic information to the graphic information buffer 270.
  • the disk management metadata processing unit 220 determines whether the image displaying mode of the image displaying device 300 is the 3D image displaying mode or the 2D image displaying mode. If it is determined that the image displaying device 300 is configured in the 3D image displaying mode, the disk management metadata processing unit 220 generates a mode switching control signal instructing the image displaying device 300 to be switched to the 2D image displaying mode and transmits the mode switching control signal to the image displaying device 300.
  • the video image buffer 250 and the graphic information buffer 270 temporarily store video images and graphic information, respectively.
  • STC presentation time stamp
  • the video image buffer 250 and the graphic information buffer 270 transmit the video images and the graphic information to a blender 280.
  • the blender 280 overlays the video images and the graphic information and transmits them to the image displaying device 300.
  • the disk management metadata processing unit 220 determines the location of particular metadata for generating a depth map corresponding to video data of the predetermined title.
  • the disk management metadata processing unit 220 controls the unit for decoding metadata for generating the depth map 230 to decode the metadata for generating a depth map corresponding to video data of the predetermined title.
  • the unit 230 decodes the metadata for generating a depth map and extracts depth information.
  • the 3D image converting unit 240 uses the depth information extracted by the unit 230 to generate a depth map corresponding to the 2D images decoded by the video data decoding unit 210, and converts the 2D images to 3D images by using the depth map.
  • the disk management metadata processing unit 220 controls the graphic information processing unit 260 to generate graphic information indicating that the images are 3D images.
  • the graphic information processing unit 260 transmits the generated graphic information to the graphic information buffer 270.
  • the disk management metadata processing unit 220 determines whether the image displaying mode of the image displaying device 300 is the 3D image displaying mode or the 2D image displaying mode. If it is determined that the image displaying device 300 is configured in the 2D image displaying mode, the disk management metadata processing unit 220 generates a mode switching control signal instructing the image displaying device 300 to be switched to the 3D image displaying mode.
  • the image processing device 200 transmits the images to be overlaid with graphic information indicating that the images are 3D images and the mode switching control signal to the image displaying device 300.
  • a predetermined title may include both the frames to be displayed as 2D images and the frames to be displayed as 3D images.
  • metadata for generating a depth map corresponding to the predetermined title only includes depth information with respect to frames to be displayed as 3D images.
  • the unit decoding metadata for generating the depth map 230 decodes metadata for generating a depth map and extracts shot information regarding frames of video data classified into a predetermined shot.
  • the unit 230 uses shot type information to determine whether the frames classified into a predetermined shot are to be displayed as 2D images or 3D images. When it is necessary to convert frames classified into a predetermined shot to 3D images, the unit 230 extracts depth information regarding the frames and transmits the depth information to the 3D image converting unit 240.
  • the unit 220 controls the graphic information processing unit 260 to generate graphic information indicating whether the images to be displayed are 2D images or 3D images, generates a mode switching control signal for changing the image displaying mode of the image displaying device 300, and transmits them to the image displaying device 300.
  • a user can recognize whether images currently displayed are 2D images or 3D images by using graphic information displayed together with the images. Furthermore, the image displaying mode of the image displaying device 300 may be automatically switched even if a user does not manually change the image displaying mode.
  • FIG. 6 is a block diagram of the 3D image converting unit 240 of FIG. 5 in closer detail, according to an embodiment of the present invention.
  • the 3D image converting unit 240 includes a background depth map generating unit 610, an object depth map generating unit 620, a filtering unit 630, and a depth map buffer unit 640, and a stereo rendering unit 650.
  • the background depth map generating unit 610 receives type information of a background, coordinate values of the background, depth values of the background corresponding to the coordinate values, and panel position values, which are included in depth information for the background, from the unit decoding metadata for generating the depth map 230, and generates a depth map corresponding to the background by using the received values.
  • the background depth map generating unit 610 transmits the depth map generated corresponding to the background to the filtering unit 630.
  • the object depth map generating unit 620 receives object identification information and object type information, which are included in depth information for an object, from the unit 230 and generates a depth map corresponding to the object.
  • the object identification information is information regarding a mask
  • the object depth map generating unit 620 receives the mask to be applied to a corresponding frame and generates a depth map corresponding to the object by using the mask.
  • the object depth map generating unit 620 transmits the depth map to the filtering unit 630.
  • the filtering unit 630 applies filters to the depth map corresponding to the background and the depth map corresponding to an object.
  • the depth map, corresponding to the object has a depth value corresponding to the surface of a screen.
  • the filtering unit 630 may apply filters to the object to provide a 3D effect to the object having depth values corresponding to the surface of a screen.
  • filters may be applied thereto to provide a 3D effect to the background.
  • the depth map buffering unit 640 temporarily stores a depth map corresponding to the background transmitted from the filtering unit 630.
  • the depth map buffering unit 640 combines the depth map corresponding to the background and the depth map corresponding to the object and updates a depth map corresponding to a frame.
  • the depth map buffering unit 640 updates a depth map by sequentially overlaying depth maps corresponding to the plurality of objects.
  • the depth map buffering unit 640 transmits the completed depth map to a stereo rendering unit 650.
  • the stereo rendering unit 650 generates a left-eye image and a right-eye image by using video images received from the video data decoding unit 210 and a depth map 640 received from the depth map buffer unit 640, and generates a 3D format image including both of the left-eye image and the right-eye image. Examples of 3D formats include top and down format, side by side format, and interlaced format.
  • the stereo rendering unit 650 transmits the 3D format image to the image displaying device 300.
  • FIG. 7 is a module view of a server transmitting metadata for disk management, according to an embodiment of the present invention.
  • a server 100 includes a plurality of application modules 740 including a transmitting/receiving processing module 741, a disk management metadata storage module 743, and a metadata for disk management searching module 745.
  • the transmitting/receiving processing module 741 processes communication with the image processing device 200
  • the disk management metadata storage module 743 stores and manages metadata for disk management, depth information for the background, video data, and other similar data and information.
  • the metadata for disk management searching module 745 uses disk identification information, which is transmitted from a user, to search for metadata for disk management having disk identification information requested by the user.
  • the server 100 may use various operating systems (OS) as a system OS.
  • the OS provides high level commands to an application program interface (API) 701 to control operations of each of the application modules 740.
  • the server 100 which includes a high level command processing unit 710, identifies the corresponding application modules 740 based on high level commands provided by the API 701, and decodes the high level commands and provides the decoded high level commands to a corresponding module of an application module control unit 720.
  • the application module control unit 720 controls operations of the application modules 740 according to a command provided by the high level command processing unit 710.
  • the high level command processing unit 710 determines whether an application module 740 corresponding to a high level command provided via the API 710 exists. If the corresponding application module 740 exists, the high level command processing unit 710 decodes the high level command so that it may be understood by the application module 740 and either transmits the decoded high level command to a corresponding mapping unit or controls message transmission.
  • the application module control unit 720 includes mapping units 721, 725, and 729 and interface units 723, 727, and 731 corresponding to the transmitting/recording processing module 741, the metadata for disk management storage module 743 for storing metadata for disk management, and the metadata for disk management searching module 745 for searching metadata for disk management, respectively.
  • the transmitting/receiving processing module mapping unit 721 receives a high level command for performing communication with a recording device 100, maps the high level command to a command in a device level recognizable by the transmitting/receiving processing module 741 can process, and provides the mapped high level command to the transmitting/receiving processing module 741 via the transmitting/receiving processing module interface unit 723.
  • the disk management metadata storage module mapping unit 725 and the disk management metadata searching module interface unit 727 store metadata for disk management.
  • the disk management metadata storage module mapping unit 725 receives a high level command for using the disk management metadata storage module 743 from the high level command processing unit 710, maps the high level command to a command in a device level, and provides the mapped command to the disk management metadata storage module 743 via the disk management metadata storage module interface unit 727.
  • the disk management metadata searching module 745 searches for metadata for disk management requested by a user.
  • the disk management metadata searching module mapping unit 729 receives a high level command applied via the high level command processing unit 710 and maps the high level command to a command at a device level recognizable by the disk management metadata searching module 745.
  • the command at the device level is provided to the disk management metadata searching module 745 via the disk management metadata searching module interface unit 731.
  • FIGS. 8A and 8B illustrate images, on which graphic information is overlaid, displayed by the image displaying device 300.
  • Examples of methods whereby the image displaying device 300 displays images as 3D images include a method of displaying 3D images by using goggles synchronized with the image displaying device 300. In this case, a user can view 3D images by wearing the goggles.
  • Methods of embodying 3D images without using goggles include a method of displaying images such that 3D images can only be viewed at a predetermined point, also known as a sweet spot, by using a display device including a lenticular lens, a parallax barrier, parallax illumination, etc. In this case, a user can view 3D images at a predetermined sweet spot.
  • FIG. 8A illustrates graphic information displayed corresponding to a method using goggles for displaying 3D images
  • FIG. 8B illustrates graphic information displayed corresponding to a method not using goggles, respectively.
  • the image processing device 200 In a method in which the image displaying device 300 displays 3D images on a screen in synchronization with goggles, the image processing device 200 generates goggle-shaped graphic information, overlays the graphic information on an image, and transmits the image to the image displaying device 300.
  • FIG. 8A shows an image overlaid with graphic information indicating whether the image is a 2D image or a 3D image.
  • the left image of FIG. 8A illustrates goggle-shaped graphic information overlaid on an image.
  • the goggle-shaped graphic information for indicates that the image displayed by the image displaying device 300 is a 3D image.
  • a user can recognize that the currently displayed image is a 3D image from the goggle-shaped graphic information, and can view the image three-dimensionally by wearing goggles.
  • the right image of FIG. 8A illustrates crossed-out goggle-shaped graphic information overlaid on an image.
  • the crossed-out goggle-shaped graphic information indicates that the image displayed by the image displaying device 300 is a 2D image.
  • a user can recognize that the currently displayed image is a 2D image from the crossed-out goggle-shaped graphic information, and can view the image without wearing goggles.
  • the image processing device 200 In the case of a method in which the image displaying device 300 displays 3D images on a screen such that a user can view the images three-dimensionally at a predetermined sweet spot, the image processing device 200 generates graphic information indicating that a user has to be at a predetermined sweet spot, overlays the graphic information on an image, and transmits the image to the image displaying device 300.
  • the left image of FIG. 8B shows an upright figure-shaped graphic information overlaid on an image.
  • the upright figure-shaped graphic information indicates that the image displayed by the image displaying device 300 is a 3D image.
  • a user can recognize that the image displayed by the image displaying device 300 is a 3D image, and can view the image three-dimensionally at a predetermined sweet spot.
  • the horizontally aligned figure-shaped graphic information indicates that the image displayed by the image displaying device 300 is a 2D image.
  • a user can recognize that the image displayed by the image displaying device 300 is a 2D image, and can view the image not only at a predetermined sweet spot but also at other spots.
  • FIG. 9 is a flowchart for describing a method of displaying images, according to an embodiment of the present invention.
  • the image processing device 200 generates 2D images by decoding video data, and determines whether the image displaying device 300 is a device capable of displaying both 2D images and 3D images (operation 910). If the image displaying device 300 is capable of displaying 2D images only, the image processing device 200 transmits 2D images to the image displaying device 300 (operation 980).
  • the image processing device 200 extracts a unique identifier of a disk from the disk and transmits the unique identifier to the server 100 (operation 920).
  • the image processing device 200 downloads metadata for disk management corresponding to the unique identifier from the server 100 (operation 930).
  • the image processing device 200 uses the metadata for disk management to determine whether metadata for generating a depth map corresponding to a predetermined title exists for each of a plurality of titles recorded in the disk (operation 940).
  • the image processing device 200 If metadata for generating a depth map corresponding to a predetermined title does not exist, the image processing device 200 generates graphic information indicating that images in the title are 2D images, overlays the graphic information on the 2D images, and transmits the images to the image displaying device 300 (operation 990). If images in a previous title transmitted to the image displaying device 300 are 3D images, the image processing device 200 generates a mode switching control signal for changing the image displaying mode of the image displaying device 300 to the 2D image displaying mode, and transmits the mode switching control signal to the image displaying device 300 (operation 1000).
  • the image processing device 200 converts the 2D images to 3D images by using the metadata for generating a depth map (operation 950).
  • the image processing device 200 generates graphic information indicating that the converted images are 3D images, overlays the graphic information on the 3D images, and transmits the images to the image displaying device 300 (operation 960).
  • images in a previous title transmitted to the image displaying device 300 are 2D images
  • the image processing device 200 generates a mode switching control signal for changing the image displaying mode of the image displaying device 300 to the 3D image displaying mode, and transmits the mode switching control signal to the image displaying device 300 (operation 970).

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Library & Information Science (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
  • Processing Or Creating Images (AREA)
PCT/KR2009/006067 2008-10-27 2009-10-21 Methods and apparatuses for processing and displaying image WO2010050691A2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP09823771A EP2319247A4 (de) 2008-10-27 2009-10-21 Verfahren und vorrichtungen zur verarbeitung und anzeige von bildern

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR10-2008-0105485 2008-10-27
KR1020080105485A KR20100002048A (ko) 2008-06-24 2008-10-27 영상 처리 방법, 영상 출력 방법 및 그 장치

Publications (2)

Publication Number Publication Date
WO2010050691A2 true WO2010050691A2 (en) 2010-05-06
WO2010050691A3 WO2010050691A3 (en) 2010-07-29

Family

ID=42130319

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2009/006067 WO2010050691A2 (en) 2008-10-27 2009-10-21 Methods and apparatuses for processing and displaying image

Country Status (2)

Country Link
EP (1) EP2319247A4 (de)
WO (1) WO2010050691A2 (de)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103179411A (zh) * 2011-12-21 2013-06-26 株式会社东芝 图像处理装置及方法、以及图像显示装置
US8687470B2 (en) 2011-10-24 2014-04-01 Lsi Corporation Optical disk playback device with three-dimensional playback functionality
EP2400769A3 (de) * 2010-06-23 2014-04-23 Samsung Electronics Co., Ltd. Anzeigevorrichtung und Anzeigeverfahren dafür
EP2742693A4 (de) * 2011-08-12 2015-04-08 Motorola Mobility Inc Verfahren und vorrichtung zur kodierung und übertragung von 3d-videosequenzen in einem drahtlosen kommunikationssystem

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020009137A1 (en) 2000-02-01 2002-01-24 Nelson John E. Three-dimensional video broadcasting system
US20040032488A1 (en) 1997-12-05 2004-02-19 Dynamic Digital Depth Research Pty Ltd Image conversion and encoding techniques
EP1633148A1 (de) 2003-05-30 2006-03-08 Sharp Kabushiki Kaisha Bildempfangsvorrichtung und bildwiedergabevorrichtung
WO2007066868A1 (en) 2005-12-09 2007-06-14 Electronics And Telecommunications Research Institute Method for providing dmb-based 3d image service, and decoding apparatus and method for dmb-based 3d image service

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001016609A (ja) * 1999-06-05 2001-01-19 Soft Foo Deii:Kk Ｍｐｅｇデータを用いた立体映像生成装置及び方法
EP1529400A4 (de) * 2002-07-16 2009-09-23 Korea Electronics Telecomm Verfahren und vorrichtung zur anpassung eines zwei- und dreidimensionalen videosignals
JP4227076B2 (ja) * 2004-05-24 2009-02-18 株式会社東芝 立体画像を表示する表示装置及び立体画像を表示する表示方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040032488A1 (en) 1997-12-05 2004-02-19 Dynamic Digital Depth Research Pty Ltd Image conversion and encoding techniques
US20020009137A1 (en) 2000-02-01 2002-01-24 Nelson John E. Three-dimensional video broadcasting system
EP1633148A1 (de) 2003-05-30 2006-03-08 Sharp Kabushiki Kaisha Bildempfangsvorrichtung und bildwiedergabevorrichtung
WO2007066868A1 (en) 2005-12-09 2007-06-14 Electronics And Telecommunications Research Institute Method for providing dmb-based 3d image service, and decoding apparatus and method for dmb-based 3d image service

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP2319247A4

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2400769A3 (de) * 2010-06-23 2014-04-23 Samsung Electronics Co., Ltd. Anzeigevorrichtung und Anzeigeverfahren dafür
EP2742693A4 (de) * 2011-08-12 2015-04-08 Motorola Mobility Inc Verfahren und vorrichtung zur kodierung und übertragung von 3d-videosequenzen in einem drahtlosen kommunikationssystem
US8687470B2 (en) 2011-10-24 2014-04-01 Lsi Corporation Optical disk playback device with three-dimensional playback functionality
CN103179411A (zh) * 2011-12-21 2013-06-26 株式会社东芝 图像处理装置及方法、以及图像显示装置
CN103179411B (zh) * 2011-12-21 2016-02-03 株式会社东芝 图像处理装置及方法、以及图像显示装置

Also Published As

Publication number Publication date
EP2319247A2 (de) 2011-05-11
WO2010050691A3 (en) 2010-07-29
EP2319247A4 (de) 2012-05-09

Similar Documents

Publication Publication Date Title
US20100103168A1 (en) Methods and apparatuses for processing and displaying image
WO2011005025A2 (en) Signal processing method and apparatus therefor using screen size of display device
KR101634569B1 (ko) 3ｄ 이미지 데이터의 전송
US8937648B2 (en) Receiving system and method of providing 3D image
JP5809064B2 (ja) ３ｄ画像データの転送
US8810563B2 (en) Transmitting apparatus, stereoscopic image data transmitting method, receiving apparatus, and stereoscopic image data receiving method
WO2010143820A2 (ko) 3차원 pip 영상 제공 장치 및 그 방법
WO2010041905A2 (ko) 수신 시스템 및 데이터 처리 방법
US20070247477A1 (en) Method and apparatus for processing, displaying and viewing stereoscopic 3D images
US20110063422A1 (en) Video processing system and video processing method
WO2009157707A2 (en) Image processing method and apparatus
WO2010085074A2 (en) Three-dimensional subtitle display method and three-dimensional display device for implementing the same
WO2010095835A2 (ko) 영상 처리 방법 및 장치
WO2011129566A2 (ko) 이미지 디스플레이 방법 및 장치
WO2012002690A2 (ko) 디지털 수신기 및 디지털 수신기에서의 캡션 데이터 처리 방법
JP2012518317A (ja) ３ｄ観察者メタデータの転送
WO2013100376A1 (en) Apparatus and method for displaying
WO2011099780A2 (ko) 이미지 디스플레이 방법 및 창치
WO2010095838A2 (ko) 그래픽 화면 처리 방법 및 장치
WO2014014263A2 (en) Image data scaling method and image display apparatus
JP2011228959A (ja) ３次元映像再生方法、および３次元映像再生装置
WO2010050691A2 (en) Methods and apparatuses for processing and displaying image
WO2010137849A2 (ko) 영상 처리 방법 및 장치
WO2015178576A1 (ko) 영상 디스플레이 장치 및 영상 디스플레이 방법
JP2012186652A (ja) 電子機器、画像処理方法及び画像処理プログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 09823771

Country of ref document: EP

Kind code of ref document: A2

WWE Wipo information: entry into national phase

Ref document number: 2009823771

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE