KR20120089604A - Method and system for error protection of 3d video - Google Patents

Abstract

PURPOSE: A method for preventing an error in a 3D video and a system thereof are provided to variously protect areas of a 3D video frame with different depths from errors. CONSTITUTION: A 3D video camera(202) encodes a plurality of areas of a captured 3D video frame. The areas are related to different depths. The encoding process variously protects the areas from errors based on the different depths. At least one interest area is identified from the areas of the 3D video frame. A higher level for error prevention uses an error-correction code longer than an error-correction code used for providing a lower level of error prevention.

Description

METHOD AND SYSTEM FOR ERROR PROTECTION OF 3D VIDEO}

Integration by cross-reference / reference to related applications

This patent application refers to U.S. Provisional Application No. 61 / 377,867, filed Aug. 27, 2010 and U.S. Provisional Application No. 61 / 439,130, filed Feb. 3, 2011, and claims priority therefrom. Claim the interests of

This application also refers to:

US Provisional Application No. 61 / 439,193, filed February 3, 2011;

United States Patent Provisional Application ______ (Attorney Docket No. 23461US03), filed March 31, 2011;

United States Patent Provisional Application 61 / 439,274, filed February 3, 2011;

US Patent Provisional Application ______ (Attorney Docket No. 23462US03), filed March 31, 2011;

United States Patent Provisional Application 61 / 439,283, filed February 3, 2011;

United States Patent Provisional Application ______ (Attorney Docket No. 23463US03), filed March 31, 2011;

United States Patent Provisional Application 61 / 439,130, filed February 3, 2011;

United States Patent Provisional Application ______ (Attorney Docket No. 23464US03), filed March 31, 2011;

US Provisional Application No. 61 / 439,290, filed February 3, 2011;

United States Patent Provisional Application ______ (Attorney Docket No. 23465US03), filed March 31, 2011;

US Provisional Application No. 61 / 439,119, filed February 3, 2011;

United States Patent Provisional Application ______ (Attorney Docket No. 23466US03), filed March 31, 2011;

US Provisional Application No. 61 / 439,297, filed February 3, 2011;

United States Patent Provisional Application ______ (Attorney Docket No. 23467US03), filed March 31, 2011;

United States Patent Provisional Application 61 / 439,201, filed February 3, 2011;

US Provisional Application No. 61 / 439,209, filed February 3, 2011;

United States Patent Provisional Application 61 / 439,113, filed February 3, 2011;

United States Patent Provisional Application ______ (Attorney Docket No. 23472US03), filed March 31, 2011;

United States Patent Provisional Application 61 / 439,103, filed February 3, 2011;

United States Patent Provisional Application ______ (Attorney Docket No. 23473US03), filed March 31, 2011;

US Provisional Application No. 61 / 439,083, filed February 3, 2011;

United States Patent Provisional Application ______ (Attorney Docket No. 23474US03), filed March 31, 2011;

US Provisional Application No. 61 / 439,301, filed February 3, 2011; And

United States patent provisional application ______, filed March 31, 2011 (Attorney Docket No. 23475US03).

Each of the applications mentioned above is hereby incorporated by reference in its entirety.

Some embodiments of the present invention relate to video processing. More specifically, some embodiments of the present invention relate to methods and systems for error protection of 3D video.

Digital video capabilities can be included in a wide range of devices such as, for example, digital televisions, digital direct broadcast systems, digital recording devices, and the like. Digital video devices can process and transmit video sequences to provide significant improvements over conventional analog video systems with increased bandwidth efficiency.

Video content may be recorded in two-dimensional (2D) format or three-dimensional (3D) format. For example, in various applications such as DVD movies and digital TV (DTV), 3D video is often desirable because it is more realistic for viewers than the corresponding 2D. The 3D video includes a left view video and a right view video.

Various video encoding standards, for example MPEG-1, MPEG-2, MPEG-4, MPEG-C Part 3, H.263, H.264 / MPEG-4 Advanced video coding (AVC), Multiview video coding and scalable video coding (SVC) have been established for encoding digital video sequences in a compression scheme. For example, the MVC standard, an extension of the H.264 / MPEG-4 AVC standard, can provide efficient coding of 3D video. The SVC standard, which is an extension of the H.264 / MPEG-4 AVC standard, also has lower or fidelity of temporary or spatial resolution while maintaining reconstruction quality similar to that obtained using H.264 / MPEG-4 AVC. It is possible to enable transmission and decoding of partial bitstreams to provide video services with reduced fidelity.

Additional limitations and disadvantages with conventional and general approaches will become apparent to those skilled in the art through a comparison of these systems with the present invention described herein below with reference to the drawings of the present invention.

Therefore, it is an object of the present invention to provide a method and system for error protection of 3D video in which various error protection is applied to a plurality of regions having different depths in a 3D video frame for error protection of 3D video. To provide.

A system and / or method is provided for error protection of 3D video, as more fully shown in the claims, as substantially shown and / or described in connection with at least one of the figures.

According to one aspect of the invention, a method for processing video is provided, the method comprising:

In a three-dimensional (3D) video generation device,

Encoding a plurality of regions of the captured 3D video frame, wherein the plurality of regions are associated with different depths, and the encoding provides various error protections to the plurality of regions based on the associated different depths. Apply.

Advantageously, the method further comprises identifying one or more regions of interest from said plurality of regions of said captured 3D video frame.

Advantageously, the higher level of error protection uses a longer error-correction code than the error-correction code used to provide a lower level of error protection.

Advantageously, said 3D video generation device comprises a monocular 3D video generation device having one or more depth sensors, said 3D video frame comprising a two-dimensional (2D) video frame and corresponding depth information.

Advantageously, one or more higher levels of error protection apply to one or more areas of interest in the 2D video frame, and one or more lower levels of error protection apply to one or more other areas in the 2D video frame. do.

Advantageously, one or more higher levels of error protection are applied to the corresponding depth information associated with one or more regions of interest in the 2D video frame, and one or more lower levels of error protection are within the 2D video frame. Applies to the corresponding depth information associated with one or more other areas.

Advantageously, a higher level of error protection is applied to each of the plurality of regions within the 2D video frame, and a lower level of error protection is associated with each of the plurality of regions within the 2D video frame. Applied to depth information.

Advantageously, said first type of error protection is applied to one or more regions of interest in said captured 2D video frame, and said second type of error protection is applied to one or more other regions in said captured 2D video frame. do.

Advantageously, said first type of error protection is applied to said corresponding depth information associated with one or more regions of interest in said captured 2D video frame, and said second type of error protection is within said captured 2D video frame. Applies to the corresponding depth information associated with one or more other areas.

Advantageously, the method further comprises transmitting said error protected 3D video frame to a 3D video rendering device for 3D video rendering and / or display.

According to one aspect, a system for processing video includes one or more processors and / or circuits for use in a three-dimensional (3D) video generation device, wherein the one or more processors and / or circuits are:

May be operable to encode a plurality of regions of a captured 3D video frame, wherein the plurality of regions are associated with different depths, and the encoding is in error in the plurality of regions based on the associated different depths. Apply protection.

Advantageously, said one or more processors and / or circuits may be operable to identify one or more regions of interest from said plurality of regions of said captured 3D video frame.

Advantageously, the higher level of error protection uses a longer error-correction code than the error-correction code used to provide a lower level of error protection.

Advantageously, said 3D video generation device comprises a monocular 3D video generation device having one or more depth sensors, said 3D video frame comprising a two-dimensional (2D) video frame and corresponding depth information.

Advantageously, one or more higher levels of error protection apply to one or more areas of interest in the 2D video frame, and one or more lower levels of error protection apply to one or more other areas in the 2D video frame. do.

Advantageously, one or more higher levels of error protection are applied to corresponding depth information associated with one or more regions of interest in the 2D video frame, and one or more lower levels of error protection are one in the 2D video frame. The above applies to the corresponding depth information relating to the other areas.

Advantageously, a higher level of error protection is applied to each of the plurality of regions within the 2D video frame, and a lower level of error protection is associated with each of the plurality of regions within the 2D video frame. Applied to depth information.

Advantageously, said first type of error protection is applied to one or more regions of interest in said captured 2D video frame, and said second type of error protection is applied to one or more other regions in said captured 2D video frame. do.

Advantageously, said first type of error protection is applied to said corresponding depth information associated with one or more regions of interest in said captured 2D video frame, and said second type of error protection is within said captured 2D video frame. Applies to the corresponding depth information associated with one or more other areas.

Advantageously, said one or more processors and / or circuits may be operable to transmit said error protected 3D video frame to a 3D video rendering device for 3D video rendering and / or display.

The various advantages, aspects and novel features of the present invention as well as the details of the described embodiment will be fully understood from the following description and drawings.

Therefore, the present invention can provide a large-scale 3D video frame with a high resolution quality at a limited bandwidth by applying various error protection for a plurality of regions having different depths in the 3D video frame for error protection of the 3D video. There is an advantage.

1A is a block diagram illustrating an exemplary monocular 3D video camera implementing aspects of the present invention as compared to a conventional stereoscopic video camera.
1B is a block diagram illustrating exemplary processing of depth information and 2D color information for generating a 3D image, in accordance with an embodiment of the present invention.
2 is a block diagram illustrating an example video communication system that can operate to provide error protection for 3D video, in accordance with an embodiment of the invention.
3 is a block diagram illustrating an exemplary monocular 3D video camera that can be operable to provide error protection for 3D video generated using a single view and depth information, in accordance with an embodiment of the invention. .
4A-4D are block diagrams illustrating exemplary error protection for 2D video frames and corresponding depth information, respectively, in accordance with one embodiment of the present invention.
5 is a flow diagram illustrating exemplary steps for providing error protection for 3D video, in accordance with an embodiment of the present invention.

Certain embodiments of the present invention may be found in a method and system for 3D video error protection. In various embodiments of the invention, the three-dimensional (3D) video generation device may be operable to encode or compress a plurality of regions of the captured 3D video frame. In this regard, the plurality of regions may be associated with different depths. The encoding may apply various error protections to the plurality of regions based on the associated different depths. The error protection may include, for example, forward error correction (FEC). The higher level of error protection may use a longer error-correcting code than the error-correcting code used to provide a lower level of error protection. In this regard, the longer error-correcting code may provide stronger error protection than the shorter error-correcting code. The 3D video generation device may be operable to identify one or more regions of interest from the plurality of regions of the captured 3D video frame.

In one exemplary embodiment of the invention, the 3D video generation device may comprise, for example, a monocular 3D video generation device having one or more depth sensors. The 3D video frame may comprise a two-dimensional (2D) video frame and corresponding depth information. The corresponding depth information may be captured by the one or more depth sensors at the monocular 3D video generation device.

In one exemplary embodiment of the invention, one or more higher levels of error protection may be applied to one or more regions of interest in the 2D video frame, wherein one or more lower levels of error protection are considered to be of less interest. It can be applied to one or more other areas within the 2D video frame that are considered. Higher levels of the error protection may be applied to the corresponding depth information associated with one or more regions of interest in the 2D video frame, and one or more lower levels of the error protection are considered less interested. May be applied to the corresponding depth information associated with one or more other areas in the apparatus. In another example, a higher level of error protection may be applied to each of the plurality of areas within the 2D video frame, and a lower level of error protection may be applied to each of the plurality of areas within the 2D video frame, for example. May be applied to the corresponding depth information associated with the.

In one exemplary embodiment of the present invention, the first type of error protection can be applied to one or more regions of interest in the captured 2D video frame, and the second type of error protection is within the captured 2D video frame. It can be applied to one or more other areas. The first type of error protection can be applied to the corresponding depth information associated with one or more regions of interest in the captured 2D video frame, and the second type of error protection is one or more other in the captured 2D video frame. May be applied to the corresponding depth information associated with the areas.

In one exemplary embodiment of the invention, the 3D video generation device may be operable to send the error protected 3D video frame to a 3D video rendering device for 3D video rendering and / or display.

1A is a block diagram illustrating an exemplary monocular 3D video camera implementing aspects of the present invention as compared to a conventional stereoscopic video camera. Referring to FIG. 1A, a stereoscopic video camera 100 and a monocular 3D video camera 102 are shown. The stereoscopic video camera 100 may include two lenses 101a and 101b. Each of the lenses 101a and 101b can capture images from different viewpoints, and the images captured through the two lenses 101a and 101b can be combined to produce a 3D image. . In this regard, electromagnetic (EM) waves in the visible spectrum may be focused on one or more first image sensors by the lenses 101a (and associated optics), and EM waves in the visible spectrum may be focused on the lens. Can be focused on one or more second image sensors by means of the 101b (and associated optics).

The monocular 3D video camera 102 may include a processor 104, a memory 106, one or more depth sensors 108, and one or more image sensors 114. The monocular 3D or single-view video camera 102 may capture images through a single perspective corresponding to lens 101c. In this regard, EM waves in the visible spectrum may be focused on one or more image sensors 114 by the lens 101c. The monocular 3D video camera 102 may also capture depth information through the lens 101c (and associated optics).

The processor 104 is operable to manage the operation of various components of the monocular 3D video camera 102 and to perform various computing and processing tasks. And / or code.

The memory 106 may include, for example, DRAM, SRAM, flash memory, hard drive or other magnetic storage, or some other suitable memory device. For example, an SRAM may be used to store data used and / or generated by the processor 104, and a hard-drive and / or flash memory may be used to store recorded image data and depth data. Can be.

The depth sensor (s) 108 may each be operable to detect EM waves within the infrared spectrum and to determine depth information based on reflected infrared light. May contain code. For example, depth information is transmitted by an emitter (not shown) in the monocular 3D video camera 102 and is time-of-flight of infrared light reflected back to the depth sensor (s) 108. Can be determined based on Depth information may also be determined using a structured light method, for example. As such an example, a light pattern such as a grid of infrared waves may be projected at a known angle to the object by a light source such as a projector. The depth sensor (s) 108 may detect deformation of a light pattern, such as the infrared light pattern on the object. Thus, depth information for a scene can be determined or calculated using, for example, triangulation techniques.

The image sensor (s) 114 may each include suitable logic, circuitry, interfaces, and / or code, which may be operable to convert optical signals into electrical signals. Each image sensor 114 may include, for example, a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor. Each image sensor 114 may capture brightness, brightness and / or chromaticity information.

1B is a block diagram illustrating exemplary processing of depth information and 2D color information for generating a 3D image, in accordance with an embodiment of the present invention. Referring to FIG. 1B, a depth information frame 130, a 2D color information frame 134, and a 3D image frame 136 are shown. The depth information frame 130 may be captured by the depth sensor (s) 108, and the 2D color information frame 134 may be captured by the image sensor (s) 114. The depth information frame 130 may be used while processing the 2D color information frame 134 by the processor 104 to generate the 3D image frame 136. The dotted line 132 may indicate a reference plane for representing the 3D image. In the depth information frame 130, the thickness of the line is used to indicate the depth. In this regard, for example, the thicker the line, the closer the portion of frame 130 is to the monocular 3D video camera 102. Thus, object 138 is furthest from the monocular 3D video camera 102, object 142 is closest to the monocular 3D video camera, and object 140 is located at a medium depth. In various embodiments of the present invention, the depth information may be mapped by the processor 104 into a grayscale or pseudo-grayscale image.

The image in the frame 134 is a typical 2D image. The viewer of the frame 134 perceives the viewer and the respective objects 138, 140, and 142 at the same depth. That is, each of the objects 138, 140, and 142 appear to be on the reference plane 132. The image of the frame 136 is a 3D image. The observer of frame 136 recognizes that object 138 is further away from the observer, object 142 is closest to the observer, and object 140 is at intermediate depth. In this regard, object 138 appears to be behind reference plane 132, object 140 appears to be present on reference plane 132, and object 142 is in front of reference plane 132. It seems to be located.

2 is a block diagram illustrating an example video communication system that can operate to provide error protection for 3D video, in accordance with an embodiment of the invention. 2, a video communication system 200 is shown. The video communication system 200 may include a 3D video camera 202 and a 3D video rendering device 204.

The 3D video camera 202 may include suitable logic, circuitry, interfaces and / or code that may be operable to capture 3D video frames. In an exemplary embodiment of the invention, the 3D video camera 202 may comprise, for example, a monocular 3D video camera 202a. In this regard, the monocular 3D video camera 202a may be substantially similar to the monocular 3D video camera 102 of FIG. 1A. In one exemplary embodiment of the invention, the 3D video camera 202 is encoded 3D video frame based on different depths involved for transmission to a 3D video rendering device, such as the 3D video rendering device 204. Operate to apply various error protections to a plurality of regions of a.

Appropriate logic, circuitry, interfaces, the 3D video rendering device 204 may be operable to receive an encoded or compressed video stream that may include the error protected 3D video frames from the 3D video camera 202. And / or code. For example, the 3D video rendering device 204, such as a set top box (STB) and / or a digital TV (DTV), can process the received video stream for rendering and / or displaying in a 3D video format.

In operation, the 3D video camera 202 may be operable to capture a sequence of 3D video frames. The captured 3D video frame may be encoded or compressed by the 3D video camera 202. The encoding may apply various error protections to a plurality of regions of the 3D video frame. The plurality of regions may be associated with different depths, and the encoding may apply various error protection to the plurality of regions based on the associated different depths.

The error protection may include, for example, forward error correction. The error protection may be differential error protection wherein the 3D video camera 202 receives the 3D video bitstreams in such a way that the region of interest or region of interest may receive a higher level of error protection than other portions. Add redundant data or error-correction codes to. As such an example, higher levels of error protection may include longer error-correcting codes than error-correcting codes for lower levels of error protection. The error-correction code may be, for example, a Reed-Solomon (RS) code, a low-density parity-check (LDPC) code, a hamming code, a turbo code, a convolutional code. Code and / or other types of error-correcting codes.

The 3D video camera 202 may identify one or more regions of interest from a plurality of regions of the captured 3D video frame. In this regard, the region of interest may be identified or selected based on the associated depth information and / or the region or object that is the primary focus of the video frame. For example, based on the depth information, the region having the shortest depth and closest to the 3D video camera 202 may be considered or selected as the region of interest. Areas including the body, cars or runners may also be considered areas of interest, for example.

In one exemplary embodiment of the invention, the 3D video camera 202 may be operable to send the error protected 3D video frame to the 3D video rendering device 204 for 3D video rendering and / or display. have.

In one exemplary embodiment of the invention, the 3D video camera 202 may comprise, for example, a monocular 3D video camera 202a. As such, the captured 3D video frame may include a 2D video frame and corresponding depth information.

In one exemplary embodiment of the present invention, the same type of error protection of various levels may be applied to a plurality of regions of the captured 2D video frame and / or corresponding depth information for the plurality of regions. In this regard, for example, one or more higher levels of the error protection may be applied to one or more regions of interest in the 2D video frame, and one or more lower levels of the error protection may be applied to one or more within the 2D video frame. It can be applied to other areas. One or more higher levels of the error protection may be applied to the corresponding depth information associated with one or more regions of interest in the 2D video frame, and one or more lower levels of the error protection may be applied to one or more other in the 2D video frame. May be applied to the corresponding depth information associated with the areas. In another example, a higher level of the error protection may be applied to each of the plurality of areas within the 2D video frame, and a lower level of the error protection may be applied, for example, to the plurality of areas within the 2D video frame. May be applied to the corresponding depth information associated with each.

In another exemplary embodiment of the invention, various types of error protection may be applied to the corresponding depth information for the plurality of regions and / or the plurality of regions of the captured 2D video frame. In this regard, for example, a first type of error protection may be applied to one or more regions of interest in the captured 2D video frame, and a second type of error protection may be applied to one or more other areas in the captured 2D video frame. Can be applied. A first type of error protection can be applied to the corresponding depth information associated with one or more regions of interest in the captured 2D video frame, and a second type of error protection is applied to one or more other areas in the captured 2D video frame. May be applied to the corresponding depth information associated with the.

Although a 3D video camera 202 such as the monocular 3D video camera 202a is shown in FIG. 2, the present invention may not be limited thereto. Thus, any type of 3D video generation device may be illustrated that produces 3D video frames having associated depths without departing from the spirit and scope of various embodiments of the present invention. For example, a 3D video generation device, such as a monocular 3D camcorder, that generates 3D video content in 2D-plus-depth formats may be illustrated. Also, a stereoscopic video camera such as stereoscopic video camera 100 may be illustrated, for example.

3 is a block diagram illustrating an exemplary monocular 3D video camera that can be operable to provide error protection for 3D video generated using a single view and depth information, in accordance with an embodiment of the invention. . Referring to FIG. 3, a monocular 3D video camera 300 is shown. The monocular 3D video camera 300 includes a processor 304, a memory 306, one or more depth sensors 308, an emitter 309, an image signal processor (ISP) 310, input / output (I / I). O) module 312, one or more image sensors 314, optics 316, speakers 311, microphone 313, video / audio encoder 307, video / audio decoder 317, audio module ( 305, error protection module 315, lens 318, a plurality of controls 322, optical viewfinder 324, and display 320. The monocular 3D video camera 300 may be substantially similar to the monocular 3D video camera 102 of FIG. 1A.

The processor 304 may include suitable logic, circuitry, interfaces and / or code that may be operable to coordinate the operation of various components of the monocular 3D video camera 300. The processor 304, for example, runs an operating system of the monocular 3D video camera 300 and controls the communication of information and signals between components of the monocular 3D video camera 300. can do. The processor 304 may execute code stored in the memory 306. In one exemplary embodiment of the present invention, the processor 304 may identify or select one or more regions of interest within the 2D video frame. The identification or selection may, for example, be based on the captured corresponding depth data and / or the main focal region or object within the video frame.

The memory 306 may include, for example, DRAM, SRAM, flash memory, hard drive or other magnetic storage or any other suitable memory devices. For example, an SRAM may be used to store data used and / or generated by the processor 304, and a hard-drive and / or flash memory may be used to store recorded image data and depth data. Can be.

The depth sensor (s) 308 may each be adapted to suitable logic, circuitry, interfaces and / or code that may be operable to sense EM waves within the infrared spectrum and determine depth information based on the reflected infrared waves. It may include. For example, depth information may be determined based on the propagation time of the infrared light transmitted by the emitter 309 and reflected back to the depth sensor (s) 308. Depth information can also be determined using, for example, structured light methods. As such an example, a light pattern such as a grid of infrared waves may be projected at a known angle to the object by a light source such as a projector. The depth sensor (s) 308 may detect deformation of a light pattern, such as the infrared light pattern on the object. Thus, depth information for the scene can be determined or calculated using, for example, triangulation techniques.

The image signal processor or image sensor processor (ISP) 310 includes suitable logic, circuitry, interfaces and / or code that can be operable to perform complex processing of captured image data and corresponding corresponding depth data. can do. The ISP 310 may, for example, filter, demosaiic, Bayer interpolation, lens shading correction, incomplete pixel correction, white balance, image compensation, color distortion, and / or post-processing. A plurality of processing techniques may be performed including post filtering.

The audio module 305 may include suitable logic, circuitry, interfaces and / or code that may be operable to perform various audio functions of the monocular 3D video camera 300. In one exemplary embodiment of the present invention, the audio module 305 may perform noise reduction and / or audio volume level adjustment for 3D scenes.

The video / audio encoder 307 may include suitable logic, circuitry, interfaces and / or code that may be operable to perform video encoding and / or audio encoding functions. For example, the video / audio encoder 307 may be configured to transmit captured 2D video frames and corresponding depth information and / or for transmission to a 3D video rendering device, such as the 3D video rendering device 204. Audio data can be encoded or compressed.

The video / audio decoder 317 may include suitable logic, circuitry, interfaces and / or code that may be operable to perform video decoding and / or audio decoding functions.

The error protection module 315 may include suitable logic, circuitry, interfaces and / or code that may be operable to perform error protection functions on the monocular 3D video camera 300. For example, the error protection module 315 may be configured to encode encoded 2D video frames and corresponding depth information and / or encoded audio data for transmission to a 3D video rendering device, such as the 3D video rendering device 204. Error protection can be provided. In an exemplary embodiment of the invention, the error protection module 315 applies various error protections to a plurality of regions of the captured 2D video frame and / or to corresponding depth information or data for the plurality of regions. can do.

The input / output (I / O) module 312 allows the monocular 3D video camera 300 to include USB, PCI-X, IEEE 1394, HDMI, DisplayPort, and / or analog audio and / or analog video standards. Suitable logic, circuitry, interfaces, and / or code, which may enable interfacing with other devices in accordance with one or more standards, such as; For example, the I / O module 312 may transmit and receive signals from the controls 322; Output video to display 320; Output audio to the speaker 311; Process audio input from microphone 313; Read and write to a cassette, flash card, solid state drives, hard disk drives or other external memory attached to the monocular 3D video camera 300; And / or output audio and / or video externally through one or more ports, such as an IEEE 1394 port, an HDMI, and / or a USB port for transmission and / or rendering.

The image sensor (s) 314 may each include suitable logic, circuitry, interfaces and / or code, which may be operable to convert optical signals into electrical signals. Each image sensor 314 may include, for example, a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor. Each image sensor 314 may capture brightness, brightness and / or chromaticity information.

The optics 316 may include various optical devices for conditioning and directing EM waves received through the lens 318. The optics 316 may direct EM waves in the visible spectrum to the image sensor (s) 314 and guide EM waves in the ultraviolet spectrum to the depth sensor (s) 308. The optics 316 may include, for example, one or more lenses, prisms, luminance and / or color filters and / or mirrors.

The lens 318 may be operable to collect and fully focus electromagnetic (EM) waves in the visible and infrared spectra.

The display 320 may include an LCD display, an LED display, an organic LED (OLED) display, and / or other digital display on which images recorded via the monocular 3D video camera 300 may be displayed. In one embodiment of the invention, the display 320 may be operated to display 3D images.

The controls 322 may include suitable logic, circuitry, interfaces, and / or code to enable a user to interact with the monocular 3D video camera 300. For example, the controls 322 allow a user to control recording and playback. In one embodiment of the invention, the controls 322 allow the user to select the recording mode of the monocular 3D video camera 300 as a 2D mode or a 3D mode.

The optical viewfinder 324 may allow a user to view or see what the lens 318 is "seeing", ie, "on a frame."

In operation, the image sensor (s) 314 may capture brightness, luminance and / or chromaticity information associated with a 2D video frame, and the depth sensor (s) 308 may capture corresponding depth information. have. In various embodiments of the present invention, various color formats may be used, such as RGB or YCrCb. The depth information may be stored in the memory 306 as metadata or as an additional layer of information, which may be used when rendering a 3D video frame from the 2D video frame information.

In one exemplary embodiment of the present invention, the processor 304 may be operable to identify one or more regions of interest within the captured 2D video frame. The region of interest may be identified or selected based on, for example, the captured corresponding depth information and / or the main focal region or objects within the video frame. The captured 2D video frame and the captured corresponding depth information may be encoded or compressed by the video / audio encoder 307. The encoding may use the error protection module 315 to apply various error protections to a plurality of regions and / or corresponding depth information for the plurality of regions of the 2D video frame. In this regard, for example, the error protection module 315 may apply one or more higher levels of error protection to one or more regions of interest within the 2D video frame. One or more lower levels of error protection may be applied to one or more other areas within the 2D video frame that are considered of less interest. The error protection module 315 may apply one or more higher levels of error protection to corresponding depth information associated with one or more regions of interest in the 2D video frame. One or more lower levels of error protection may be applied to corresponding depth information associated with one or more other regions within the 2D video frame that are considered to be of less interest. In another example, the error protection module 315 may apply a higher level of error protection to each area within the 2D video frame, and a lower level of error protection, for example, each area within the 2D video frame. Apply to the corresponding depth information associated with the.

Various types of error protection can be applied to different regions of the captured 2D video frame and / or the corresponding depth information for the different regions. For example, for regions with high details in the region of interest, a first type of error-correction code may be used, while a region with little or no change may use a second type of error-correction code. Can be. In this regard, the error protection module 315 may apply the first type of error protection to one or more regions of interest in the captured 2D video frame. The second type of error protection may be applied to one or more other areas within the captured 2D video frame. The error protection module 315 may apply the first type of error protection to the corresponding depth information associated with one or more regions of interest in the captured 2D video frame. The second type of error protection may be applied to the corresponding depth information associated with one or more other regions within the captured 2D video frame.

In one exemplary embodiment of the present invention, the I / O module 312 is configured to render the error protected 2D video frame and the error protected corresponding depth information for the 3D video rendering device for 3D video rendering and / or display. And send to or output to 204.

4A-4D are block diagrams illustrating exemplary error protection for 2D video frames and corresponding depth information, respectively, according to one embodiment of the invention. These scenarios are provided by way of example and not for purposes of limitation. Referring to each of FIGS. 4A-4D, a monocular 3D video camera 402 and a 3D video rendering device 404 are shown. Also shown is a 2D video frame 434 and a corresponding depth information frame 430. The monocular 3D video camera 402 may be substantially similar to the monocular 3D video camera 202 of FIG. 2. The 3D video rendering device 404 may be substantially similar to the 3D video rendering device 204 of FIG. 2. The 2D video frame 434 may include an identified region of interest 434a and another region 434b. The depth information frame 430 may include an area 430a corresponding to the ROI 434a in the 2D video frame 434. An area 430b in the depth information frame 430 corresponds to another area 434b in the 2D video frame 434.

4A shows a first level of error protection for the monocular 3D video camera 402 to transmit the 2D video frame 434 and the corresponding depth information frame 430 to the 3D video rendering device 404. 411 is applied to the regions 434a and 430a and a first scenario can be operated to apply a second level of error protection 412 to the regions 434b and 430b. In such a scenario, the first level of error protection 411 may be a higher level of error protection, and the second level of error protection 412 may be a lower level of error protection. In this regard, the first level of error protection 411 may use, for example, a longer error-correction code than the error-correction code used by the second level of error protection 412.

4B shows a first level of error protection 411 for the monocular 3D video camera 402 to send the 2D video frame 434 and the corresponding depth information frame 430 to the 3D video rendering device 404. ) Is applied to the area 434a, a second level of error protection 412 is applied to the area 434b, a third level of error protection 413 is applied to the area 430a, and A second scenario may be operative to apply four levels of error protection 414 to the area 430b. In such a scenario, the first level of error protection 411 may be the highest level of error protection, while the fourth level of error protection 414 may be, for example, the lowest level of error protection or no error protection. You may not. The second level of error protection 412 may be a lower level of error protection than the first level of error protection 411. The third level of error protection 413 may be, for example, a lower level of error protection than the second level of error protection 412.

4C shows a first level of error protection for the monocular 3D video camera 402 to transmit the 2D video frame 434 and the corresponding depth information frame 430 to the 3D video rendering device 404. A third scenario is shown that may be operative to apply 411 to the regions 434a and 434b and to apply a second level of error protection 412 to the regions 430a and 430b. In such a scenario, the first level of error protection 411 may be a higher level of error protection, and the second level of error protection 412 may be a lower level of error protection. In this regard, for example, the 2D video frame 434 and the corresponding depth information frame 430 may be transmitted separately to the 3D video rendering device 404 via two different or layered bitstreams. have. The 2D video frame 434 may be protected with the first level of error protection 411, which is a higher level of error protection, while the corresponding depth information frame 430 is the lower level of error protection. Two levels of error protection 412 may be protected.

4D shows a first type of error protection 414 for the monocular 3D video camera 402 to send the 2D video frame 434 and the corresponding depth information frame 430 to the 3D video rendering device 404. ) Is applied to the regions 434a and 430a, and a fourth scenario may be operated to apply a second type of error protection 415 to the regions 434b and 430b. In such a scenario, for example, the first type of error protection 414 uses a Reed-Solomon (RS) code, while the second type of error protection 415 uses other types of errors. Correction code can be used.

5 illustrates exemplary steps for providing error protection for 3D video, in accordance with an embodiment of the present invention. It is a flow chart. Degree Referring to 5, the exemplary steps begin at step 501. In step 502, the 3D video camera 202 may be operable to identify one or more regions of interest associated with one or more depths within the captured 3D video frame. The identification may be based, for example, on the relevant depth information and / or the main focal region or object within the video frame.

In step 503, the captured 3D video frame may be encoded or compressed by the 3D video camera 202. In step 504, during the encoding, the 3D video camera 202 may apply various error protections to a plurality of regions of the 3D video frame, for example based on the associated depths. For example, the 3D video camera 202 may apply one or more higher levels of error protection to one or more regions of interest within the 3D video frame, and one or more other regions within the 3D video frame. Lower levels of fault protection can be applied. In another example, for example, the 3D video camera 202 may apply a first type of error protection to one or more regions of interest within the captured 3D video frame, and one or more within the captured 3D video frame. A second type of error protection can be applied to other areas.

In step 505, the error protected 3D video frame is transmitted or output by the 3D video camera 202 to a 3D video rendering device such as the 3D video rendering device 204 for 3D video rendering and / or display. Can be. The exemplary steps may proceed to final step 506.

In various embodiments of the invention, a 3D video generation device, such as the 3D video camera 202, may be operable to encode or compress a plurality of regions of the captured 3D video frame. In this regard, the plurality of regions may be associated with different depths. The encoding may apply various error protections to the plurality of regions based on the associated different depths. The error protection may include, for example, forward error correction (FEC). Higher levels of error protection may use longer error-correcting codes than error-correcting codes used to provide lower levels of error protection. In this regard, the longer error-correcting code can provide stronger error protection than the shorter error-correcting code. The 3D video camera 202 may be operable to identify one or more regions of interest from a plurality of regions of the captured 3D video frame. In this regard, the region of interest may be identified or selected, for example, based on the relevant depth information and / or the region or object of primary focus.

In one exemplary embodiment of the invention, the 3D video camera 202 transmits the error protected 3D video frame to a 3D video rendering device such as 3D video rendering device 204 for 3D video rendering and / or display. To operate.

In one exemplary embodiment of the invention, the 3D video camera 202 may comprise a monocular 3D video camera, such as the monocular 3D video camera 300 having one or more depth sensors 308, for example. . In this regard, the captured 3D video frame may include a 2D video frame 434 and corresponding depth information 430. The corresponding depth information 430 may be captured by one or more depth sensors 308 in the monocular 3D video camera 300.

In one exemplary embodiment of the present invention, the error protection module 315 of the monocular video camera 300 provides one or more higher levels of error protection, such as an area 434a in the 2D video frame 434. And to apply the one or more lower levels of error protection to one or more other areas, such as area 434b in the 2D video frame 434. The error protection module 315 applies one or more higher levels of error protection to one or more areas of interest, such as area 430a in the corresponding depth information 430, and applies one or more lower levels of error protection. And apply to one or more other areas, such as area 430b in the corresponding depth information 430. In another example, the error protection module 315 applies a higher level of error protection to each area within the 2D video frame 434 and uses a lower level of error protection, e. 430 may be adapted to apply to each area within.

In one exemplary embodiment of the present invention, the error protection module 315 applies the first type of error protection to one or more areas of interest, such as area 434a in the 2D video frame 434, and It may be operable to apply a second type of error protection to one or more other areas, such as area 434b in the 2D video frame 434. The error protection module 315 applies the first type of error protection to one or more areas of interest, such as area 430a in corresponding depth information 430, and applies the second type of error protection to a corresponding depth. Operate to apply to one or more other areas, such as area 430b in information 430.

Other embodiments of the invention are non-transitory computer readable media and / or storage media, and / or non-transitory machine readable media and / or storage media, machine code and / or machines and / or storing therein. Or a computer program having one or more sections of code executable by a computer, thereby allowing the machine and / or computer to perform the steps as described herein for error protection of 3D video. .

Thus, the present invention can be realized in hardware, software or a combination of hardware and software. The invention can be realized in a centralized manner in at least one computer system or in a distributed manner in which different components are spread across several interconnected computer systems. Any kind of computer system or other apparatus adapted to carry out the methods described herein is tailored. A typical combination of hardware and software may be a general purpose computer system having a computer program that, when loaded or executed, controls the computer system to execute the methods described herein.

In addition, the present invention may be embedded in a computer program product that includes all the features that make it possible to implement the methods described herein and that may perform these methods when loaded into a computer system. In this context, a computer program is a set of instructions that allows an information processing system to perform a particular function either directly or after a) conversion to another language, code or notation, b) playback in other data formats, or both. Means any expression in any language, code or notation.

While the invention has been described in connection with some embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted without departing from the scope of the invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the invention without departing from its scope. Therefore, it is intended that the invention not be limited to the particular embodiment disclosed, but rather that the invention will include all embodiments falling within the scope of the appended claims.

Claims (15)

As a method for video processing,
In a three-dimensional (3D) video generation device,
Encoding a plurality of regions of a captured 3D video frame, wherein the plurality of regions are associated with different depths, the encoding applying various error protection to the plurality of regions based on the associated different depths. And the encoding.
The method of claim 1, further comprising identifying one or more regions of interest from the plurality of regions of the captured 3D video frame.
The method of claim 1, wherein the higher level of error protection uses a longer error-correction code than the error-correction code used to provide a lower level of error protection.
The method of claim 1, wherein the 3D video generation device comprises a monocular 3D video generation device having one or more depth sensors, wherein the 3D video frame includes a two-dimensional (2D) video frame and corresponding depth information.
The method of claim 4, wherein one or more higher levels of error protection are applied to one or more regions of interest in the 2D video frame, and one or more lower levels of error protection are applied to one or more other regions in the 2D video frame. Applied, method.
The method of claim 4, wherein the one or more higher levels of error protection are applied to the corresponding depth information associated with one or more regions of interest in the 2D video frame, and the one or more lower levels of error protection are applied to the 2D video frame. Apply to the corresponding depth information associated with one or more other regions in the apparatus.
The method of claim 4, wherein a higher level of error protection is applied to each of the plurality of regions within the 2D video frame, and a lower level of error protection is associated with each of the plurality of regions within the 2D video frame. Method applied to corresponding depth information.
The method of claim 4, wherein the first type of error protection is applied to one or more areas of interest in the captured 2D video frame, and the second type of error protection is applied to one or more other areas in the captured 2D video frame. Applied, method.
The method of claim 4, wherein the first type of error protection is applied to the corresponding depth information associated with one or more regions of interest in the captured 2D video frame, and the second type of error protection is the captured 2D video frame. Apply to the corresponding depth information associated with one or more other regions in the apparatus.
The method of claim 1, further comprising transmitting the error protected 3D video frame to a 3D video rendering device for 3D video rendering and / or display.
A system for processing video,
The system includes one or more processors and / or circuits for use in a three-dimensional (3D) video generation device,
The one or more processors and / or circuits,
May be operable to encode a plurality of regions of the captured 3D video frame,
And the plurality of regions are associated with different depths, and the encoding applies various error protections to the plurality of regions based on the associated different depths.
The system of claim 11, wherein the one or more processors and / or circuits are operable to identify one or more regions of interest from the plurality of regions of the captured 3D video frame.
The system of claim 11, wherein the higher level of error protection uses a longer error-correction code than the error-correction code used to provide a lower level of error protection.
The system of claim 11, wherein the 3D video generation device comprises a monocular 3D video generation device having one or more depth sensors, wherein the 3D video frame includes a two-dimensional (2D) video frame and corresponding depth information.
15. The method of claim 14, wherein one or more higher levels of error protection are applied to one or more regions of interest in the 2D video frame, and one or more lower levels of error protection are applied to one or more other regions in the 2D video frame. Applied, system.

Images