Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/106—Processing image signals
  • H04N13/139—Format conversion, e.g. of frame-rate or size
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/106—Processing image signals
  • H04N13/144—Processing image signals for flicker reduction
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
  • G06T19/20—Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10016—Video; Image sequence
  • G06T2207/10021—Stereoscopic video; Stereoscopic image sequence
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/106—Processing image signals
  • H04N13/128—Adjusting depth or disparity
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N2013/0074—Stereoscopic image analysis
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N2013/0074—Stereoscopic image analysis
  • H04N2013/0085—Motion estimation from stereoscopic image signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N2213/00—Details of stereoscopic systems
  • H04N2213/002—Eyestrain reduction by processing stereoscopic signals or controlling stereoscopic devices

Definitions

• the present invention relates generally to video processing, and more particularly, to a method and apparatus for processing 3D video.
• the output of the FRC process is a new (middle) frame that is placed between successive source frames.
• a problem with the new frame can arise if there is missing information regarding an obscured or hidden object in the background. If there is missing information, FRC cannot accurately produce the middle frame. This is referred to as the occlusion problem.
• the middle frame is produced by motion estimation (determining a motion vector) and motion compensation (applying the motion vector to reduce the effects of motion), i.e., by the FRC process, then some immaterial images may be produced. This problem exists in both 2D images and in 3D images, where the hidden material can be in the left image, the right image, or both.
• FIG. 1 An example of an occlusion region is shown in Figure 1 , which shows five frames - a current frame (N), two previous frames (ND2, NDl), and two later frames (N+ 1, N+2).
• an occlusion region shows no good matching motion vectors. If there is a motion vector field discontinuity in a homogeneous region, this could lead to potential occlusion regions because the correct motion vector could not be estimated due to the discontinuity.
• the motion vector discontinuity due to occlusion could be detected be comparing the motion vectors between frames ND2 and NDl and between frames NDl and N+l . To be able to handle the occlusion region correctly, more than four consecutive frames of data will be needed, creating a complex solution. [0010] There is therefore a need to reduce judder when converting a film source, in particular with a 3D film source. There is also a need to address the occlusion problem in 3D sources.
• the judder problem can be addressed by separating a 3D image into its component left image and right image, performing FRC on the individual images, and then reordering the images for display.
• the separation and FRC processes can be reversed, in that FRC can be performed on the 3D image and the original and motion compensated images can be separated into their component left and right images.
• One way to address the occlusion problem in 3D sources is to utilize the disparity information (the difference between the left image and the right image) to help generate the hidden information.
• Figure 1 is a diagram showing an example of an occlusion region
• Figure 2 is a flowchart of a method for performing FRC on a 3D input source
• Figure 4 is a flowchart of an alternate method for performing FRC on a 3D input source
• Figure 6 is a flowchart of a method for performing 2D to 3D conversion
• Figure 8 is a flowchart of an alternate method for performing 2D to 3D conversion
• Figure 9 is a diagram showing an example application of the method of
• Figure 10 is a block diagram of an apparatus configured to perform any of the methods of Figures 2, 4, 6, or 8;
• Figure 11 is a diagram showing how 3D images assist in addressing the occlusion problem
• Figure 12 is a diagram showing an example of disparity between what a viewer's left eye and right eye sees in a 3D image
• Figure 13 is a diagram showing how background information in a 3D image can be obtained by using the disparity.
• Figure 14 is a flowchart of a method for addressing the occlusion problem.
• the motion estimation and motion compensation needs to be performed on both the left image and the right image of a 3D video frame.
• the left image and the right image need to be separated at some point during the process; the separation can occur before FRC (as shown in Figures 2 and 3) or after FRC (as shown in Figures 4 and 5).
• each left image and right image is half the size of a full frame. Processing (e.g., performing FRC) on a smaller size image is easier, and the method shown in Figure 2 may complete faster than the method shown in Figure 4.
• FIG. 2 is a flowchart of a method 200 for performing FRC on a 3D input source.
• a 3D frame is separated into a left image and a right image (step 202).
• a motion vector is calculated for the left image (step 204) and for the right image (step 206). It is noted that calculating the motion vectors for the left image and the right image (steps 204 and 206) can be reversed.
• only one motion vector may be calculated, for either the left image or the right image, and that motion vector can be applied to the other image.
• the reasoning behind this possibility is that in most cases (approximately 95% on the time), the motion vector between the left image and the right image is very similar. A slight difference between the respective motion vectors (if calculated separately) should not have an effect. If the motion vectors between the left image and the right image are substantially different, then the disparity will be different and the motion vectors should be calculated separately.
• the hardware complexity can be reduced; the determination whether only one motion vector is calculated can be based on the hardware present and is implementation- specific.
• FRC is then performed on the left image (step 208) and on the right image
• step 210 Performing the FRC (steps 208 and 210) can also be reversed, provided that the motion vectors are determined before FRC is performed. After FRC has been performed on both the left image and the right image, the converted images are reordered for display (step 212).
• a 3D frame 300 is separated into a left image 302 and a right image 304.
• a motion vector (MV]) 310 is calculated for the left image 302 and is applied to determine a motion compensated image (L2) 320.
• a motion vector (MV 2 ) 312 is calculated for the right image 304 and is applied to determine a motion compensated image (R2) 322.
• the original images 302, 304 and the motion compensated images 320, 322 are then reordered for display.
• FIG. 4 is a flowchart of an alternate method 400 for performing FRC on a 3D input source.
• a motion vector is calculated for the 3D image (step 402) and FRC is performed on the 3D image (step 404). Both the original 3D image and the motion compensated image are separated into left and right images (step 406). All of the images are then reordered for display (step 408).
• FIG. 6 is a flowchart of a method 600 for performing 2D to 3D conversion.
• a 2D image is extracted into a 3D image, including a left image and a right image (step 602).
• a motion vector is calculated for the left image (step 604) and for the right image (step 606). It is noted that calculating the motion vectors for the left image and the right image (steps 604 and 606) can be reversed. It is further noted that only one motion vector may be calculated, for either the left image or the right image, and that motion vector can be applied to the other image.
• FRC is then performed on the left image (step 608) and on the right image
• Performing the FRC (steps 608 and 610) can also be reversed, provided that the motion vectors are determined before FRC is performed. After FRC has been performed on both the left image and the right image, the converted images are reordered for display (step 612).
• FIG. 7 is a diagram showing an example application of the method of
• Figure 8 is a flowchart of an alternate method 800 for performing 2D to
• FIG. 8 is a diagram showing an example application of the method of
• Figure 8 FRC is performed on two 2D images 902, 904 to produce motion compensated images 1* 910 and 2* 912.
• the original 2D images 902, 904 and the motion compensated 2D images 910, 912 undergo 3D extraction into respective left and right images, to generate left and right images 920-934.
• Figure 10 is a block diagram of an apparatus 1000 configured to perform any of the methods 200, 400, 600, or 800. While the apparatus 1000 as shown is capable of performing any of the methods 200, 400, 600, or 800, the apparatus 1000 can be specifically configured to perform only one of the methods be removing unused components.
• the apparatus 1000 When performing the method 200, the apparatus 1000 operates as follows.
• a 3D input 1020 is provided to the 3D image separating device 1006, which separates the 3D input 1020 into left and right images.
• Motion vectors for the left and right images are calculated by the motion vector calculating device 1004.
• the frame rate conversion device 1008 performs FRC on the left and right images using the calculated motion vectors to produce motion compensated images.
• the left and right images and the motion compensated images are passed to the image reordering device 1010, where the images are reordered for display and are output 1022.
• the apparatus 1000 When performing the method 400, the apparatus 1000 operates as follows.
• a 3D input 1030 is provided to the motion vector calculating device 1004, which calculates the motion vector for the 3D input 1030.
• the frame rate conversion device 1008 performs FRC on the 3D input 1030 using the calculated motion vector to produce a motion compensated 3D image.
• the 3D input 1030 and the motion compensated 3D image are passed to the 3D input 1030 separating device 1006, which generates left and right images from the 3D image and left and right images from the motion compensated 3D image.
• the original left and right images and the motion compensated left and right images are passed to the image reordering device 1010, where the images are reordered for display and are output 1022.
• the apparatus 1000 When performing the method 600, the apparatus 1000 operates as follows.
• a 2D input 1040 is provided to the 2D to 3D image extracting device 1002, which extracts the 2D input 1040 into left and right 3D images.
• the motion vector calculating device 1004 calculates motion vectors for the left and right images.
• the frame rate conversion device 1008 performs FRC on the left and right images using the calculated motion vectors to produce motion compensated left and right images.
• the left and right images and the motion compensated left and right images are passed to the image reordering device 1010, where the images are reordered for display and are output 1022.
• the apparatus 1000 When performing the method 800, the apparatus 1000 operates as follows.
• a 2D input 1050 is provided to the motion vector calculating device 1004, which calculates the motion vector for the 2D input 1050.
• the frame rate conversion device 1008 performs FRC on the 2D input 1050 using the calculated motion vector to produce a motion compensated image.
• the 2D input 1050 and the motion compensated image are passed to the 2D to 3D image extracting device 1002, which extracts the 2D input 1050 and the motion compensated image into left and right 3D images and produces an output 1052.
• the occlusion problem in 3D images can be addressed by using both the left image and the right image together during FRC.
• the two images may contain the information missing from a 2D image due to the different information contained in the left image and the right image (also referred to as disparity information).
• Figure 11 is a diagram showing how 3D images assist in addressing the occlusion problem.
• the viewer's left eye is at a first coordinate dj along the X-axis and the viewer's right eye is at a second coordinate d 2 along the X-axis.
• By changing the angle D or the position of the left or right eye (dj or d 2 ) more information can be obtained for images in the background (plane z 2 ) as compared to images in the foreground (plane z,).
• the additional information in a 3D image (on plane Z 2 ) can be used to address the occlusion problem.
• motion estimation in block matching can be stated as, for example, the calculation of the minimization of the mean absolute difference (MAD) between the viewer's left and right eyes (d] and d 2 ).
• MAD mean absolute difference
• B denotes an Ni x N 2 block for a set of candidate motion vectors (d,, d 2 ) and signal s is at a pixel (n u n 2 ) in frame k (shown in Figure 11 as frame Z 1 ).
• T represents the transpose of the motion vectors d ⁇ d 2 .
• Figure 12 is a diagram showing an example of disparity between what a viewer's left eye and right eye sees in a 3D image.
• the viewer can see at least a portion of block A 1202, block B 1204, and block C 1206. Based on the relative positioning of block C 1206 between block A 1202 and block B 1204, the viewer can see more of block C 1206 with their left eye than with their right eye.
• the angle D can be defined as:
• FIG. 13 is a diagram showing how background information in a 3D image can be obtained by using the disparity.
• the covered portion in the image (block C 1206) is handled in a similar manner as an occlusion is handled in a normal FRC process.
• frame NDl there is an occlusion region 1302.
• the occlusion region 1302 is the amount of block C 1206 that is hidden at time t.
• the occlusion region 1304 the amount of block C 1206 that is hidden at time t+1).
• FIG. 14 is a flowchart of a method 1400 for addressing the occlusion problem.
• a stereo stream (the left and right images of a 3D video stream) is provided (step 1402).
• a disparity analysis is performed on the received stream (step 1404).
• the motion vector is estimated based on the result of the disparity analysis (step 1406) and motion compensation is performed, utilizing the motion vector (step 1408).
• FRC is then performed on the 2D stream (step 1410).
• This step includes separately performing FRC on the left image and the right image (each a 2D stream).
• 2D FRC can also be used in cases where there is a 3D input, but only a 2D display, and therefore, the FRC output needs to be a 2D stream.
• the present invention can be implemented in a computer program tangibly embodied in a computer-readable storage medium containing a set of instructions for execution by a processor or a general purpose computer; and method steps can be performed by a processor executing a program of instructions by operating on input data and generating output data.
• Suitable processors include, by way of example, both general and special purpose processors.
• a processor will receive instructions and data from a read-only memory (ROM), a random access memory (RAM), and/or a storage device.
• Storage devices suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs).
• non-volatile memory including by way of example semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs).
• the illustrative embodiments may be implemented in computer software, the functions within the illustrative embodiments may alternatively be embodied in part or in whole using hardware components such as Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), or other hardware, or in some combination of hardware components and software components.
• ASICs Application Specific Integrated Circuits
• FPGAs Field Programmable Gate Arrays

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
• Television Systems (AREA)
• Processing Or Creating Images (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (1)

Family

ID=42677898

Family Applications (1)

Country Status (6)

Families Citing this family (15)

Citations (7)

Family Cites Families (21)

• 2009
  • 2009-03-04 US US12/397,448 patent/US8395709B2/en active Active
• 2010
  • 2010-03-04 EP EP18210059.4A patent/EP3512196B1/en active Active
  • 2010-03-04 KR KR1020117023309A patent/KR20120006498A/ko not_active Ceased
  • 2010-03-04 WO PCT/CA2010/000307 patent/WO2010099616A1/en not_active Ceased
  • 2010-03-04 CN CN2010800131135A patent/CN102362503A/zh active Pending
  • 2010-03-04 JP JP2011552290A patent/JP2012519431A/ja active Pending
  • 2010-03-04 EP EP10748268.9A patent/EP2404452B1/en active Active
• 2013
  • 2013-03-05 US US13/785,274 patent/US9270969B2/en active Active

Patent Citations (7)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events