WO2014168614A1 - Two-dimensional video to three-dimensional video conversion method and system - Google Patents
Two-dimensional video to three-dimensional video conversion method and system Download PDFInfo
- Publication number
- WO2014168614A1 WO2014168614A1 PCT/US2013/035843 US2013035843W WO2014168614A1 WO 2014168614 A1 WO2014168614 A1 WO 2014168614A1 US 2013035843 W US2013035843 W US 2013035843W WO 2014168614 A1 WO2014168614 A1 WO 2014168614A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- frame
- video
- viewing
- movement
- video frame
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
- G06T7/248—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving reference images or patches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/285—Analysis of motion using a sequence of stereo image pairs
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/261—Image signal generators with monoscopic-to-stereoscopic image conversion
- H04N13/264—Image signal generators with monoscopic-to-stereoscopic image conversion using the relative movement of objects in two video frames or fields
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/223—Analysis of motion using block-matching
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
Definitions
- the present invention is directed to a method and system for converting two-dimensional
- Humans have a binocular vision system that uses two eyes spaced approximately two and a half inches (approximately 6.5 centimeters) apart. Each eye sees the world from a slightly different perspective. The brain uses the difference in these perspectives to calculate or gauge distance.
- This binocular vision system is responsible for the ability to determine with relatively good accuracy the distance of an object up to approximately twenty feet away. The relative distance of multiple objects in a field of view can also be determined. Using only one eye would significantly decrease the accuracy of this distance determination.
- Videos Traditional three-dimensional movies or videos (herein after referred to generically as "videos") are made using two video sources (e.g. cameras) that are mounted side-by-side, about three (e.g. at the same distance as the separation of human eyes) to eight inches apart. This distance is often referred to as the interaxial or interoccular distance.
- the two video sources actually create two videos; one for the left eye and one for the right eye.
- Each video is made up of a series of "frames” (referred to as “frames” or "video frames”).
- projection or display of a three-dimensional video has been accomplished by projecting or displaying a plurality of videos using, for example, a color differentiation system or a polarization system. This can be done using a plurality of projection or display devices (e.g. projectors) that each display one of the videos. This has also been done using single projection or display devices (e.g. the digital or analog display system of a computer, BETAMAX® player, VCR, DVD player, blue-ray player, television) that display the videos in an overlapping or interleaving fashion.
- the human binocular vision system is able to correlate these overlapping displays automatically because each eye sees only one of the overlapping or interleaving displays.
- FIG. 1 shows an exemplary traditional color differentiatim system 20 for projecting or displaym three-dimensional video 22 that uses color for differentiation.
- the first video source 24 projects through a first color filter (e.g. red) and a second video source 26 projects through a second color filter (e.g. blue).
- the video 22 in this figure is shown with an exaggerated double image.
- Viewers wear special glasses 28 with corresponding colored lenses 30, 32.
- the first lens 30 would be a color that is the same color as one of the color filters (e.g. blue— shown as horizontal lines parallel to the bottom of the drawing page) and the second lens 32 would be a color that is the same color as the other color filter (e.g. red— shown as lines vertical lines parallel to the side of the drawing page).
- the screen display would have both colors (shown as arrows projecting from the screen).
- the eye covered by the first lens 30 would view the picture projected or displayed by the video source 24 projecting or displaying the opposite color.
- the eye covered by the second lens 32 would view the picture projected or displayed by the video source 26 projecting or displaying the opposite color.
- ChromaDepth® produces glasses (using micro-prisms with similar two-color technology) that work on the same basic principle.
- FIG. 2 shows an exemplary polarization differentiation system 40 for projecting or displaying three-dimensional video 42 that uses polarization for differentiation.
- the video 42 in this figure is shown with an exaggerated double image.
- This system takes advantage of the fact that polarized light will pass through polarized glass only if they are both polarized in the same direction.
- the first video source 44 projects through a first polarized filter (e.g. horizontal) and a second video source 46 projects through a second polarized filter (e.g. vertical).
- Viewers wear special glasses 48 with corresponding polarized lenses 50, 52.
- the first lens 50 would have the same polarization as one of the polarized filters (e.g.
- the second lens 52 would have the same polarization as the other polarized filter (e.g. shown as horizontal dashed lines).
- the eye covered by the first lens 50 would view the picture projected or displayed by the video source 44 projecting the horizontally polarized picture and the eye covered by the second lens 52 would view the picture projected or displayed by the video source 46 projecting the vertically polarized picture.
- LCD shutter glasses Another technology that is used for showing three-dimensional movies uses LCD shutter glasses.
- LCD shutter glasses have lenses that use liquid crystals and a polarizing filter that is transparent until a voltage is applied, at which time they become dark.
- An IR emitter sends an IR signal to trigger the voltage so that the lenses switch between transparent and dark in an alternating fashion, first one eye and then the other.
- This transparent dark alternating is synchronized with the refresh rate of a specialized display screen that alternates between the display of a first perspective for a first eye and a second display for the second eye using a technique called alternate-frame sequencing.
- alternate-frame sequencing used together, the LCD shutter glasses and the specialized display screen create the illusion of a three-dimensional picture (or at least three-dimensional elements of a picture).
- Three-dimensional movies have been around for a long time. But after their heyday in the 1950s, three-dimensional movies as a medium fell into decline and movie producers turned their attention to other technologies. But new technologies (including polarization differentiation systems) have made this medium more attractive and new movies are being made and released as three- dimensional movies. A primary reason for this is that there has been a significant improvement in the quality of three- dimensional movies. Another reason that three-dimensional movies are becoming popular is that the movie viewing public appears willing to pay a premium for this special effect.
- One preferred embodiment of the present invention is directed to a method for converting two-dimensional video to three-dimensional video.
- the method includes the steps of comparing at least part of video frame x to a corresponding at least part of video frame y to determine movement therebetween, calculating a movement direction and movement extent based on the determined movement, determining viewing frame L and viewing frame R based on the movement direction, and modifying viewing frame R based on the movement direction and the movement extent to create modified viewing frame R.
- the step of comparing further includes comparing a predeteimined number of pixels of video frame x to a corresponding predeteimined number of pixels of video frame y to deteimine movement therebetween.
- the step of comparing further includes the step of comparing at least one pixel at an edge of video frame x to a corresponding at least one pixel at an edge of video frame y, the step of comparing at least one pixel in the middle of video frame x to a corresponding at least one pixel in the middle of video frame y, and/or the step of comparing at least one pixel at at least two comers of video frame x to a corresponding at least one pixel at at least two comers of video frame y to determine movement therebetween.
- the step of deteimining viewing frame L and viewing frame R based on the movement direction further includes the steps of (1) setting video frame x to be viewing frame L and setting video frame y to be viewing frame R if movement direction is rightward and (2) setting video frame x to be viewing frame R and setting video frame y to be viewing frame L if movement direction is leftward.
- the step of modifying viewing frame R further includes the step of digitally distorting viewing frame R to create modified viewing frame R.
- the digital distortion may be digitally distorting viewing frame R into a stretched trapezoid to create modified viewing frame R.
- the digital distortion may be digitally distorting viewing frame R into a stretched trapezoid with edges beyond the original frame size cropped off to create modified viewing frame R.
- the present invention may also be directed to a video display device for converting two- dimensional video to three-dimensional video.
- the present invention may also be directed to one or more device-readable media storing executable instructions that, when executed, configure a video display device to convert two- dimensional video to three-dimensional video.
- FIG. 1 is a simplified perspective view of an exemplary traditional color differentiation system.
- FIG. 2 is a simplified perspective view of an exemplary traditional polarization differentiation system.
- FIG. 3 is a flowchart showing an exemplary preferred embodiment of a 10 method or system for converting two-dimensional video to three-dimensional video.
- FIG. 4 is a simplified block diagram of an exemplary preferred embodiment
- FIG. 5 is a simplified block diagram of an exemplary preferred embodiment of a system that combines pairs of consecutive video frames to create' viewing frames using an exemplary preferred embodiment of a polarization differentiation system.
- FIG. 6 is a simplified view of a set of exemplary video frames x and y each having an array of pixels (P) and graphic indications of an exemplary edge area, an exemplary middle area, and two exemplary corners in video frame x and with a corresponding exemplary edge area, a corresponding exemplary middle area, and two corresponding exemplary corners in video frame y.
- P pixels
- FIG. 7 is a flow chart of an exemplary preferred embodiment of how 25 various areas are checked in succession, one after another.
- FIGS. 8A-8D are a series of graphic representations of an exemplary preferred embodiment of the present invention when the movement is rightward.
- FIGS. 9A-9D are a series of graphic representations of an exemplary preferred embodiment of the present invention when the movement is leftward.
- FIG. 10 is a simplified view of an array of pixels (P) with elements from two consecutive frames overlaid thereon, the elements moving a few pixels (P) between 5 the two frames.
- FIG. 11 is a simplified view of an array of pixels (P) with elements from two consecutive frames overlaid thereon, the elements moving many pixels (P) between the two frames.
- FIG. 12 is a simplified screen view of an array of the evenly spaced. 10 components (X) of viewing frame R in its original form.
- FIG. 13 is a simplified screen view of an array of the components (X) of viewing frame R after elongating stretching distortion.
- FIG. 14 is a simplified screen view of an array of the components (X) of viewing frame R after trapezoidal stretching distortion.
- the present invention is directed to a method and system for converting two-dimensional (monoscopic) video to three-dimensional (stereoscopic) video.
- This invention uses minimal computational resources so that the conversion can occur in real time.
- the present invention greatly conserves resources. This makes it possible for the present invention to converting two-dimensional video to three-dimensional video in real time.
- video is used to describe movies or videos that are made up of a series of “frames” (referred to as “frames” or as “video frames”).
- frames consecutive video frames will be referred to as video frame x and video frame y.
- video frame y consecutive video frames will be referred to as video frame x and video frame y.
- video frame y would become video frame x to the frame that followed it.
- viewing frame L the video frame that is displayed to the left eye
- viewing frame R the video frame that is displayed to the right eye
- viewing frame R the video frame that is displayed to the right eye
- Each video frame and/or viewing frame displays a "picture” that includes elements or objects (referred to generally as “elements”). For example, in a “picture” of a sky, the “element” of a plane may fly across the screen. Elements may be moving elements or stationary elements.
- the term “pixel” is generally used to describe the smallest item of information in an image. Pixels are normally arranged in a two-dimensional grid. The term “pixel” is used in the present invention primarily in the form of pixels on a display device that are used to display the picture or the elements therein. It may also be used to describe the digital data of the original video frames x and y. Pixels (P) are shown and discussed, for example, in relation to FIGS. 6, 10, and 11.
- the picture may also be described as being comprised of an array of components (X) that represent small parts of the picture arranged in an array.
- the components (X) may be, for example, electronic data and/or pixels.
- analog media e.g. film and video tape
- the components (X) may be the actual celluloid.
- Components (X) are shown and discussed, for example, in relation to FIGS. 12-14. Although in some circumstances they may be the interchangeable, for purposes of the present invention what distinguishes components (X) from pixels (P) is that the components (X) can be displaced in the process of distortion. This might be understood better by looking at FIG. 12 and FIG. 14. In FIG.
- the upper right hand corner component (X) might be the same as the upper right hand corner pixel (P) (not shown), but after the frame has been distorted as shown in FIG. 14, the upper right hand corner component (X) would be out of the frame and would not be the same as the upper right hand corner pixel (P) (not shown) which would remain in the same position as it was in FIG. 12.
- video display systems e.g. VCRs, movie projectors, televisions, and other projection or display devices
- computers e.g. workstations, handheld technical devices or other programmable apparatuses
- dedicated or general purpose devices capable of projecting or displaying video
- virtually any current or future technology means capable of projecting or displaying video all of which are referred to in this specification as "video display devices.”
- Video display devices may be made up of one or more processing units.
- a video display device may also be a combination of devices such as a television and an attached "box” that together function as a video display device of the present invention.
- a video display device of the present invention may be implemented in two temporally and/or physically distinct stages, for example having a recording stage (e.g. burning or recording a CD, DVD, or tape) and a playback stage (e.g. playing the CD, DVD, or tape).
- the video display devices may be built specifically for the present invention and/or may be programmed or otherwise adapted for use with the present invention.
- a method of the present invention may be encoded and/or stored on a medium capable of being "read” by a video display device or a device working in conjunction with a video display device.
- This medium includes, but not limited to memory media (e.g. RAM, PROM, EPROM, or FLASH-EPROM), magnetic media (e.g. floppy disks, flexible disks, hard disks, or magnetic tapes), optical media (e.g. CDROMs and DVDs), physical media (e.g.
- the memory may be volatile and/or non-volatile.
- the memory may be integral with and/or distinct from the video display device. As an example, if the video display device was a computer, the memory could be computer-readable media having computer-executable instructions thereon that, when executed, direct the video display device to convert two-dimensional video to three-dimensional video in accordance with the present invention.
- the present invention may be implemented as a method for converting two-dimensional video to three-dimensional video. It should be noted that the present invention may be implemented as a system for converting two-dimensional video to three-dimensional video. It should be noted that a "system" may be a video display device and/or one or more device-readable media storing executable instructions that, when executed, configure a video display device to convert two- dimensional video to three-dimensional video.
- FIGS. 3 and 7 are flow charts illustrating methods and systems. Even if described only in terms of steps in a method, it will be understood that each block of these flow charts, and combinations of blocks in these flow charts, may be implemented by software (e.g. program instructions, software programs, and subprograms), by hardware (e.g. processors and memory), by firmware, and/or a combination of these forms.
- program instructions may be loaded onto a video display device (or the memory of the video display device) to produce a machine, such that the instructions that execute on the video display device create structures for implementing the functions specified in the flow chart block or blocks.
- program instructions may also be stored in a memory that can direct a video display device to function in a particular manner, such that the instructions stored in the memory produce an article of manufacture including instruction structures that implement the function specified in the flow chart block or blocks.
- the program instructions may also be loaded onto a video display device to cause a series of operational steps to be performed on or by the video display device to produce a video display device implemented process such that the instructions that execute on the video display device provide steps for implementing the functions specified in the flow chart block or blocks.
- blocks of the flow charts support combinations of steps, structures, and/or modules for performing the specified functions. It will also be understood that each block of the flow charts, and combinations of blocks in the flow charts, may be divided and/or joined with other blocks of the flow charts without affecting the scope of the invention.
- the present invention begins with a single two-dimensional video.
- the prior art discussed in connection with FIGS. 1 and 2 begins with two two-dimensional videos, each made from a separate one of two video sources.
- FIG. 3 is a flowchart showing a method for converting two-dimensional video to three-dimensional video, which includes four basic steps.
- the first step as shown in step 100, is comparing at least part of video frame x to a corresponding at least part of video frame y to determine movement therebetween.
- the second step as shown in step 102, is calculating a movement direction and movement extent based on the determined movement.
- the third step as shown in step 104, is determining viewing frame L and viewing frame R based on the movement direction.
- the fourth step is modifying viewing frame R based on the movement direction and the movement extent to create modified viewing frame R'.
- steps 100, 102, 104, and 106 may be repeated for multiple consecutive frames.
- the method may be implemented not only by software, but as a system that uses hardware, firmware, and/or a combination of hardware, firmware, and/or software.
- FIGS. 4 and 5 are simplified block diagrams that may be used to explain two exemplary embodiments of the present invention.
- FIG. 4 might be used in an exemplary preferred embodiment using a color differentiation system
- FIG. 5 might be used in an exemplary preferred embodiment using an exemplary polarization differentiation system that takes advantage of modern systems that can display more frames per second than prior systems. For example, traditional systems only display 24 frames per second, but more modern (and future) systems can display 60, 120, 144, or more frames per second.
- F1 -F5 are used to represent consecutive frames in a video. Two consecutive frames would be designated as video frame x (the first) and video frame y (the second).
- both the viewing frame L and the modified viewing frame R' may be modified (viewing frame L) or further modified (modified viewing frame R') so that they are viewable only by a respective eye (e.g. they may have colored filters applied to them). It should be noted that this modification or further modification may occur before or after the distortion modification of modified viewing frame R.
- the bottom row of boxes in FIG. 5 represents the interleavingly displayed viewing frame L and modified viewing frame R. It should be noted that both the viewing frame L and the modified viewing frame R' may be modified (viewing frame L) or further modified (modified viewing frame R') so that they are viewable only by a respective eye (e.g. as shown by the directional arrows below the bottom row of boxes, the frames may be displayed in alternating polarities). It should be noted that this modification or further modification may occur before or after the distortion modification of modified viewing frame R'.
- Step 100 in FIG. 3 is to compare at least part of video frame x to a corresponding at least part of video frame y to determine movement therebetween. Since preferred embodiments of the present invention only compare part of video frame x to a corresponding part of video frame y, computational resources are conserved. As compared to traditional comparison schemes that compare all the pixels in a first frame to all the pixels in a second frame, the present invention only uses significantly less than 1% of the computational resources used by a system that compares all of video frame x to all of video frame y.”
- FIG. 6 shows a simplified set of video frames x and y with exaggerated pixels (P).
- One exemplary screen might have a 1920x1080 array of pixels (P).
- step 100 might be accomplished using pseudo-pattern recognition to determine movement of pixels (P) therebetween.
- pseudo-pattern recognition scheme could be used for this purpose, for the purpose of providing examples and/or enablement, exemplary schemes that could be used for this purpose are disclosed in U.S. Patent Publication No. 20070217685 to Kaneko et al., U.S. Patent Publication No. 20070146380 to Nystad et al, U.S. Patent Publication No. 20090028425 to Cavallaro et al, U.S. Patent No. 5,406,501 to Florent, and U.S. Patent No.
- step 100 might be accomplished by comparing a predetermined number of pixels (P) of video frame x to a corresponding predetermined number of pixels (P) of video frame y to determine movement therebetween.
- the predetermined number of pixels (P) might be between 5 and 30 pixels (P) for the exemplary display having a 1920x1080 array of pixels (P).
- the predetermined number of pixels (P) may be a simple preset number, a number determined based on characteristics of the screen (e.g.
- the predetermined number of pixels (P) will be less than 10% of the total number of pixels of the screen. It should be noted that in alternative preferred embodiments of the present invention, the predetermined number of pixels (P) will be less than 1 % of the total number of pixels of the screen.
- step 100 might be accomplished by comparing at least one pixel (P) at an edge of video frame x to a corresponding at least one pixel (P) at an edge of video frame y to determine movement therebetween, by comparing at least one pixel (P) in the middle of video frame x to a corresponding at least one pixel (P) in the middle of video frame y to determine movement therebetween, and/or by comparing at least one pixel (P) at at least two comers of video frame x to a corresponding at least one pixel (P) at at least two comers of video frame y to determine movement therebetween.
- FIG. 6 shows video frame x with an exemplary edge area 120, an exemplary middle area 122, and two exemplary comers 124a, 124b and video frame y with a corresponding exemplary edge area 120', a corresponding exemplary middle area 122', and two corresponding exemplary comers 124a', 124b'.
- These exemplary areas are not meant to limit the scope of the invention.
- step 100 might be accomplished by comparing at least one pixel (P) at an edge of video frame x to a corresponding at least one pixel (P) at an edge of video frame y to determine movement therebetween, by comparing at least one pixel (P) in the middle of video frame x to a corresponding at least one pixel (P) in the middle of video frame y to determine movement therebetween, and/or by comparing at least one pixel (P) at at least two comers of video frame x to a corresponding at least one pixel (P) at at least two comers of video frame y to determine movement therebetween (the latter comparison meaning at least one pixel at at least one comer, and at least one pixel at at least a second comer—using the two comers would be particularly effective if the scene were zooming in, i.e.
- FIG. 7 is a flow chart that shows an example of how this might work.
- At 130 at least one pixel (P) at an edge of video frame x is compared to a corresponding at least one pixel (P) at an edge of video frame y to determine movement therebetween.
- Decision 132 asks whether movement has been found at 130. If it has, the analysis would be complete and the next step would be step 102 of FIG. 3 (calculating a movement direction and movement extent based on the determined movement). On the other hand, if no movement has been found, as shown at 134, at least one pixel (P) in the middle of video frame x would be compared to a corresponding at least one pixel (P) in the middle of video frame y to determine movement therebetween. Decision 136 asks whether movement has been found at 134.
- step 102 of FIG. 3 calculating a movement direction and movement extent based on the determined movement.
- at least one pixel (P) at at least two comers of video frame x would be compared to a corresponding at least one pixel (P) at at least two comers of video frame y to determine movement therebetween.
- Decision 140 asks whether movement has been found at 138. If it has, the analysis would be complete and the next step would be step 102 of FIG. 3 (calculating a movement direction and movement extent based on the determined movement). On the other hand, if no movement has been found there areseveral possible scenarios.
- video frame x and video frame y are compared.
- all the pixels (P) of video frame x and video frame y are compared.
- the analysis ends and video frame x and video frame y are displayed.
- step 134 the middle area
- the choice of which area is considered first may be "intelligent.” For example, if motion is not found in step 130 (edge area), but is found in step 134 (middle area), the next set video frames might be compared using step 134 first. This option takes advantage of the fact that motion is likely to be similar in a series of frames. For example, if a movie scene has people moving in the center of the screen, then the series of frames will have motion in that area.
- FIGS. 4 and 5 show simplified block diagrams of a series of video frames and how consecutive video frames are combined to create viewing frames using a color differentiation system (FIG. 4) and a polarization differentiation system (FIG. 5). More detail on the steps (e.g. steps 102, 104, and 106 from FIG. 3) between the video frames x and y of a video and the viewing frames L and R', is shown in FIGS. 8A-8D and FIGS. 9A-9D. The most significant difference between FIGS. 8A-8D and FIGS. 9A-9D is that FIGS. 8A-8D show movement in a rightward (left to right) direction and FIGS. 9A- 9D show movement in a leftward (right to left) direction.
- FIGS. 8A-8D show movement in a rightward (left to right) direction
- FIGS. 9A- 9D show movement in a leftward (right to left) direction.
- the second step 102 (FIG. 3) of the preferred embodiment of the exemplary process of the present invention is calculating a movement direction and movement extent based on the determined movement. This step is performed using the information obtained from step 100 (comparing at least part of video frame x to a corresponding at least part of video frame y to determine movement therebetween).
- two sequential video frames (labeled as x and y) have movement in their middle areas (shown as rectangles 150a, 150b).
- the movement direction of rectangle 150a is rightward in that the rectangle 150a in video frame x is in the left half of the frame and the rectangle 150a in video frame y has moved towards the right to a more central location of the frame.
- the movement direction of rectangle 150b is leftward in that the rectangle 150b in video frame x is in the right half of the frame and the rectangle 150b in video frame y has moved towards the left to a more central location of the frame.
- Leftward or rightward would be the movement direction,
- the other directions e.g. upward and downward
- still other directions e.g. at an angle
- the movement extent is how far the movement has gone between video frame x and video frame y.
- the movement extents might be measured in pixels (P).
- Slow movements e.g. a cloud element floating gently across a picture in a series of many frames
- Fast movements e.g. a race car element speeding across a picture in a series of frames
- "Few" and "many” are relative based on a predetermined number of pixels (P). This predetermined number of pixels (P) would be the "ideal" number of pixels (P) that would be needed to create a "shadow! effect.
- the ideal number of pixels (P) might be between 10 and 20 pixels (P).
- the predetermined number of pixels (P) will be discussed as fifteen (15) pixels (P).
- FIG. 10 shows an array of pixels (P) (that would only be a small section of the exemplary screen) with elements 152, 154 from two consecutive frames overlaid thereon.
- the elements 152, 154 are shown as circles with plus signs therein (element 154 is shown in phantom). Using the center of the plus signs and ignoring upward/downward movement, only a few pixels (P) (shown as four (4)) are between the elements 152, 154 of the frames. If element 152 is the first element (viewing frame x) and element 154 is the second element (viewing frame y), then the movement direction would be leftward and the movement extent would be four (4) pixels (P). If element 154 is the first element (viewing frame x) and element 152 is the second element (viewing frame y), then the movement direction would be rightward and the movement extent would still be four (4) pixels (P).
- FIG. 11 shows an array of pixels (P) (that would only be a small section of the exemplary screen) with elements 156, 158 from two consecutive frames overlaid thereon.
- the elements 156, 158 are shown as circles with plus signs therein (element 158 is shown in phantom).
- pixels (P) shown as eleven (11) between the elements 156, 158 of the frames. If element 156 is the first element (viewing frame x) and element 158 is the second element (viewing frame y), then the movement direction would be leftward and the movement extent would be eleven (11) pixels (P). If element 158 is the first element (viewing frame x) and element 156 is the second element (viewing frame y), then the movement direction would be rightward and the movement extent would still be eleven (11) pixels (P).
- the third step 104 is determining viewing frame L and viewing frame R based on the movement direction.
- video frame x is set to be viewing frame L and video frame y is set to be viewing frame R. This is shown as the transition between FIG. 8A and FIG. 8B.
- video frame x is set to be viewing frame R and video frame y is set to be viewing frame L. This is shown as the transition between FIG. 9A and FIG. 9B.
- the fourth step 106 (FIG. 3) of the preferred embodiment of the exemplary process of the present invention is modifying viewing frame R based on the movement direction and the movement extent to create modified viewing frame R'.
- this step in its original form, the components (X) of the picture shown in viewing frame R are evenly spaced in an array as shown in FIG. 12.
- the movement direction determines which direction viewing frame R is modified to create viewing frame R.
- the movement extent determines how much viewing frame R is modified to create viewing frame R.
- step 106 includes digitally "distorting" viewing frame R to create modified viewing frame R'.
- the movement direction determines the direction of the "distortion" of the picture shown in the frame.
- the term “distorting” may be any type of distortion, but is shown in the examples as elongating stretching (FIG. 13) and trapezoidal stretching (FIG. 14) (which will be discussed below).
- FIG. 12 As shown by the change from the original picture (FIG. 12) to the modified viewing frame R' picture (FIG. 13) that has elongated stretched distortion, because the movement direction is rightward, the left edge is held in place and the right edge is digitally pulled.
- components (X) are in their original form, substantially evenly spaced.
- FIG. 13 shows the viewing frame distorted (elongated stretched) such that components (X) of the picture closer to the left edge would be less distorted and components (X) of the picture closer to the right side would be more distorted. It is not individual elements (such as the rectangle 150a of FIG. 8B) that are distorted, but components of the entire "picture” that are distorted (although the individual elements are distorted proportionally with the rest of the components (X) of the picture). It should be noted that if the movement direction had been leftward the right edge would have been held in place and the left edge would have been digitally pulled (elongated stretched) similar to how it is shown in FIG. 9C. How much the picture is distorted depends on the movement extent. If the movement extent is small (e.g.
- the viewing frame R could bedisplayed as modified viewing frame R without additional distortion. Edges beyond the original frame size (shown in phantom) are preferably "cropped off or otherwise not displayed as part of modified viewing frame R.
- step 106 includes digitally "distorting" viewing frame R to create trapezoidally stretched modified viewing frame R' such as that shown in FIG. 14.
- FIG. 14 shows the viewing frame distorted (elongated stretched) such that components (X) of the picture closer to the left edge would be less distorted and components (X) of the picture closer to the right side would be more distorted.
- the vertical distortion is the same as or similar to (e.g. 50% to 150%) the horizontal distortion.
- the horizontal distortion is 10 pixels (P) (or a length equivalent to 10 pixels (P))
- the rectangle 150a has been distorted to a trapezoid 150a' substantially proportional to the distortion of the rest of the picture. It should be noted that if the movement direction had been leftward, the right edge would have been held in place and the left edge would have been digitally distorted as shown in FIG. 9C. How much the picture is distorted depends on the movement extent as discussed in the previous example. If the movement extent is more than the ideal number, in preferred embodiments the viewing frame R could be displayed as modified viewing frame R' without additional distortion or only with vertical distortion. Edges beyond the original frame size (shown in phantom) may be "cropped off or otherwise not displayed as part of modified viewing frame R.
- FIG. 8D and FIG. 9D show an unmodified viewing frame L and the primary modification discussed for modified viewing frame R
- both viewing frame L and modified viewing frame R may have additional modifications that enable the viewing frames to be viewable by specific eyes.
- frame L may be modified for viewing with a user's left eye
- modified frame R may be modified for viewing with a user's right eye.
- This modification may be, for example, removing a first color (e.g. red) from viewing frame L and removing a second color (e.g. blue or green) from modified viewing frame R.
- this modification may be polarizing in a first direction (e.g. vertical) from viewing frame L and polarizing in a second direction (e.g. horizontal) from viewing frame R.
- viewing frame L and modified viewing frame R could be modified for other technologies known and yet to be developed (e.g.. those using LCD shutter glasses).
- step of modifying viewing frame L and modified viewing frame R so that they are viewable by specific eyes may occur at several different points in the method of the present invention.
- this modification may occur before, after, or simultaneously with step 102 (calculating a movement direction and movement extent based on the determined movement).
- this modification may occur before, after, or simultaneously with step 104 (determining viewing frame L and viewing frame R based on the movement direction).
- this modification may occur before, after, or simultaneously with step 106 (modifying viewing frame R based on the movement direction and the movement extent to create modified viewing frame R").
- Preferred embodiments of the present invention preferably include a step of displaying viewing frame L for viewing by a user's left eye and modified viewing frame R for viewing by a user's right eye. This "displaying" may occur simultaneously as discussed in connection with FIG. 4 or interleavingly as discussed in connection with FIG. 5.
- Alternative technologies e.g. LCD shutter glasses
- This step of displaying may be accomplished using a projection or display device.
- Another prefened embodiment of the present invention preferably includes the step of storing viewing frame L for viewing by a user's left eye and modified viewing frame R for viewing by a user's right eye.
- memory e.g. memory mediums, magnetic mediums, optical mediums
- the newly created three-dimensional video can be saved to a CD, DVD, or tape for later display.
- Preferred embodiments of the method discussed herein are designed to be implemented on a video display device that includes at least one processing unit for implementing the steps discussed.
- Processing units include, but are not limited to CPUs (central processing unit), GPUs (graphics processing unit), math processors, and/or specially adapted processors known or yet to be developed. Multiple processing units may be used in combination.
- a CPU may send instructions and geometry data to the GPU located on a graphics card.
- the instructions that a GPU uses are in the form of a compiled shading language program ("shader").
- shader in the field of computer graphics, is a set of software instructions that is used primarily to calculate rendering effects on graphics hardware with a high degree of flexibility.
- shaders are vertex shaders, pixel shaders, and geometry shaders. Shaders are able to apply transformations to a large set of elements at a time, for example, to each pixel (P) in an area of the screen, or for every vertex of a model. Shaders may be used, for example, for trapezoidal stretching or distortion and/or for applying filters (e.g. color and/or polarization).
- filters e.g. color and/or polarization
- the term “or” is used in its nonexclusive form (e.g. "A or B” includes A, B, A and B, or any combination thereof, but it would not have to include all of these possibilities). It should be noted that, unless otherwise specified, “and/or” is used similarly (e.g. “A and/or B” includes A, B, A and B, or any combination thereof, but it would not have to include all of these possibilities). It should be noted that, unless otherwise specified, the term “includes” means “comprises” (e.g. a device that includes or comprises A and B contains A and B but optionally may contain C or additional components other than A and B). It should be noted that, unless otherwise specified, the singular forms "a,” “an,” and “the” refer to one or more than one, unless the context clearly dictates otherwise.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Signal Processing (AREA)
- Computer Graphics (AREA)
- Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
- Transforming Electric Information Into Light Information (AREA)
- Controls And Circuits For Display Device (AREA)
- Processing Or Creating Images (AREA)
Abstract
Description
Claims
Priority Applications (9)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2016507524A JP6333952B2 (en) | 2013-04-09 | 2013-04-09 | Convert 2D video to 3D video based on motion between video frames |
KR1020157031808A KR101729883B1 (en) | 2013-04-09 | 2013-04-09 | Two-dimensional video to three-dimensional video conversion method and system |
RU2015147541A RU2642367C2 (en) | 2013-04-09 | 2013-04-09 | System and method of transforming two-dimensional videorecord to three-dimensional videorecord |
CN201380077274.4A CN105531997B (en) | 2013-04-09 | 2013-04-09 | Method for transformation and system of the two-dimensional video to 3 D video |
EP13881474.4A EP2984820A4 (en) | 2013-04-09 | 2013-04-09 | Two-dimensional video to three-dimensional video conversion method and system |
AU2013385831A AU2013385831B2 (en) | 2013-04-09 | 2013-04-09 | Two-dimensional video to three-dimensional video conversion method and system |
CA2909245A CA2909245C (en) | 2013-04-09 | 2013-04-09 | Two-dimensional video to three-dimensional video conversion method and system |
PCT/US2013/035843 WO2014168614A1 (en) | 2013-04-09 | 2013-04-09 | Two-dimensional video to three-dimensional video conversion method and system |
SG11201508332YA SG11201508332YA (en) | 2013-04-09 | 2013-04-09 | Two-dimensional video to three-dimensional video conversion method and system |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
PCT/US2013/035843 WO2014168614A1 (en) | 2013-04-09 | 2013-04-09 | Two-dimensional video to three-dimensional video conversion method and system |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2014168614A1 true WO2014168614A1 (en) | 2014-10-16 |
Family
ID=51689873
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2013/035843 WO2014168614A1 (en) | 2013-04-09 | 2013-04-09 | Two-dimensional video to three-dimensional video conversion method and system |
Country Status (9)
Country | Link |
---|---|
EP (1) | EP2984820A4 (en) |
JP (1) | JP6333952B2 (en) |
KR (1) | KR101729883B1 (en) |
CN (1) | CN105531997B (en) |
AU (1) | AU2013385831B2 (en) |
CA (1) | CA2909245C (en) |
RU (1) | RU2642367C2 (en) |
SG (1) | SG11201508332YA (en) |
WO (1) | WO2014168614A1 (en) |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO1999012127A1 (en) * | 1997-09-02 | 1999-03-11 | Dynamic Digital Depth Research Pty Ltd | Image processing method and apparatus |
KR20100127423A (en) * | 2009-05-26 | 2010-12-06 | 엘지전자 주식회사 | Apparatus and method for displaying in image display device |
KR20110120170A (en) * | 2010-04-28 | 2011-11-03 | 주식회사 알티캐스트 | Apparatus and method for processing image data |
US20120002279A1 (en) * | 2010-06-01 | 2012-01-05 | Jain Sunil K | 2d quality enhancer in polarized 3d systems for 2d-3d co-existence |
KR20130010613A (en) * | 2011-07-19 | 2013-01-29 | 엘지전자 주식회사 | Apparatus and method for converting 2 dimensional image into 3 dimensional image |
Family Cites Families (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH07307961A (en) * | 1994-05-13 | 1995-11-21 | Pioneer Electron Corp | Stereoscopic display method and device therefor |
JP2951230B2 (en) * | 1994-09-22 | 1999-09-20 | 三洋電機株式会社 | Method for generating 3D image from 2D image |
JPH08149510A (en) * | 1994-11-18 | 1996-06-07 | Sanyo Electric Co Ltd | Two-dimensional three-dimensional video image conversion method |
JP4226730B2 (en) * | 1999-01-28 | 2009-02-18 | 株式会社東芝 | Object region information generation method, object region information generation device, video information processing method, and information processing device |
WO2001039512A1 (en) * | 1999-11-26 | 2001-05-31 | Sanyo Electric Co., Ltd. | Device and method for converting two-dimensional video to three-dimensional video |
US8325220B2 (en) * | 2005-12-02 | 2012-12-04 | Koninklijke Philips Electronics N.V. | Stereoscopic image display method and apparatus, method for generating 3D image data from a 2D image data input and an apparatus for generating 3D image data from a 2D image data input |
RU2454025C2 (en) * | 2006-12-19 | 2012-06-20 | Конинклейке Филипс Электроникс Н.В. | Method and system to convert two-dimensional video into three dimensional video |
RU92595U1 (en) * | 2009-12-24 | 2010-03-20 | Михаил Михайлович Слепко | DEVICE FOR FORMING 3D STEREOSCOPIC IMAGE FROM 2D MONOSCOPIC IMAGE |
US10158846B2 (en) * | 2010-12-06 | 2018-12-18 | Teranex Systems, Inc. | Pseudo-3d forced perspective methods and devices |
US9300947B2 (en) * | 2011-03-24 | 2016-03-29 | Kodak Alaris Inc. | Producing 3D images from captured 2D video |
CN102724530B (en) * | 2012-05-29 | 2014-10-22 | 清华大学 | Three-dimensional method for plane videos based on feedback control |
-
2013
- 2013-04-09 SG SG11201508332YA patent/SG11201508332YA/en unknown
- 2013-04-09 RU RU2015147541A patent/RU2642367C2/en not_active IP Right Cessation
- 2013-04-09 WO PCT/US2013/035843 patent/WO2014168614A1/en active Application Filing
- 2013-04-09 AU AU2013385831A patent/AU2013385831B2/en not_active Ceased
- 2013-04-09 CN CN201380077274.4A patent/CN105531997B/en not_active Expired - Fee Related
- 2013-04-09 EP EP13881474.4A patent/EP2984820A4/en not_active Withdrawn
- 2013-04-09 CA CA2909245A patent/CA2909245C/en not_active Expired - Fee Related
- 2013-04-09 KR KR1020157031808A patent/KR101729883B1/en active IP Right Grant
- 2013-04-09 JP JP2016507524A patent/JP6333952B2/en not_active Expired - Fee Related
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO1999012127A1 (en) * | 1997-09-02 | 1999-03-11 | Dynamic Digital Depth Research Pty Ltd | Image processing method and apparatus |
KR20100127423A (en) * | 2009-05-26 | 2010-12-06 | 엘지전자 주식회사 | Apparatus and method for displaying in image display device |
KR20110120170A (en) * | 2010-04-28 | 2011-11-03 | 주식회사 알티캐스트 | Apparatus and method for processing image data |
US20120002279A1 (en) * | 2010-06-01 | 2012-01-05 | Jain Sunil K | 2d quality enhancer in polarized 3d systems for 2d-3d co-existence |
KR20130010613A (en) * | 2011-07-19 | 2013-01-29 | 엘지전자 주식회사 | Apparatus and method for converting 2 dimensional image into 3 dimensional image |
Also Published As
Publication number | Publication date |
---|---|
AU2013385831B2 (en) | 2016-09-01 |
CA2909245A1 (en) | 2014-10-16 |
EP2984820A4 (en) | 2017-03-22 |
SG11201508332YA (en) | 2015-11-27 |
KR101729883B1 (en) | 2017-04-24 |
CN105531997A (en) | 2016-04-27 |
KR20160022295A (en) | 2016-02-29 |
CN105531997B (en) | 2018-07-13 |
CA2909245C (en) | 2018-02-27 |
RU2642367C2 (en) | 2018-01-24 |
AU2013385831A1 (en) | 2015-11-12 |
EP2984820A1 (en) | 2016-02-17 |
JP2016519905A (en) | 2016-07-07 |
RU2015147541A (en) | 2017-05-12 |
JP6333952B2 (en) | 2018-05-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6611837B2 (en) | Video presentation device, method and program | |
US8922628B2 (en) | System and process for transforming two-dimensional images into three-dimensional images | |
Devernay et al. | Stereoscopic cinema | |
KR101583289B1 (en) | Method for image correction at ovelapped region of image, computer readable medium and executing device thereof | |
JP2015043573A (en) | Correction method of video overlapping area, recording medium, and starting device | |
US20140306954A1 (en) | Image display apparatus and method for displaying image | |
KR101713875B1 (en) | Method and system for generation user's vies specific VR space in a Projection Environment | |
US20180184066A1 (en) | Light field retargeting for multi-panel display | |
CN104702929A (en) | Method of correcting distortion of image overlap area, recording medium, and execution apparatus | |
US11375179B1 (en) | Integrated display rendering | |
US10595000B1 (en) | Systems and methods for using depth information to extrapolate two-dimentional images | |
US9172940B2 (en) | Two-dimensional video to three-dimensional video conversion based on movement between video frames | |
US20120120190A1 (en) | Display device for use in a frame sequential 3d display system and related 3d display system | |
EP3057316A1 (en) | Generation of three-dimensional imagery to supplement existing content | |
CN110870304B (en) | Method and apparatus for providing information to a user for viewing multi-view content | |
US20120120191A1 (en) | Image processor for use in a frame sequential 3d display system and related 3d display system | |
AU2013385831B2 (en) | Two-dimensional video to three-dimensional video conversion method and system | |
JP6457964B2 (en) | Apparatus, projection apparatus, display apparatus, image generation apparatus, method thereof, and program | |
US11245889B1 (en) | Perspective based green screening | |
WO2012157459A1 (en) | Stereoscopic view image generating system | |
WO2019208143A1 (en) | Image generation device, image generation method and program | |
JP2019013012A (en) | data structure |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 201380077274.4 Country of ref document: CN |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 13881474 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 2909245 Country of ref document: CA Ref document number: 2016507524 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2013881474 Country of ref document: EP |
|
ENP | Entry into the national phase |
Ref document number: 20157031808 Country of ref document: KR Kind code of ref document: A |
|
ENP | Entry into the national phase |
Ref document number: 2015147541 Country of ref document: RU Kind code of ref document: A |
|
ENP | Entry into the national phase |
Ref document number: 2013385831 Country of ref document: AU Date of ref document: 20130409 Kind code of ref document: A |