Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/20—Image signal generators
  • H04N13/204—Image signal generators using stereoscopic image cameras
  • H04N13/239—Image signal generators using stereoscopic image cameras using two 2D image sensors having a relative position equal to or related to the interocular distance
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/20—Image signal generators
  • H04N13/261—Image signal generators with monoscopic-to-stereoscopic image conversion
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/106—Processing image signals
  • H04N13/161—Encoding, multiplexing or demultiplexing different image signal components
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/106—Processing image signals
  • H04N13/167—Synchronising or controlling image signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/20—Image signal generators
  • H04N13/296—Synchronisation thereof; Control thereof
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/30—Image reproducers
  • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
  • H04N13/341—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] using temporal multiplexing
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/30—Image reproducers
  • H04N13/363—Image reproducers using image projection screens
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/106—Processing image signals
  • H04N13/15—Processing image signals for colour aspects of image signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/189—Recording image signals; Reproducing recorded image signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
  • H04N13/194—Transmission of image signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/20—Image signal generators
  • H04N13/204—Image signal generators using stereoscopic image cameras
  • H04N13/207—Image signal generators using stereoscopic image cameras using a single 2D image sensor
  • H04N13/218—Image signal generators using stereoscopic image cameras using a single 2D image sensor using spatial multiplexing
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/20—Image signal generators
  • H04N13/204—Image signal generators using stereoscopic image cameras
  • H04N13/246—Calibration of cameras
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/20—Image signal generators
  • H04N13/286—Image signal generators having separate monoscopic and stereoscopic modes
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/30—Image reproducers
  • H04N13/302—Image reproducers for viewing without the aid of special glasses, i.e. using autostereoscopic displays
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/30—Image reproducers
  • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
  • H04N13/334—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] using spectral multiplexing
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/30—Image reproducers
  • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
  • H04N13/337—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] using polarisation multiplexing
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/30—Image reproducers
  • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
  • H04N13/344—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/30—Image reproducers
  • H04N13/398—Synchronisation thereof; Control thereof
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
  • H04N19/597—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding specially adapted for multi-view video sequence encoding

Definitions

• This invention relates to systems and methods for processing and displaying video imagery. More specifically, this invention relates to systems and methods that receive a two-dimensional video signal and synthesize a three-dimensional video signal which is displayed on a display device.
• Realistic three-dimensional video is useful in entertainment, business, industry, and research. Each area has differing requirements and differing goals. Some systems that are suitable for use in one area are totally unsuitable for use in other areas due to the differing requirements. In general, however, three-dimensional video imagery must be comfortable to view for extended periods of time without having the viewing system impart stress and eye strain. In addition, system should be of sufficient resolution and quality to allow for a pleasing experience. However, prior art systems have not always accomplished these goals in a sufficient manner.
• any approach designed to produce three-dimensional video images relies on the ability to project a different video stream to each eye of viewer.
• the video streams contain visual clues that are interpreted by the viewer as a three-dimensional image.
• Many different systems have been developed to present these two video streams to different eyes of an individual. Some systems utilize twin screen displays using passive polarized or differently colored viewing lenses and glasses that are worn by the viewer in order to allow each eye to perceive a different video stream.
• Other approaches use field or frame multiplexing which utilizes a single display screen that quickly switches between the two video streams. These systems typically have a pair of shuttered glasses that are worn by an individual and the shutters alternately cover one eye and then the other in order to allow each eye to perceive a different video stream.
• some systems such as those commonly used in virtual reality systems, use dual liquid crystal or dual CRT displays that are built into an assembly worn on the viewers head. Other technologies include projection systems and various auto stereoscopic systems that do not require the wearing of glasses.
• Prior art systems that generate and display three-dimensional video imagery have typically taken one of two approaches.
• the first approach has been to employ a binocular system, e.g., two lenses or two cameras to produce two channels of visual information.
• the spacial offset of the two channels creates a parallax effect that mimics the effect created by an individual's eyes.
• the key factor in producing high quality stereoscopic video that uses two cameras is the maintenance of proper alignment of the two channels of image data.
• the alignment of the camera lenses must be maintained and the video signals generated by the cameras must maintain a proper temporal alignment as they are processed by system electronics or optics. Misalignment will be perceived as distortion to a viewer.
• Twin screen viewing systems are known to be particularly prone to misalignment, tend to be bulky and cumbersome, and tend to be rather expensive due to the cost of multiple displays.
• Single screen solutions which multiplex fields or frames tend to minimize the problems associated with dual display monitors, yet these systems also rely on the accuracy of alignment of the input video data.
• the second approach taken by various systems has been an attempt to convert an input two-dimensional video signal into a form that is suitable for stereoscopic display.
• the problems of the prior art have been successfully overcome by the present invention which is directed to systems and methods for synthesizing a simulated three-dimensional video image from a two-dimensional input video signal.
• the present invention is relatively inexpensive, produces high quality video, and has high user tolerance.
• the systems of the present invention do not rely on temporal shifting in order to create a simulated three-dimensional scene.
• certain embodiments may use temporal shifting in combination with other processing to produce simulated three-dimensional video from a two-dimensional video source.
• Traditional video sources, such as an NTSC compatible video source is composed of a sequence of frames that are displayed sequentially to a user in order to produce a moving video image.
• the frame rate for NTSC video is thirty frames per second.
• Frames are displayed on a display device, such as a monitor or television, by displaying the individual horizontal scan lines of the frame on the display device.
• televisions have been designed to display the frame by interlacing two different fields. In other words, the television first displays all the odd numbered scan lines and then interlaces the even numbered scan lines in order to display a complete frame.
• a frame is typically broken down into an even field which contains the even numbered scan lines and an odd field which contains the odd numbered scan lines.
• the present invention takes a two-dimensional video input signal and digitizes the signal so that it can be digitally processed. The digitized frame is separated into the even field and the odd field.
• the even field and/or the odd field are then processed through one or more transformations in order to impart characteristics to the field that, when combined with the other field, and properly displayed to a viewer will result in a simulated three-dimensional video stream.
• the fields are then placed in a digital memory until they are needed for display. When the fields are needed for display, they are extracted from the digital memory and sent to the display device for display to the user.
• the fields are displayed to the user in such a manner that one field is viewed by one eye and the other field is viewed by the other eye.
• Many mechanisms may be used to achieve this, including the various prior art mechanisms previously discussed.
• the system utilizes a pair of shuttered glasses that are synchronized with the display of the different fields so that one eye is shuttered or blocked during the display of one field and then the other eye is shuttered or blocked during the display of the other field.
• three-dimensional video may be viewed on a conventional display device, such as a conventional television.
• the mind when receiving signals from the eyes, will interpret the visual clues included in the video stream and will fuse the two fields into a single simulated three-dimensional image.
• the processing used to impart various characteristics to a field that will be interpreted as three-dimensional visual clues may comprise one or more transformations that occur in the horizontal and/or vertical dimension of a field.
• the fields are comprised of a matrix of sampled video data. This matrix of video data may be transformed through shifting, scaling, and other spatial transformations in order to impart appropriate visual clues that will be interpreted by the brain of a viewer in order to create the simulated three-dimensional images that are desired.
• the skewing transformation begins with a particular row or column of information and then shifts each succeeding row or column by a specified amount relative to the row or column immediately preceding it. For example, each line may be shifted a certain number of data samples in a horizontal direction relative to the row above. Data samples that extend beyond the boundary of the matrix may be dropped or may be wrapped back to the front of the row.
• transformations that have proven useful in imparting visual clues are shifting transformations where all rows or columns are shifted by a designated amount, and scaling transformations which scale rows or columns to increase or decrease the number of data samples in the rows or columns of the field.
• scaling transformations which scale rows or columns to increase or decrease the number of data samples in the rows or columns of the field.
• fill data samples may be inserted as needed through the use of interpolation or simply by picking a fixed value to insert.
• the processing of various fields through transformations occur within a single frame.
• no temporal transformation or delay is introduced into the system.
• a frame is simply broken into its component fields, the fields are transformed appropriately, and then the frame is reassembled.
• a field may be transformed and then held and recombined with other fields of a later frame.
• Figure 1 is a diagram illustrating the conceptual processing that occurs in one embodiment of the present invention
• FIG. 2 illustrates the conceptual processing that takes place in another embodiment of the present invention
• Figures 3A through 3D illustrate various transformations that may be used to impart visual clues to the synthesized three-dimensional scene
• Figures 4A through 4D illustrate a specific example using a scaling transformation
• Figure 5 illustrates temporal transformation
• Figures 6 A through 8B illustrate the various circuitry of one embodiment of the present invention.
• the present invention is directed to systems and methods for synthesizing a three-dimensional video stream from a two-dimensional video source.
• the video source may be any source of video such as a television signal, the signal from a VCR, DVD, video camera, cable television, satellite TV, or any other source of video. Since the present invention synthesizes a three-dimensional video stream from a two-dimensional video stream no special video input source is required. However, if a video source produces two video channels, each adapted to be viewed by an eye of a user, then the present invention may also be used with appropriate modification. From the discussion below, those skilled in the art will quickly recognize the modifications that should be made.
• a video signal is comprised of a plurality of frames that are intended to be displayed in a sequential fashion to the user or viewer of a display device in order to provide a moving scene for the viewer.
• Each frame is analogous to the frame on a movie film in that it is intended to be displayed in its entirety before the next frame is displayed.
• Traditional display devices such as television sets or monitors, may display these video frames in a variety of ways. Due to limitations imposed by early hardware, televisions display a frame in an interlaced manner. This means that first one sequence of lines is scanned along the monitor and the then another sequence of lines is scanned along the monitor. In this case, a television will scan the odd numbered lines first and then return and scan the even numbered lines.
• the persistence of the phosphor on the television screen allows the entire frame to be displayed in such a manner that the human eye perceives the entire frame displayed at once even though all lines are not displayed at once.
• the two different portions of the frame that are displayed in this interlaced manner are generally referred to as fields.
• the even field contains the even numbered scan lines
• the odd field contains the odd numbered scan lines.
• FIG. 1 a general diagram of the processing of one embodiment of the present invention is illustrated.
• an input video stream shown generally as 20 is comprised of a plurality of frames 22 labeled FI through F8.
• frame 24 is extracted for processing.
• frame 24 is comprised of a plurality of scan lines.
• the even scan lines of frame 24 are labeled 26 and the odd scan lines of frame 24 are labeled 28. This is done simply for notational purposes and to illustrate that a frame, such as frame 24, may be divided into a plurality of fields.
• the frame is digitized by encoder 30.
• Encoder 30 samples the video data of frame 24 and converts it from analog format to a digital format.
• Encoder 30 may also perform other processing functions relating to color correction/translation, gain adjustments, and so forth. It is necessary that encoder 30 digitize frame 24 with a sufficient number of bits per sample in order to avoid introducing unacceptable distortion into the video signal. In addition, it may be desirable to sample various aspects of the video signal separately. In NTSC video, it may be desirable to sample the luminescence and chrominance of the signal separately. Finally, the sample rate of encoder 30 must be sufficient to avoid introducing aliasing artifacts into the signal. In one embodiment, a 13.5 MHz sample rate using sixteen bits to represent the signal has been found to be sufficient for standard NTSC video. Other video sources may require different sample rates and sample sizes. In Figure 1, the digitized frame is illustrated as 32
• Digitized frame 32 is processed by modification processing component 34.
• Modification processing component 34 performs various transformations and other processing on digitized frame 32 in order to introduce visual clues into the frame that, when displayed to a viewer, will cause the frame to be interpreted as a three-dimensional image.
• a wide variety of processing may be utilized in modification processing component 34 to introduce appropriate visual clues.
• Various transformations and other processing are discussed below. In general, however, modification processing component 34 will prepare the frame to be displayed to a user so that the frame is interpreted as a three-dimensional object.
• the transformations and other processing performed by modification processing component 34 often entail separating frame 32 into two or more components and transforming one component relative to the other.
• the resultant modified frame is illustrated in Figure 1 as 36.
• controller 38 stores modified frame 36 in memory 40 until it is needed.
• modified frame 36 is extracted and sent to the appropriate display device to be displayed. This may require controller 38, or another component, to control the display device or other systems so that the information is displayed appropriately to the viewer.
• the brain will take the visual clues introduced by modification processing component 34 and fuse the two fields into a single image that is interpreted in a three-dimensional manner.
• Other mechanisms may also be utilized. These mechanisms include multidisplay systems where one eye views one display and the other eye views the other display.
• the traditional polarized or colored approach which utilizes a pair of passive glasses may also be used, as previously described.
• controller 38 is illustrated as controlling a shuttering device 42 in order to allow images multiplexed on monitor 44 to be viewed appropriately.
• decoder 46 converts modified frame 36 from a digital form to an analog form appropriate for display on monitor 44. Decoder 46 may also generate various control signals necessary to control monitor 44 in conjunction with shuttering device 42 so that the appropriate eye views the appropriate portion of frame 36. Decoder 46 may also perform any other functions necessary to ensure proper display of frame 36 such as retrieving the data to be displayed in the appropriate order.
• FIG. 2 a more detailed explanation of one embodiment of the present invention is presented.
• the embodiment of Figure 2 has many elements in common with the embodiment illustrated in Figure 1. However, a more detailed explanation of certain processing that is performed to modify the frame from two-dimensional to three-dimensional is illustrated.
• a video frame such as frame 48
• Encoder 50 represents an example of means for receiving a frame from a two-dimensional video stream and for digitizing the frame so that the frame can be processed. Encoder 50, therefore, digitizes frame 48 among other things.
• the digitized frame is illustrated in Figure 2 as digitized frame 52. Encoder 50 may also perform other functions as previously described in conjunction with the encoder of Figure 1.
• Digitized frame 52 is split by splitter 54 into odd field 56 and even field 58.
• Splitter 54 represents an example of means for separating a frame into a plurality of fields.
• Odd field 56 and even field 58 are simply representative of the ability to split a frame, such as digitized frame 52, into multiple fields. When interlaced display devices are utilized, it makes sense to split a frame into the even and odd fields that will be displayed on the device. In progressively scanned display devices, even and odd fields may be used, or other criteria may be used to split a frame into multiple fields. For example, at one time it was proposed that an advanced TV standard may use vertical scanning rather than the traditional horizontal scanning. In such a display device, the criteria may be based on a vertical separation rather than the horizontal separation as illustrated in Figure 2. All that needs happen is that splitter 54 separate frame 52 into at least two fields that will be processed separately.
• Odd field 56 and even field 58 are processed by modification processing components 60 and 62, respectively.
• Modification processing component 60 and 62 represent the conceptual processing that occurs to each of the fields separately. In actuality, the fields may be processed by the same component.
• Modification processing component 60 and 62 represent but one example of means for transforming at least one field using a selected transform. Such a means may be implemented using various types of technologies such as a processor which digitally processes the information or discrete hardware which transforms the information in the field. Examples of one implementation are presented below.
• modified odd field 64 and modified even field 66 represent the fields that are transformed by modification processing components 60 and 62, respectively. Note that although Figure 2 illustrates modified field 64 and 66, in various embodiments one, the other, or both fields may be modified.
• the fields may be transformed in any manner that is desirable to introduce appropriate visual clues into the field, as previously explained. Examples of some transforms that have been found useful to introduce visual clues in order to convert a two-dimensional video stream into a three-dimensional video stream are presented and discussed below. In general, such transforms involve shifting, scaling, or otherwise modifying the information contained in one or both fields. Note that the transforms performed by modification processing components 60 and 62 may be performed either in the horizontal direction, the vertical direction, or both. Modified fields 64 and 66 are then stored by controller 68 in memory 70 until they are needed for display. Once they are needed for display, controller 68 will extract the information in the desired order and transfer the information to decoder 72.
• controller 68 will transfer one field and then the other field for appropriate display. If, however, the display is progressively scanned, then controller 68 may supply the information in a different order.
• controller 68 represents an example of means for recombining fields and for transferring the recombined fields to a display device.
• certain of this functionality may be included in decoder 72. Decoder 72 is responsible for taking the information and converting it from a digital form to an analog form in order to allow display of the information. Decoder 72 may also be responsible for generating appropriate control signals that controls the display. In the alternative, controller 68 may also supply certain control signals in order to allow proper display and inte ⁇ retation of the information.
• a separate device such as a processor or other device, may be responsible for generating control signals that control the display device so that the information is properly displayed. From the standpoint of the invention, all that is required is that the information be converted from a digital format to a format suitable for use with the display device. Currently, in most cases this will be an analog format, although other display devices may prefer to receive information in a digital format. The display device is then properly controlled so that the information is presented to the viewer in an appropriate fashion so that the scene is interpreted as three-dimensional. This may include, for example, multiplexing one field and then the other on the display device while, simultaneously, operating a shuttering device which allows one eye to view one field and the other eye to view the other field.
• any of the display devices previously discussed may also be used with appropriate control circuitry in order to allow presentation to an individual.
• all these display systems are premised on the fact that one eye views a certain portion of the information and another eye views a different portion of the information. How this is accomplished is simply a matter of choice, given the particular implementation and use of the present invention.
• FIG. 3 A through 3D some of the transforms that have been found useful for providing visual clues that are included in the data and inte ⁇ reted by a viewer as three-dimensional.
• the examples illustrated in Figures 3 A through 3D present transformations in the horizontal direction. Furthermore, the examples illustrate transformation in a single horizontal direction. Such should be taken as exemplary only.
• a skew transform is presented. This transform skews the data in the horizontal or vertical direction.
• a field that is to be transformed is illustrated generally as 74. This field has already been digitized and may be represented by a matrix of data points. In Figure 3 this matrix is five columns across by three rows down.
• the transformations used in the present invention will shift or otherwise modify the data of the field matrix.
• Typical field matrices are hundreds of columns by hundreds of rows. For example, in NTSC video an even or odd field may contain between eight and nine hundred columns and two to three hundred rows.
• the skew transform picks a starting row or column and then shifts each succeeding row or column by an amount relative to the column or row that precedes it.
• each row is shifted by one data point relative to the row above it.
• the transformed field illustrated generally as 76
• the transformed field has row 78 being unshifted, row 80 being shifted by one data point, and row 82 being shifted by two data points.
• the data points of original matrix are thus bounded by dashed lines 84 and takes on a skew shape.
• the total shift from the beginning row to the ending row is a measure of the amount of skew added to the frame.
• the data points begin to move outside the original matrix boundaries, illustrated in Figure 3A by solid lines 86.
• holes begin to develop in the field matrix as illustrated by data points 88.
• Several options may be utilized.
• the data points are shifted they are wrapped around and placed in the holes created at the beginning of the row or column.
• row 80 when the last data point was shifted outside the field matrix boundary it would be wrapped and placed at the beginning of the row.
• the process would be similar for any other rows.
• the holes opened in the field matrix lie outside the normal visual range presented on the display, then they may simply be ignored or filled with a fixed value, such as black.
• various interpolation schemes may be used to calculate a value to place in the holes. As previously mentioned, this transformation may be performed in the horizontal direction, the vertical direction, or a combination of both.
• each row or column in the field matrix is shifted by a set amount.
• the unshifted field matrix is illustrated as 90, while the shifted field matrix is illustrated as 92.
• this again places certain data points outside the boundaries of the field matrix.
• the data points may be wrapped to the beginning of the row and placed in the holes opened up, or the holes that opened up may be filled with a different value and the data points that fall beyond the boundaries of the field matrix may simply be ignored.
• various schemes may be used to fill the holes such as filling with a fixed data point or using a myriad of inte ⁇ olation schemes.
• Figures 3C and 3D illustrate various scaling transformations.
• Figure 3C illustrates a scaling transformation that shrinks the number of data points in the field matrix while
• Figure 3D illustrates a scaling transformation that increases the number of data points. This would correspond to making something smaller and larger respectively.
• the unsealed matrix is illustrated as 96 while the scaled field matrix is illustrated by 98.
• appropriate data points are simply dropped and the remainder of the data points are shifted to eliminate any open space for data points that were dropped.
• values must be placed in the holes that are opened by the reduced number of data points. Again, such values may be from a fixed value or may be derived through some interpolation or other calculation. In one embodiment, the holes are simply filled with black data points.
• Figure 3D represents a scaling that increases the number of data points in a field matrix.
• the unsealed field matrix is illustrated by 100 and the scaled field matrix is illustrated by 102.
• the scaled field matrix is illustrated by 102.
• holes open up in the middle of the data points.
• a decision must be made as to what values to fill in the holes.
• it is typically adequate to interpolate between surrounding data values to arrive at a particular value to put in a particular place.
• any data points that fall outside the size of the field matrix are simply ignored. This means that the only values that must be interpolated and filled are those that lie within the boundaries of the field matrix.
• an untransformed frame 104 is illustrated.
• This frame comprises six rows, numbered 105 through 110 and seven columns.
• the rows of the frame are first separated into an even field and an odd field.
• Odd field 112 contains rows 105, 107, and 109 while even field 114 contains rows 106, 108 and 110.
• Such a function may be performed, for example, by a splitter or other means for separating a frame into a plurality of fields.
• Splitter 54 of Figure 2 is but one example.
• FIG 4B the process of transforming one or both fields is illustrated. In the example illustrated in 4B, odd field 112 will be transformed while even field 114 remains untransformed.
• FIG. 4C the alignment issues that can be created when a transform is applied are illustrated.
• a transform is applied that changes the number of data points in a field.
• transformed odd field 116 has ten columns instead of the normal seven.
• the fields would appropriately shifted as shown on the right hand side of Figure 4C.
• the edge of the field matrix is then indicated by dashed lines 120 and any data points that fall outside those lines can simply be discarded.
• Figure 4D the process of recombining the fields to create a simulated three-dimensional frame is illustrated.
• the left-hand side of Figure 4D illustrates transformed odd field 116 that has been cropped to the appropriate size.
• Figure 4D also illustrates even field 114.
• the frame is reconstructed by interleaving the appropriate rows as indicated on the right-hand side of Figure 4D.
• the reconstructed frame is illustrated generally as 122.
• Such a reconstruction may take place, for example, when the fields are displayed on a display device. If the display device is an interlaced display, as for example a conventional television set, then the odd field may be displayed after which the even field is displayed in order to create the synthesized three-dimensional frame.
• the synthesized three-dimensional frame is referred to as being constructed from a recombining of the various fields of the frame.
• the reconstructed frame is then illustrated as being displayed on a display device.
• the embodiments presented above have processed a frame and then displayed the same frame.
• the frame rate of the output video stream is equal to the frame rate of the input video stream. Technologies exist, however, that either increase or decrease the output frame rate relative to the input frame rate. It may be desirable to employ such technologies with the present invention.
• the first approach is simply to send the data of a frame more often. For example, if the output frame rate is doubled, the information of a frame may simple be sent twice.
• an input video stream comprising a plurality of frames is illustrated generally as 124.
• a single frame is extracted for processing.
• This frame is illustrated in Figure 5 as 126.
• the frame is broken down into a plurality of fields, as for example field 128 and 130. As previously discussed, although two fields are illustrated, the frame may be broken into more than two fields if desired.
• Modified field 130 is illustrated as field 136.
• the embodiment illustrated in Figure 5 introduces a temporal shift as illustrated by delay 138.
• Delay 138 simply holds the transformed field for a length of time and substitutes a transformed field from a previous frame.
• a field from frame 1 may not be displayed until frames 2 or 3.
• a delayed field, illustrated in Figure 5 as 140, is combined with field 136 to create frame 142.
• Frame 142 is then placed in the output video stream 144 for proper display. Referring next to Figures 6A through Figures 8B, one embodiment of the present invention is presented.
• FIGS. 6A through 8B are designed to operate with a conventional display, such as a television and shuttered glasses which operate to alternatively block one eye and the other so that one field of the frame is seen by one eye and another field of the frame is seen by the other eye.
• a conventional display such as a television and shuttered glasses which operate to alternatively block one eye and the other so that one field of the frame is seen by one eye and another field of the frame is seen by the other eye.
• processor 144 is responsible for overall control of the system.
• processor 144 is responsible for receiving various user input commands, as from a remote control or other input devices in order to allow user input for various parameters of the system.
• Such inputs may, for example, adjust various parameters in the transforms that are used to produce the synthesized three-dimensional images.
• Such an ability allows a user to adjust the synthesized three- dimensional scene to suit his or her own personal tastes.
• Processor 144 will then provide this information to the appropriate components.
• processor 144 may help perform various transformations that are used in producing the synthesized three-dimensional scenes.
• Figure 6A also illustrates a schematic representation of shuttered glasses 150, which is discussed in greater detail below.
• FIG. 6B illustrates a block level connection diagram of video board 146.
• Video board 146 will be more particularly described in conjunction with Figures 7 A through 71 below.
• Video board 146 contains all necessary video circuitry to receive a video signal, digitize the video signal, store and receive transformed fields in memory, reconvert transformed fields back to analog signals, and provide the analog signals to the display device.
• video board 146 may contain a logic to generate control signals that are used to drive shuttered glasses used by this embodiment to produce a synthesized three-dimensional effect when worn by a viewer.
• Block 148 of Figure 6C contains a schematic representation of the drivers which are used to drive the shuttered glasses.
• the shuttered glasses are illustrated schematically in Figure 6A by block 150.
• Figures 6D - 6F contain various types of support circuitry and connectors as for example, power generation and filtering, various ground connectors, voltage converters, and so forth.
• the support circuitry is labeled generally as 152.
• FIGs 7A through 71 a more detailed schematic diagram of video board 146 of Figure 6B is presented.
• Video board 146 comprises decoder 154 (Figure 7A), controller 156 ( Figure 7B), memory 158 ( Figures 7C and 7D), and encoder 162 ( Figure 7E).
• Figure 7F an alternate memory configuration is illustrated as block 160.
• Various support circuitry is illustrated in Figures 7G through 71.
• Block 164 of Figure 7G contains various input circuitry that receives video and other data from a variety of sources.
• Block 165 of Figure 7G illustrates how the pinouts of video board 146 of Figure 6B translate into signals of Figures 7 A through 71.
• Block 166 of Figures 7H and 71 contains output and other support circuitry.
• Decoder 154 ( Figure 7 A) is responsible for receiving the video signal and for digitizing the video signal. The digitized video signal is stored in memory 158 ( Figures 7C and 7D) under the control of controller 156 ( Figure 7B).
• Controller 156 is a highly sophisticated controller that basically allows information to be written into memory 158 while, information is being retrieved from memory 158 by encoder 162 ( Figure 7E) for display.
• the various frames and fields of an input video received by decoder 154 may be identified from the control signals in the video data. The fields may then be separated out for processing and transformation, as previously described.
• transformations occur in the horizontal direction, then the transformation may be applied line by line as the field is received. If, on the other hand, a transformation occurs in the vertical direction, it may be necessary to receive the entire field before transformation can occur. The exact implementation of the transformations will be dependent upon various design choices that are made for the embodiment.
• controller 156 of Figure 7B in addition to storing and retrieving information from memory 158, controller 156 also generates the control signals which drive the shuttered glasses. This allows controller 156 to synchronize the shuttering action of the glasses with the display of information that is retrieved from memory 158 and passed to encoder 162 for display on the display device. Encoder 162 ( Figure 7E) takes information retrieved from memory 158 and creates the appropriate analog signals that are then sent to the display device.
• Alternate memory 160 (Figure 7F), which is more fully illustrated in Figures 8 A and 8B, is an alternate memory configuration using different component parts that may be used in place of memory 158.
• Figure 8A illustrates the various memory chips used by alternate memory 160.
• Figure 8B illustrate how the pinouts of Figure 7F translate into the signals of Figures 8 A and 8B in pinout block 161.
• Figure 8B also illustrates filtering circuitry 163.
• the present invention produces high-quality, synthesized, three-dimensional video. Because the present invention converts a two-dimensional video source into a synthesized three-dimensional video source, the present invention may be used with any video source.
• the system will work, for example, with television signals, cable television signals, satellite television signals, video signals produced by laser disks,
• the present invention retrieves the video source, digitizes it, splits the video frame into a plurality of fields, transforms one or more of the fields, and then reassembles the transformed fields into a synthesized, three-dimensional video stream.
• the synthesized three-dimensional video stream may be displayed on any appropriate display device.
• Such display devices include, but are not limited to, multiplexed systems that use a single display to multiplex two video streams and coordinate the multiplexing with a shuttering device such as a pair of shutter glasses worn by a viewer. Additional display options may be multiple display devices which allow each eye to independently view a separate display. Other single or multidisplay devices are also suitable for use with the present invention and have been previously discussed.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
• Processing Or Creating Images (AREA)
• Controls And Circuits For Display Device (AREA)
• Inorganic Insulating Materials (AREA)

Applications Claiming Priority (5)

Publications (1)

Family

ID=26710614

Family Applications (1)

Country Status (7)

Families Citing this family (19)

Family Cites Families (1)

• 1997
  • 1997-12-24 BR BR9713629-8A patent/BR9713629A/pt not_active Application Discontinuation
  • 1997-12-24 CA CA002276190A patent/CA2276190A1/en not_active Abandoned
  • 1997-12-24 EP EP97953466A patent/EP1012822A1/en not_active Withdrawn
  • 1997-12-24 WO PCT/US1997/023941 patent/WO1998029860A1/en not_active Application Discontinuation
  • 1997-12-24 JP JP53023898A patent/JP2001507890A/ja active Pending
  • 1997-12-24 CN CN97181060A patent/CN1244278A/zh active Pending
  • 1997-12-24 AU AU57206/98A patent/AU5720698A/en not_active Abandoned

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events