Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/20—Image signal generators
  • H04N13/204—Image signal generators using stereoscopic image cameras
  • H04N13/239—Image signal generators using stereoscopic image cameras using two 2D image sensors having a relative position equal to or related to the interocular distance
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/30—Image reproducers
  • H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
  • H04N13/341—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] using temporal multiplexing
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
  • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
  • H04N23/61—Control of cameras or camera modules based on recognised objects
  • H04N23/611—Control of cameras or camera modules based on recognised objects where the recognised objects include parts of the human body
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B30/00—Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images
  • G02B30/20—Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images by providing first and second parallax images to an observer's left and right eyes
  • G02B30/22—Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images by providing first and second parallax images to an observer's left and right eyes of the stereoscopic type
  • G02B30/24—Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images by providing first and second parallax images to an observer's left and right eyes of the stereoscopic type involving temporal multiplexing, e.g. using sequentially activated left and right shutters
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N2213/00—Details of stereoscopic systems
  • H04N2213/001—Constructional or mechanical details

Definitions

• the present invention relates to stereoscopic 3D image acquisition and methods and apparatus. More particularly, the present invention relates to stereoscopic 3D image capture and viewing devices incorporating radio frequency communications.
• Another drawback includes that such systems typically reply upon expensive silver or metalized reflective screens, that maintain the appropriate polarization of light from the projector to the right and left eye images. Such screens are often too expensive for the average consumer. Yet another drawback is that because both left and right eye images are displayed to the user at the same time, despite the polarized glasses, right eye images are often visible to the left eye and left eye images are often visible to the right eye. This light pollution degrades the quality of the 3D images and can be termed as "ghosting" of 3D images. [0004] The inventors are aware of a number of techniques that may be used to reduce this ghosting effect.
• Some techniques may include deliberate degradation of left eye images to account for right eye image ghosting and the deliberate degradation of right eye images to account for left eye image ghosting.
• the inventors believe that such techniques are disadvantageous as they tend to reduce the contrast of objects in an image, and they may result in a visible halo around objects in the image.
• the inventors have recognized that 3D versions of features often do not appear as aesthetically pleasing as 2D versions of such features.
• FIG. 1 Another approach to 3D visualization has included the use of stereoscopic shutter glasses that are based upon physical shutters, or more commonly liquid crystal display (LCD) technology.
• LCD liquid crystal display
• left and right images are alternatively displayed to the user, and the right and left LCD lenses alternate between a dark and transparent state.
• the shutter glasses quickly alternate between transparency in the left then right eyes in synchronicity with an image which presents alternating left and right points of view, the left and right eyes receive different 2D images and the observer experiences the perception of depth in a 3D image.
• Fig. 1 illustrates a typical stereoscopic system.
• such systems typically include a computer 1, an infrared transmitter 3, a display 12, and a pair of liquid crystal display glasses (LCD shutter glasses) 8.
• computer 1 alternatively provides left eye images and right eye images on signal line 2, in addition to a signal that distinguishes when the left eye image or right eye image is displayed.
• IR transmitter 3 In response to the signal, IR transmitter 3 outputs infrared data 6 that indicate when the right eye image is being output and when the left eye image is being output.
• infrared data 6 For example, one simple format for infrared data is a simple square wave with a high signal indicating left and a low signal indicating right; and another format includes a 8-bit word. Because of these different data formats, IR transmitters from one manufacturer often cannot be used with LCD glasses from another manufacturer.
• infrared data 6 is received by LCD glasses 8, and in response, for example, the right LCD of the LCD glasses 8 becomes opaque and the left LCD becomes translucent (e.g. clear, neutral density), or the left LCD of the LCD glasses 8 becomes opaque and the right LCD becomes translucent.
• the right LCD becomes translucent display 12 is displaying a right eye image
• the left LCD becomes translucent display 12 is displaying a left eye image.
• One such limitation includes the difficulty in synchronizing the glasses to the images that are displayed. Synchronization data is typically based upon when the images are provided to the 3D display. Limitations to such approaches, determined by the inventors includes that both latency and timing jitter are introduced as it is processed and rendered by the 3D display device. As a result of such latency and jitter information, the LCD lenses or shutters are often opened and closed often at improper times, e.g. out of phase, with some of the image intended for the left eye being shown to the right eye and vice versa. This is perceivable by the user as ghosting effects. Additionally, as the inventors have determined that the phase difference is not constant and is subject to jitter, the user may see the image brightness change or flicker undesirably.
• One approach to reduce such latency or jitter effects has been to reduce the amount of time the left LCD shutter and the amount of time the right LCD shutter are translucent.
• the left shutter instead of the left shutter being open for example 50% of the time, the left shutter may be open 35% of the time, or the like. This reduction in open time should reduce the amount of ghosting.
• the inventors recognize drawbacks to such approach to image ghosting.
• One such drawback is the reduction in net amount of light transmitted to the user's eyes.
• the user will perceive a darkening of the images for each eye. Accordingly, a 3D version of a feature will appear darker and duller compared to a 2D version of the feature when using IR-type shutter glasses.
• IR LCD glasses based upon IR receivers often lose synchronization with the display as a result of stray reflections. For example, it has been observed by the inventors that IR LCD glasses may become confused as a result of sunlight reflecting from household objects; heat sources such candles, open flames, heat lamps; other IR remote controls (e.g. television remotes, game controllers); light sources (e.g. florescent lights); and the like. Additionally, it has been observed by the inventors that IR LCD Glasses may also lose synchronization as a result of clothing, hair, portions of other users bodies (e.g. head), or the like, that temporarily obscure an IR receiver of the LCD glasses. The loss of synchronization may lead the user to seeing a series of flickering or rolling images and / or the left eye seeing the right eye image. The inventors believe these types of anomalies are highly disturbing to most users and should be inhibited or minimized.
• manufacturers of such devices specifically instruct users to use IR LCD glasses in highly controlled environments. For example, they suggest that the 3D displays and glasses be used only in darkened rooms. The inventors believe such a solution limits the applicability and attractiveness of such 3D display devices to typical consumers. This is believed to be because most consumers do not have the luxury of a dedicated, light- controlled room for a home theater, and that most consumer entertainment rooms are multipurpose family rooms.
• each 3D display system includes its own IR transmitter and 2D field timing and phase data. Then, when two such systems are in close proximity, a user's IR LCD glasses may receive IR transmissions from either of the 3D display systems. Because of this, although a user is viewing a first 3D display, the user's 3D glasses may be synchronizing to a different 3D display, causing the user to undesirably view flickering and rolling images.
• the inventors of the present invention thus recognize that multiple 3D display systems cannot easily be used in applications such as for public gaming exhibitions, tournaments, or contests, trade shows, in stadiums, in restaurants or bars, or the like.
• stereoscopic 3D image capture the inventors have recognized that methods for capturing such images have been limited and have been beyond the reach of the average consumer.
• the inventors of the present invention recognize that capturing of stereoscopic 3D images currently requires complex 3D imaging hardware and processing software. These systems are typically dedicated to acquiring or generating stereoscopic images. For example, the left and right 2D images that are used to form stereoscopic 3D images are generated entirely by computation (e.g. animation software) or by a pair of professional grade cameras which have been firmly affixed to each other with tightly manufactured proximity and spacing, not necessarily in the prior art.
• the inventors believe that because current systems provide such narrow and specialized functionality, they are too "expensive" for typical consumers to purchase. The inventors believe that if the cost of 3D hardware and software capturing systems could be reduced, consumers would more readily embrace stereoscopic 3D imaging.
• FIG. 10 illustrates a number of devices 601-603 including cameras 605, 607 and 609, that may or may not exist, that might use such software.
• device 604 includes an off-center camera 605; device 606 includes centered camera 607, and device 603 includes camera 609.
• portions 610 and 608 of device 603 may be reoriented or repositioned with respect to each other, as shown in dotted positions 611 and 612.
• the present invention relates to stereoscopic 3D image capture and viewing methods and apparatus. More particularly, the present invention relates to stereoscopic 3D image viewing devices incorporating radio frequency communications.
• a stereoscopic 3D image viewing device is based upon liquid crystal display (LCD) shutters that are synchronized to a source of 3D images.
• the synchronization is based upon rf protocols such as Bluetooth, ZigBee radio (ZigBee Alliance), IEEE Standard 802.15.1, IEEE Standard 802.15.4, or any other type of rf communications protocol.
• the stereoscopic 3D image viewing device may transmit data back to the source of 3D images, via the rf communications mechanism or protocol, to increase the level of synchronization between the two devices.
• a system, method, and apparatus of perceiving stereoscopic 3D can be generated which improves the level of synchronization between the alternating images and the alternating action of shutter glasses.
• a system, apparatus, method, and computer-readable media are provided to enable stereoscopic viewing.
• the physical method of connecting the display system to stereoscopic glasses is the IEEE 802.15.4 wireless radio, ZigBee radio or Bluetooth technology. This allows a user to move one's head into positions that would normally lose reception of wireless transmissions (e.g. IR) thus simplifying the user experience of wearing stereoscopic glasses.
• the wireless radio connection also has the advantage of replacing the infra-red light transmission method and its associated interference with remote controls and tendency to accept interference from natural and artificial light sources, thus enhancing the user experience.
• a method for synchronization between the video transmitter and the shutter glasses. Synchronization is provided via a protocol that provides timing information such as a beacon offset or any series of packets that is used as the energy to excite a clock. A precision timing protocol may be utilized to provide synchronization between the transmitter and the receiver.
• the shutter glasses and the transmission device may include executable computer programs resident in a memory that instructs a respective processor to perform specific functions or operations, such as to transmit data, to determine a latency, or the like.
• a device for providing 3D synchronization data to a user 3D viewing device includes a receiving portion configured to receive 3D image data indicating timing of left and right images from a source of 3D data.
• a system may include a radio frequency transmitter coupled to the receiving portion, wherein the radio frequency transmitter is configured to output 3D synchronization signals to the 3D viewing device in response to the 3D image data.
• a method for transmitting stereoscopic display information includes receiving a plurality of 3D video synchronization signals from a source of 3D image data, and converting the plurality of 3D video synchronization signals into a plurality of wireless radio signals.
• a method may include outputting the plurality of wireless radio signals to a pair of shutter glasses associated with a user that are adapted to receive the wireless radio signals, wherein the plurality of wireless radio signals are adapted to change the states for a pair of LCD shutters on the pair of shutter glasses, in response to the wireless radio signals.
• a method for operating a pair of shutter glasses including a right LCD shutter and a left LCD shutter includes receiving synchronization data via radio frequency transmissions from a radio frequency transmitter, and determining shutter timings for the right LCD shutter and the left LCD shutter in response to the synchronization data.
• a technique may include applying the shutter timings to the right LCD shutter and the left LCD shutter to enable the viewer to view right-eye images via the right LCD shutter and left-eye images via the left LCD shutter.
• a method for transmitting stereoscopic display information includes converting one or more video synchronization signals into wireless radio signals; and decoding the wireless radio signal in a pair of shutter glasses or other device.
• a method for transmitting stereoscopic display information includes: converting one or more video synchronization signals into wireless radio signals; and decoding the wireless radio signal in a pair of shutter glasses or other device; wherein the wireless radio is the IEEE Standard 802.15.1, Bluetooth, or components thereof.
• a method for transmitting stereoscopic display information includes: converting one or more video synchronization signals into wireless radio signals; and decoding the wireless radio signal in a pair of shutter glasses or other device; wherein the wireless radio is the IEEE 802.15 (802.15.1-4) ZigBee radio, or components thereof.
• a method for transmitting stereoscopic display information includes: converting one or more video synchronization signals into wireless radio signals; and decoding the wireless radio signal in a pair of shutter glasses or other device; wherein the wireless radio is the IEEE Standard 802.11, WiFi, or components thereof.
• a method for transmitting stereoscopic display information includes: a pair of shutter glasses or other consumer electronics device which contains a localized clock such that the device remains synchronous to the video display system even when the connection to the source transmitting the synchronization information is interrupted or is not present.
• the synchronization information between the display system and the glasses or other device are determined by a precision timing protocol in which bidirectional communication of timing information occurs.
• a method for transmitting stereoscopic display information includes: a pair of shutter glasses or other consumer electronics device which receives synchronous information from the video display system, and a means and method of storing the delay and synchronization information in the transmitter or the video source generating computer, home theater system, or device.
• the delay and synchronization information are stored and then transmitted to multiple devices to allow multiple users to simultaneously use the same system.
• a method for transmitting stereoscopic display information includes: a pair of shutter glasses or other consumer electronics device which receives synchronous information from the video display system, and a means of determining the delay and synchronization information through information contained in the display and transmitter from the display via the video signal cable.
• a method for transmitting stereoscopic display information includes: a transmitter of synchronization information and a pair of shutter glasses or other consumer electronics device which is capable of receiving synchronization
• a method for transmitting stereoscopic display information includes: a transmitter of synchronization information and a pair of shutter glasses or other consumer electronics device which is capable of receiving synchronization information from both infrared and visible light sources.
• a method for transmitting stereoscopic display information includes: a transmitter of synchronization information and a pair of shutter glasses or other consumer electronics device which is capable of receiving synchronization information from both infrared and radio sources.
• a method for transmitting stereoscopic display information including: a transmitter of synchronization information and a pair of shutter glasses or other consumer electronics device which is capable of receiving synchronization information from both infrared and radio sources.
• a method for transmitting stereoscopic display information including: a transmitter of synchronization information and a pair of shutter glasses or other consumer electronics device which is capable of receiving synchronization information from both visible light and radio sources.
• a method for transmitting stereoscopic display information including: a transmitter of synchronization information and a pair of shutter glasses or other consumer electronics device which is capable of receiving synchronization information from infrared, visible light and radio sources.
• the shutter glasses or other receiving device can incorporate a computer program which allows the device to automatically determine which source or sources of synchronization information are available and automatically use the best source or sources.
• the shutter glasses or other receiving device can incorporate a computer program which allows the device to automatically determine which source or sources of synchronization information are available and automatically use the best source or sources.
• the shutter glasses or other receiving device can incorporate a computer program which allows the device to automatically determine which source or sources of synchronization information are available and automatically use the best source or sources. In other aspects, the shutter glasses or other receiving device can incorporate a computer program which allows the device to automatically determine which source or sources of synchronization information are available and automatically use the best source or sources. In other aspects, the shutter glasses or other receiving device can incorporate a computer program which allows the device to automatically determine which source or sources of synchronization information are available and automatically use the best source or sources. I other aspects, the shutter glasses or other receiving device can incorporate a computer program which allows the device to automatically determine which source or sources of synchronization information are available and automatically use the best source or sources.
• a method for transmitting stereoscopic display information including: a transmitter of synchronization information and a pair of shutter glasses or other consumer electronics device which is capable of receiving synchronization information from both visible light and another source, and the visible light information is also used to deduce that correct image is going to the correct eye and that the information is not reversed such that the left image is going to the right eye and vice versa.
• a method for transmitting stereoscopic display information including: a transmitter of synchronization information and a pair of shutter glasses or other consumer electronics device which is capable of receiving synchronization information where the system is capable of dynamically changing whether all viewers are sharing the same set of images or different sets of images.
• a method for transmitting stereoscopic display information including: a transmitter of synchronization information and a pair of shutter glasses or other consumer electronics device which is capable of receiving synchronization information where the system is capable of displaying a sequence such that the wearers of shutter glasses or other consumer electronics devices see stereoscopic content while viewers without glasses or without other consumer electronics devices see only the left or right image, a non-stereoscopic version of the content, a blank or solid colored screen, or an arbitrary piece of content such as an advertisement.
• anti left, anti right, or combined anti left and anti right images are incorporated into the video frame sequence.
• a method for transmitting stereoscopic display information including: a pair of shutter glasses or other consumer electronics device and a transmitter of synchronization information which has been incorporated into a mobile device either by using an unused wireless or infrared technology on the phone or by adding addition information to a protocol the phone is already using.
• the wireless technology is the IEEE Standard 802.15.1, Bluetooth, or components thereof.
• the wireless technology is the IEEE Standard 802.15.3, ZigBee radio (compliant with IEEE 802.15.4), or components thereof.
• a method for transmitting stereoscopic display information including: a pair of shutter glasses or other consumer electronics device and a transmitter of synchronization information which has been incorporated into a mobile device which has been augmented by an external cradle, dongle or device which contains additional hardware and means of providing synchronization information or image viewing.
• the cradle, dongle or device contains an image projector.
• a method of using stereoscopic glasses as ordinary sunglasses is disclosed.
• the stereoscopic glasses incorporate a visible light sensor and make automatic decisions about the appropriate level of perceived darkening.
• the level of perceived darkening is based on computer algorithms, information about the user and the environment stored on a mobile device, information retrieved from a computer network via the mobile device, and other deductions.
• the stereoscopic glasses and ordinary sunglasses are combined with a wireless headset, Bluetooth headset, or stereo Bluetooth headset.
• a method of combining stereoscopic glasses with a wireless headset, Bluetooth headset, or stereo Bluetooth headset is disclosed.
• a method of combining ordinary or automatically darkening sunglasses with a wireless headset, Bluetooth headset, or stereo Bluetooth headset is disclosed.
• Embodiments of the present invention include an imaging device including one or more image sensors (e.g. cameras) and a communications channel.
• the imaging device may be physically coupled to a general purpose consumer device such as a personal media player (e.g. iPod), a communications device (e.g. iPhone, Android-based phone), a mobile internet device, a processing device (e.g. netbook, notebook, desktop computers), or the like.
• the imaging device may utilize the communications channel (e.g. Bluetooth, Wi-Fi, ZigBee radio, IR, USB, IEEE 802.15.1, IEEE 802.15.4) to provide image data from the imaging device to the consumer device.
• the communications channel e.g. Bluetooth, Wi-Fi, ZigBee radio, IR, USB, IEEE 802.15.1, IEEE 802.15.4
• the imaging device may be used independently of the consumer device to acquire stereoscopic images, and such images may be provided to the consumer device via the communications channel.
• the consumer device may process and / or retransmit the stereoscope images to a remote server.
• stereoscopic images may be viewed on the consumer device and / or uploaded to the web (e.g. Facebook, MySpace, TwitPic), sent via e-mail, IM, or the like.
• the imaging device may capture one of the left or right pair of 2D images, and an image sensor on the general purpose consumer device may be used to capture the other 2D image.
• the imaging device may include two or more image sensors (e.g. embedded therein) and be used to capture the left and right stereoscopic pair of 2D images.
• pair of images are typically captured simultaneously or within a short amount of time apart (e.g. less than 1 second) to facilitate proper 3D image capture. This time period may increase when photographing still life, landscapes, or the like.
• users e.g.
• Embodiments for methods of stereoscopic 3D image capture could incorporate an existing phone or device, a new piece of hardware such as a cradle or dongle for an existing phone or device.
• Other embodiments may include a piece of software or computer readable method of using an existing or new device to capture stereoscopic 3D images.
• Still other embodiments may include a system and method that combines these aspects in the capture of stereoscopic 3D images.
• a consumer device for capturing stereoscopic images includes a plurality of image acquisition devices, wherein a first image acquisition device and a second image acquisition device are both approximately directed in a common direction, wherein the first image acquisition device and the second image acquisition device are displaced by a displacement, wherein the first image acquisition device is configured to capture a first image, and wherein the second image acquisition device is configured to capture a second image.
• a system may include a user input device configured to receive an input from a user, a memory configured to store the first image and the second image, and a wired or wireless communications portion configured to transmit data to a remote device.
• Various devices may include a processor coupled to the first image acquisition device, to the second image acquisition device, to the user input device, and to the communications portion, wherein the processor is configured to approximately contemporaneously direct acquisition of the first image by the first image acquisition device and of the second image by the second image acquisition device in response to the input from the user, wherein the processor is configured direct storage the first image and the second image in the memory, and wherein the processor is configured to direct the communications portion to transmit at least a portion of the first image and at least a portion of the image to a remote device.
• a method for capturing stereoscopic images, photos or videos on a mobile computing device wherein the mobile computing device includes at least a first camera and a second camera, and wherein a distance and an orientation between the first and the second cameras are determinable.
• Techniques may include receiving an initiation signal from a user, while the user points the first and the second cameras in a direction of interest, and substantially simultaneously acquiring a first image with first camera, a second image with a second camera and camera parameters, in response to the initiation signal.
• One process may include storing the first image, the second image and the camera parameters in a memory, and uploading at least a portion of the first image, at least a portion of the second image, and the camera parameters to a remote server.
• FIG. 1 is a block diagram illustrating aspects of the prior art
• FIGS. 2A-D include block diagrams of various embodiments of the present invention illustrating the process of elements of a system in which stereoscopic glasses are synchronized with the display device by incorporation of a wireless radio into the system;
• FIG. 3 illustrates a block diagram of a process according to various embodiments of the present invention.
• FIG. 4 is a timing diagram of various embodiments of the present invention illustrating a method of sending image information to a display in which the frames which compose the image are sent sequentially;
• FIG. 5 illustrates various embodiments incorporated into a mobile phone's hardware, firmware, and software and into a pair of stereoscopic shutter glasses;
• FIG. 6 illustrates various embodiments incorporated into a mobile phone, some of the methods are incorporated into a pair of stereoscopic shutter glasses, and other methods are incorporated into a cradle or other device that attaches to the mobile phone;
• FIG. 7 illustrates various embodiments incorporated into a pair of stereoscopic shutter glasses combined with a mobile phone headset;
• FIG. 8 illustrates various embodiments of the present invention
• Fig. 9 illustrates various embodiments of the present invention
• FIG. 10 is a diagram illustrating aspects of the prior art
• FIG. 11 is a diagram illustrating embodiments of the present invention including methods of incorporating stereoscopic capture capabilities into a mobile device by embedded multiple cameras;
• FIG. 12 is a diagram illustrating embodiments of the present invention including methods of incorporating stereoscopic capture capabilities into a mobile phone or other device by attaching an external dongle or fitting the device to a cradle;
• FIG. 13 is a diagram illustrating embodiments of the present invention where multiple cameras have been incorporated into a mobile device in which the relative orientation of the cameras can be changed by way of actuating a hinge which is part of the mobile device;
• FIG. 14 is a diagram illustrating embodiments of the present invention where the field of view of two cameras changes as a hinge is manipulated;
• FIG. 15 is a diagram illustrating embodiments of the present invention where the field of view of two obliquely oriented cameras are rotated and/ or cropped to specific regions of interest.
• FIGS. 2A-D illustrate various embodiments of the present invention.
• FIGS. 2A-D illustrate various arrangements of embodiments of the present invention.
• FIG. 2A includes a 3D source 34 of image data, a transmission device 37, a display 43, and shutter glasses 42.
• 3D source 34 may be a computer, a Blu-ray or DVD player, a gaming console, a portable media player, set-top-box, home theater system, preamplifier, a graphics card of a computer, a cable box, or the like
• 3D display 43 may be any 3D display device such as an LCD/Plasma/OLED display, a DLP display, a projection display, or the like.
• transmission device 37 and shutter glasses 42 may be embodied by a product developed by the assignee of the current patent application, Bit Cauldron Corporation of Gainesville, FL.
• shutter glasses 42 may be implemented with mechanical shutters or LCD shutters. For example, LCD shutters based upon pi-cell technology may be used.
• 3D source 34 sends 3D display signals to display 43 through a video cable 35, typically through a standards-based interface such as VGA, DVI, HDMI, Display Port (DP), or the like.
• Such 3D display signals are often configured as one or more interleaved full right-eye images then full left-eye images (e.g. field sequential); double wide (e.g. side by side) or double height (e.g. stacked) images including both left and right images; images interleaved with right-eye images and left-eye images on a pixel by pixel basis; or the like.
• a transmission device 37 e.g. a radio transmitter may be inserted between the 3D source 34 or other video source and 3D display 43.
• transmission device 37 determines 3D timing information by decoding the 3D display signals as they pass through to display 43 on signal line or cable 44.
• transmission device 37 includes a transmitter based upon radio frequency (rf) signals.
• the rf signals may use or may be combined with any conventional transmission protocol such as IEEE Standard 802.15.1 (e.g. Bluetooth), Wi-Fi, IEEE Standard 802.15.4 (e.g. ZigBee Alliance radio), or the like.
• IEEE Standard 802.15.1 e.g. Bluetooth
• Wi-Fi Wireless Fidelity
• IEEE Standard 802.15.4 e.g. ZigBee Alliance radio
• transmission device 37 may be a stand-alone device, e.g. a dongle, a USB "key,” or the like and transmission device 37 may be powered by power source 36 and 38, self-powered, powered from 3D data source, USB powered, or the like. In other embodiments, transmission device 37 may incorporated into another device, such as 3D source 34, 3D display 43, a pre-amplifier, or the like.
• FIG. 2B illustrates additional embodiments of the present invention.
• Fig. 2B includes a source of 3D images 100, a transmission device 110, and a 3D display 120.
• 3D image source 100 provides 3D images (e.g. double- wide or double- height images including both right and left images) to 3D display 120 via a signal line 130 such as a VGA, DVI, Display Port (DP), cable, or the like.
• 3D image source 100 provides a synchronization signal along signal line 140 to transmission device 110.
• 3D image source 100 includes an industry standard interface such as a VESA miniDIN-3 connector, VESA 1997.11, USB connector, or the like, to which transmission device 110 may be coupled.
• FIG. 2C illustrates additional embodiments of the present invention.
• Fig. 2C includes a source of 3D images 160, a transmission device 170, and a 3D display 180.
• 3D image source 160 provides 3D images (e.g. double-wide or double- height images including both right and left images) to 3D display 180 via a signal line 190 such as a VGA, DVI, HDMI cable, Display Port (DP), or the like.
• 3D display 180 provides a synchronization signal along signal line 200 to transmission device 170.
• 3D display 180 includes an industry standard interface such as a VESA miniDIN-3 connector, USB connector, or the like, to which transmission device 170 may be coupled.
• FIG. 2D illustrates other additional embodiments of the present invention.
• Fig. 2D includes a source of 3D images 220, a transmission device 230, and a 3D display 240.
• 3D image source 220 provides 3D images (e.g. double-wide or double-height images including both right and left images) to 3D display 240 via a signal line 250 such as a VGA, DVI, HDMI cable, Display Port (DP), or the like.
• transmission device 230 may be disposed within 3D display 240.
• transmission device 230 may be installed within the manufacturing facility of 3D display 240, or the like.
• 2D display 240 may also power transmission device 230.
• 3D display 240 provides a (derived) synchronization signal along signal line 260 to transmission device 230.
• shutter glasses 42 include a radio receiver 41 that receives the synchronization signals 40.
• shutter glasses 42 alternatively changes the properties of one lens from translucent to opaque (e.g. dark) to translucent and of the other lens from opaque to translucent (e.g. clear) to opaque.
• a user / viewer views 3D display images 45 from display 43 at the proper timing. More particularly, the user's right eye is then exposed to a right-eye image from 3D display images 45, and then the user's left eye is then exposed to a left-eye image from 3D display images 45, etc.
• transmission device 37 based upon a radio frequency transmitter has several advantages over an infrared transmitter.
• One advantage recognized is that radio signals can be received in many situations where an infrared signal would be blocked. For example this allows the user of a pair of 3D shutter glasses or the like, to move their head much farther away from the 3D display or transmission device than if IR were used, and allows the user to move throughout a room with a larger range of motion while maintaining synchronization with the 3D display.
• rf transmitters allow other people or objects to pass in front the user / viewer with out interrupting the signal.
• shutter glasses 42 may include its own localized clock. Benefits to such a configuration include that it allows shutter glasses 42 to remain approximately synchronized to display 43 even though the connection to transmission device 37 is interrupted and / or synchronization signals 40 are not received.
• a precision timing protocol can be used so that the clock that is local to shutter glasses 42 is synchronized with a clock within transmission device 37 and / or the 3D display signals.
• a precision timing protocol may include the transmission of data packets with a time stamp time associated with the 3D display signals to shutter glasses 42.
• the protocol may include transmission of a data packet with a time stamp associated with shutter glasses 42 to transmission device 37.
• shutter glasses 42 receive the time stamp from the 3D data source, compares the received time stamp to its local clock and returns a data packet with its local time stamp. Using this information, transmission device 37 can determine a round-trip time for data between transmission device 37 and shutter glasses 42.
• the round-trip time offset is evenly divided between transmission device 37 and shutter glasses 42.
• one or both devices are capable of determining a difference in speed or lag between the two transmissions, then a more precise determination of the relative values of both clocks (offsets) can be determined. As a result, in various embodiments, more precise synchronization between the two clocks can be established.
• the difference in rate (e.g. frequency) between the two clocks (transmission device 37 or 3D source 34 and shutter glasses 42) can be more precisely determined, hi some embodiments if there is a low degree of consistency in the latencies, the period of time between the determination of a latency process may be made small, e.g. once a minute; and if there is a higher degree of consistency in the latencies, the period of time between the determination of a latency process may be increased, e.g. once every ten minutes.
• Embodiments of the present invention enable the use of multiple pairs of shutter glasses 42. hi such embodiments, a single pair of shutter glasses 42 may be used to determine delay and jitter as discussed was discussed above. Next, a simpler protocol, such as a unidirectional or broadcast protocol, may be used by transmission device 37 to communicate this synchronization information to the remaining pairs of shutter glasses, hi various embodiments, the delay and jitter information can be stored in transmission device 37, in 3D source 34, or other consumer electronics device generating the 3D data, either in a volatile or non- volatile manner.
• a simpler protocol such as a unidirectional or broadcast protocol
• this data may be determined using bidirectional communications on cable 44, such as the DisplayPort protocol, or the like, as illustrated in FIG. 2C.
• Communications protocols such as display data channel (DDC and DDC2) protocols, PanelLink serial protocol or a similar protocols allows the display to communicate information back to the computer, home theater system, video source, or the like, hi various embodiments, this serial protocol can be enhanced to provide the appropriate latency and synchronization characteristics of 3D display 43 back to 3D source 34 and / or transmission device 37.
• these protocols can be used to determine the manufacturer, vendor, or other identifying information for 3D display 34, and a table of pre-determined synchronization information can be retrieved, either locally, across a local area network, across a network, or the like This information may include an appropriate delay and synchronization information for respective 3D displays.
• FIG. 3 illustrates a block diagram of a process according to various embodiments of the present invention. More specifically, FIG. 3 illustrates a process for synchronizing shutter glasses to a source of 3D images.
• a 3D data source provides 3D images, step L.
• the 3D images may be provided in any number of specific formats, such as right and left images: sequentially transmitted, packed vertically or horizontally into a single image and transmitted, combined on a pixel by pixel basis into a single image and transmitted, or the like, hi other embodiments, as illustrated in Fig. 2B, 3D data source may provide specific timing data.
• synchronization data such as an identifier of a timing clock resident on 3D data source is determined, step 310. hi various embodiments, this may include a packet of data including a source time stamp, or the like.
• the synchronization data may then be transmitted through radio frequency transmissions to a first pair of shutter glasses, step 320.
• the shutter glasses receive the source time stamp and synchronizes the operation of the right / left shutters to the synchronization data, step 330.
• the synchronization data can then be maintained within the shutter glasses by an internal clock within such glasses, step 340.
• the internal clock can be resynchronized.
• Such embodiments are believed to be advantageous as the glasses need not wait for synchronization data from the 3D data source to be able to switch. Accordingly, synchronization data from the transmission device may be dropped or lost while the shutter glasses continue to operate properly. When synchronization data is reestablished, the synchronization described above may be performed.
• rf communications using the ZigBee radio occur at 2.4 GHz, the same band as most Wi-Fi transmissions.
• embodiments of the present shutter glasses are designed to inhibit communications, and defer to such Wi-Fi signals.
• the shutter glasses will continue to operate autonomously, until the interference stops and new synchronization data is received from the transmission server.
• the shutter glasses may transmit data back to the rf transmission device. More specifically, the shutter glasses may transmit the received source time stamp and / or the glasses time stamp back to the transmission device via the same rf communications channel, or the like, step 350. [0095] In FIG. 3, in response to the received source time stamp and / or the glasses time stamp, and the source time stamp when these data are received, the transmission device may determine adjustments to subsequent synchronization data that will be sent to the shutter glasses, step 360. As an example, the transmission device may determine that it should output synchronization data to the shutter glasses, even before the synchronization data is determined or received from the 3D data source. As a numeric example, if it is determined that the shutter glasses lag the 3D data source by 100 microseconds, the shutter glasses may trigger its shutters 100 microseconds before the expected arrival of a synchronization pulse.
• this adjustment to synchronization data may be used to drive 3D glasses of other viewers of the 3D image.
• 3D glasses of other viewers in the room may also have synchronization data adjusted using the process described above.
• the transmission device may output the synchronization data at different times for different 3D glasses.
• the shutter glasses may verify that they are in sync. If not, the shutter glasses may adjust the frequency of its own internal clocks until they are kept in a higher amount of synchronization.
• the process may be repeated.
• the synchronization process may be performed periodically, with the period dependent upon how well the 3D data source and the shutter clock stays remain in synchronization - if highly synchronized, the synchronization process may be performed at longer time periods apart (e.g. 2 minutes) than if these devices continually have synchronization problems (e.g. every 10 seconds).
• Various embodiments of the present invention may include shutter glasses or other devices that includes multiple physical methods for receiving synchronization information.
• some embodiments may contain both an infrared and radio receiver; an infrared and visible light receiver; a radio or visible light receiver; a combination of infrared, visible light and radio receivers; or the like.
• the shutter glasses or other receiving device may include executable computer program that instructs a processor to automatically determine which communications channel or channels are available, and automatically use the communications channel having the strongest signal, lowest number of dropped data packets, or the like.
• a visible light receiver e.g. IR
• another synchronization transmission technology e.g. rf
• the information transmitted via visible light and the synchronization information transmitted via another transmission technology may be combined within shutter glasses 42 to deduce unknown elements of the delay in 3D display 43 and other synchronization information.
• the data from the different communication channels are compared to more precisely synchronize 3D display 43 and shutter glasses 42.
• the two communications channels can be used to verify that a left image displayed on 3D display 43 is going to the left eye and the right image displayed on 3D display 43 is going to the right eye. Li such an example, this would preventing the error of a reversal of synchronization information somewhere in the system that results in the sending the left image to the right eye and vice versa.
• shutter glasses 42 may be used to provide a variety of new functions.
• FIG. 4 illustrates typical video output timing where frame one 26, frame two 28, frame three 30 and frame four 32 are output sequentially.
• left images (frames) and right images (frames) are alternatively output.
• frame one 26 is left
• frame two 28 is right
• frame three 30 is left
• frame four 32 is right, creating the sequence L, R, L, R images to the user.
• Various embodiments of the present invention may be applied to 3D displays having display rates on the order of 120Hz and higher.
• the refresh rate is 120 Hz
• right and left images will be displayed and refreshed at 60 Hz. Accordingly, the viewer should not be able to detect significant flickering, however, the viewer may detect a darkening of the images.
• the higher refresh rate may enable new features, as described below.
• more than one left image and right image may be output.
• multiple viewers may view a 3D display, and different viewers may see different 3D images.
• a two viewer sequence of output images may be user 1 left, user 1 right, user 2 left, user 2 right, etc. This could be represented as: Ll, Rl, L2, R2.
• shutter glasses of a first viewer will allow the first viewer will see images Ll and Rl and a shutter glasses of a second viewer will allow the second viewer will see images L2 and R2.
• other sequences are contemplated, such as Ll, L2, Rl, R2, and the like.
• refresh rate for a 3D display having a 240 Hz refresh rate, a viewer will see the respective right and left images at a refresh rate of 60Hz. As noted above, this frequency should be above the typical sensitivity of the eye, however, viewers may detect a darker image. Such artifacts may be mitigated by increasing the brightness of the images, or the like.
• FIG. 1 Other embodiments may be extended to additional (e.g. three or more viewers). Applications of such embodiments may include for computer or console gaming, or the like. As an example, two or more viewers may initially see the same 3D image, and subsequently one or more viewers "break off to view a different 3D image. For example, three people could be playing a multiplayer game in which all three are traveling together and see the same 3D images. Next, one player then breaks away from the other players. Using the additional communications protocols disclosed in various embodiments of the present invention, the player's glasses can be reprogrammed to allow the third person to see a different 3D image. Subsequently, the third person may return to the group, and then see the same 3D image.
• additional communications protocols disclosed in various embodiments of the present invention
• a sequence of images output by the 3D display could begin with L0-R0-L0-R0, where 0 indicates everyone in the party.
• the 3D display could switch and output images in a sequence such as L1&2, R1&2, L3, R3; L1&2, L3, R1&2, R3; or the like.
• the sequence may revert to LO, RO, LO, RO. hi various embodiments, switching back and forth may occur with little, if any, visible interruption in the 3D images viewed by the viewer.
• the inventors recognize that the brightness of each frame may have to be adjusted to correct for the changes in overall viewing time.
• sequences of images enable still other types functionality.
• separate anti-left, anti-right images or both may also be sent.
• theater-goers can decide whether they care to watch the same movie or feature with or without 3D glasses; game players can play in 3D while viewers watch the same display in 2D.
• users not utilizing embodiments of the 3D glasses may view other arbitrary images.
• the viewer with 3D glasses may see the left image in the left eye and the right image in the right eye, and may not see the Arbitrary image.
• FIG. 5 illustrates additional embodiments of the present invention. More specifically, FIG. 5 illustrates a general purpose consumer device (e.g. mobile phone, personal media player, laptop, or the like) capable of 3D image output, hi such embodiments, the synchronization information to the shutter glasses may be provided by with the consumer device including embodiments of the rf transmitter described above, or unused or available transmitters available in the consumer device. Various examples may use infrared, WiFi, Bluetooth, or the like, to provide synchronization signals to shutter glasses according to embodiments of the present invention.
• a general purpose consumer device e.g. mobile phone, personal media player, laptop, or the like
• the synchronization information to the shutter glasses may be provided by with the consumer device including embodiments of the rf transmitter described above, or unused or available transmitters available in the consumer device.
• Various examples may use infrared, WiFi, Bluetooth, or the like, to provide synchronization signals to shutter glasses according to embodiments of the present invention.
• Fig. 6 illustrates additional embodiments of the present invention wherein existing consumer devices (e.g. mobile phone) may be augmented to better support stereoscopic 3D viewing.
• a cradle or dongle which attaches to the mobile device or holds the mobile device may be used, hi such examples, the cradle or dongle may incorporate a projection system such that the image may be projected at a larger size than the screen on the mobile device.
• the cradle or dongle or consumer device may also provide the synchronization signals to the shutter glasses.
• the cradle or dongle may include a ZigBee radio-type transmitter (IEEE 802.15.4) that transmits the synchronization data to the shutter glasses, or the like.
• IEEE 802.15.4 ZigBee radio-type transmitter
• stereoscopic shutter glasses that are to be used with the consumer device described above, can be used for other purposes.
• glasses incorporate a visible light sensor, they can be worn as ordinary sunglasses but make improved automatic decisions about the appropriate level of perceived darkening.
• This information can be based on computer algorithms, information about the user and the environment that is stored on a mobile device; information retrieved from a computer network via the mobile device, and the like.
• Fig. 7 illustrates yet another embodiment of the present invention.
• a user of the consumer device may desire to perform multiple functions at the same time, such as: talk on a Bluetooth headset, view stereoscopic 3D content, and wear sunglasses.
• Embodiments illustrated in Fig. 7 may include a pair of shutter glasses 57 combined with a pair of sunglasses and a Bluetooth or stereo mobile Bluetooth headset with a left earpiece 58 and a right earpiece 55, or the like.
• Fig. 8 illustrates various embodiments of the present invention, hi particular, Fig. 8 illustrates a block diagram of various embodiments of a dongle 400 providing rf transmissions, as described above.
• a physical interface 410 is illustrated.
• physical interface 410 may be a DVI port, HDMI port, Display Port (DP), USB, VESA 1997.11, or the like, for coupling to a source of 3D data (e.g. computer, DVD / BluRay player, HD display, monitor, etc.).
• the 3D data may include 3D image data
• the 3D data may include 3D timing data.
• an interface chip or block 420 may provide the electronic interface to physical interface 410.
• a processing device such as a CPLD (complex programmable logic device) 430 may be used to decode 3D synchronization data from 3D image data or 3D timing data.
• 3D synchronization data 440 is then provided to an rf interface device 450 that references a clock 440.
• rf interface device 450 is a TI CC2530 System on a Chip, that includes a 8051 MCU (processor), RAM, Flash memory, and a IEEE 802.14.4 ZigBee RF transceiver.
• the flash memory is configured to store executable computer code or instructions that directs the processor to perform various functions, as described herein.
• the flash memory includes computer code that directs the processor to transmit the 3D synchronization data to the 3D glasses, to receive timing data back from the 3D glasses, to determine a round- trip communication latency, to adjust 3D synchronization data in response to the round-trip communication latency, and the like, as described above.
• dongle 400 may include an output port or 460 driven by an output interface 470.
• the output port may be a DVI port, HDMI port, Display Port (DP), or the like providing 3D image data to a 3D display (e.g. an display, projector, etc.).
• a 3D display e.g. an display, projector, etc.
• Fig. 9 illustrates various embodiments of the present invention.
• Fig. 9 illustrates a block diagram of a pair of shutter glasses 500 according to various embodiments of the present invention.
• Shutter glasses 500 is illustrated to include an rf interface device 510 that references a clock 540 and a pair of electronically controlled LCD shutter elements 520 and 530.
• 3D synchronization data 550 is received in rf interface device 510.
• rf interface device 510 is also an TI CC2530 System on a Chip, that includes a 8051 MCU (processor), RAM, Flash memory, and a IEEE 802.14.4 ZigBee RF transceiver.
• the flash memory is configured to store executable computer code or instructions that directs the processor to perform various functions, as described herein.
• the flash memory includes computer code that directs the processor to receive the 3D synchronization data, to change the states of/ drive shutter elements 520 and 530 at the appropriate timing (e.g. Ll and Rl in the sequence Ll, L2, Rl, R2), to send clock or timing data back to a transmission device via rf communications, and the like.
• Embodiments described above may be useful for hand-held consumer devices such as cell-phones, personal media players, mobile internet devices, or the like. Other embodiments may also be applied to higher-end devices such as laptop computers, desktop computers, DVRs. BluRay players, gaming consoles, hand-held portable devices, or the like. Other embodiments may take advantage of existing IR transmission devices for IR shutter glasses. More specifically, in such embodiments, an IR to RF conversion portion may be added to receive the IR 3D output instructions and to convert them to RF 3D transmission signals, described above. In some embodiments, an RF receiver is thus used.
• the RF 3D transmission signals are then transmitted to the RF 3D shutter glasses, described above.
• Such embodiments can therefore be a simple upgrade to available IR 3D glasses transmitters.
• feedback from shutter glasses to the transmitter device described above with regards to synchronization may be used for additional purposes.
• One such embodiment may allow the 3D image source (e.g. a cable box, computer, or the like) to take the indication that a pair of shutter glasses are currently synchronized to mean a person is viewing the 3D content, and to provide that data back to a marketing company such as Media Metrics, Nielsen Ratings, or the like. By doing this, such market research companies may determine the number of viewers of specific 3D features, or the like.
• FIG. 11 illustrates various embodiments of the present invention. More specifically, Fig. 11 illustrates incorporation of more than one image sensor onto a consumer device to provide stereoscopic capture capabilities as described herein.
• a consumer device 613 such as a mobile telephone, personal media player, mobile internet device, or the like includes two imaging sensors 616 and 617 coupled to body 615.
• imaging sensors 616 and 617 are configured to acquire left and right 2D image pairs at substantially the same time (e.g. within a second or less).
• stereoscopic cameras 616 and 617 may be embedded directly into consumer device 613, and imaging sensors 616 and 617 may have a fixed position and orientation with respect to body 615.
• imaging sensors 616 and 617 may be movable within body 615 (e.g. along a track or point of rotation) or may be removable from body 615.
• cameras 616 and 617 may be affixed at a known displacement (e.g. offset or location) relative to each other. In other embodiments, the displacement between cameras 616 and 617 may be modified by the user and the displacement may be determined by consumer device 613. In various embodiments, cameras 616 and 617 may alternatively capture left and right 2D images or videos or simultaneously capture such images at any other speed fast enough to approximate simultaneity.
• the acquisition of such images may be initiated by the user via software and / or hardware portions of consumer device 613.
• the acquisition may be initiated via depression of a physical switch or the selection of a "soft" button on a display of the consumer device 613.
• executable software code stored within a memory in consumer device 613 may instruct one or more processors within consumer device 613 to acquire at least a pair of images from image sensors 616 and 617 and to store such images into a memory.
• the acquired left and / or right images may be displayed to back to the user on a display of consumer device 613. Additionally, the left / right images may processed by the one or more processors for display as a stereoscopic image pair.
• the images may be combined into a static stereoscopic image, and when a lenticular lens (e.g. prismatic) is disposed on top the display, the user may simultaneously see the left / right images with their respective left / right eyes.
• a lenticular lens may be provided in various embodiments of the present invention, in the form of a removable sheet the user places over a display of the consumer device to view 3D images.
• the lens may be part of a removable sleeve the user slips onto the consumer device to view 3D images.
• the left / right images may be uploaded to another consumer device, such as a laptop, desktop, cloud storage system, television, HD monitor, or the like, hi such embodiments, the right / left images may be displayed on a display in a time- interleaved manner and viewed by the viewer.
• another consumer device such as a laptop, desktop, cloud storage system, television, HD monitor, or the like
• the right / left images may be displayed on a display in a time- interleaved manner and viewed by the viewer.
• the optical settings and characteristics of cameras 616 and 617 may also be recorded in the memory and / or referenced.
• Such parameters or settings may be made available to various processing software (resident upon consumer device 613, or other processing device) to further deduce, capture, or process information from cameras 616 and 617.
• processing software resident upon consumer device 613, or other processing device
• estimates of distances and other measurements may be performed in three-dimensions.
• parameters or settings may include camera parameters, e.g. shutter speed, aperture, gain, contrast, and the like may be measured from a left camera and be applied to the right camera to normalize the captured left / right 2D images.
• consumer device 613 may be vertically oriented when acquiring a left / right images.
• a consumer device may be horizontally oriented when acquiring left / right image pairs.
• An example of this is illustrated by consumer device 614 (e.g. mobile phone) in Fig. 11, where cameras 618 and 620 are distributed along the long axis of device 621.
• image sensors 618 and 620 may capture right / left images at substantially the same time (or the like) to enable the generation / viewing of stereoscopic images.
• one or more additional image sensors such as camera 619 may be provided as part of the consumer device.
• camera 619 may be directed towards the user, while cameras 618 and 620 are directed away from the user.
• Such embodiments may be provided to capture not only a right / left image pair, but a reaction of the user.
• the user may use consumer device 613 to record a video of a roller coaster ride in "3D" and to contemporaneously record their reactions
• camera 619 may be installed or rotated such that cameras 618, 619 and 620 are all pointed in approximately the same direction, toward the same plane, line, or the like, towards or away from the user, or the like
• display of consumer device may also be directed towards or away from the user.
• images and camera parameters captured by cameras 618-620 may also be used as a source of stereoscopic image data, such as for 3D scene reconstruction, or the like.
• the consumer device may include one or more segments which move with respect to each other such that the stereoscopic cameras remain horizontally oriented (e.g. level) while the display or other sections of the consumer device are rotated or manipulated, hi various embodiments, the cameras may be manually leveled by the user to be horizontal disposed, and in other embodiments, the cameras may be automatically manipulated by the consumer device via feedback from one or more tilt sensors or accelerometers provided in the consumer device.
• Fig. 12 illustrates embodiments of the present invention directed towards supplementing consumer devices having an image sensor with right / left image acquisition capabilities, hi particular, FIG. 12. illustrates embodiments where stereoscopic image capture capabilities can be added to an existing consumer device (e.g. mobile phone) by means of an external cradle, dongle, or other device.
• a consumer device 622 includes a body portion 624 coupled to a dongle 626.
• Dongle 626 may include one or more image sensors, such as cameras 627 and 629.
• dongle 626 provides image data, camera parameter data, or the like to consumer device 622 via a physical and / or data communications channel, such as USB or microUSB connector, wireless (e.g. IR, Bluetooth, Wi-Fi, ZigBee radio (ZigBee Alliance), IEEE Standard 802.15.4, IEEE Standard 802.15.1), docking (e.g.
• any other method for physically restraining dongle 626 with respect to consumer device 622 is contemplated, additionally, any other transfer protocol for providing data from dongle 626 to consumer device 622 is also contemplated.
• a user may initiate capture of right / left images on dongle 626 via one or more physical buttons on dongle 626 or consumer device 622 or soft buttons on a display of consumer device 622. Similar to the embodiments described above, executable software code operating upon consumer device 622 may direct a processing device within consumer device 622 or dongle 626 to initiate acquisition of images by cameras 627 and 629. It is contemplated that consumer device 622 may send one or more instruction signals to dongle 626 via the same physical and / or data communications channel as described above. Alternatively, other communications methods and mechanisms for instructing dongle 626 are contemplated.
• dongle 626 initiates the capturing of one or more images from image sensors 627 and 629. Additionally, dongle 626 may capture image parameters from one or both of image sensors 627 and 629 to assist in capturing, normalizing, and / or generating of stereoscopic images. In various embodiments, such information may include the fixed or relative locations of cameras 627 and 629, optical parameters (e.g. aperture, shutter speed, focal length, iso, focal point in the images, and the like), level or orientation information from tilt sensor 28, and the like, hi various embodiments of the present invention, consumer device 622 may include functionality described above for dongle 626, such as tilt sensor 628, or the like.
• optical parameters e.g. aperture, shutter speed, focal length, iso, focal point in the images, and the like
• dongle 626 may be capable of using more than one communication protocol and may connect to other devices than consumer device 622.
• dongle 626 may provide right / left images directly to other users' mobile phones, computers, televisions, or the like, via Bluetooth, Wi-Fi, ZigBee radio, IEEE 802.15.1, IEEE 802.15.4, IR, or the like.
• dongle 626 may communicate with such devices either one at a time, in an interleaved manner, simultaneously, or the like.
• Fig. 12 also illustrates additional embodiments of the present invention. More particularly, a consumer device 623 is illustrated including an external device such as cradle 634 physically holding or enveloping the body 631 of consumer device 623.
• cradle 634 includes a single image sensor 632, although in other embodiments, more than one image sensor may be provided.
• cradle 634 is operated such that the image sensor 630 of consumer device 623 operates in coordination with image sensor 632 to capture right / left image pairs.
• cradle 634 may be physically coupled to consumer device 623, as illustrated, or may be physically coupled in any manner contemplated or described in the embodiments above (e.g. iPod connector, USB). As illustrated in Fig. 12, consumer device 623 is placed into an undersized opening of cradle 634, and thus consumer device 623 and cradle 634 are physically restrained with respect to each other. Further, cradle 634 and device 623 may communicate image data, sensor data, instructions to and from consumer device 623 in any manner described in the embodiments above (e.g. Bluetooth, Wi-Fi, IR).
• additional camera information such as camera parameters and / or information from a tilt sensor 633 maybe determined by cradle 634.
• cradle 634 may also communicate such information about the optical characteristics and properties of image sensor 632 to consumer device 623. Such data may be used by consumer device 623 to coordinate the actions of image sensors 630 and 632.
• camera or lens parameters from image sensor 632 may be used to set the parameters of image sensor 630.
• a gain setting from image sensor 630 may be used to set a gain setting of image sensor 632
• a shutter speed of image sensor 632 may be used to set a shutter speed of image sensor 630, and the like.
• Fig. 13 illustrates various embodiments of the present invention.
• a consumer device 635 is illustrated including two sections 639 and 642 and at least a pair of image sensors 640 and 641.
• section 635 and 642 are coupled together by a hinge or other conveyance.
• consumer device 635 may be "folded-up", or consumer device 636 may be partially opened, consumer device 637 may be fully opened, or the like, as shown.
• the displacement between sensors may vary.
• image sensor 643 is disposed upon the side or end of section 644 and image sensor (e.g. camera) 647 is disposed upon the end of side or end of section 646.
• cameras 643 and 647 are laterally displaced with respect to each other; in another case image, sensor 640 is adjacent to 41; and in another case, cameras 648 and 652 are far away from each other.
• the orientation of the two cameras in terms of their distance relative to each other, their rotation relative to each other, the tilt of the entire system and the like, are variable. Because of this, in various embodiments, the displacements between the cameras, camera parameters, image parameters and the like may be recorded. As described in the various embodiments above, such data may be used for many purposes by the consumer device, external device (e.g. desktop computer), or the like, such as determining stereoscopic images, 3D image reconstruction, or the like.
• an additional image sensor such as image sensor 45 may also be included and may provide all the benefits of more than two cameras described herein.
• the additional image sensors may be fixed or rotated such that the three cameras are pointed in the same direction (e.g. toward the same plane, line, or other geometric construction) such that stereoscopic information can be deduced for multiple orientations of the device, in opposite directions, or the like.
• Fig. 14 illustrates additional features of embodiments of the present invention illustrated in Fig. 13.
• a field of view 663 is shown for camera 664
• a field of view 671 is illustrated for camera 670, and the like.
• cameras 659 and 660 that are adjacent in consumer device 657 are "pulled apart," in consumer device 654, and cameras 664 and 670 are then separated and rotated relative to each other.
• cameras 664 and 670 may remain in the same plane as they move, however they may also change plane with respect to each other.
• tilt sensors 666 and 668 may be used to determine the tilt of each camera. These measurements may be referenced to determine a separation angle between cameras 664 and 670. Then, using the known geometry of the device, the linear displacement, or the like between cameras 664 and 670 can be determined, hi various embodiments, such information may be deduced by other means, such as installing a single tilt sensor and directly measuring the angle of a hinge 667, by deduction from the camera image data, or the like.
• Fig. 15 illustrates an example of the result of various embodiments of the present invention. More particularly, FIG.
• FIG. 15 shows an example of image data 682 and 684 from two image sensors of a consumer device that have been separated by an arbitrary distance, hi this example, image data 682 and 684 are rotated and tilted relative to each other as a result of being captured on a consumer device similar to cameras 664 and 670, in Fig. 14.
• images captured by a user are expected to be rectangular in shape and parallel to the ground.
• rectangles 681 and 683 represent level rectangular image information available from image data 682 and 684.
• lines 685 and 686 illustrate that rectangles 681 and 683 are level.
• rectangles 681 and 683 may be used to represent the right / left image pair for generating a stereoscopic 3D image.
• a consumer device 678 may display one or both of image data 681 and 683 to the user on a display as 2D images or as a 3D stereoscopic image, before or while acquiring or storing image and / or video data, hi such embodiments, the user can be provided feedback as to how to reorient the image sensors with respect to each other, to capture the desired 2D image(s).
• the inventors have determined that if images 681 and 683 do not have sufficiently overlapping subject matter, or if images 681 and 683 have narrow fields of view, a stereoscopic image formed from images 681 and 683 will not convey a significant 3D effect to the viewer.
• the feedback from consumer device 678 may be provided in real-time to the user.
• consumer device 678 may provide feed back in the form of tilt sensor feedback, to encourage the user to hold the device such that both cameras are more level, as illustrated by consumer device 655 in Fig. 14. hi practice, the inventors have determined that if the cameras are more level with respect to each other, the rectangular size of the region of interest increases.
• face recognition technology can also be used, either to override or to coordinate with the tilt sensor, to increase or maximize the area of a face which is captured by images 681 and 683.
• the system can encourage the user to reorient the system manually or the system may be able to do so automatically.
• consumer device 678 may increase the region of interest (e.g. image data 681 and 683) by encouraging the user to manually level the cameras with respect to each other; encouraging the user to manually open or close the hinge completely; encouraging the user to "zoom out," pan upwards; etc.
• the inventors have determined that automatic zooming out or panning upwards are particularly useful if face recognition technology is also included. As an example, such techniques would be useful to prevent image 681 and 683 from cropping out the eyebrows of the person illustrated in Fig. 15.
• stereoscopic 3D image data may be deduced without the need for multiple camera image capture systems, such as those shown in 613, 614, 622, 623, 635, 636 or 637 if the subject is still enough that the user can generate one image such as 682 and then translate or rotate the camera to produce another image such as 684. If the subject is not sufficiently still, multiple images can still be used to deduce the stereoscopic data that is correct for a particular image.
• a graphical display similar to the one shown in 678 may be displayed to the user, either as a flat (e.g. 2D) or stereoscopic 3D image.
• the consumer device may then provide the user with information which allows the user to translate or rotate the camera of the consumer device into a new position to take another image.
• the multiple images from a single camera may substitute for a set of single images from multiple cameras.
• the consumer device determines the above-mentioned horizontal region of interest on the first image and displays information to guide the user in taking the next picture, to increase the 3D overlap or effect.
• the user may be given written, verbal, or other instructions to take both pictures.
• Such embodiments can also assist the user in taking the first image offset to one side instead of centered. This allows the second image to be offset to the other side, the result of which is the subject is centered in the deduced stereoscopic 3D image.
• An example of such instructions is "Take a picture with the left eye on the viewfinder, then move the camera such that the right eye is on the viewfinder and take another picture.”
• the feedback that the user has generated both images appropriately can be provided after both images are taken and after the resulting stereoscopic 3D image is determined.
• the camera image data is examined to provide the user with graphical feedback to assist the user in capturing the second image.
• the user is presented with a display, audible or other information to help select the second image after the first image has been taken.
• facial recognition technology may be useful to encourage the user to translate the entire camera such that a stereoscopic image data of an entire face can be ensured, as opposed to a stereoscopic 3D image of landscape or still life, for example
• the dongle described above may be operated to acquire 2D images when semi-permanently affixed to a consumer device.
• the dongle may be operated to acquire 2D images apart from the consumer device. Subsequently, the 2D images may be provided to the consumer device using one or more of the transmission protocols described above. In such embodiments, the dongle may be stored semi-permanently affixed to consumer

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
• Stereoscopic And Panoramic Photography (AREA)

Applications Claiming Priority (6)

Publications (2)

Family

ID=42542635

Family Applications (1)

Country Status (2)

Families Citing this family (11)

Citations (4)

Family Cites Families (3)

• 2010
  • 2010-02-03 EP EP10739079.1A patent/EP2394195A4/de not_active Withdrawn
  • 2010-02-03 WO PCT/US2010/023091 patent/WO2010091113A2/en active Application Filing

Patent Citations (4)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events