New! View global litigation for patent families

US20100314631A1 - Display-pixel and photosensor-element device and method therefor - Google Patents

Display-pixel and photosensor-element device and method therefor Download PDF

Info

Publication number
US20100314631A1
US20100314631A1 US12828280 US82828010A US2010314631A1 US 20100314631 A1 US20100314631 A1 US 20100314631A1 US 12828280 US12828280 US 12828280 US 82828010 A US82828010 A US 82828010A US 2010314631 A1 US2010314631 A1 US 2010314631A1
Authority
US
Grant status
Application
Patent type
Prior art keywords
photosensor
display
element
device
light
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US12828280
Inventor
Lester F. Ludwig
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NRI R&d Patent Licensing LLC
Original Assignee
Avistar Communications Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N3/00Scanning details of television systems
    • H04N3/10Scanning details of television systems by means not exclusively optical-mechanical
    • H04N3/12Scanning details of television systems by means not exclusively optical-mechanical by switched stationary formation of lamps, photocells or light relays
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration, e.g. from bit-mapped to bit-mapped creating a similar image
    • G06T5/001Image restoration
    • G06T5/003Deblurring; Sharpening
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, TV cameras, video cameras, camcorders, webcams, camera modules for embedding in other devices, e.g. mobile phones, computers or vehicles
    • H04N5/225Television cameras ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, video cameras, camcorders, webcams, camera modules for embedding in other devices, e.g. mobile phones, computers or vehicles
    • H04N5/235Circuitry or methods for compensating for variation in the brightness of the object, e.g. based on electric image signals provided by an electronic image sensor
    • H04N5/238Circuitry or methods for compensating for variation in the brightness of the object, e.g. based on electric image signals provided by an electronic image sensor by influencing the optical part of the camera, e.g. diaphragm, intensifier, fibre bundle
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00Television systems
    • H04N7/14Systems for two-way working

Abstract

A display-pixel and photosensor-element device for use as a display and a camera, the device comprising a plurality of light-emitting diode (LED) display elements and a plurality of light-sensitive photosensor devices, together fabricated onto an essentially planar surface so as to create a device that can be used as a display and a camera. In an implementation, a plurality of micro-optic structures can be associated with the plurality of light-sensitive devices. In various exemplary implementations, the LEDs may comprise organic light emitting diodes (OLEDs) or stacked organic light emitting diode (SOLED). In an implementation, the light-emitting diode and the light-sensitive device are integrated into a single element. In an implementation, the device is configured to serve as either or both of a color light sensor array and a display comprising color LEDs.

Description

    CROSS-REFERENCE TO RELATED PATENT APPLICATIONS
  • [0001]
    The present application claims the benefit of and is a continuation application of U.S. patent application Ser. No. 12/419,229 filed on Apr. 6, 2009, which is a Divisional of U.S. patent application Ser. No. 09/601,384 filed on Jul. 27, 2000, which was an application filed under 35 U.S.C. §371 of International Patent Application No. PCT/US1999/001789 filed Jan. 27, 1999, which claims the benefit of U.S. Provisional Application No. 60/072,762 filed on Jan. 27, 1998, the disclosures of all of which are incorporated by reference herein.
  • FIELD OF THE INVENTION
  • [0002]
    The present invention relates generally to multimedia conferencing systems, and more particularly to multimedia-enabled communication and computing devices. Still more particularly, the present invention is a device for providing real-time multimedia conferencing capabilities to one or more companion computers or on a stand-alone basis.
  • BACKGROUND
  • [0003]
    Early computers were large, clumsy, difficult-to-operate and unreliable room-sized systems shared within a single location. Similarly, early video and graphics teleconferencing systems suffered from the same drawbacks, and were also shared within a single location. With regard to computers, technological innovations enabled the advent of desktop “personal computers.” Relative to teleconferencing systems, new technologies were also introduced, such as those described in U.S. Pat. No. 5,617,539, entitled “Multimedia Collaboration System with Separate Data Network and A/V Network Controlled by Information Transmitting on the Data Network,” that brought high-quality, reliable video and graphics teleconferencing capabilities to a user's desktop. In both early desktop personal computers and conferencing systems, there were and remain many incompatible implementations.
  • [0004]
    Digital technology innovations targeted at working in conjunction with market forces gave rise to standardized desktop computer platforms, such as Microsoft/Intel machines and Apple machines, which have existing and strengthening unifying ties between them. The standardization of converging platforms unified fragmentations that existed within the computer hardware and software industries, such that immense economies of scale lowered the per-desktop development and manufacturing costs. This in turn greatly accelerated desktop computer usage and promoted the interworking between applications such as work processing, spreadsheet, and presentation tool applications that freely exchange data today. As a result, businesses employing such interworking applications became more efficient and productive. The push for greater efficiency has fueled the development of additional innovations, which further led to developments such as the explosion in electronic commerce as facilitated by the world-wide Internet.
  • [0005]
    Relative to present-day desktop conferencing, there are many networking approaches characterized by varying audio/video (A/V) quality and scalability. In recent years, customers have assumed a wide range of positions in their investments in such technologies. At one end of this range, various types of dedicated analog A/V overlay networks exist that deliver high-quality A/V signals at a low cost. At another end of this range are local area data network technologies such as switched Ethernet and ATM data hubs that function with high-performance desktop computers. These desktop computers and data networking technologies currently support only lower-quality A/V capabilities at a relatively high cost. Despite this drawback, these desktop computers and data networking technologies are believed to be the preferred path for eventually providing high-quality A/V capabilities at a low cost. Other A/V networking solutions, such as ISDN to the desktop, also lie in this range.
  • [0006]
    Within each of many separate networked A/V technology “islands,” various approaches toward providing multimedia applications such as teleconferencing, video mail, video broadcast, video conference recording, video-on-demand, video attachments to documents and/or web pages, and other applications can be performed only in fragmented ways with limited interworking capability. For many years, it has been projected that the desktop computer industry and the data networking industry will solve such fragmentation and interworking problems, and eventually create a unified, low-cost solution. Several generations of these technologies and products have consistently fallen short of satisfying this long-felt need. Furthermore, it is likely to be disadvantageous to continue to rely upon the aforementioned industries to satisfy such needs. For example, if the introduction of today's standardized multi-method fax technology had been held back by those who maintain that the idea that all electronic text should only be computer ASCII (as advocated, for example, by M.I.T. Media Lab Director Negroponte), a great amount of the fax-leveraged domestic and international commerce that has occurred since the early 1980's may not have occurred. Desktop multimedia technologies and products are currently in an analogous position, as it is commonly accepted that it will be only the desktop computer and data networking industries that at some point in the future will make high-quality networked A/V widely and uniformly available, and at the same time it is doubtful that this will occur any time soon.
  • [0007]
    What is sorely needed, given the pace and market strategies of the desktop computer and data networking industries, is an integration of separate technology and application islands into a single low-cost, manufacturable, reliable real-time multimedia collaboration apparatus capable of supporting a wide range of A/V networking technologies; A/V applications; and A/V and data networking configurations in a wide variety of practical environments. A need also exists for a design or architecture that makes such an apparatus readily adaptable to future technological evolution, such that the apparatus may accommodate evolving or new families of interrelated standards.
  • SUMMARY OF THE INVENTION
  • [0008]
    The inventive methodology is directed to methods and systems that substantially obviate one or more problems associated with conventional photo sensors and display devices.
  • [0009]
    In accordance with one aspect of the inventive methodology, there is provided a display-pixel and photosensor-element device for use as a display and a camera. The inventive display-pixel and photosensor-element device incorporates multiple display elements, each display element including a light-emitting diode; and multiple photosensor elements, each photosensor element including a light-sensitive device. In the inventive device, the multiple display elements and multiple photosensor elements are fabricated onto an essentially planar surface so as to create a device that is usable as a display and a camera.
  • [0010]
    Additional aspects related to the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Aspects of the invention may be realized and attained by means of the elements and combinations of various elements and aspects particularly pointed out in the following detailed description and the appended claims.
  • [0011]
    It is to be understood that both the foregoing and the following descriptions are exemplary and explanatory only and are not intended to limit the claimed invention or application thereof in any manner whatsoever.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • [0012]
    FIG. 1 is a high-level block diagram of a multimedia collaboration device constructed in accordance with the present invention.
  • [0013]
    FIG. 2 is a high-level perspective view illustrating a box package for the multimedia collaboration device.
  • [0014]
    FIG. 3 is a high-level drawing of a plug-in card package for the multimedia collaboration device, which also includes a bus interface.
  • [0015]
    FIG. 4 is a perspective view of a stand-alone package for the multimedia collaboration device, which includes a camera, a display, a microphone array, and speakers.
  • [0016]
    FIG. 5 is a block diagram of a first embodiment of a multimedia collaboration device constructed in accordance with the present invention, and which provides primary and auxiliary (AUX) support for analog audio/video (A/V) input/output (I/O), and further provides support for networked digital streaming.
  • [0017]
    FIG. 6 is a block diagram of a second embodiment of a multimedia collaboration device, which provides primary support for analog audio I/O and digital visual I/O, and further supports analog and digital auxiliary A/V I/O, plus networked digital streaming.
  • [0018]
    FIG. 7 is a block diagram of a third embodiment of a multimedia collaboration device, which provides primary support for analog audio I/O and digital visual I/O, support for digital auxiliary A/V I/O, and support for networked digital streaming.
  • [0019]
    FIG. 8 is a block diagram of an adaptive echo-canceled stereo microphone and stereo speaker arrangement within an audio signal conditioning unit of the present invention.
  • [0020]
    FIG. 9 is a block diagram of an adaptive echo-canceled mono-output synthetic aperture microphone arrangement, assuming stereo speakers, within the audio signal conditioning unit, which is of particular value in noisy environments such as office cubicles or service depot areas.
  • [0021]
    FIG. 10 is an illustration showing an exemplary localized primary hot-spot, within which the synthetic aperture microphone has enhanced sensitivity to sound waves produced by a user.
  • [0022]
    FIG. 11 is an illustration showing exemplary primary hot-spot directivity, where the synthetic aperture microphone captures or rejects directionally-specific sound energy from a user within a primary hot-spot that is offset relative to that shown in FIG. 10.
  • [0023]
    FIG. 12 is an illustration showing exemplary reflected speech energy rejection by the synthetic aperture microphone.
  • [0024]
    FIG. 13 is an illustration showing exemplary ambient audio noise rejection by the synthetic aperture microphone.
  • [0025]
    FIG. 14 is a block diagram of a first embodiment of a first and a second multimedia network interface provided by the present invention.
  • [0026]
    FIG. 15 is a block diagram of a second embodiment of a first and a second multimedia network interface provided by the present invention.
  • [0027]
    FIG. 16 is an illustration of a first photosensor and display element planar interleaving technique.
  • [0028]
    FIG. 17 is an illustration of an exemplary photosensor element color and display element color distribution scheme.
  • [0029]
    FIG. 18 is an illustration of a second alternating photosensor and display element interleaving technique, in which photosensor and display element geometries and size differentials aid in minimizing pixel pitch and maximizing displayed image resolution.
  • [0030]
    FIG. 19 is a cross-sectional view showing a full-color pixel array integrated with a photosensor element array upon a common substrate.
  • [0031]
    FIG. 20 is a cross-sectional view showing an integrated full-color pixel/photosensor element, which may form the basis of an integrated display element/photosensor element array.
  • [0032]
    FIG. 21 is a cross-sectional view of a first full-color emitter/detector.
  • [0033]
    FIG. 22 is a cross-sectional view of a second full-color emitter/detector.
  • [0034]
    FIG. 23 is a cross-sectional view of a third full-color emitter/detector.
  • [0035]
    FIG. 24 is a top-view of an exemplary microoptic layer having different optical regions defined therein.
  • [0036]
    FIG. 25 is an illustration showing individually-apertured photosensor elements capturing light from portions of an object and outputting signals to an imaging unit.
  • DETAILED DESCRIPTION 4.1 General Provisions
  • [0037]
    The present invention comprises a device that provides analog audio/video and/or digital audio/visual (both referred to herein as A/V) multimedia collaboration capabilities to a user coupled to a multimedia network, such as a Multimedia Local Area Network (MLA/V) as described in U.S. Pat. No. 5,617,539 the disclosure of which is incorporated herein by reference.
  • [0038]
    The present invention may operate either in conjunction with one or more user's computers or in a stand-alone manner, and may support two-way videoconferencing, two-way message publishing, one-way broadcast transmission or reception, one-way media-on-demand applications, as well as other audio, video, and/or multimedia functionality or operations. The present invention may support such multimedia functionality across a wide range of multimedia network implementations, including mixed analog and digital and/or all-digital multimedia networks. When used in conjunction with a companion computer (i.e., desktop, laptop, special-purpose workstation or other type of computer), the present invention may operate as a high-performance multimedia processing device that offloads potentially computation-intensive multimedia processing tasks from the companion computer.
  • [0039]
    The present invention unifies several previously segregated or disparate audio-, video-, and/or multimedia-related technologies in a single physical device that supports multiple multimedia applications and multiple network signal formats and standards. Such technologies may include hardware and/or software that provide audio signal processing, analog-to-digital (A-D) and digital-to-analog (D-A) conversion, compression and decompression, signal routing, signal level control, video conferencing, stored video-on-demand, internet browsing, message publishing, and data networking capabilities. Heretofore, these technologies were typically implemented via separate devices and/or systems that may have operated in accordance with different data or signal formats and/or standards, and that offered limited ability (if any) to interface or operate together.
  • [0040]
    In particular, the multimedia collaboration device described herein supports functionality that may include the following:
  • [0041]
    1. Audio signal handling:
  • [0042]
    a) stereo speakers—to provide realistic audio reproduction capabilities needed for multimedia presentations, music, and multiport teleconferencing, including support for three-dimensional sound and audio positioning metaphors;
  • [0043]
    b) adaptive echo-canceled stereo speakers for the environment and mono or stereo microphone—to provide high-quality, realistic audio interactions and eliminate echo and/or feedback in conferencing situations; and
  • [0044]
    c) adaptive echo-canceled mono synthetic aperture microphone—to significantly improve audio capture performance in noise-prone or poorly-controlled audio environments, such as office cubicles or public kiosks.
  • [0045]
    2. One or more data networking protocols, where such protocols may span a range of technological generations. In one embodiment, the present invention includes built-in support for 10 and 100 Megabit-per-second (MBPS) Ethernet, as well as Gigabit Ethernet, via Unshielded Twisted Pair (UTP) wiring. Other embodiments could include support for other or additional networking protocols, such as Asynchronous Transfer Mode (ATM) networking and Integrated Services Digital Network (ISDN).
  • [0046]
    3. One or more analog A/V signal transmission/reception formats, where such formats may span various means of:
  • [0047]
    a) Analog A/V signal transfer via a separate pair of wires for each of audio transmit, audio receive, video transmit, and video receive (i.e., a total of four sets of UTP wires);
  • [0048]
    b) Analog A/V signal transfer via a single set of UTP wires for audio/video transmit, plus a single set of UTP wires for audio/video receive (i.e., a total of two twisted-pairs carrying analog A/V signals), through frequency modulation (FM) or other multiplexing techniques;
  • [0049]
    c) Analog A/V signal transfer via encoding both audio and video signals on a single set of UTP wires, for example, through FM or other multiplexing methods and perhaps 2-wire/4-wire electronic hybrids; and
  • [0050]
    d) Any of the above approaches that carry the analog A/V signals on the same wire pairs as used by data networking circuits (through the use of FM or other modulation techniques).
  • [0051]
    Either of the above analog A/V signal transfer formats allow the use of a single conventional data network connector for carrying both analog A/V and data networking signals. For example, a standard 8-wire RJ-45 connector can support 10 and/or 100 MBPS Ethernet in conjunction with analog A/V signal transfer, using two twisted pairs for Ethernet networking and two twisted pairs for A/V signal transfer. In the event that data networking is implemented via a protocol for which a sufficient number of connector pins or wires are unavailable for A/V signal transfer, such as Gigabit Ethernet, which conventionally utilizes the entire physical capacity of an RJ-45 connector, the present invention may include an additional connector or coupling for analog A/V signal transfer.
  • [0052]
    4. Digital multimedia streaming I/O, transmitted to and/or received from a multimedia network and/or a companion computer, as further described below.
  • [0053]
    5. Internal A/V signal encoding and decoding capabilities to support A/V compression formats such as MPEG 1/2/4, JPEG, H.310, H.320, H.323, QuickTime, etc.
  • [0054]
    6. Internal data routing capabilities, through which data packets, cells, or streams may be selectively transferred among a multimedia network, the present invention, and/or a set of companion computers.
  • [0055]
    7. Multimedia call and connection control protocols, such as described in U.S. Pat. No. 5,617,539.
  • [0056]
    8. Internet browsing and multimedia internet message transfer capabilities.
  • [0057]
    9. Data sharing and/or application sharing protocols.
  • [0058]
    10. Network configuration and/or network traffic monitoring capabilities.
  • [0059]
    Through the combination of the data routing, internal encoding/decoding, and/or digital streaming capabilities, the present invention may operate as a multimedia processing device that offloads potentially computationally-intensive multimedia processing tasks from a companion computer. Use of the present invention to reduce a companion computer's processing burden can be particularly advantageous in real-time multimedia situations. The present invention may further provide an older or outdated computer with comprehensive real-time multimedia collaboration capabilities, as described below. Additionally, the present invention may operate as a stand-alone device, such as a self-contained internet or intranet appliance having real-time multimedia capabilities, and/or an ISDN video teleconferencing terminal.
  • [0060]
    The present invention also may advantageously incorporate new technologies, including an integrated camera/display device as described in detail below. Furthermore, the present invention provides support for technology and standards evolution by 1) facilitating the use of standard plug-in and/or replaceable components, which may be upgraded or replaced over time; 2) providing designed-in support for recently-developed technologies that are likely to gain widespread use, such as switched 10 MBPS full-duplex internet, 100 MBPS switched Ethernet, ATM, or Gigabit Ethernet (as well as interim-value networks such as ISDN); and 3) providing for upgradability via software and/or firmware downloads. The present invention may additionally implement particular capabilities via reconfigurable or reprogrammable logic devices, such as Field Programmable Gate Arrays (FPGAs). Updated configuration bitstreams can be downloaded into these reconfigurable devices to provide hardware having upgraded or new capabilities.
  • 4.2 High-Level Architecture and Packaging Options
  • [0061]
    FIG. 1 is a high-level block diagram of a multimedia collaboration device 100 constructed in accordance with the present invention. The multimedia collaboration device 100 comprises a preamplifier and buffer unit 102; an audio signal conditioning unit 104; a switching unit 106; an Unshielded Twisted Pair (UTP) transceiver 108; a pair splitter 110; a routing unit 112; an encoding/decoding unit 116; a processor set 118; a memory 120; an input device interface 130; a companion computer port 136; and a building or premises network port 138.
  • [0062]
    The premises network port 138 facilitates coupling to premises- or building-based UTP wiring that forms a portion of a multimedia network 60. In one embodiment, the premises network port 138 comprises a conventional network coupling, such as an RJ-45 connector. The companion computer port 136 facilitates coupling to one or more host or companion computers 50, such that the present invention can offload real-time multimedia processing tasks from a companion computer 50 and/or provide a pass-through for data packet exchange between a host computer 50 and the multimedia network 60. In one embodiment, the companion computer port 136 comprises a conventional network coupling that is compatible with the premises network port 138. In another embodiment, the premises network port 138 may employ a more sophisticated or modern protocol than that used by the companion computer port 136. In yet another embodiment, a host or companion computer may access the multimedia collaboration device 100 via the premises network port 138, and hence such an embodiment may not include a separate companion computer port 136. It is also possible for the present invention to communicate with a host or companion computer 50 over the data networking ports 136, 138 for use in running Graphical User Interfaces (GUIs) or coordinating with application processes executing on the host or companion computer 50.
  • [0063]
    The preamplifier and buffer unit 102 receives A/V signals from a left and a right microphone 140.1, 140.2 and a camera 142, and transmits A/V signals to a left and a right speaker 144.1, 144.2 and a display device 146. The preamplifier and buffer unit 102 can additionally send and receive A/V signals via a set of auxiliary (AUX) A/V ports 148, which could couple to a device such as a Video Cassette Recorder (VCR).
  • [0064]
    As elaborated upon below, the audio signal conditioning unit 104 provides volume control functionality in conjunction with echo-canceled stereo microphone or mono synthetic aperture microphone capabilities. In one embodiment, the echo-canceled stereo microphone and mono synthetic aperture microphone capabilities may be implemented in a single mode-controlled Digital Signal Processor (DSP) chip, in a manner that may facilitate user-selectivity between these two types of microphone functionality. If the microphone array 140.1, 140.2 includes more than two microphones, it may be desirable to employ DSP techniques to synthesize a stereo synthetic aperture microphone. Further multiple microphone processing modes, such as stochastic noise suppression for extreme noise environments, can also be included.
  • [0065]
    In the present invention, transfer of incoming and/or outgoing A/V signals between a variety of sources and/or destinations is required, including the microphones 140.1, 140.2, the camera 142, the speakers 144.1, 144.2, the display device 146, other A/V or I/O devices, the premises network port 138, and/or the companion computer port 136. Signal transfer pathways for such sources and destinations may ultimately be analog or digital in nature. To meet these switching needs, the multimedia collaboration device employs the switching unit 106, which selectively routes analog A/V signals associated with the microphones 140.1, 140.2, the camera 142, the speakers 144.1, 144.2, the display device 146, and/or other devices to or from the analog A/V UTP transceiver 108 and/or the encoding/decoding unit 116. The encoding/decoding unit 116 may also perform any required conversion between analog and digital formats.
  • [0066]
    As further described below, the analog A/V UTP transceiver 108 provides an analog signal interface to the pair splitter 110, which separates data networking and analog A/V signals. In many cases this signal separation is most easily accomplished by selectively separating wires or wire pairs, but may also include the use of passive (or equivalent) wire switching arrangements and programmable Frequency Division Multiplexing (FDM) modulators and demodulators. As indicated earlier, the encoding/decoding unit 116 performs conversions between analog and digital signal formats, and as such also compresses and decompresses A/V signals. Although not shown, those skilled in the art will understand that an ISDN transceiver, inverse multiplexer, network connector, Q.931 call control, etc. . . . can be introduced into the architecture to add support for ISDN. The processor set 118 controls the operation of the multimedia collaboration device 100, and performs data network communication operations. In conjunction with operating system and other software resident within the memory 120, the processor set 118 may provide graphic overlay capabilities on a video image so as to implement any GUI capabilities. These GUIs may facilitate control over the operations of the present invention, and may further provide internet browsing capabilities, as described in detail below. The routing unit 112 performs network packet exchange operations between the premises network port 138, the companion computer port 136, and the processing unit 118, where such packets may include data, portions of, or entire digital AV streams, and/or network configuration or traffic monitoring information. Finally, the input device interface 130 may provide auxiliary mouse and keyboard ports 132, 134, and may also support an internal local geometric pointing input device as described below.
  • [0067]
    Particular groupings of the aforementioned elements may be packaged in various manners so as to match particular deployment settings. For example, selected element groupings may reside within or upon a peripheral box package, computer-bus-compatible card, or housing 150, where such element groupings may include various A/V transducers. The nature of the selected package 150, and the manner in which the aforementioned elements are incorporated therein or thereupon as integrated, modular, plug-in, and/or other types of components, is dependent upon the manner in which the present invention is employed, and may be subject to or adaptive to evolving market forces and embedded legacy equipment investments. Three exemplary types of packages are described in detail hereafter.
  • [0068]
    FIG. 2 is a high-level perspective view illustrating a box package 160 for the multimedia collaboration device 100. This illustrative box package 160 comprises a housing 162 having a control panel 164 and a cable panel 182. The control panel 164 includes an audio mode control 166; a microphone/speaker/headset selector 168; a microphone mute control 170; a hold/resume control 172; AUX video and audio inputs 174, 176; a telephone add/remove control 178; and a speaker/earphone volume control 180. The audio mode control 166 facilitates user-selection between stereo microphone and synthetic aperture microphone operation, as further described below. The microphone/speaker/headset selector 168 provides for user-selection of different audio input/output interfaces, and the microphone mute control 170 facilitates user control over audio input muting. The hold/resume control 172 pauses or resumes audio inputs in response to user-selection. The AUX video and audio inputs 174, 176 respectively facilitate video and audio input from various sources. The telephone add/remove control 178 provides control of the insertion of an optional bridge or coupling to a telephone line for two-way audio contact with an addition of third-party telephone user. The supporting electrical couplings would provide for standard telephone loop-through. In one embodiment, the telephone add/remove control 178 includes conventional telephone line echo cancellation circuitry to remove the undesired transmit/receive coupling effects introduced by telephone loops. Finally, the speaker/earphone volume control 180 controls the amplitude of an audio signal delivered to speakers or an earphone (in accordance with the setting of the microphone/speaker/headset selector 168). Some implementations may include separate volume controls for speakers, earphones, and/or auxiliary audio I/O.
  • [0069]
    The cable panel 182 on the box package 160 includes inputs and outputs that facilitate coupling to a camera/microphone cable 184; a premises UTP cable 186; left and right speaker cables 188, 190; a video monitor or video overlay card cable 192; and a UTP computer networking cable 194.
  • [0070]
    The box package 160 is suitable for use with a companion desktop or portable computer, and could reside, for example, underneath, atop, or adjacent to a computer or video monitor. Furthermore, a single box package 160 may be used to provide a plurality of companion computers 50 with multimedia collaboration capabilities, for example, in a small office environment.
  • [0071]
    Those skilled in the art will understand that the above combination of features is illustrative and can be readily altered. Those skilled in the art will also understand that in an alternate embodiment, the box package 160 could include a built-in microphone or microphone array, as well as one or more speakers. Furthermore, those skilled in the art will understand that one or more controls described above could be implemented via software.
  • [0072]
    FIG. 3 is a suggestive high-level drawing showing the format of a plug-in card package 200 for the multimedia collaboration device 100. The plug-in card package 200 comprises a circuit board or card 202 having a standard interface 204 that facilitates insertion into an available slot within a computer. For example, the standard interface 204 could comprise plated connectors that form a male Peripheral Component Interface (PCI) connector, for insertion into a female PCI slot coupled to a PCI bus. The elements comprising the multimedia collaboration device 100 may be disposed upon the card 202 in the form of discrete circuitry, chips, chipsets, and/or multichip modules. The card 202 includes inputs and outputs for coupling to a camera/microphone cable 214; left and right speaker cables 206, 208; a premises UTP cable 210; and a UTP-to-computer cable 212 that facilitates pass-through of data networking signals to an existing data networking card. It is understood that conventional PCI bus interface electronics and firmware may be added to this configuration. Alternatively, the PCI bus may simply be used to provide power and electrical reference grounding.
  • [0073]
    The multimedia collaboration device 100 may include more extensive data networking capabilities, capable in fact of supporting essentially all the networking needs of one or more companion or host computers, as described in detail below. In this variation, the plug-in card package 200 may therefore be used to provide a computer into which it is inserted with complete data networking capabilities in addition to multimedia collaboration capabilities via transfer of data networking packets between the interface 204 and the computer, in which case the UTP-to-computer cable 212 may not be necessary. The presence of the plug-in-card package 200 may therefore obviate the need for a separate network interface card (NIC) in market situations in which sufficient evolution stability in data networking technologies exists.
  • [0074]
    The plug-in card package 200 may be used to provide older or less-capable computers with comprehensive, up-to-date real-time multimedia collaboration capabilities. Alternatively, the plug-in card package 200 can provide video overlay multimedia capabilities to computer systems having a monitor for which a video overlay card is unavailable or difficult to obtain. In the event that video overlay multimedia capabilities are to be delivered to a display or video monitor other than that utilized by the companion computer 50, the plug-in card package 200 may include a port that facilitates coupling of a video monitor or video overlay card cable 192 in a manner analogous to that shown in FIG. 2. A host computer 50 that incorporates a plurality of plug-in card packages 200 could be used as a multimedia collaboration server for other computers, in a manner understood by those skilled in the art.
  • [0075]
    Those skilled in the art will additionally understand that one or more of the physical panel controls described above with reference to the box package 160 would be implemented via software control for the plug-in card package 200.
  • [0076]
    FIG. 4 is a perspective view of a stand-alone package 300 for the multimedia collaboration device 100 that includes a range of advantageous internal A/V transducer configurations. In one deployment, the stand-alone package may be attached, mounted, or placed proximate to the side of a computer monitor or laptop/palmtop computer, and hence is referred to herein as a “side-kick” package 300.
  • [0077]
    The side-kick package 300 provides users with a self-contained highly-localized multimedia communication interface. The incorporation of the microphone array 304 into the side-kick package 300 assists in controlling the present invention's superior audio performance relative to adaptive echo-canceled stereo microphone and adaptive echo-canceled mono synthetic aperture microphone capabilities described below. The placement of the camera 306 in close proximity to the flat display device 312 aids in maintaining good user eye contact with a displayed image, which in turn better simulates natural person-to-person interactions during videoconferencing. The eye contact can be further improved, and manufacturing further simplified, by an integrated camera/display device as described below with reference to FIGS. 16 through 25.
  • [0078]
    The side-kick package 300 can be used in conjunction with a companion computer 50, or in a stand-alone manner. When used with a companion computer 50, the side-kick package 300 eliminates the need to consume companion computer screen space with a video window. As a stand-alone device, the side-kick package 300 can be used, for example, in office reception areas; public kiosks; outside doorways; or alongside special-purpose equipment for which explicatory, possibly interactive assistance may be useful, such as a photocopier.
  • [0079]
    Relative to FIG. 2, like reference numbers designate like elements. The side-kick package 300 comprises a housing 302 in which the multimedia collaboration device 100 described above plus additional elements such as an internal shock-mounted microphone array 304; a camera 306 that may include auto-focus, auto-iris, and/or electronic-zoom features; acoustically-isolated stereo speakers 308; a thumbstick mouse or similar type of geometric input device 310; and a flat display device 312 may reside. The side-kick package 300 may further include display brightness and contrast controls 314, 316, and/or one or more auxiliary audio level controls 180. Additionally, the side-kick package 300 may include a control panel having physical panel controls such as an audio mode control 166; a microphone/speaker/headphone selector 168; a microphone mute control 170; a hold/resume control 172; AUX video and audio inputs 174, 176; and a telephone add/remove control 178, which function in the manner previously described. Those skilled in the art will understand that the functions of one or more of the physical controls shown in FIG. 4 could be implemented so as to be controlled remotely via software. In some arrangements, there might not be any physical controls, in which case control is facilitated by GUIs executing on one or more companion computers 50. Ideally, this embodiment may include both physical and remote software controls so that it can operate as a fully stand-alone device as well as a slave device supporting applications running on the companion computer 50.
  • [0080]
    The side-kick package 300 has ports for coupling to a premises UTP cable 336 and an optional UTP-to-computer cable 338. The side-kick package 300 may also include another connector set 334, which, for example, facilitates coupling to a headset, an auxiliary mouse, and/or an auxiliary keyboard. FIG. 4 additionally depicts an overlay window 340 upon the flat display device 312, which may be realized via graphics overlay capabilities. The graphics overlay capabilities can implement menus or windows 340 that can provide a user with information such as text or graphics and which may be selectable via the input device 310, creating internal stand-alone GUI capabilities.
  • [0081]
    Relative to each package 160, 200, 300 described herein, use of the multimedia collaboration device 100 with one or more companion computers 50 to effect digital networked A/V communication advantageously spares each companion computer 50 the immense computational and networking burdens associated with transceiving and encoding/decoding A/V streams associated with A/V capture and presentation. The invention may also incorporate additional video graphics features in any of the packages 160, 200, 300 described above, such as telepointing over live video and/or video frame grab for transference to or from a companion or host computer 50.
  • [0082]
    While FIG. 1 provides a broad overview of the architecture of the present invention, specific architectural details and various embodiments are elaborated upon hereafter, particularly with reference to FIGS. 5, 6, and 7.
  • 4.3 Architectural Details
  • [0083]
    FIG. 5 is a block diagram of a first embodiment of a multimedia collaboration device 10 constructed in accordance with the present invention, and which provides primary and auxiliary (AUX) support for analog A/V, and further provides support for networked digital streaming. With reference to FIG. 1, like reference numbers designate like elements. The embodiment shown in FIG. 5 supports analog A/V, and comprises the preamplifier and buffer unit 102; the audio signal conditioning unit 104; the A/V switch 106; the analog A/V UTP transceiver 108; the pair splitter 110; a first and a second digital transceiver 111, 135; the routing unit 112; a network interface unit 114; an analog-to-digital (A/D) and digital-to-analog (D/A) converter 116 a; an A/V compression/decompression (codec) unit 116 b; at least one, and possibly multiple, processors 118.1, 118.n; the memory 120; the I/O interface 130; and the companion and premises network ports 136, 138. An internal bus 115 couples the network interface unit 114, the A/V codec 116 b, each processor 118.1, 118.n, the memory 120, and the I/O interface 130. Each of the audio signal conditioning unit 104, the A/V switch 106, the analog A/V UTP transceiver 108, the routing unit 112, and the A/D-D/A converter 116 a may also be coupled to the internal bus 115, such that they may receive control signals from the processors 118.1, 118.n.
  • [0084]
    The preamplifier and buffer unit 102 is coupled to receive left and right microphone signals from a left and right microphone 140.1, 140.2, respectively; and a camera signal from the camera 142. It is understood that additional microphones 140.3 . . . 140.x and processing 118 and/or switching capabilities 106 may be included to enhance the synthetic aperture microphone capabilities described below. The preamplifier and buffer unit 102 may further receive AUX A/V input signals from one or more auxiliary A/V input devices such as an external VCR, camcorder, or other device. The preamplifier and buffer unit 102 respectively outputs left and right speaker signals to a left and a right speaker 144.1, 144.2; and a display signal to the display device 146. The preamplifier and buffer unit 102 may also deliver AUX A/V output signals to one or more auxiliary devices.
  • [0085]
    The audio signal conditioning unit 104 facilitates the adjustment of outgoing audio signal volume in conjunction with providing adaptive echo cancelled stereo microphone or mono synthetic aperture microphone processing operations upon audio signals received from the preamplifier and buffer unit 102. FIG. 8 is a block diagram of an adaptive echo-canceled stereo microphone unit 103 within the audio signal conditioning unit 104. The adaptive echo-canceled stereo microphone unit 103 comprises a stereo echo canceller 310 and a stereo volume control unit 350.
  • [0086]
    The stereo echo canceller 310 comprises conventional monaural echo canceller subsystems that function in a straightforward manner readily apparent to those skilled in the art. This arrangement includes a left microphone/left speaker (LM/LS) adaptive acoustic echo filter model 312; a left microphone/right speaker (LM/RS) adaptive acoustic echo filter model 314; a right microphone/left speaker (RM/LS) adaptive acoustic echo filter model 316; and a right microphone/right speaker (RM/RS) adaptive acoustic echo filter model 318. It will be readily understood by those skilled in the art that linear superposition results in stereo echo canceling capabilities for stereo microphones and stereo speakers.
  • [0087]
    The stereo volume control unit 350 is coupled to a volume adjustment control such as described above with reference to the various package embodiments 160, 200, 300 shown in FIGS. 2, 3, and 4, and is further coupled to receive the left and right speaker signals. The stereo volume control unit 350 is also coupled to each model 312, 314, 316, 318 in order to maximize the utilization of DSP arithmetic and dynamic range throughout the full range of speaker volume settings. It is understood that stereo balance controls can be implemented using the same stereo volume control elements operating in complimentary increments.
  • [0088]
    The LM/LS and LM/RS models 312, 314 are coupled to receive the left and right speaker signals, respectively. Similarly, the RM/LS and RM/RS models 316, 318 are respectively coupled to receive the left and right speaker signals 300. Each of the LM/LS, LM/RS, RM/LS, and RM/RS models 312, 314, 316, 318 incorporates an adaptive coefficient tapped delay line weighting element coupled to its corresponding microphone 140.1, 140.2 and speaker 144.1, 144.2 in a conventional manner. Additionally, the LM/LS and LM/RS models 312, 314 maintain conventional couplings to the left microphone 140.1 to facilitate initial acoustic environment and subsequent adaptive acoustic training operations. Similarly, the RM/LS and RM/RS models 316, 318 maintain couplings to the right microphone 140.2 to facilitate these types of training operations.
  • [0089]
    The stereo echo canceller 310 additionally includes a first signal summer 320 coupled to outputs of the left microphone 140.1, the LM/LS model 312, and the LM/RS model 314; plus a second signal summer 322 coupled to outputs of the right microphone 140.2, the RM/LS model 316, and the RM/RS model 318. The first signal summer 320 delivers a left echo-canceled signal to the A/V switch 106, and the second signal summer 322 delivers a right echo-canceled signal to the A/V switch 106, in a manner readily understood by those skilled in the art.
  • [0090]
    In one embodiment, the stereo echo canceller 310 and stereo volume control unit 350 are implemented together via DSP hardware and software. Furthermore, a single DSP may be used to implement the stereo echo canceller 310, the stereo volume control unit 350, and the adaptive echo-canceled mono synthetic aperture microphone unit 105, which is described below. In an exemplary embodiment, such a DSP may comprise a Texas Instruments TMS320C54x generation processor (Texas Instruments Incorporated, Dallas, Tex.).
  • [0091]
    In the event that a user employs an earphone, headphone set, or AUX audio device in conjunction with the present invention, as described above with reference to the box, card, and side-kick packages 160, 200, 300, the stereo echo canceller 310 is placed in a bypassed, inactive, or quiescent state and the DSP and stereo volume control unit 350 facilitate normalization and/or volume adjustment in a conventional manner as understood by those skilled in the art. Alternatively, separate volume control and/or normalization circuitry could be provided when stereo microphones or the stereo echo canceller 310 is not needed. These may be implemented in various ways with respect to the paths connecting to the A/V switch.
  • [0092]
    FIG. 9 is a block diagram of an adaptive echo-canceled mono synthetic aperture microphone unit 105 within the audio signal conditioning unit 104. With reference to FIG. 8, like reference numbers designate like elements. The adaptive echo-canceled mono synthetic aperture microphone unit 105 comprises the volume control unit 350 plus a synthetic aperture microphone processing unit 330, which may include hardware and/or software. The synthetic aperture microphone processing unit 330 comprises a synthetic aperture microphone unit 340 which may include hardware and/or software to implement synthetic aperture microphone processing algorithms; a synthetic microphone/left speaker (SM/LS) model 332; a synthetic microphone/right speaker (SM/RS) model 334; and a signal summing circuit 336, each coupled in the manner shown.
  • [0093]
    The synthetic aperture microphone unit 330 is coupled to receive the left and right microphone signals, and additionally includes conventional adaptive coefficient weighting and training couplings. Taken together, the synthetic aperture microphone unit 330, the left microphone 140.1, and the right microphone 140.2 (plus one or more additional microphones that may be present) form a mono-output synthetic aperture microphone. The synthetic aperture microphone unit 330 performs delay and/or frequency dispersion operations upon the left and right microphone signals to internally create or define an audio reception sensitivity distribution pattern in a manner readily understood by those skilled in the art. The audio reception sensitivity distribution pattern includes one or more spatial regions referred to as “hot-spots,” as well as a set of spatial regions referred to as “rejection regions.” Typically, a set of one or more “hot-spots” includes a primary hot-spot of maximal audio reception sensitivity that has a particular position or orientation relative to the geometry of the microphone array 140.1, 140.2. The rejection regions comprise spatial positions in which the synthetic aperture microphone has minimal audio reception sensitivity.
  • [0094]
    FIG. 10 is an illustration showing an exemplary localized primary hot-spot 10-3 and a surrounding rejection region 10-8. Within the primary hot-spot 10-3, the synthetic aperture microphone 10-2 can detect sound waves produced by a speaker 10-1. The location of the primary hot-spot may be adjusted in accordance with particular conditions in an acoustic environment. In one embodiment, the position or orientation of the primary hot-spot may be modified under software control. This in turn could facilitate user-directed hot-spot positioning for optimizing audio performance in different acoustic situations. FIG. 11 is an illustration showing exemplary primary hot-spot directivity, where the synthetic aperture microphone 11-2 captures directionally-specific speech energy from a user 11-1 within a primary hot-spot 11-3 that is offset relative to that shown in FIG. 10. A rejection region 11-8 exists outside the primary hot-spot 11-3 in a conventional manner.
  • [0095]
    The synthetic aperture microphone can additionally reject reflected speech energy that originated within the primary hot-spot and that approaches the microphone array 140.1, 140.2 from angles beyond those that span the primary hot-spot. FIG. 12 is an illustration showing exemplary reflected speech energy rejection. The synthetic aperture microphone 12-2 detects sound waves produced by a user 12-1 within a primary hot-spot 12-3. The synthetic aperture microphone 12-2 rejects sound waves 12-5, 12-6 originating within the primary hot-spot 12-3 and reflected from nearby surfaces because the reflected sound waves are likely to travel through one or more rejection regions 12-8 along their reflection path.
  • [0096]
    The synthetic aperture microphone is further advantageous by virtue of good ambient acoustical noise rejection performance. FIG. 13 is an illustration showing exemplary ambient audio noise rejection, in which a synthetic aperture microphone 13-2 rejects conversational noise 13-4 and various forms of outside or environmental noise 13-5, 13-6, 13-7. The noise and noise reflections traveling towards the microphone array 140.1, 140.2 enter a rejection region 13-8 through various directions, and hence are strongly attenuated via the synthetic aperture microphone's directional rejection behavior. This is in contrast to a user 13-1 within a primary hot-spot 13-3, who produces sound waves that the synthetic aperture microphone 13-2 readily detects with high sensitivity.
  • [0097]
    Referring also now to FIGS. 5 and 9, the synthetic aperture microphone unit 330 outputs a mono microphone signal having a magnitude that most directly corresponds to the amount of audio energy present within the set of hot-spots, and in particular the primary hot-spot. The synthetic aperture microphone output signal has little contribution from audio energy entering from any rejection region directions. Those of ordinary skill in the art will understand that multiple microphones can be used to extract voice information from background noise that is in fact louder than the actual speech using adaptive cancellation techniques such as those described by Boll and Pulsipher in IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. ASSP-28, No. 6, December 1980. This could be incorporated as a third operational mode for the audio DSP, for supporting extreme noise environments as might be found on public streets or repair depots, for example.
  • [0098]
    The volume control unit 350 is coupled to the left and right speaker signals, as are the SM/LS and SM/RS models 332, 334. The signal summing circuit 336 is coupled to the output of the synthetic aperture microphone unit 340, as well as outputs of the SM/LS and SM/RS models 332, 334, and delivers an echo-canceled mono synthetic aperture microphone signal to the A/V switch 106.
  • [0099]
    In one embodiment, the adaptive echo-canceled synthetic aperture microphone unit 105 comprises DSP hardware and/or software. The present invention can thus provide either adaptive echo-canceled stereo microphone or adaptive echo-canceled mono synthetic aperture microphone capabilities in response to user selection. In an exemplary embodiment, the adaptive echo-canceled synthetic aperture microphone unit 105 is implemented in a DSP such as the Texas Instruments TMS320C54x processor referenced above. Those skilled in the art will recognize that a single DSP system can be configured to provide both the adaptive echo-canceled stereo and mono synthetic aperture microphone capabilities described herein as distinct or integrated operating modes.
  • [0100]
    In the event that a user employs an earphone, headphone set, or AUX audio devices in conjunction with the present invention, the synthetic aperture microphone unit 330 is placed in a bypassed, inactive, or quiescent state and the DSP and/or volume control unit 350 facilitate conventional normalization and adjustment of output signal amplitude, in a manner understood by those skilled in the art. Alternatively, separate normalization and/or volume control circuitry could be provided to accommodate the aforementioned devices.
  • [0101]
    Referring again to FIG. 5, the A/V switch 106 comprises conventional analog switching circuitry that is coupled to the preamplifier and buffer unit 102, the audio signal conditioning unit 104, the analog A/V UTP transceiver 108, and the A/D-D/A converters 116 a. The A/V switch 106 further maintains a coupling to the internal bus 115, thereby facilitating processor control over A/V switch operation.
  • [0102]
    The A/V switch 106 routes incoming signals generated by the left and right microphones 140.1, 140.2 (or larger microphone array), the camera 142, and/or any AUX A/V input devices to the analog A/V UTP transceiver 108 or the A/D-D/A converters 116 a under the direction of a control signal received via the internal bus 115. Similarly, the A/V switch 106 selects either the analog A/V UTP transceiver 108 or the A/D-D/A converters 116 a as a source for outgoing signals directed to the left and right speakers 144.1, 144.2, the display device 146, and/or any AUX A/V output devices.
  • [0103]
    The analog A/V UTP transceiver 108 comprises a conventional analog A/V transceiver that provides a signal interface to a first set of UTP wires that carry analog A/V signals and which couple the analog A/V UTP transceiver 108 to the pair splitter 110. The pair splitter 110 is further coupled to the first digital transceiver 111 via a second set of UTP wires that carry digital A/V signals. The analog A/V UTP transceiver 108 may be reconfigurable, supporting a range of analog 4-pair, 2-pair, or 1-pair signal transmission methodologies. The selection of any particular signal transmission methodology may be performed under processor control or by physical configuration switching. Similarly, distance compensation adjustments may be performed under processor control or via physical switching, or alternatively through automatic compensation techniques in a manner understood by those skilled in the art.
  • [0104]
    The first and second digital transceivers 111, 135 provide conventional digital interfaces to UTP wiring, and are coupled to the routing unit 112 in the manner shown. The second digital transceiver 135 is further coupled to the companion computer port 136. The first and second digital transceivers 111, 135 may be implemented using portions of a standard NIC, as described below, or by other means. In addition to the aforementioned couplings, the routing unit 112 is coupled to the network interface unit 114. The routing unit 112 comprises conventional network hub or mini-hub circuitry. In one embodiment, the routing unit 112 performs hard-wired signal distribution and merge functions. In an alternate embodiment, the routing unit 112 performs data packet delivery path selection operations.
  • [0105]
    The network interface unit 114 comprises conventional network interface circuitry, for exchanging data with the internal bus 115 and data packets with either the multimedia network 60 or a companion computer 50 via the premises and companion computer network ports 138, 136 in accordance with a conventional networking protocol. In one embodiment, the network interface unit 114 is implemented as at least one standard NIC. The NIC may typically include built-in data packet address examination or screening capabilities, and hence simplify the routing unit's function to one of communications distribution and merge functions in such an embodiment. These distribution and merge functions serve to provide simultaneous signal or packet exchange among each of the premises network port 138, the NIC 114, and the companion computer port 136. One advantage of an embodiment employing a standard NIC is that the NIC could be easily replaced or upgraded to accommodate technological evolution. This range of possibilities is further enhanced by the switching arrangement described below with reference to FIG. 15. Although not shown, it is again understood that should ISDN support be deemed valuable, network connectors, interface electronics, inverse multiplexers, and Q.931 call control can be introduced through, for example, connection to the internal bus 115 in a manner familiar to those skilled in the art.
  • [0106]
    Taken together, the premises network port 138, the pair splitter 110, the analog A/V UTP transceiver 108, the digital transceiver 111, the routing unit 112, the network interface unit 114, and the companion computer port 136 form 1) a first multimedia network interface for handling analog A/V signals; and 2) a second multimedia network interface for handling digital A/V and data networking signals. FIG. 14 is a block diagram of a first embodiment of a first 400 and a second 410 multimedia network interface provided by the present invention. The first multimedia network interface 400 comprises the aforementioned first set of UTP wires plus the analog A/V UTP transceiver 108. The first multimedia network interface 400 facilitates the exchange of analog A/V signals between the premises network port 138 and the analog A/V UTP transceiver 108. The second multimedia network interface 410 comprises the second set of UTP wires, the digital transceiver 111, the routing unit 112, and the network interface unit 114, which are coupled in the manner shown. In some implementations, the digital transceiver 135 may also be a NIC that may be either similar to or different from a NIC employed in the network interface unit 114. The second multimedia interface 410 facilitates the exchange of digital A/V and data networking signals between the premises network port 138, the network interface unit 114, and the companion computer port 136.
  • [0107]
    FIG. 15 is a block diagram of a second embodiment of first and second multimedia network interfaces provided by the present invention. The first and second multimedia network interfaces are implemented via a passive switching arrangement and/or an active analog switching matrix 420 that includes low-capacity, high-frequency analog protection devices. Such protection devices may comprise three-terminal, back-to-back diode arrangements, as employed in a Motorola BAV99LT1 (Motorola, Inc., Schaumberg, Ill.). In this arrangement, the analog transceiver 108 may support a number of 4, 2, and 1 pair formats, which may be dictated by the marketplace. Alternatively, the analog transceiver 108 can be a replaceable module.
  • [0108]
    In the event that data networking is implemented via Gigabit Ethernet or other network protocol that conventionally consumes the entire physical capacity of an entire RJ-45 connector, the present invention may employ an additional RJ-45 or other type of connector for carrying analog A/V signals.
  • [0109]
    Via the second multimedia network interface, the present invention provides internal data communication transmit, receive, and routing capabilities. An external or companion computer 50 can therefore issue control signals directed to the present invention in accordance with standard data networking protocols. The second multimedia network interface can also provide “loop-through” signal routing between the premises network port 138 and the companion computer port 136. Additionally, the data routing capabilities provided by the second multimedia network interface facilitate coupling to both existing broadcast or switching hubs. The second multimedia network interface also supports the transfer of digital A/V streams. Thus, the second multimedia network interface cleanly separates data communications directed to one or more companion computers 50, the multimedia network 60, and the multimedia collaboration device 10.
  • [0110]
    Once again referring to FIG. 5, each of the A/V switch 106, the analog A/V UTP transceiver 108, the routing unit 112, the network interface unit 114, the A/V codec 116 b, the set of processors 118.1, 118.n, the memory 120, and the I/O interface 130 is coupled to the internal bus 115. The A/V codec 116 b is further coupled to the A/D-D/A converters 116 a, which are coupled to the A/V switch 106. It is noted that the A/D-D/A converters 116 a may include color-space conversion capabilities to transform between RGB and YUV or other advantageous color spaces.
  • [0111]
    The memory 120 comprises Random Access Memory (RAM) and Read-Only Memory (ROM), and stores operating system and application software 122, 124. Depending upon the nature of the processors 118.1, 118.n, the operating system 122 could comprise a scaled-down, conventional, or enhanced version of commercially-available operating system software, and/or special-purpose software. In an exemplary embodiment, the operating system 122 comprises Windows CE (Microsoft Corporation, Redmond, Wash.) or another commercial product selected in accordance with the particular environment in which the present invention is employed. The application software 124 may comprise programs for performing videoconferencing, messaging, publishing, broadcast reception, and media-on-demand operations, and internet browsing using programs such as Netscape Navigator (Netscape Communications Corporation, Mountain View, Calif.). Depending upon the nature of the processors 118.1, 118.n, the internet browser program could be a scaled down, conventional, or augmented version of a commercially-available browser.
  • [0112]
    The processors 118.1, 118.n manage communication with the network interface unit 114, and control the overall operation of the multimedia collaboration device 10 in accordance with control signals received via the network interface unit 114. The processors 118.1, 118.n additionally provide graphics overlay capabilities, and may further provide internet browsing capabilities in conjunction with application software 124 as previously described. Relative to managing communication with the network interface unit 114, the processors 118.1, 118.n may manage protocol stacks and/or state machines. With regard to controlling the overall operation of the multimedia collaboration device 10, the processors 118.1, 118.n issue control signals to the A/V switch 106 and execute application software resident within the memory 120. The graphics overlay capabilities facilitate the placement of fonts, cursors, and/or graphics over video present upon the display device 146. With sufficient processing power, the present invention can serve as a stand-alone, real-time video-capable internet appliance.
  • [0113]
    As described above, the A/D-D/A converters 116 a may comprise conventional circuitry to perform color-space conversion operations in addition to analog-to-digital and digital-to-analog signal conversion. The A/V codec 116 b comprises conventional A/V signal encoding and decoding circuitry, and provides the present invention with compression and decompression capabilities. Together these enable the present invention to encode and decode A/V streams without loading down a companion computer's processing and networking power. Either of the first or second multimedia network interfaces described above can route digital A/V signals to the A/V codec 116 b, while routing non-A/V signals to the companion computer 50. The present invention's ability to encode and decode A/V signals independent of a companion or external computer is particularly advantageous in situations in which video signal encoding and decoding must occur simultaneously, such as in 2-way teleconferencing or network-based video editing applications. The present invention may support network-based video editing applications based upon a high bandwidth near-zero-latency compression approach, which can be implemented, for example, through JPEG or wavelet compression operations; or an interim compression approach.
  • [0114]
    In one embodiment, the A/V codec 116 b comprises a chip or chipset. In another embodiment, the A/V codec 116 b comprises a processor 118.k capable of performing compression and decompression operations. In more advanced implementations, the A/V codec 116 b could comprise a single processor 118.m capable of performing user interface functions in addition to A/V compression and decompression operations. Such an implementation could also provide an Application Program Interface (API) in conjunction with operating system software 122. In an exemplary embodiment of such an implementation, the A/V codec 116 b may comprise a NUON processor (VM Labs, Mountain View, Calif.).
  • 4.4 Additional Embodiments
  • [0115]
    FIG. 6 is a block diagram of a second embodiment of a multimedia collaboration device 20, which provides primary support for analog audio I/O and digital visual I/O, and further supports analog and digital auxiliary A/V I/O, plus networked digital streaming. Relative to FIG. 5, like reference numbers designate like elements.
  • [0116]
    The second embodiment of the multimedia collaboration device 20 includes a digital camera 152, a digital display device 154, a digital AUX A/V interface 156, and a stream selector 158. The digital camera 152 and the digital display device 154 respectively capture and display images in a conventional manner. The digital AUX A/V interface 156 facilitates bidirectional coupling to auxiliary digital A/V devices, such as an external computer, a digital VCR, or Digital Versatile Disk (DVD) player. Each of the digital camera 152, the digital display device 154, and the digital AUX A/V interface 156 is coupled to the stream selector 158, which is coupled to the A/V codec 116 b.
  • [0117]
    The stream selector 158 comprises conventional circuitry that selectively routes digital streams between the A/V codec 116 b and the digital camera 152, the digital display device 154, and the digital AUX A/V interface 156. The stream selector 158 may route incoming digital image streams received from either of the digital camera 152 or the digital AUX A/V interface 156 to the A/V codec 116 b. In one embodiment, the stream selector 158 may be capable of multiplexing between these two incoming digital stream sources. Undersampling may also be used to facilitate the compositing of multiple video images. Relative to outgoing digital image streams, the stream selector 158 may route such streams to either or both of the digital display device 154 and digital AUX A/V interface 156, where such routing may occur in a simultaneous or multiplexed manner. The stream selector 158 additionally facilitates the exchange of digital audio streams between the A/V codec 116 b and the digital AUX A/V interface 156.
  • [0118]
    The A/V codec 116 b and the A/D-D/A converters 116 a together facilitate the conversion of digital A/V signals associated with the digital camera 152, the digital display device 154, and/or auxiliary digital A/V devices into analog A/V signals. The A/V switch 106 facilitates exchange of these analog A/V signals with AUX A/V devices and/or the premises network port 138.
  • [0119]
    Because the A/V codec 116 b is also coupled to the internal bus 115 and hence to the network interface unit 114, digital A/V signals captured from the digital camera 152 or directed to the digital display 154 or received from the digital AUX A/V interface 156 may be packetized and exchanged via the premises network port 138 and/or the companion computer port 136.
  • [0120]
    FIG. 7 is a block diagram of a third embodiment of a multimedia collaboration device 30, which provides primary support for analog audio I/O and digital visual I/O, support for digital auxiliary A/V I/O, and support for networked digital streaming. Relative to FIGS. 5 and 6, like reference numbers designate like elements.
  • [0121]
    The third embodiment of the multimedia collaboration device 30 includes a digital camera 152, a digital display device 154, a digital AUX A/V interface 156, and a stream selector 158 in the manner described above. Analog audio signals associated with the microphones 140.1, 140.2 and speakers 144.1, 144.2 are routed through the A/D-D/A converters 116 a and A/V codec unit 116 b. Thus, the third embodiment of the present invention manages digital A/V streams, and may exchange such streams with the multimedia network 60 and/or a companion computer 50. The third embodiment of the multimedia collaboration device 30 does not transmit analog A/V signals over the multimedia network 60, and hence the analog switching unit 106, the analog A/V UTP transceiver 108, and the pair splitter 110 described above relative to the first and second multimedia collaboration device embodiments are not required.
  • 4.5 Camera and Display Device Integration
  • [0122]
    As previously indicated, placement of the camera 142 in close proximity to the display device 146 aids in maintaining good user eye-contact with a displayed image, thereby closely approximating natural face-to-face communication in videoconferencing situations. Essentially perfect eye-contact can be achieved by integrating a large-area photosensor array with a large-area array of emissive or transmissive devices that form the basis for display device pixels.
  • [0123]
    Multiple photosensor and display element integration techniques exist. In general, the formation of an image using a photosensor array necessitates the use of optical elements in conjunction with photosensor elements. Photosensor and display element integration techniques are described in detail hereafter, followed by image formation considerations relative to integrated photosensor/display element arrays.
  • 4.6 Display Pixel and Photosensor Element Interleaving
  • [0124]
    One way of integrating photosensor elements with emissive or transmissive display elements is via element interleaving. FIG. 16 is an illustration showing a first photosensor and display element interleaving technique, in which display elements 510 and photosensor elements 520 populate a viewing screen 502 in an alternating manner. Each display element 510 generates or transmits light corresponding to a particular color or set of colors. Similarly, each photosensor element 520 detects light corresponding to a particular color. As described in detail below, display elements 510 and photosensor elements 520 operate in a temporally and/or spatially separated manner relative to each other to ensure that image capture is essentially unaffected by image display.
  • [0125]
    Display and photosensor elements 510, 520 corresponding to a particular color are interleaved in accordance with a color distribution scheme. FIG. 17 is an illustration of an exemplary photosensor element color and display element color distribution scheme. In FIG. 17, display elements 510 corresponding to the colors red, green, and blue are identified via the uppercase letters R, G, and B, respectively. Photosensor elements 520 corresponding to red, green, and blue are respectively identified by the lowercase letters r, g, and b. Display elements 510 corresponding to a particular color are offset relative to each other, and interleaved with display and photosensor elements 510, 520 corresponding to other colors. Similarly, photosensor elements 520 corresponding to a particular color are offset relative to each other, and interleaved with display and photosensor elements 510, 520 corresponding to other colors. Those skilled in the art will recognize that a variety of photosensor and display element color distribution schemes are possible.
  • [0126]
    The presence of photosensor elements 520 interleaved with display elements 510 reduces image resolution, and increases pixel pitch (i.e., the spacing between pixels). To minimize the effect that the photosensor elements 520 have upon the appearance of a displayed image, photosensor elements 520 having or consuming a smaller area than the display elements 510 are employed. Furthermore, various display and photosensor element layout geometries may be used to produce an interleaving pattern that closely approximates display element pitch found in conventional display devices. FIG. 18 is an illustration of a second photosensor and display element interleaving technique, in which photosensor and display element geometries and size differentials aid in minimizing pixel pitch and maximizing displayed image resolution. Since a viewer's eye will integrate or average the light output by groups of display elements 510, interleaving techniques of the type shown in FIG. 18 ensure that the viewer will perceive a high-quality image. Those skilled in the art will understand that various microoptic structures or elements, such as microlenses, could be employed in the nonluminent spaces between display elements 510 and/or photosensor elements 520 to reduce or minimize a viewer's perception of nonluminent areas in a displayed image. Such microoptic structures are elaborated upon below.
  • [0127]
    The display elements 510 referred to herein may comprise essentially any type of conventional light emitting or transmitting device, such as a Light Emitting Diode (LED) or Liquid Crystal Display (LCD) pixel element. Similarly, the photosensor elements 520 may comprise essentially any type of conventional light sensing or detecting device. For example, the photosensor elements 520 could comprise photodiodes, such as Schottky or p-i-n photodiodes; phototransistors; capacitive or charge-coupled devices (CCDs); charge modulated devices (CMDs); or other types of light-sensitive devices. The photosensor elements 520 could be fabricated, for example, using standard semiconductor processing techniques employed during the manufacture of flat panel displays.
  • [0128]
    In a typical display device, a single display element 510 is used to output light of a particular color. Display elements 510 based upon organic electroluminescence are capable of simultaneously generating light comprising multiple wavelengths in the visible spectrum, and form the basis for full-color LED arrays. In particular, a single Stacked Organic Light Emitting Diode (SOLED) pixel element can produce red, green, and blue light. The intensity of each color is independently tunable, as is each color's mean wavelength. Thus, a single SOLED can form a full-color pixel. As an alternative to organic electroluminescent materials, the present invention may employ other full-color transparent or semitransparent luminescent materials, such as light-emitting and/or light-responsive polymer films.
  • [0129]
    FIG. 19 is a cross-sectional view showing a full-color pixel array integrated with a photosensor element array upon a common substrate 702 such as glass or plastic. As an example, a SOLED 710 is considered as the full-color pixel technology in the discussion that follows. Those skilled in the art will understand that the concepts described herein can be applied to other full-color pixel technologies. Each SOLED 710 comprises a first, second, and third semitransparent electrode 712, 714, 716; a first, second, and third organic electroluminescent layer 722, 724, 726; and a reflecting contact layer 730, in a manner understood by those skilled in the art. Each electroluminescent layer 722, 724, 726 emits light in a particular wavelength range in response to an applied electric field. For example, the first, second, and third organic electroluminescent layers 722, 724, 726 could respectively output blue, green, and red light.
  • [0130]
    A color filter 750, an optional microoptic structure 760, and a photosensor element 520 form a color-specific photosensor element 770 that is fabricated adjacent to each SOLED 710. The microoptic 760 may comprise one or more microlenses, apertures, and/or other types of planar optic structures, and serves to focus incoming light onto the photosensor element 520 to aid image formation in the manner described below. The microoptic structure 760 may be formed through the application of conventional microlens or planar optic fabrication techniques during photosensor element fabrication steps. For example, the microoptic structure 760 may be formed by depositing a selectively-doped dielectric or dielectric stack prior to or during photosensor element fabrication, in a manner well understood by those skilled in the art.
  • [0131]
    The color-specific photosensor element 770 may also include one or more antireflection layers, which are deposited in a conventional manner. Additionally, one or more types of passivation or isolation materials, such as Silicon Dioxide, Silicon Nitride, Polyimide, or spin-on-glass may be deposited in between each SOLED 710 and color-specific photosensor element 770 in a manner understood by those skilled in the art.
  • [0132]
    Each color-specific photosensor element 770 detects light characterized by a specific wavelength interval. Thus, while any given SOLED 710 may simultaneously output red, green, and/or blue light, separate color-specific photosensor elements 770 are used to individually detect red, green, and blue light. Because each SOLED 710 forms a full-color pixel, integration of a SOLED array with a photosensor array in the manner shown in FIG. 19 is particularly advantageous relative to providing a high-resolution display having image capture capabilities.
  • 4.7 Display and Photosensor Element Stacking
  • [0133]
    a) Integrated SOLED/Photosensor Element
  • [0134]
    A full-color pixel element such as a SOLED 710 and a color-specific photosensor element 770 can be integrated together, such that the incorporation of a photosensor element array into a display element array can be accomplished essentially without a resolution or pixel pitch penalty. FIG. 20 is a cross-sectional view showing an integrated full-color pixel/photosensor element 800, which may form the basis of an integrated display element/photosensor element array. For purpose of example, the full-color pixel element is considered to be a SOLED 810 in the description that follows. Those skilled in the art will understand that other types of full-color pixel technologies could be used to produce the integrated full-color pixel/photosensor element 800 described hereafter.
  • [0135]
    Relative to FIG. 19, like reference numbers designate like elements. The full-color pixel/photosensor element 800 comprises a SOLED 810 having a color-specific photosensor element 770 fabricated thereupon. The full-color pixel/photosensor element 800 is fabricated upon a substrate 702 such as glass. The SOLED 810 comprises a first, a second, a third, and a fourth semitransparent electrode 712, 714, 716, 812; a first, second, and third organic electroluminescent layer 722, 724, 726; and a patterned reflecting contact layer 830.
  • [0136]
    With the exception of the fourth semitransparent electrode 812 and the patterned reflecting contact layer 830, the SOLED 810 shown in FIG. 20 is essentially the same as that depicted in FIG. 19. The fourth semitransparent electrode 812 serves as one of the electrodes for the photosensor element 520 within the color-specific photosensor element 770, in a manner readily understood by those skilled in the art. Deposition of the fourth semitransparent electrode 812 may not be required under the patterned reflecting contact layer 830, and as such the SOLED 810 and color-specific photosensor element 770 may not share a common electrical interface layer. The patterned reflecting contact layer 830 comprises conventional contact materials or metals that have been patterned to include a gap or opening.
  • [0137]
    The color-specific photosensor element 770 is fabricated on top of the fourth semitransparent electrode 812, in the opening defined in the patterned reflecting contact layer 830. The color-specific photosensor element 770 thus detects light that has been transmitted through the substrate 702 and each of the first through fourth semitransparent electrodes 712, 714, 716, 812. Those skilled in the art will understand that the location of the opening defined in the patterned reflecting contact layer 830, and hence the location of the color-specific photosensor element 770 upon the SOLED 810, may vary among adjacent full-color pixel/photosensor elements to ensure that the human observer perceives a high-quality displayed image. The SOLED 810 and the color-specific photosensor element 770 may operate in a temporally-separated manner to ensure that image capture is essentially unaffected by image display, as further elaborated upon below.
  • [0138]
    b) Stacked Full-Color Emitter/Full-Color Detector Structures
  • [0139]
    A full-color pixel element, such as a stacked organic electroluminescent (SOE) structure, may also be used to detect light. Thus, a single structure based upon full-color materials technology may be used for both RGB light emission and RGB light detection, thereby advantageously facilitating the integration of a photosensor element array and a display element array while maintaining small pixel pitch and high image resolution.
  • [0140]
    FIG. 21 is a cross-sectional view of a first full-color emitter/detector 900. In the description that follows, the first full-color emitter/detector 900 is considered to be an SOE-based device. Those skilled in the art will recognize that other full-color technologies could be employed to produce the first full-color emitter/detector 900 in alternate embodiments.
  • [0141]
    Relative to FIGS. 19 and 20, like reference numbers designate like elements. The first full-color emitter/detector 900 is fabricated upon a substrate 702 such as glass, and comprises first through seventh semitransparent electrodes 712, 714, 716, 812, 912, 914, 916; first through sixth organic electroluminescent layers 722, 724, 726, 922, 924, 926; an optional microoptic structure 920; and a reflecting contact layer 730.
  • [0142]
    In the first full-color emitter/detector 900, the first through third organic electroluminescent layers 722, 724, 726 serve as RGB light emitters controlled by voltages applied to the first through fourth semitransparent electrodes 712, 714, 716, 812, and thus form a SOLED 902. The microoptic structure 920 comprises one or more microlenses, apertures, and/or other planar microoptic structures that focus incoming light into the fourth, fifth, and sixth organic electroluminescent layers 922, 924, 926, which in turn produces or induces pairwise voltage differences across the fifth, sixth, and seventh semitransparent electrodes 912, 914, 916 and the reflecting contact layer 730. The microoptic structure 920, the fourth through sixth organic electro-luminescent layers 922, 924, 926, the fifth through seventh semitransparent electrodes 912, 914, 916, and the reflecting contact layer 730 therefore form a first SOE photosensor 904 for detecting RGB light.
  • [0143]
    Light emitted by the SOLED 902 may travel through the substrate 702 toward a viewer, or through the first SOE photosensor 904, where it is reflected back toward the substrate 702 by the reflecting contact layer 730. The first SOE photosensor 904 detects incoming light that has traveled through the substrate 702 and the SOLED 902. As described in detail below, SOLED light emission and SOE photosensor light detection may occur in a temporally and/or spatially separated manner, such that image capture is essentially unaffected by image display.
  • [0144]
    Those skilled in the art will recognize that the SOLED 902 and the first SOE photosensor 904 may be able to share a single semitransparent electrode at their interface in an alternate embodiment (i.e., the first full-color emitter/detector 900 may be fabricated without one of the fourth or fifth semitransparent electrodes 812, 912) since SOLED and SOE photosensor operation within a single first full-color emitter/detector 900 may be temporally separated). Those skilled in the art will also understand that in addition to the layers described above, the first full-color emitter/detector 900 may include additional microoptic layers and/or one or more antireflective layers. Those skilled in the art will further recognize that in an alternate embodiment, the first full-color emitter/detector 900 could be fabricated such that the first SOE photosensor 904 resides in contact with the substrate 702, and the SOLED 902 resides on top of the first SOE photosensor 904. In such an embodiment, the reflecting contact layer 730 would be incorporated into the SOLED 902. Those skilled in the art will also recognize that either or both of the SOLED 902 and the first SOE photosensor 904 could be implemented using other types of transparent or semitransparent full-color device and/or materials technologies in alternate embodiments.
  • [0145]
    FIG. 22 is a cross-sectional view of a second full-color emitter/detector 1000. For ease of understanding, the second full-color emitter/detector is considered to be based upon SOE technology in the following description. Those skilled in the art will recognize that other full-color materials technologies could be employed to produce the second full-color emitter/detector 1000 in alternate embodiments.
  • [0146]
    Relative to FIG. 20, like reference numbers designate like elements. The second full-color emitter/detector 1000 is fabricated upon a substrate 702 such as glass, and comprises a first through fifth semitransparent electrode 712, 714, 716, 812, 1012; a first through sixth organic electroluminescent layer 722, 724, 726, 1022, 1024, 1026; an optional microoptic structure 1020; a first, a second, and a third reflecting contact layer 1032, 1034, 1036; and a first and a second boundary structure 1042, 1044.
  • [0147]
    The first through third organic electroluminescent layers 722, 724, 726, in conjunction with the first through fourth semitransparent electrodes 712, 714, 716, 812, form a SOLED 902 in a manner analogous to that described above with reference to FIG. 21. The microoptic structure 1020, the first through third organic electroluminescent layers 1022, 1024, 1026, the reflecting contact layers 1032, 1034, 1036, and the first and second boundary structures 1042, 1044 form a second SOE photosensor 1004.
  • [0148]
    Taken together, the fourth, fifth, and sixth organic electroluminescent layers 1022, 1024, 1026 and the boundary structures 1042, 1042 span an area essentially equal to that of any semitransparent electrode 712, 714, 716, 812, 1012. The first boundary structure 1042 separates the fourth and fifth organic electroluminescent layers 1022, 1024. Similarly, the second boundary structure 1044 separates the fifth and sixth organic electroluminescent layers 1024, 1026. The first, second, and third reflecting contact layers 1032, 1034, 1036 respectively reside upon or atop the fourth, fifth, and sixth organic electroluminescent layers 1022, 1024, 1026.
  • [0149]
    The microoptic structure 1020 may comprise one or more microlenses, apertures, and/or other planar microoptic structures that focus incoming light into the fourth, fifth, and sixth organic electroluminescent layers 1022, 1024, 1026. The fourth organic electroluminescent layer 1022 detects incoming photons having a wavelength range associated with a particular color, for example, red. The presence of such photons in the fourth organic electroluminescent layer produces or induces a voltage difference between the fourth semitransparent electrode 1012 and the first reflecting contact layer 1032. Similarly, the fifth and sixth organic electroluminescent layers 1024, 1026 each detect incoming light corresponding to a particular wavelength range, for example, green and blue, respectively. The presence of blue and green light respectively induces a voltage difference between the second and third reflecting contact layers 1034, 1036 and the fourth semitransparent electrode 1012.
  • [0150]
    Those skilled in the art will recognize that the thickness of each of the fourth, fifth, and sixth organic electroluminescent layers 1022, 1024, 1026 may be varied in accordance with the particular wavelength range that each such layer is to detect. Those skilled in the art will additionally recognize that the microoptic structure 1020 may be fabricated such that its characteristics vary laterally from one organic electroluminescent layer 1022, 1024, 1026 to another, and that one or more antireflection layers may be incorporated into the second full-color emitter/detector 1000. Moreover, the SOLED 902 and the second SOE photosensor 1004 may be able to share a single semitransparent electrode at their interface a manner analogous to that described above relative to the first SOE photosensor 904. Finally, those skilled in the art will recognize that either or both of the SOLED 902 and the second SOE photosensor 1004 could be implemented using other types of transparent or semitransparent full-color technologies in alternate embodiments.
  • 4.8 Other Integrated Emitter/Detector Structures
  • [0151]
    As indicated above, a light detecting element may be similar, nearly, or essentially identical in structure and/or composition to a light emitting element. Because any given emitter/detector structure may be used for light emission during one time interval and light detection during another time interval as described below, a single light emitting structure may also be used for light detection.
  • [0152]
    FIG. 23 is a cross-sectional diagram of a third full-color emitter/detector 1100. For ease of understanding, the third full-color emitter/detector is described hereafter in the context of SOE technology. Those skilled in the art will understand that other full-color materials and/or technologies could be employed to produce the third full-color emitter/detector 1100 in alternate embodiments.
  • [0153]
    Relative to FIG. 19, like reference numbers designate like elements. The third full-color emitter/detector 1100 is fabricated upon a substrate 702 such as glass or plastic. The third full-color emitter/detector 1100 comprises a SOLED 710 having a first through a third semitransparent electrode 712, 714, 716; a first, a second, and a third organic electroluminescent layer 722, 724, 726; a reflecting top contact layer 730. The third full-color emitter/detector 1000 may additionally include a microoptic layer 1120. During a first time interval, the SOLED 710 may operate in a light emitting mode in a conventional manner. During a second time interval, the SOLED 710, in conjunction with the microoptic layer 1120, operates as a photosensor to detect incoming light in a manner analogous to that described above relative to the SOE photosensors 904.
  • [0154]
    The microoptic layer 1120 may comprise a microlens and/or other type of planar optic structure, and may be fabricated such that different portions of the microoptic layer 1120 affect light in different manners. This in turn could aid in providing particular light detection responsivity while minimally affecting the manner in which light emitted by the third full-color emitter detector 1100 will be perceived by a human eye.
  • [0155]
    FIG. 24 is a top-view of an exemplary microoptic layer 1120 having different optical regions 1190, 1192 defined therein. A first optical region 1190 may allow light to pass in an essentially unaffected manner. A second optical region 1192 serves as a focusing element that produces a desired spatial or modal light intensity pattern within the third full-color emitter/detector. As the second optical region 1192 occupies a smaller area than the first optical region 1190, its affect upon human perception of light emitted by the third full-color emitter/detector may be small or minimal. Those skilled in the art will understand that the location of the second optical region 1192 may vary among adjacent third full-color emitter/detectors 1100, to further enhance the quality of a displayed image seen by a human eye.
  • [0156]
    In an alternate embodiment, the microoptic layer 1120 could include additional optical regions. For example, one or more portions of the first optical region 1190 could be designed or fabricated to compensate for any effects the second optical region 1192 has upon human perception of light emitted by the third full-color emitter/detector 1100. As another example, the second optical region 1192 could be replaced or augmented with other, possibly smaller, optical regions distributed across the plane of the microoptic layer 1120 to further optimize light detection and emission characteristics.
  • 4.9 Image Formation
  • [0157]
    A simple or compound lens is conventionally used to focus an image onto an array of photosensors. FIG. 25 illustrates a simple or compound lens 600 that receives or collects light 602 reflected or emanating from an object 604, and focuses such light onto a photosensor element array 606.
  • [0158]
    Relative to a single array that integrates both display and photosensor elements 510, 520, the use of a conventional simple or compound lens would adversely affect the characteristics of the displayed image. To facilitate image detection in such an integrated array, photosensor elements 520 may incorporate microoptic structures and/or apertures, as described above, on an individual basis. Each aperture and/or microoptic structure focuses light received from a small portion of an object onto a photosensor element 520. As depicted in FIG. 25, sets of microoptic-equipped photosensor elements 520 within a photosensor array 620 receive light 622, 624 emanating from different parts of an object 626. Those skilled in the art will recognize that the present invention could employ microoptic structures or elements that focus light onto multiple photosensor elements 520 in alternate embodiments, where such microoptic elements may be incorporated onto separates substrates. Signals output by the microoptic-equipped photosensor elements 520 are transferred to an image processing unit 628 for further processing, as described in detail below.
  • [0159]
    Conventional display devices comprise multiple rows or lines of display elements 510, and produce a displayed image on a line-by-line basis. Similarly, conventional photosensor arrays comprise multiple rows of photosensor elements 520, which may be scanned on a line-by-line basis during image capture operations. The integrated display element/photosensor element arrays considered herein may also 1) produce a displayed image by activating display elements 510 on a line-by-line basis; and 2) capture light received from an object by detecting photosensor element output signals on a line-by-line basis.
  • [0160]
    In one embodiment, the present invention includes a display control circuit for performing display line scans that produce a displayed image on a line-by-line basis, and a capture control circuit for performing photosensor line scans that read photosensor element output signals on a line-by-line basis. Each of the display and capture control circuits include conventional clocking, address decoding, multiplexing, and register circuitry. In order to ensure that image capture is essentially unaffected by image display (i.e., to prevent light emitted or transmitted by display elements 510 from affecting incoming light detection by adjacent photosensor elements 520), the display line scans and photosensor line scans may be temporally and/or physically separated relative to each other. This separation may be controlled via conventional clocking and/or multiplexing circuitry.
  • [0161]
    In one embodiment, photosensor line scans are initiated after a display line scan has generated fifty percent of an image (i.e., after fifty percent of the display element lines have been activated during a single full-screen scan cycle), such that the photosensor line scan trails the display line scan by a number of display element rows equal to one-half of the total number of display element rows present in the integrated display element/photosensor element array. More generally, the capture line scan could trail the display line scan by a particular time interval or a given number of completed display line scans.
  • [0162]
    In another embodiment, one-half of the display element lines define a first display field, and one-half of the display element lines define a second display field, in a manner well understood by those skilled in the art. Similarly, one-half of the photosensor element lines define a first photosensor field, and the remaining photosensor element lines define a second photosensor field. The first display field and either of the first or second photosensor fields may be scanned either simultaneously or in a time-separated manner, after which the second display field and the remaining photosensor field may be scanned in an analogous manner. Those skilled in the art will recognize that the display and photosensor field scanning can be performed in a manner that supports odd and even field scanning as defined for NTSC and PAL television standards.
  • [0163]
    In yet another embodiment, a single full-screen display line scan cycle is completed, after which a single full-screen photosensor line scan cycle is completed, after which subsequent full-screen display line and photosensor line scans are separately performed in a sequential manner.
  • [0164]
    The set of photosensor element output signals received during any given photosensor line scan are transferred to an image processing unit 628. The image processing unit 628 comprises signal processing circuitry, such as a DSP, that performs conventional digital image processing operations such as two-dimensional overlap deconvolution, decimation, interpolation, and/or other operations upon the signals generated during each photosensor line scan. Those skilled in the art will understand that the number and types of digital image processing operations performed upon the signals generated during each photosensor line scan may be dependent upon the properties of any microoptic structures associated with each photosensor element 520. Those skilled in the art will further understand that signal conditioning circuitry may additionally be present to amplify photosensor element signals or eliminate noise associated therewith. Such signal conditioning circuitry, or a portion thereof, may be integrated with each photosensor element 520.
  • [0165]
    The image processing unit 628 forms a conventional final output image array using signal processing methods, and outputs image array signals to a buffer or memory, after which such signals may be compressed and incorporated into data packets and/or converted into analog video signals for subsequent transmission, where the compression and/or conversion may occur in conjunction with associated audio signals.
  • [0166]
    The signal processing algorithms employed in image formation are determined by the nature of any microoptic elements employed in conjunction with the photosensor elements 520. Such algorithms may perform deconvolution, edge-effect handling, decimation, and/or interpolation operations in a manner understood by those skilled in the art.
  • [0167]
    For example, if the microoptic elements amount to tiny apertures that limit detector pixel source light to non-overlapping segments in the principal area of view, the signal processing amounts to aggregating the pixels into an array and potentially performing interpolation and/or decimation operations to match the resolution of the pixel detector array to that of the final desired image.
  • [0168]
    As detection pixels overlap by increasing amounts, the applied signal processing operations can advantageously sharpen the image by deconvolving the impulse response of the pixel overlap function. Depending upon the microoptic arrangement employed, which may be dictated by device cost and fabrication yield or reliability, the overlap impulse response takes on varying characteristics, affecting the algorithm the image processing unit 628 is required to perform. In general, the deconvolution can be handled as either a set of two-dimensional iterated difference equations, which are readily addressed by standard numerical methods associated with the approximate solution of differential equations, or through conversions to the frequency domain and appropriate division operations. Further, if the overlap function is highly localized, which can be a typical situation, the difference equations can be accurately approximated by neglecting higher-order terms, which greatly simplifies the resulting operations. This is in contrast to frequency domain techniques for this case, as localization in the impulse response implies immense nonlocalization in the transform domain. However, should the overlap impulse response itself be far less localized, frequency domain deconvolution methods may be advantageous. Care must be taken in limiting the division to relevant areas when there are zeros in the frequency-domain representation of the overlap impulse response (transfer function).
  • [0169]
    Edge effects at the boundaries of the pixel detector array can be handled by various methods, but if the overlap impulse response is kept localized by apertures and/or other microoptic elements, then undesirable edge effects in the final image formation (that may result from “brute-force” treatment of the edges) quickly vanish within a few pixels from the boundary of the final formed image. Cropping can then be employed to avoid such edge effect altogether. Thus, by creating a slightly-oversized pre-final image formation array and eliminating edge effect by cropping, a final image array of desired resolution having no edge effects induced by overlap impulse responses can be readily produced.
  • [0170]
    It is known to those skilled in the art that in general, aperture effects invoked by actual apertures and/or microoptic elements can create diffraction patterns or spatial intensity modes in the light transmitted through the optical structure. Such optical structures may be designed to enhance or eliminate particular modes or diffraction effects, in a manner readily understood by those skilled in the art.
  • [0171]
    While the teachings presented above have been described in relation to a display device having a camera or image capture capabilities integrated therein or thereupon, the above teachings relating to 1) various photosensor element, microoptic and/or apertured structures; and 2) image processing requirements for creating an array of image signals that correspond to a captured image can be applied to effectively create a camera disposed or integrated upon any one of a wide variety of surfaces or substrates, including glass, plastic, partially-silvered mirrors, or other materials. Photosensor elements 520 disposed upon such substrates may be organized or distributed in a manner similar to that shown above with reference to FIGS. 16, 17, and 18, with the exception that display elements 510 shown in those figures may not be present.
  • [0172]
    The principles of the present invention have been discussed herein with reference to certain embodiments thereof. Study of the principles disclosed herein will render obvious to those having ordinary skill in the art certain modifications thereto. The principles of the present invention specifically contemplate all such modifications.

Claims (13)

  1. 1. A display-pixel and photosensor-element device for use as a display and a camera, the device comprising:
    a plurality of display elements, each display element comprising a light-emitting diode; and
    a plurality of photosensor elements, each photosensor element comprising a light-sensitive device,
    wherein the plurality of display elements and the plurality of photosensor elements are fabricated onto an essentially planar surface so as to create a device that is usable as a display and a camera.
  2. 2. The display-pixel and photosensor-element device of claim 1 wherein the light-sensitive device comprises a diode that is light-sensitive.
  3. 3. The display-pixel and photosensor-element device of claim 2 wherein the diode is configured to act as a photodiode.
  4. 4. The display-pixel and photosensor-element device of claim 1 wherein the light-sensitive device is fabricated employing standard semiconductor manufacturing employed during the manufacture of flat panel displays.
  5. 5. The display-pixel and photosensor-element device of claim 1 wherein some of the plurality of display elements comprise a light-emitting diode that is configured to emit red light.
  6. 6. The display-pixel and photosensor-element device of claim 1 wherein some of the plurality of display elements comprise a light-emitting diode that is configured to emit green light.
  7. 7. The display-pixel and photosensor-element device of claim 1 wherein some of the plurality of display elements comprise a light-emitting diode that is configured to emit blue light.
  8. 8. The display-pixel and photosensor-element device of claim 1 wherein light-emitting diode comprises is an organic light emitting diode.
  9. 9. The display-pixel and photosensor-element device of claim 1 wherein light-emitting diode is comprised by a stacked organic light emitting diode (SOLED).
  10. 10. The display-pixel and photosensor-element device of claim 1 wherein the light-emitting diode and the light-sensitive device are comprised by a stacked organic electroluminescent (SOE) structure.
  11. 11. The display-pixel and photosensor-element device of claim 10 wherein the light-emitting diode and the light-sensitive device are a single structure comprising full-color materials technology for RGB light emission and RGB light detection, thereby integrating a photosensor element and a display element into a single element.
  12. 12. The display-pixel and photosensor-element device of claim 1 wherein the light-emitting diode and the light-sensitive device are a single structure comprising full-color materials technology for light emission and light detection, thereby integrating a photosensor element and a display element into a single element.
  13. 13. The display-pixel and photosensor-element device of claim 1 further comprising a plurality of micro-optic structures associated with at least the plurality of light-sensitive devices.
US12828280 1998-01-27 2010-06-30 Display-pixel and photosensor-element device and method therefor Abandoned US20100314631A1 (en)

Priority Applications (5)

Application Number Priority Date Filing Date Title
US7276298 true 1998-01-27 1998-01-27
PCT/US1999/001789 WO1999038324A1 (en) 1998-01-27 1999-01-27 Multifunction video communication service device
US60138400 true 2000-07-27 2000-07-27
US12419229 US8754842B2 (en) 1998-01-27 2009-04-06 Combined display and image capture without simple or compound lenses for video conferencing eye-contact and other applications
US12828280 US20100314631A1 (en) 1998-01-27 2010-06-30 Display-pixel and photosensor-element device and method therefor

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US12828280 US20100314631A1 (en) 1998-01-27 2010-06-30 Display-pixel and photosensor-element device and method therefor

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US12419229 Continuation US8754842B2 (en) 1998-01-27 2009-04-06 Combined display and image capture without simple or compound lenses for video conferencing eye-contact and other applications

Publications (1)

Publication Number Publication Date
US20100314631A1 true true US20100314631A1 (en) 2010-12-16

Family

ID=22109602

Family Applications (4)

Application Number Title Priority Date Filing Date
US12419229 Active 2020-07-30 US8754842B2 (en) 1998-01-27 2009-04-06 Combined display and image capture without simple or compound lenses for video conferencing eye-contact and other applications
US12828280 Abandoned US20100314631A1 (en) 1998-01-27 2010-06-30 Display-pixel and photosensor-element device and method therefor
US14105085 Active US9160894B2 (en) 1998-01-27 2013-12-12 Lensless imaging camera performing image formation in software and employing micro-optic elements that impose light diffractions
US14105123 Active US9172850B2 (en) 1998-01-27 2013-12-12 Lensless imaging camera performing image formation in software employing micro-optic elements creating overlap of light from distant sources over multiple photosensor elements

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US12419229 Active 2020-07-30 US8754842B2 (en) 1998-01-27 2009-04-06 Combined display and image capture without simple or compound lenses for video conferencing eye-contact and other applications

Family Applications After (2)

Application Number Title Priority Date Filing Date
US14105085 Active US9160894B2 (en) 1998-01-27 2013-12-12 Lensless imaging camera performing image formation in software and employing micro-optic elements that impose light diffractions
US14105123 Active US9172850B2 (en) 1998-01-27 2013-12-12 Lensless imaging camera performing image formation in software employing micro-optic elements creating overlap of light from distant sources over multiple photosensor elements

Country Status (4)

Country Link
US (4) US8754842B2 (en)
EP (1) EP1064786A4 (en)
CA (1) CA2318395C (en)
WO (1) WO1999038324A1 (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110032369A1 (en) * 2008-05-25 2011-02-10 Avistar Communications Corporation Vignetted optoelectronic array for use in synthetic image formation via signal processing, lensless cameras, and integrated camera-displays
EP2717568A2 (en) * 2012-10-04 2014-04-09 Sony Computer Entertainment Europe Limited Imaging device and method
US9496922B2 (en) 2014-04-21 2016-11-15 Sony Corporation Presentation of content on companion display device based on content presented on primary display device
DE102015016714A1 (en) * 2015-12-22 2017-06-22 Catharina Casper communication device

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB0015943D0 (en) 2000-06-30 2000-08-23 Smith Graeme R Modular software definable pre-amplifier
EP2144443B1 (en) * 2008-07-11 2017-08-23 Axis AB Video over ethernet
US9632344B2 (en) * 2010-07-09 2017-04-25 Lester F. Ludwig Use of LED or OLED array to implement integrated combinations of touch screen tactile, touch gesture sensor, color image display, hand-image gesture sensor, document scanner, secure optical data exchange, and fingerprint processing capabilities
US9626023B2 (en) * 2010-07-09 2017-04-18 Lester F. Ludwig LED/OLED array approach to integrated display, lensless-camera, and touch-screen user interface devices and associated processors
US8848054B2 (en) * 2010-07-29 2014-09-30 Crestron Electronics Inc. Presentation capture with automatically configurable output
US8890143B2 (en) * 2010-09-22 2014-11-18 Qualcomm Incorporated Method to optimize and reduce integrated circuit, package design, and verification cycle time
KR20120064545A (en) * 2010-12-09 2012-06-19 한국전자통신연구원 Video communicating apparatus having function of eye-to-eye communication and method thereof
US9578336B2 (en) * 2011-08-31 2017-02-21 Texas Instruments Incorporated Hybrid video and graphics system with automatic content detection process, and other circuits, processes, and systems
JP5924943B2 (en) * 2012-01-06 2016-05-25 キヤノン株式会社 Control method for an imaging apparatus and an imaging apparatus
US20140185989A1 (en) * 2013-01-02 2014-07-03 Broadcom Corporation Connector, Cabling And Signaling For Communication Protocols
US9413854B1 (en) * 2013-07-15 2016-08-09 Amazon Technologies, Inc. Network-accessible signal processing service

Citations (65)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3958235A (en) * 1974-07-26 1976-05-18 Duffy Francis A Light emitting diode display apparatus and system
US4496981A (en) * 1981-02-26 1985-01-29 Matsushita Electric Industrial Co., Ltd. Video camera with a monitor
US4583122A (en) * 1982-06-02 1986-04-15 Hitachi, Ltd. Imaging display device
US4794634A (en) * 1985-12-24 1988-12-27 Kabushiki Kaisha Komatsu Seisakusho Position-sensitive photodetector and light transmissive tablet and light-emitting pen
US4879250A (en) * 1988-09-29 1989-11-07 The Boeing Company Method of making a monolithic interleaved LED/PIN photodetector array
US4935812A (en) * 1987-11-12 1990-06-19 Etat Francais Method for broadcasting high definition television programs using temporal attenuation
US4991020A (en) * 1989-02-17 1991-02-05 Hughes Aircraft Company Imaging system for providing separate simultaneous real time images from a singel image sensor
US5101270A (en) * 1990-12-13 1992-03-31 The Johns Hopkins University Method and apparatus for radon transformation and angular correlation in optical processors
US5251263A (en) * 1992-05-22 1993-10-05 Andrea Electronics Corporation Adaptive noise cancellation and speech enhancement system and apparatus therefor
US5278673A (en) * 1992-09-09 1994-01-11 Scapa James R Hand-held small document image recorder storage and display apparatus
US5313055A (en) * 1991-09-30 1994-05-17 Fuji Xerox Co., Ltd. Two-dimensional image read/display device
US5319182A (en) * 1992-03-04 1994-06-07 Welch Allyn, Inc. Integrated solid state light emitting and detecting array and apparatus employing said array
US5331149A (en) * 1990-12-31 1994-07-19 Kopin Corporation Eye tracking system having an array of photodetectors aligned respectively with an array of pixels
US5335011A (en) * 1993-01-12 1994-08-02 Bell Communications Research, Inc. Sound localization system for teleconferencing using self-steering microphone arrays
US5340978A (en) * 1992-09-30 1994-08-23 Lsi Logic Corporation Image-sensing display panels with LCD display panel and photosensitive element array
US5389965A (en) * 1993-04-01 1995-02-14 At&T Corp. Video telephone station having variable image clarity
US5410595A (en) * 1992-11-12 1995-04-25 Motorola, Inc. Apparatus and method for noise reduction for a full-duplex speakerphone or the like
US5446564A (en) * 1992-06-03 1995-08-29 Casio Computer Co., Ltd. Liquid crystal display device having photosensor at each pixel
US5473701A (en) * 1993-11-05 1995-12-05 At&T Corp. Adaptive microphone array
US5555310A (en) * 1993-02-12 1996-09-10 Kabushiki Kaisha Toshiba Stereo voice transmission apparatus, stereo signal coding/decoding apparatus, echo canceler, and voice input/output apparatus to which this echo canceler is applied
US5572643A (en) * 1995-10-19 1996-11-05 Judson; David H. Web browser with dynamic display of information objects during linking
US5587928A (en) * 1994-05-13 1996-12-24 Vivo Software, Inc. Computer teleconferencing method and apparatus
US5617539A (en) * 1993-10-01 1997-04-01 Vicor, Inc. Multimedia collaboration system with separate data network and A/V network controlled by information transmitting on the data network
US5625684A (en) * 1993-02-04 1997-04-29 Local Silence, Inc. Active noise suppression system for telephone handsets and method
US5664021A (en) * 1993-10-05 1997-09-02 Picturetel Corporation Microphone system for teleconferencing system
US5703436A (en) * 1994-12-13 1997-12-30 The Trustees Of Princeton University Transparent contacts for organic devices
US5708659A (en) * 1993-10-20 1998-01-13 Lsi Logic Corporation Method for hashing in a packet network switching system
US5812659A (en) * 1992-05-11 1998-09-22 Jabra Corporation Ear microphone with enhanced sensitivity
US5812109A (en) * 1994-08-23 1998-09-22 Canon Kabushiki Kaisha Image input/output apparatus
US5825898A (en) * 1996-06-27 1998-10-20 Lamar Signal Processing Ltd. System and method for adaptive interference cancelling
US5828756A (en) * 1994-11-22 1998-10-27 Lucent Technologies Inc. Stereophonic acoustic echo cancellation using non-linear transformations
US5835627A (en) * 1995-05-15 1998-11-10 Higgins; Eric W. System and method for automatically optimizing image quality and processing time
US5848146A (en) * 1996-05-10 1998-12-08 Rane Corporation Audio system for conferencing/presentation room
US5920401A (en) * 1993-12-29 1999-07-06 Xerox Corporation Compact document imager
US5932895A (en) * 1997-05-20 1999-08-03 The Trustees Of Princeton University Saturated full color stacked organic light emitting devices
US6489992B2 (en) * 1996-04-15 2002-12-03 Massachusetts Institute Of Technology Large field of view CCD imaging system
US6504143B2 (en) * 1996-05-29 2003-01-07 Deutsche Telekom Ag Device for inputting data
US6535610B1 (en) * 1996-02-07 2003-03-18 Morgan Stanley & Co. Incorporated Directional microphone utilizing spaced apart omni-directional microphones
US20030193599A1 (en) * 2002-03-27 2003-10-16 Marconi Communications, Inc. Method and apparatus for displaying images in combination with taking images
US20040125053A1 (en) * 2002-09-10 2004-07-01 Sony Corporation Information processing apparatus and method, recording medium and program
US20040247484A1 (en) * 2002-09-20 2004-12-09 Yerazunis William S. Multi-way LED-based surface reflectance sensor and spectrophotometer
US20050169551A1 (en) * 2004-02-04 2005-08-04 Dean Messing System for improving an image displayed on a display
US20060001914A1 (en) * 2004-06-30 2006-01-05 Mesmer Ralph M Color scanner display
US20060007222A1 (en) * 2004-06-21 2006-01-12 Apple Computer, Inc. Integrated sensing display
US20060012986A1 (en) * 2004-07-19 2006-01-19 Joseph Mazzochette LED array package with internal feedback and control
US7034866B1 (en) * 2000-11-22 2006-04-25 Koninklijke Philips Electronics N.V. Combined display-camera for an image processing system
US20060086896A1 (en) * 2004-10-22 2006-04-27 New York University Multi-touch sensing light emitting diode display and method for using the same
US20060145326A1 (en) * 2004-04-06 2006-07-06 Available For Licensing NANO IC packaging
US20060157640A1 (en) * 2005-01-18 2006-07-20 Perlman Stephen G Apparatus and method for capturing still images and video using coded aperture techniques
US20070187794A1 (en) * 2005-06-17 2007-08-16 Toppan Printing Co., Ltd. Imaging device
US7265784B1 (en) * 2002-08-19 2007-09-04 Pixim, Inc. Image processor with noise reduction circuit
US7348946B2 (en) * 2001-12-31 2008-03-25 Intel Corporation Energy sensing light emitting diode display
US20080150848A1 (en) * 2006-12-26 2008-06-26 Lg. Philips Lcd Co., Ltd. Organic light-emitting diode panel and touch-screen system including the same
US7420608B2 (en) * 2003-06-18 2008-09-02 Canon Kabushiki Kaisha Display device with image sensing device
US20080231564A1 (en) * 2007-03-16 2008-09-25 Sony Corporation Display apparatus and method for controlling the same
US7532772B2 (en) * 2004-07-20 2009-05-12 Duke University Coding for compressive imaging
US7697053B2 (en) * 2006-11-02 2010-04-13 Eastman Kodak Company Integrated display having multiple capture devices
US7714923B2 (en) * 2006-11-02 2010-05-11 Eastman Kodak Company Integrated display and capture apparatus
US7808540B2 (en) * 2007-01-09 2010-10-05 Eastman Kodak Company Image capture and integrated display apparatus
US7821561B2 (en) * 2003-12-12 2010-10-26 Canon Kabushiki Kaisha Display apparatus with camera and communication apparatus
US20110032369A1 (en) * 2008-05-25 2011-02-10 Avistar Communications Corporation Vignetted optoelectronic array for use in synthetic image formation via signal processing, lensless cameras, and integrated camera-displays
US7920171B2 (en) * 2007-05-18 2011-04-05 Aptina Imaging Corporation Methods and apparatuses for vignetting correction in image signals
US20110164165A1 (en) * 2008-11-14 2011-07-07 Canon Kabushiki Kaisha Focus detection apparatus, focus detection method, and image sensing apparatus
US20110243430A1 (en) * 2008-11-04 2011-10-06 Konica Minolta Opto, Inc. Image input apparatus
US20120153153A1 (en) * 2007-07-25 2012-06-21 Eminent Electronic Technology Corp. Integrated Ambient Light Sensor and Distance Sensor

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0778704A3 (en) * 1991-07-15 1997-08-20 Hitachi Ltd Teleconference module
US5802281A (en) * 1994-09-07 1998-09-01 Rsi Systems, Inc. Peripheral audio/video communication system that interfaces with a host computer and determines format of coded audio/video signals
EP0705037B1 (en) * 1994-09-28 2002-12-04 Canon Kabushiki Kaisha Image input unit
US5761318A (en) * 1995-09-26 1998-06-02 Nippon Telegraph And Telephone Corporation Method and apparatus for multi-channel acoustic echo cancellation
US5717691A (en) * 1995-10-30 1998-02-10 Nec Usa, Inc. Multimedia network interface for asynchronous transfer mode communication system
DE19545484C2 (en) * 1995-12-06 2002-06-20 Deutsche Telekom Ag Image recording device
US5929845A (en) * 1996-09-03 1999-07-27 Motorola, Inc. Image scanner and display apparatus

Patent Citations (65)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3958235A (en) * 1974-07-26 1976-05-18 Duffy Francis A Light emitting diode display apparatus and system
US4496981A (en) * 1981-02-26 1985-01-29 Matsushita Electric Industrial Co., Ltd. Video camera with a monitor
US4583122A (en) * 1982-06-02 1986-04-15 Hitachi, Ltd. Imaging display device
US4794634A (en) * 1985-12-24 1988-12-27 Kabushiki Kaisha Komatsu Seisakusho Position-sensitive photodetector and light transmissive tablet and light-emitting pen
US4935812A (en) * 1987-11-12 1990-06-19 Etat Francais Method for broadcasting high definition television programs using temporal attenuation
US4879250A (en) * 1988-09-29 1989-11-07 The Boeing Company Method of making a monolithic interleaved LED/PIN photodetector array
US4991020A (en) * 1989-02-17 1991-02-05 Hughes Aircraft Company Imaging system for providing separate simultaneous real time images from a singel image sensor
US5101270A (en) * 1990-12-13 1992-03-31 The Johns Hopkins University Method and apparatus for radon transformation and angular correlation in optical processors
US5331149A (en) * 1990-12-31 1994-07-19 Kopin Corporation Eye tracking system having an array of photodetectors aligned respectively with an array of pixels
US5313055A (en) * 1991-09-30 1994-05-17 Fuji Xerox Co., Ltd. Two-dimensional image read/display device
US5319182A (en) * 1992-03-04 1994-06-07 Welch Allyn, Inc. Integrated solid state light emitting and detecting array and apparatus employing said array
US5812659A (en) * 1992-05-11 1998-09-22 Jabra Corporation Ear microphone with enhanced sensitivity
US5251263A (en) * 1992-05-22 1993-10-05 Andrea Electronics Corporation Adaptive noise cancellation and speech enhancement system and apparatus therefor
US5446564A (en) * 1992-06-03 1995-08-29 Casio Computer Co., Ltd. Liquid crystal display device having photosensor at each pixel
US5278673A (en) * 1992-09-09 1994-01-11 Scapa James R Hand-held small document image recorder storage and display apparatus
US5340978A (en) * 1992-09-30 1994-08-23 Lsi Logic Corporation Image-sensing display panels with LCD display panel and photosensitive element array
US5410595A (en) * 1992-11-12 1995-04-25 Motorola, Inc. Apparatus and method for noise reduction for a full-duplex speakerphone or the like
US5335011A (en) * 1993-01-12 1994-08-02 Bell Communications Research, Inc. Sound localization system for teleconferencing using self-steering microphone arrays
US5625684A (en) * 1993-02-04 1997-04-29 Local Silence, Inc. Active noise suppression system for telephone handsets and method
US5555310A (en) * 1993-02-12 1996-09-10 Kabushiki Kaisha Toshiba Stereo voice transmission apparatus, stereo signal coding/decoding apparatus, echo canceler, and voice input/output apparatus to which this echo canceler is applied
US5389965A (en) * 1993-04-01 1995-02-14 At&T Corp. Video telephone station having variable image clarity
US5617539A (en) * 1993-10-01 1997-04-01 Vicor, Inc. Multimedia collaboration system with separate data network and A/V network controlled by information transmitting on the data network
US5664021A (en) * 1993-10-05 1997-09-02 Picturetel Corporation Microphone system for teleconferencing system
US5708659A (en) * 1993-10-20 1998-01-13 Lsi Logic Corporation Method for hashing in a packet network switching system
US5473701A (en) * 1993-11-05 1995-12-05 At&T Corp. Adaptive microphone array
US5920401A (en) * 1993-12-29 1999-07-06 Xerox Corporation Compact document imager
US5587928A (en) * 1994-05-13 1996-12-24 Vivo Software, Inc. Computer teleconferencing method and apparatus
US5812109A (en) * 1994-08-23 1998-09-22 Canon Kabushiki Kaisha Image input/output apparatus
US5828756A (en) * 1994-11-22 1998-10-27 Lucent Technologies Inc. Stereophonic acoustic echo cancellation using non-linear transformations
US5703436A (en) * 1994-12-13 1997-12-30 The Trustees Of Princeton University Transparent contacts for organic devices
US5835627A (en) * 1995-05-15 1998-11-10 Higgins; Eric W. System and method for automatically optimizing image quality and processing time
US5572643A (en) * 1995-10-19 1996-11-05 Judson; David H. Web browser with dynamic display of information objects during linking
US6535610B1 (en) * 1996-02-07 2003-03-18 Morgan Stanley & Co. Incorporated Directional microphone utilizing spaced apart omni-directional microphones
US6489992B2 (en) * 1996-04-15 2002-12-03 Massachusetts Institute Of Technology Large field of view CCD imaging system
US5848146A (en) * 1996-05-10 1998-12-08 Rane Corporation Audio system for conferencing/presentation room
US6504143B2 (en) * 1996-05-29 2003-01-07 Deutsche Telekom Ag Device for inputting data
US5825898A (en) * 1996-06-27 1998-10-20 Lamar Signal Processing Ltd. System and method for adaptive interference cancelling
US5932895A (en) * 1997-05-20 1999-08-03 The Trustees Of Princeton University Saturated full color stacked organic light emitting devices
US7034866B1 (en) * 2000-11-22 2006-04-25 Koninklijke Philips Electronics N.V. Combined display-camera for an image processing system
US7348946B2 (en) * 2001-12-31 2008-03-25 Intel Corporation Energy sensing light emitting diode display
US20030193599A1 (en) * 2002-03-27 2003-10-16 Marconi Communications, Inc. Method and apparatus for displaying images in combination with taking images
US7265784B1 (en) * 2002-08-19 2007-09-04 Pixim, Inc. Image processor with noise reduction circuit
US20040125053A1 (en) * 2002-09-10 2004-07-01 Sony Corporation Information processing apparatus and method, recording medium and program
US20040247484A1 (en) * 2002-09-20 2004-12-09 Yerazunis William S. Multi-way LED-based surface reflectance sensor and spectrophotometer
US7420608B2 (en) * 2003-06-18 2008-09-02 Canon Kabushiki Kaisha Display device with image sensing device
US7821561B2 (en) * 2003-12-12 2010-10-26 Canon Kabushiki Kaisha Display apparatus with camera and communication apparatus
US20050169551A1 (en) * 2004-02-04 2005-08-04 Dean Messing System for improving an image displayed on a display
US20060145326A1 (en) * 2004-04-06 2006-07-06 Available For Licensing NANO IC packaging
US20060007222A1 (en) * 2004-06-21 2006-01-12 Apple Computer, Inc. Integrated sensing display
US20060001914A1 (en) * 2004-06-30 2006-01-05 Mesmer Ralph M Color scanner display
US20060012986A1 (en) * 2004-07-19 2006-01-19 Joseph Mazzochette LED array package with internal feedback and control
US7532772B2 (en) * 2004-07-20 2009-05-12 Duke University Coding for compressive imaging
US20060086896A1 (en) * 2004-10-22 2006-04-27 New York University Multi-touch sensing light emitting diode display and method for using the same
US20060157640A1 (en) * 2005-01-18 2006-07-20 Perlman Stephen G Apparatus and method for capturing still images and video using coded aperture techniques
US20070187794A1 (en) * 2005-06-17 2007-08-16 Toppan Printing Co., Ltd. Imaging device
US7714923B2 (en) * 2006-11-02 2010-05-11 Eastman Kodak Company Integrated display and capture apparatus
US7697053B2 (en) * 2006-11-02 2010-04-13 Eastman Kodak Company Integrated display having multiple capture devices
US20080150848A1 (en) * 2006-12-26 2008-06-26 Lg. Philips Lcd Co., Ltd. Organic light-emitting diode panel and touch-screen system including the same
US7808540B2 (en) * 2007-01-09 2010-10-05 Eastman Kodak Company Image capture and integrated display apparatus
US20080231564A1 (en) * 2007-03-16 2008-09-25 Sony Corporation Display apparatus and method for controlling the same
US7920171B2 (en) * 2007-05-18 2011-04-05 Aptina Imaging Corporation Methods and apparatuses for vignetting correction in image signals
US20120153153A1 (en) * 2007-07-25 2012-06-21 Eminent Electronic Technology Corp. Integrated Ambient Light Sensor and Distance Sensor
US20110032369A1 (en) * 2008-05-25 2011-02-10 Avistar Communications Corporation Vignetted optoelectronic array for use in synthetic image formation via signal processing, lensless cameras, and integrated camera-displays
US20110243430A1 (en) * 2008-11-04 2011-10-06 Konica Minolta Opto, Inc. Image input apparatus
US20110164165A1 (en) * 2008-11-14 2011-07-07 Canon Kabushiki Kaisha Focus detection apparatus, focus detection method, and image sensing apparatus

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110032369A1 (en) * 2008-05-25 2011-02-10 Avistar Communications Corporation Vignetted optoelectronic array for use in synthetic image formation via signal processing, lensless cameras, and integrated camera-displays
US8830375B2 (en) 2008-05-25 2014-09-09 Lester F. Ludwig Vignetted optoelectronic array for use in synthetic image formation via signal processing, lensless cameras, and integrated camera-displays
EP2717568A2 (en) * 2012-10-04 2014-04-09 Sony Computer Entertainment Europe Limited Imaging device and method
EP2717568A3 (en) * 2012-10-04 2014-04-16 Sony Computer Entertainment Europe Limited Imaging device and method
US9100536B2 (en) 2012-10-04 2015-08-04 Sony Computer Entertainment Europe Limited Imaging device and method
US9496922B2 (en) 2014-04-21 2016-11-15 Sony Corporation Presentation of content on companion display device based on content presented on primary display device
DE102015016714A1 (en) * 2015-12-22 2017-06-22 Catharina Casper communication device

Also Published As

Publication number Publication date Type
CA2318395C (en) 2005-03-22 grant
CA2318395A1 (en) 1999-07-29 application
EP1064786A1 (en) 2001-01-03 application
US20140098279A1 (en) 2014-04-10 application
EP1064786A4 (en) 2005-09-28 application
US9160894B2 (en) 2015-10-13 grant
US20090231474A1 (en) 2009-09-17 application
US20140104475A1 (en) 2014-04-17 application
WO1999038324A1 (en) 1999-07-29 application
US9172850B2 (en) 2015-10-27 grant
US8754842B2 (en) 2014-06-17 grant

Similar Documents

Publication Publication Date Title
US6195357B1 (en) Interactive information transaction processing system with universal telephony gateway capabilities
US7714923B2 (en) Integrated display and capture apparatus
US5802281A (en) Peripheral audio/video communication system that interfaces with a host computer and determines format of coded audio/video signals
US6972786B1 (en) Multimedia services using central office
US5774527A (en) Integrated telephone and cable communication networks
US7155679B2 (en) Digital media frame
US20030122954A1 (en) Video camera and method and device for capturing video, audio and data signals
US5048077A (en) Telephone handset with full-page visual display
US5793415A (en) Videoconferencing and multimedia system
US6452632B1 (en) Solid state image sensor and video system using the same
US5079627A (en) Videophone
US20040150748A1 (en) Systems and methods for providing and displaying picture-in-picture signals
US20070256105A1 (en) Entertainment device configured for interactive detection and security vigilant monitoring in communication with a control server
US20120083314A1 (en) Multimedia Telecommunication Apparatus With Motion Tracking
US20030160861A1 (en) Video-telephony system
US20040001137A1 (en) Integrated design for omni-directional camera and microphone array
US20070047712A1 (en) Scalable, distributed architecture for fully connected network intercom system
US20070002130A1 (en) Method and apparatus for maintaining eye contact during person-to-person video telecommunication
US6978474B1 (en) Media interface device
US6425131B2 (en) Method and apparatus for internet co-browsing over cable television and controlled through computer telephony
US20050047431A1 (en) Outlet with analog signal adapter, a method for use thereof and a network using said outlet
US20080030573A1 (en) Volumetric panoramic sensor systems
US7221386B2 (en) Camera for communication of streaming media to a remote client
US20040150751A1 (en) Systems and methods for forming picture-in-picture signals
US20020054601A1 (en) Network interface unit control system and method therefor

Legal Events

Date Code Title Description
AS Assignment

Owner name: AVISTAR COMMUNICATIONS CORPORATION, CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:LUDWIG, LESTER F.;REEL/FRAME:024912/0119

Effective date: 20100728

AS Assignment

Owner name: LUDWIG, LESTER F., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:AVISTAR COMMUNICATIONS CORPORATION;REEL/FRAME:029226/0078

Effective date: 20120816

AS Assignment

Owner name: NRI R&D PATENT LICENSING, LLC, TEXAS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:LUDWIG, LESTER F;REEL/FRAME:042745/0063

Effective date: 20170608