WO1998047291A2

WO1998047291A2 - Video teleconferencing

Info

Publication number: WO1998047291A2
Application number: PCT/IL1998/000141
Authority: WO
Inventors: Noam Sorek; Doron Adler; Ilia Vitsnudel; Shai Eisenberg; David Gidron; Micha Nisani; Ron Fridental; Alex Lishanski; Itai Otashinski; Ofra Zinat; Arieh Blumensvig; Rami Livne; Michael Skala
Original assignee: Isight Ltd.
Priority date: 1997-04-16
Filing date: 1998-03-26
Publication date: 1998-10-22
Also published as: AU6515798A; WO1998047291A3

Abstract

Apparatus (60) and methods for producing and displaying panoramic images. The apparatus includes a plurality of video sensor heads (92), each sensor head generating video signals corresponding to a partial image having a respective field of view (100), such that at least some of the fields of view include regions of substantial overlap (124) with respective neighboring fields of view, and a union of the fields of view substantially covers a scene having an angular extent beyond the field of view of any one of the sensor heads. A processor (40) receives and combines the partial images from the sensor heads to produce a panoramic image (231) of the scene. The panoramic image is displayed by defining an annular strip (230) and a point of view outside a volume bounded by the strip, and mapping the panoramic image onto the annular strip. An image of the strip, as seen from the point of view, is rendered to a video display (50).

Description

VIDEO TELECONFERENCING

RELATED APPLICATION

This patent application claims the benefit of U.S. Provisional Patent Application No. 60/043,243, filed April 16, 1997, which is assigned to the assignee of the present application, and whose disclosure is incorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates generally to video teleconferencing systems, and specifically to methods and apparatus for producing, transmitting and displaying panoramic images for use in video teleconferencing. BACKGROUND OF THE INVENTION

Panoramic imaging systems are known in the art. Such systems frequently make use of a single camera with panoramic optics, for example, a fisheye lens, or a combination of a lens and a convex mirror, as in the "OmniCam" system developed at Columbia University, New York. Image processing is applied to the images captured by the camera, in order to remove the distortion introduced by the optics. These systems generally have poor resolution, due to the inherently low resolution of standard video cameras, or require the use of a very expensive, high-resolution camera to achieve adequate picture quality.

Alternatively, panoramic images may be produced by panning a video camera over a scene and then aligning and combining multiple images from the pan into a single "mosaic" image. This approach allows only a single, relatively small region of interest (ROI) in the scene to be updated in real time (live images), while other portions of the scene, at which the camera is not pointing, are "frozen." Such cameras cannot easily accommodate rapid changes in the ROI, due, for example, to different participants speaking in a teleconference, because of the time it takes the camera to pan from one ROI to the other. Computational methods for image mosaicking are well known in the art, as described, for example, in an article by Jaillon, et al., entitled "Image Mosaicking Applied to Three-Dimensional Surfaces," published in the Proceedings of the Twelfth International Conference on Pattern Recognition (IAPR), October, 1994, pages A253-257, which is incorporated herein by reference.

U.S. patent 5,187,571, to Braun et al., which is incorporated herein by reference, describes a television system for displaying multiple views of a remote location. The system includes a sending station, a composing system and a receiving station. The sending station includes a sequence of commonly-synchronized cameras, arranged so that the fields of view of the cameras merge contiguously, without substantial overlap between the fields, to form an aggregate field of view, which is shown in preferred embodiments to be a 360 panoramic view. The composing station multiplexes the camera signals so that a viewer at the receiving station can select a subsection of the aggregate field of view to display on a local monitor.

Braun et al. state that in order for the fields of view of the cameras to be contiguous, the cameras must be arranged so that all of the fields of view have a common vertex. Two optical arrangements are proposed for this purpose: A first embodiment is shown in Figs. 2 and 3 of the patent, in which the cameras are positioned in a common plane, and two folding mirrors are associated with each camera to suitably direct its field of view, as well as serving as baffles, thus preventing substantial overlap between the fields of view of adjacent cameras. A second, substantially more compact embodiment is shown in Figs. 4 and 5, in which the cameras are positioned side-by-side, and a pyramidal mirror, having one face corresponding to each of the cameras, directs the fields of view. In the second embodiment, there is no provision made for baffling.

U.S. patent 5,444,478, to Lelong et al., which is also incorporated herein by reference, points out several limitations of the system of patent 5,187,571, and describes a method and apparatus for image processing meant to overcome these limitations. In accordance with this method, the signals from the cameras, referred to as "source images," are digitized and are then geometrically transformed, or remapped, to construct a "target image." It is pointed out that at the boundaries between the fields of view of neighboring cameras in the system of the '571 patent, the viewing angle difference between one camera and its neighbor causes great distortions of the image. In the method described by Lelong et al., a user selects a "virtual camera" angle and, optionally, a zoom. Data from the source images are then transformed to produce a target image, largely distortion-free, such as would have been captured by the virtual camera. Whenever a new virtual camera angle or zoom is chosen, the transformation is accordingly recomputed. Neither Lelong nor Braun suggests how the target image is to be chosen within the aggregate image, or how a panoramic image might be displayed representing the aggregate image. In some video conferencing systems known in the art, a camera is controlled to pan, tilt and zoom automatically, so as to capture an image of whichever meeting participant is speaking. For example, the "Limelight" speaker locating system, produced by PictureTel Corporation, of Andover, Massachusetts, uses a microphone array, adjacent to a video teleconferencing camera, to determine the direction of sounds in a conference room and directs the camera to pan and tilt toward the sound. The camera can also be directed to zoom, in order to capture multiple speakers simultaneously.

SUMMARY OF THE INVENTION

It is an object of some aspects of the present invention to provide an improved system for video teleconferencing.

It is a further object of some aspects of the present invention to provide improved panoramic video imaging apparatus and methods.

Yet another object of some aspects of the present invention is to provide methods and apparatus for conveniently displaying panoramic video images.

It is still another object of some aspects of the present invention to provide methods and apparatus for efficiently compressing and transmitting images captured by the apparatus. In some aspects of the present invention, audio signals are received, processed and transmitted in conjunction with video signals.

An additional object of some aspects of the present invention is to provide a user interface for video teleconferencing which is flexible and easy to operate.

In preferred embodiments of the present invention, a video teleconferencing system comprises panoramic image capture apparatus and a panoramic image display, as well as user controls and a communications interface for sending and receiving teleconference data over communication lines.

In preferred embodiments of the present invention, the image capture apparatus comprises a plurality of substantially stationary video sensor heads and optics coupled to the sensor heads. The optics direct the field of view of each of the sensor heads in a different direction, such that the fields of view of all the sensor heads share a substantially commonly- centered entrance pupil, and such that the field of view of each sensor head overlaps in an edge region thereof with that of its neighbors. The fields of view of all the sensor heads together cover a desired angular spread, preferably having an angular aspect ratio (width/height) of at least 2:1. Preferably, the combined fields of view cover at least a 90° azimuthal spread, more preferably at least 180°, and most preferably a 360° spread. The apparatus further includes image capture circuitry, which commonly controls the plurality of sensor heads and receives and combines video signals from the sensor heads to form a panoramic image.

The term "video" as used in the present patent application and in the claims refers to electronic signals generated responsive to and/or representative of an image. As such, the terms "video sensor head" and "video signal" may be taken to refer to sensor heads and signals conforming with common video standards, such as NTSC and PAL, but may also refer to other standard and non-standard devices and signals used in electronic imaging. It will be understood by those skilled in the art that the principles of the present invention are equally applicable to standard video systems and to non-standard electronic imaging apparatus and methods.

In preferred embodiments of the present invention, the optics cause the image captured by each of the sensor heads to be vignetted in the edge regions thereof. Preferably, the overlap and vignetting are used to improve the quality of the resultant panoramic image and simplify the image processing needed to construct the image. More preferably, the optics are designed so that in regions of overlap between neighboring fields of view, the intensity of the image received by each of the sensor heads is reduced by vignetting according to a predetermined functional form. Additionally or alternatively, image processing is applied in the edge regions of the images to adjust the intensity and add the images together as required.

The video signals generated by the neighboring sensor heads are mutually registered and are selectively summed together in each of the overlap regions. Preferably, the sum of the signals due to the reduced intensities is generally equal to the signal that would have been generated by a single sensor head viewing the region without vignetting or overlap, with a substantially smooth transition from one field of view to the next.

In some preferred embodiments of the present invention, the image capture optics comprise a multifaceted mirror, including baffles, which protrude radially outward relative to the facets of the mirror, to prevent optical "crosstalk" between the facets. The baffles are preferably located at the joints between the facets of the mirror or, alternatively or additionally, along the midlines of the facets or in any other suitable location. The facets themselves are preferably planar, but may alternatively have a suitable curved shape, preferably convex. The curved shape may be spherical or may correspond to a surface of revolution of another conic section, or may be of any other suitable aspheric shape known in the art. Such curved facets are useful in reducing the size of the image capture apparatus and in reducing the effects of keystone distortion, in the output image therefrom.

By comparison with such preferred embodiments, in image capture systems known in the art using stationary cameras, such as that described in the above-mentioned U.S. Patent 5,187,571, there is no provision for such baffling, and no account is taken of the overlap of the neighboring fields of view. In the system of the '571 patent, the neighboring fields of view are assumed to be mutually contiguous, i.e., non-overlapping, necessitating substantial post- acquisition image processing, as described in the above-mentioned U.S. Patent 5,444,478. There is also no provision made in such systems for the possibility of using multifaceted mirrors with curved facets, as described above.

In some preferred embodiments of the present invention, the image capture circuitry collectively controls and, preferably, mutually synchronizes the plurality of sensor heads and processes the video signals generated thereby to vary certain parameters of the image, preferably varying the parameters as a function of azimuthal angle. The circuitry thus optimizes the quality of the panoramic image for at least a plurality of the azimuthal angles that the image covers, while maintaining a substantially smooth variation of the parameters over the entire image. Preferably, the parameters controlled and optimized by the circuitry include the image brightness, contrast, dynamic range and color balance, inter alia.

Furthermore, in some preferred embodiments of the present invention, the panoramic image is mapped onto a substantially cylindrical image surface, centered at a central point of view. In other words, each point in a scene that is captured by the image capture apparatus is geometrically transformed, i.e., magnified, either optically or by electronic image processing, or by a combination of optical and electronic image processing, so that the panoramic image is equivalent to an image that would be captured by a cylindrical image detector (if such a detector and suitable optics were available), likewise centered at the central point of view. Preferably, the mapping is such that the geometrical transformation performed by the apparatus varies as a function of azimuthal angle, so that all the participants in a video teleconference captured by the apparatus are shown in the panoramic image at substantially the same size, even if not all the participants are equally distant from the apparatus.

Although such cylindrical mapping may cause mild distortion of the image when viewed on a flat display, it has been found that such distortion is not generally noticeable in teleconferencing applications. On the other hand, the cylindrical mapping and azimuth- dependent variable magnification allows smooth panning through the image when it is displayed, as described below, without camera movement and without the need to remap the image each time the view angle is changed, as is necessitated by systems known in the art, for example, as described in the above-mentioned '478 patent. The cylindrical mapping also reduces the effect of keystone distortion, which is characteristic of flat projection transformations.

In some preferred embodiments of the present invention, there is a microphone array associated with the image capture apparatus, and audio signals generated by the array are transmitted to a remote viewing site, along with the panoramic image. Preferably, the array comprises a plurality of omnidirectional microphones, each receiving sound in a different location or, alternatively, a plurality of directional microphones, each aimed in a different direction. A viewer of the panoramic image at the remote viewing site can choose to hear sound received from a desired direction, generally from the direction of a person of interest shown in the image. The audio signals received by the plurality of microphones are combined, preferably in a phase-dependent manner, to generate an output audio signal corresponding to the sound incident on the microphones from the desired direction. Alternatively or additionally, the audio signals from the plurality of microphones may be processed and fed to a plurality of speakers at the remote viewing site, so as to provide directional, stereophonic sound.

Further alternatively, as is known in the art, the microphone array may be used to determine the direction of sounds impinging thereon, so as to indicate or select automatically the person in the image who is speaking.

Still further additionally or alternatively, the audio signals may be combined, as described above, so that the sound impinging on the microphone array from a certain direction is muted. Such a muting function may be invoked, for example, by a person participating in a video teleconference who wishes to speak without being overheard at the remote viewing site. Similarly, a portion of the panoramic image may be frozen, blacked out or dimmed, or replaced by a video looping sequence if it is desired that the portion not be seen by participants at the remote viewing site.

In a preferred embodiment of the present invention, the image capture apparatus produces a stereoscopic panoramic image. Preferably, the plurality of sensor heads comprises a plurality of pairs of first and second sensor heads, suitably spaced. Each pair of sensor heads is positioned to receive images of a scene from a respective one of the faces of the multifaceted mirror, as described above. The image capture circuitry combines the video signals from all of the first sensor heads to generate a first panoramic image and from all of the second sensor heads to generate a second panoramic image. When the first and second panoramic images are viewed using a suitable stereoscopic display, as is known in the art, a stereoscopic image is observed, giving the illusion of depth. It will be appreciated that other methods and apparatus for panoramic image capture, as are known in the art, are generally unsuited to creating stereoscopic images, due to mutual obscuration by the at least two cameras required for stereoscopy of each other's field of view. In some preferred embodiments of the present invention, the display comprises a video monitor and a display processor, which renders the panoramic image to the monitor. Preferably, the cylindrical image surface onto which the panoramic image is mapped is rendered to the monitor in the form of an annular strip, such as would be seen generally if the cylindrical image surface were observed from a point of view outside a volume contained within the surface. Preferably, the point of view is taken sufficiently above the equatorial plane of the volume, so that the entire surface can be seen and assumes a generally elliptical shape as rendered. This rendering scheme allows the entire panoramic image to be observed on an ordinary video monitor, in a manner that maintains an intuitive correspondence between the actual positions of participants in a video conference room and the positions of the respective images of the participants in the image on the monitor. Preferably, the user at the remote viewing site can control certain aspects of the panoramic image, for example, the location of the point of view from which the image is rendered and the height and eccentricity of the strip.

The annular strip comprises a first hemiannulus relatively proximate to the point of view outside the volume and a second hemiannulus relatively distant therefrom. Whereas the second hemiannulus is rendered substantially as seen from the point of view, the first hemiannulus is shown "inside-out," so that the inner surface of the hemiannulus is displayed on the outside of the hemiannulus (since the cylindrical image surface has an image only on its inner side, and there is no image associated with the outer side). In order to maintain the images of the video conference participants in their proper spatial order, the images of the participants in the first hemiannulus are preferably left-right reversed.

As described above, because the panoramic image is mapped onto the generally cylindrical surface, the point of view from which the image is rendered can be changed without the need for remapping. Changing the azimuth of the point of view is substantially equivalent to rotating the surface about a central axis thereof. Thus, in some preferred embodiments of the present invention, a viewer observing the display may select a region of interest, specifically of a participant of interest, shown in the image, and the cylindrical surface is then "rotated" on the monitor, so that the region of interest occupies a central position in one of the hemiannuli. Alternatively, such rotation may be engendered responsive to the direction of sound received by a microphone array associated with the image capture apparatus, as described above. Preferably, an enlarged image of the region of interest is shown in a picture-in-picture (P P) window in the display. Alternatively, the region of interest within the annulus is enlarged. Further preferably, the viewer may control the tilt and zoom of the PIP window display, as well as other aspects of the image.

In the context of the present patent application and in the claims, the terms "annulus" and "annular" refer broadly to any ring-like geometrical figure, not necessarily round. In preferred embodiments of the present invention, as described above, the figure is round, corresponding to the cylindrical surface onto which the panoramic image is preferably mapped. It will be appreciated, however, that within the scope of the present invention, panoramic images may also be rendered using ring-like figures of other shapes, for example, oblong or oval. Although in the preferred embodiments described above, certain image processing functions are associated with the image capture apparatus and others with the image display, it will be understood that in other preferred embodiments, these functions may be distributed differently between image capture and display terminals. For example, in some preferred embodiments of the present invention, the image of the annular strip may be rendered by the image capture circuitry, and then transmitted to substantially any video monitor in the form of a standard video signal, rather than as a panoramic image, as described above. In some of these preferred embodiments, in which there are certain user controls associated with the video monitor, for example, for selecting a region of interest at the remote videoconference location, the image capture circuitry generates an on-screen user interface, which is transmitted and displayed as part of the standard video signal, alongside the panoramic image. A viewer of the image can use the user controls to interact with the user interface and convey instructions or responses to the remote image capture circuitry.

It will be appreciated that the principles of the present invention enable a remote viewer of a video conference to select a desired region to view and hear or, additionally or alternatively, to select a viewpoint from which to watch the conference, without the necessity of moving any of the sensor heads or altering any of the controls of the image capture apparatus. Furthermore, in order to conserve transmission bandwidth between the image capture apparatus and the remote display, the apparatus may be controlled to transmit only images generated by the sensor head that is directed toward the region selected by the viewer, or toward the participant who is speaking at a given moment. The remainder of the panoramic image is transmitted and refreshed on the display only intermittently. In other preferred embodiments of the present invention, on the other hand, the image capture apparatus conveys raw video data to the image display system, which then remaps the data to the generally cylindrical surface and generates the panoramic image. In such preferred embodiments, the image capture apparatus preferably comprises an array of stationary sensor heads, as described above. Alternatively, however, the image capture apparatus may comprise a camera having pan and, optionally, tilt and zoom functions, which are remotely-controlled from the image display system, as is known in the art. The display system controls the camera to acquire a panoramic sequence of images, which it then processes, remaps as a panoramic image, and renders to the monitor in the form of an annular strip, as described above. The display system may also perform digital zoom functions, to display enlarged images, preferably under viewer control.

In some preferred embodiments of the present invention, the image capture compresses image data for transmission to the display system, using image compression methods known in the an, for example, in accordance with the JPEG and/or MPEG standards. Preferably, the circuitry controls the sensor heads and transmits image capture parameters so as to minimize the data volume of the compressed signals and/or to optimize the quality of the images when decompressed and viewed on the display.

In another preferred embodiment, the circuitry identifies areas of the image occupied by participants in the video conference, and other areas in which only a background is seen. As long as the background remains unchanged, only the portions of the image corresponding to the participants are then transmitted to the display processor, which then receives and updates only these portions of the image that is rendered to the monitor.

Those skilled in the art will appreciate that although preferred embodiments of the present invention are described herein with reference to a complete video teleconferencing system, elements of the system, such as the panoramic image capture apparatus and the panoramic image display, may also be used independently of one another, in other teleconferencing systems and in other image capture and display applications.

It will be further appreciated that although preferred embodiments are described herein with reference to teleconferencing, the principles of the present invention may similarly be applied to create, transmit and view panoramic images for other purposes.

There is therefore provided, in accordance with a preferred embodiment of the present invention, apparatus for producing a panoramic image, including: a plurality of video sensor heads, each sensor head generating video signals corresponding to a partial image having a respective field of view, such that at least some of the fields of view include regions of substantial overlap with respective neighboring fields of view, and a union of the fields of view substantially covers a scene having an angular extent beyond the field of view of any one of the sensor heads; and a processor, which receives and combines the partial images from the sensor heads to produce a panoramic image of the scene.

Preferably, the panoramic image covers an azimuthal angle of at least 90°, more preferably at least 180°, and most preferably substantially equal to 360°. Preferably, the panoramic image has an aspect ratio of width to height of at least 2:1.

Preferably, the sensor heads are substantially stationary.

Preferably, the panoramic image is mapped onto a generally cylindrical geometrical image surface, and the processor renders the image surface to display the panoramic image in a generally elliptical form on a video screen. Further preferably, the processor is controlled so as to geometrically rotate the image surface.

Preferably, the panoramic image includes a reference image, and wherein the processor updates a portion of the reference image responsive to a video signal received from a selected one or more of the plurality of sensor heads.

In a preferred embodiment, the apparatus includes a microphone array, which receives sound from the scene and generates audio signals, and an audio processor, which processes the audio signals responsive to a direction corresponding to a selected region in the scene.

There is further provided, in accordance with a preferred embodiment of the present invention, apparatus for producing a panoramic image, including: an image capture device, which generates video signals responsive to a panoramic scene, and a processor, which receives the video signals and applies a processing operation thereto, which varies as a function of azimuthal angle, to produce a panoramic image of the scene.

Preferably, the processing operation includes an image magnification, varying as a function of the azimuthal angle, wherein the processing operation varies the magnification to adjust for differences in the distances of objects in the scene from the image capture device. Alternatively or additionally, the processing operation includes an adjustment of the image brightness and/or image contrast and/or image color balance as a function of the azimuthal angle.

There is also provided, in accordance with a preferred embodiment of the present invention, apparatus for image compression, including: at least one video sensor head, which captures an image of a scene; and a controller, which receives, digitizes and compresses the image, and which exchanges control signals with the at least one video sensor head so as to reduce the data size of the compressed image. Preferably, the controller provides timing signals to the at least one sensor head.

Further preferably, the at least one sensor head includes a plurality of sensor heads, which receive respective partial images, and the controller controls the plurality of sensor heads and combines the partial images received therefrom to produce a panoramic image.

Additionally or alternatively, the controller receives a video sensor parameter from the at least one sensor head and transmits the parameter to a remote site.

In accordance with another preferred embodiment of the present invention, there is provided apparatus for displaying a panoramic image, including: a video display; and a processor, which receives panoramic image data, geometrically maps the data onto a generally annular surface, and renders at least a portion of the surface to the display.

Preferably, the processor renders the generally annular surface in the form of two hemiannular strips, wherein the generally annular surface includes a generally cylindrical surface, and the processor renders the generally cylindrical surface to the display in a generally elliptical form. Preferably, the processor varies a point of view from which the surface is rendered.

In a preferred embodiment of the invention, the video display includes a generally annular display screen, including a back-projection screen and a video projector, which projects the image onto an inner side of the screen, so that the image is visible from the outer side thereof. There is moreover provided, in accordance with a preferred embodiment of the present invention, a method for displaying a panoramic image on a video display, including: defining an annular strip and a point of view outside a volume bounded by the strip; mapping the panoramic image onto the annular strip; and rendering an image of the strip, as seen from the point of view, to the video display.

Preferably, mapping the panoramic image onto the annular strip includes: mapping a first segment of the image, in relative proximity to the point of view, onto a near hemiannulus; mapping a second segment of the image, relatively distant from the point of view, onto a far hemiannulus; and reversing at least a portion of the first segment.

Preferably, defining the annular strip includes defining a generally cylindrical strip, and defining the point of view includes defining a point of view above an equatorial plane of the annular strip, so that the image of the strip rendered to the display is generally elliptical in shape.

Preferably, the annular strip is rotated about an axis thereof, relative to the point of view, so as to shift the image rendered of the strip as seen from the point of view, substantially without engendering mechanical motion.

There is also provided, in accordance with a preferred embodiment of the present invention, a method for displaying a panoramic image, including: providing a generally annular back-projection screen; mapping and rendering the panoramic image to the screen; projecting the image onto the screen from inside the annulus; and viewing the image from outside the annulus.

In a preferred embodiment of the invention, the methods above include: receiving audio signals from an array of microphones placed adjacent to an area shown in the panoramic image; selecting a region of interest in the image; and processing the audio signals selectively, responsive to the selected region.

In accordance with another preferred embodiment of the present invention, there is provided a method for transmitting an image to a remote site, including: generating video signals using one or more video sensor heads; receiving and compressing the video signals for transmission; and exchanging control signals with the one or more video sensor heads so as to reduce the data size of the compressed signals. Preferably, exchanging the control signals includes receiving a sensor head control parameter and transmitting the parameter to the remote site, most preferably transmitting a change in a parameter relative to an earlier value thereof.

Further preferably, generating the video signals includes generating signals using a plurality of video sensor heads, and exchanging the control signals includes selecting one or more of the plurality of sensor heads to transmit signals for use in updating a corresponding portion of a reference image, while other portions of the image corresponding to the non- selected sensor heads are substantially frozen.

There is further provided, in accordance with a preferred embodiment of the present invention, a method for producing a panoramic image of a scene, including: positioning a plurality of substantially stationary video sensor heads to capture a respective plurality of partial images of the scene, each partial image having an angular extent substantially less than a total angular extent of the scene whose image is to be produced and having a region of overlap with a neighboring partial image; receiving video partial images from the plurality of sensor heads; and combining the partial images to produce the panoramic image.

There is additionally provided, in accordance with a preferred embodiment of the present invention, a method for producing a panoramic image including: receiving video images covering a panoramic field of view; and processing the images variably as a function of azimuthal angle in the panoramic image.

There is moreover provided, in accordance with a preferred embodiment of the present invention, a system for video teleconferencing, including at least one transmitting station, which comprises: a plurality of video sensor heads, positionable at or near the center of a table around which teleconferencing participants are seated so as to capture images of the participants on opposing sides of the table; and a transmission processor, which receives and combines video signals from the sensor heads to produce and transmit a panoramic image of the scene.

Preferably, the plurality of sensor heads capture images of the participants along generally horizontal respective optical axes.

Preferably, the system also includes at least one receiving station, which includes: a video monitor; a receiving processor, which receives the transmitted panoramic image and displays the image to the monitor; and user controls, which provide user commands to both the receiving processor and the transmission processor. Preferably, the user controls of the receiving station select one or more of the plurality of sensor heads of the transmitting station, and the video display is updated using images captured by the one or more selected sensor heads.

In a preferred embodiment of the invention, the user controls of the receiving station include a menu generated by the transmitting station and displayed on the video display of the receiving station.

In another preferred embodiment, the transmission processor exchanges sensor control parameters with the plurality of video sensor heads and transmits at least some of the parameters to the receiving processor, for use in rendering the image, wherein the transmission processor compresses the image for transmission, and wherein the receiving processor decompresses the image using the transmitted parameters.

There is additionally provided, in accordance with a preferred embodiment of the present invention, apparatus for producing a stereoscopic panoramic image, including: a plurality of video sensor heads, which capture respective partial images of a scene; and a processor, which receives and combines the partial images from the sensor heads to produce a stereoscopic panoramic image of the scene.

Preferably, the plurality of video sensor heads are substantially stationary.

The present invention will be more fully understood from the following detailed description of the preferred embodiments thereof, taken together with the drawings in which:

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is a schematic illustration of a teleconferencing facility, including panoramic image acquisition and display apparatus, in accordance with a preferred embodiment of the present invention; Fig. 2 is a block diagram illustrating signal flow between elements of the facility of Fig. i;

Fig. 3 is a block diagram illustrating signal flow within the image acquisition and display apparatus of Fig. 1;

Fig. 4A is a schematic, partly sectional illustration showing a side view of panoramic image acquisition apparatus, comprising a multifaceted mirror and a plurality of sensor heads, in accordance with a preferred embodiment of the present invention;

Fig. 4B is a schematic top view of the sensor heads in the apparatus of Fig. 4A;

Fig. 5 is a schematic top view of the multifaceted mirror in the apparatus of Fig. 4 A;

Fig. 6A is a schematic, pictorial illustration showing a detail of the multifaceted mirror shown in Fig. 5;

Fig. 6B is a schematic illustration showing a head-on view of a single facet of the multifaceted mirror shown in Fig. 5;

Fig. 6C is a schematic diagram illustrating optical beam paths associated with the detail of Fig. 6 A; Fig. 6D is a schematic diagram illustrating a detail of a multifaceted mirror, in accordance with an alternative embodiment of the present invention;

Fig. 7A is a schematic illustration showing construction of a panoramic image, in accordance with a preferred embodiment of the present invention;

Fig. 7B is a graph showing an intensity transfer function characteristic of panoramic image acquisition apparatus, in accordance with a preferred embodiment of the present invention;

Fig. 8 is a schematic top view of a multifaceted mirror, in accordance with an alternative embodiment of the present invention;

Fig. 9A is a block diagram which schematically illustrates the operation of a camera control unit, for use particularly in the panoramic image acquisition apparatus of Fig. 1, in accordance with a preferred embodiment of the present invention; Fig. 9B is a block diagram which schematically illustrates details of the camera control unit of Fig. 9 A;

Fig. 10A is a block diagram which schematically illustrates the operation of a panoramic image construction unit, in accordance with a preferred embodiment of the present invention; Figs. 10B and IOC are block diagrams which schematically illustrate details of the image construction unit of Fig. 10 A;

Fig. 11 is a schematic, partly pictorial illustration useful in understanding a method of panoramic image processing, in accordance with a preferred embodiment of the present invention; Fig. 12A is a flow chart illustrating a method of image compression, in accordance with a preferred embodiment of the present invention;

Fig. 12B is a block diagram which schematically illustrates an image compression CODEC, in accordance with a preferred embodiment of the present invention;

Fig. 13 is a schematic illustration showing a panoramic image display, in accordance with a preferred embodiment of the present invention;

Fig. 14 is a schematic illustration showing a panoramic image display, in accordance with another preferred embodiment of the present invention;

Fig. 15 is a schematic illustration showing a panoramic image display, in accordance with still another preferred embodiment of the present invention; Fig. 16 is a schematic illustration showing a panoramic image display, in accordance with yet another preferred embodiment of the present invention;

Fig. 17 is a schematic, pictorial illustration showing panoramic image display apparatus, in accordance with a preferred embodiment of the present invention;

Figs. 18A-18C are schematic illustrations, each showing two teleconferencing facilities interconnected by a communication line, in accordance with preferred embodiments of the present invention; and

Fig. 19 is a schematic illustration of a stereoscopic panoramic image capture device, in accordance with a preferred embodiment of the present invention. DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Reference is now made to Fig. 1, which schematically illustrates a teleconference facility 30, based on apparatus and methods for image acquisition and display in accordance with a preferred embodiment of the present invention. Facility 30 is built around a panoramic imaging subsystem 40, including a panoramic image capture device 54, which is used to acquire a simultaneous, panoramic image of at least some, and preferably all, of participants 46 in the teleconference around a conference table 48. Device 54, which is preferably placed at or near the center of table 48, comprises a plurality of sensor heads, each of which acquires an image of a respective field of view, which images are then combined by subsystem 40 to form the panoramic image. The construction and operation of device 54 and subsystem 40 are described in detail hereinbelow.

Facility 30 also includes a display monitor 50 and a plurality of local video displays (LVDs) 78, preferably comprising flat panel displays arranged in the center of table 48, as shown. The facility optionally also includes a conventional video camera, such as a pan-tilt- zoom (PTZ) controlled camera 32, as is known in the art, which is preferably adjacent to the monitor. A microphone array 52, for acquiring voice signals from the conference participants, is coupled to subsystem 40. Other elements of the facility optionally include a personal computer (PC) 34, a mouse 38 and/or other user interface device for controlling certain functions of subsystem 40, a presentation board 42 and an overhead camera 44 (also known in the art as a document camera). A CODEC 62 receives and, preferably, compresses video and audio signals generated by subsystem 40, and transmits the signals to a remote site, as well as receiving and distributing such signals from the remote site, preferably controlled by a remote control unit 36. Such elements of facility 30 are known in the art, but certain aspects of their construction and use in conjunction with subsystem 40 differ from conventional teleconferencing systems.

Fig. 2 is a block diagram that schematically illustrates a video acquisition and display system 60, operative in the framework of facility 30. System 60 is built around acquisition subsystem 40, which is described below with reference to Fig. 3, along with CODEC 62. The CODEC may comprise any standard CODEC, known in the art, for example, a PictureTel Series 4000 CODEC, or otherwise as described in Chapter 12 of Video Demystified: A Handbook for the Digital Engineer, by Keith Jack (HighText Publications, San Diego). Alternatively, CODEC 62 may comprise a special-purpose CODEC, designed to operate with subsystem 40, as described below with reference to Figs. 12A and 12B.

The remote site may also have a panoramic imaging subsystem, like subsystem 40, or it may be based on imaging equipment known in the art. Preferably, subsystem 40 identifies whether the remote site has such a subsystem, by sending and receiving appropriate control signals via CODEC 62. The panoramic image capture and display capabilities of the subsystem, as described hereinbelow, may be realized regardless of whether the remote site has the subsystem, but certain aspects of these capabilities are enhanced when the remote site is suitably equipped. Fig. 3 is a block diagram showing details of panoramic imaging system 60 and subsystem 40. Subsystem 40 comprises a camera control unit (CCU) 74, which provides timing and control signals to image capture device (ICD) 54 and receives raw video output signals therefrom. CCU 74 preprocesses and digitizes these video signals and conveys them to an image construction unit (ICU) 64. The ICU combines the digitized video signals from the plurality of sensor heads in ICD 54 to form a single, substantially "seamless" digital panoramic image, as described in greater detail below. In the context of the present patent application, the expression "seamless" is meant to indicate that the panoramic image, which is constructed from a plurality of sub-images, as described below, appears to be substantially continuous and smooth over its entire extent, without spurious variations or artifacts at transitions between the sub-images. Preferably, ICU 64 also has RS-232 or other digital communications channels for communicating and exchanging control signals with CCU 74 and with CODEC 62, as well as with a touch screen 76 or mouse 38, for user control.

A video input/output (VIO) unit 66 receives the digital panoramic image signals from ICU 64 and conveys the signals to CODEC 62. VIO 66 also receives audio and video signals via CODEC 62 from the remote site and distributes these signals respectively to speaker 68 and to local video displays (LVD) 78, where they may be viewed by the conference participants, as well as to the display of touch screen 76. Pan/tilt/zoom (PTZ) signals may be received from the remote site, and are input to ICU 64 to control the acquisition and display of the panoramic image, as described below. VIO is preferably also coupled to receive and transmit VGA signals from PC 34 and video signals from overhead camera 44, which it conveys to ICU 64, and may include other I/O functions useful in teleconferencing, as are known in the art. A voice activation unit (VAU) 70 receives audio signals from microphone array 52 and conveys these signals to VIO 66 for transmission via CODEC 62. The VAU is further coupled by a digital data and control link to ICU 64, so as to enable the following functions:

• Voice-activated PTZ. The panoramic image produced by ICU 64 may be manipulated automatically so that the participant who is speaking is displayed in the center of the image (or in some other desired area of the image) or in an enlarged picture- in-picture (PLP) view, or both.

• Audio beam-forming. The audio signals received from the multiple microphones in array 52 may be processed and combined, using signal processing techniques known in the art, such that the audio output heard at the remote site is that which comes from a selected direction within facility 30, for example, from the direction of a participant in the teleconference who is selected or centered in the panoramic image. The beam-forming may be carried out within acquisition and display system 60 before transmission via CODEC 62. Alternatively or additionally, some or all of the multiple audio signals received from the multiple microphones may be transmitted to the remote site, and the beam-forming can be performed there, preferably under the control of a user at the remote site. Still further alternatively or additionally, the beam-forming function may similarly be used to mute sound coming from a selected direction in facility 30, so that one or more of participants 46 can speak, for example, without being heard at the remote site.

• Stereo sound. The remote site may be equipped with multiple speakers, and the audio signals from array 52 may be distributed to the speakers so as to give a stereophonic effect.

Other elements of subsystem 40 preferably include user input and control interfaces, such as touch-screen 76, pointing device 38, preferably a wireless mouse, and a "Look-At-Me" button. The "Look-At-Me" button (not shown in the figures) is a sort of acoustic pointer, which emits a special tone, preferably at a high frequency and/or short duration, substantially inaudible to humans, which is recognized by VAU 70. These interface devices are designed so that subsystem 40 can interact with a similar subsystem at the remote site or, alternatively, so that it can be used opposite and interact with other teleconferencing video systems known in the art. An internal power supply 80 provides suitable electrical power to all the elements of subsystem 40. PANORAMIC IMAGE CAPTURE DEVICE Fig. 4A is a schematic, sectional, side view of panoramic image capture device (ICD) 54, and Fig. 4B is a pictorial, sectional view of the device, as seen along line IVB-IVB in Fig. 4A, in accordance with a preferred embodiment of the present invention. ICD 54 comprises a multifaceted mirror 100, having a plurality of planar facets 102, and a plurality of sensor heads 92, preferably one for each facet of the mirror, commonly mounted on a frame 104. Each sensor head comprises a lens 96 and a detector array 98, preferably a CCD array, as is known in the art, contained in a housing 94. Although Fig. 4B shows ICD 54 comprising eight sensor heads 92, any other suitable number of sensor heads could be used. Each of the sensor heads receives an image from a respective angular field of view 110, represented in Fig. 4A by rays that converge to a respective apex 108. The effect of multifaceted mirror 100 and of the positioning of sensor heads 92 with respect thereto is such that apices 108 of all of the sensor heads are imaged to a common virtual apex 112 by respective facets 102. It will be appreciated, however, that the rays shown in Fig. 4A represent only principal rays of sensor heads 92, and the actual ray traces for ICD 54 will be substantially more complicated. As is known in the art, rays that reach CCD 98 generally pass through an entrance pupil having a substantial lateral extent, as indicated schematically by a circle 106 in the figure. The entrance pupils of all of the sensor heads are imaged to substantially overlapping virtual pupils 115. ICD 54 is preferably covered by a protective case 105, including a transparent visor section 107 corresponding to fields of view 110 of sensor heads 92. As illustrated in Fig. 4A, section 107 optionally comprises a refractive element, which functions in conjunction with lenses 96 to focus light onto CCDs 98. In this case, the visor is preferably formed so as to function as a cylindrical lens, thereby widening fields of view 110 in a vertical dimension. In addition to capturing a substantially "seamless" panoramic image, ICD 54 has a number of other advantages over teleconferencing cameras known in the art. It will be observed in Fig. 4A that the optical axes of fields of view 110 of sensor heads 92 are generally horizontal and are relatively high off table 48, near the eye level of participants 46. Because the axes are also near the top of ICD 54 and mirror 100, however, the ICD does not substantially prevent participants 46 from seeing one another across the table. Since ICD 54 has no moving parts or motors to make noise, microphone array 52 can be mounted on or in close proximity to the ICD, so that sound pickup from all sides of the table is unobstructed, and so that audio signals can be easily correlated with the video images for the purposes of beam-forming, as described hereinabove.

Fig. 5 is a schematic, bottom view of multifaceted mirror 100, as seen from the viewpoint of sensor heads 92. In this preferred embodiment, the mirror is seen to comprise eight facets 102, covering a full 360°, so that each of the sensor heads has, effectively, a 45° azimuthal field of view in the plane of Fig. 5. The outline of one of sensor heads 92 is shown schematically as a circle, superimposed on one facet 102a. It will be appreciated, however, that in other embodiments of the present invention, not shown in the figures, the multifaceted mirror may comprise more or fewer facets, and ICD 54 may be set up to cover less than the full 360°. In still other preferred embodiments, as described below, for example, with reference to Fig. 13, there may be more than one sensor head per facet of the multifaceted mirror.

Fig. 6A is a schematic, pictorial representation of a portion of mirror 100, including two neighboring facets 102a and 102b, and Fig. 6B is a schematic, head-on view of one of facets 102. As seen most clearly in Fig. 6 A, facets 102a and 102b are separated by a generally triangular baffle 120. The purpose of this baffle is to allow rays to reach CCD 98 in each of sensor heads 92 only from the sensor head's respective facet 102, and to prevent rays reflected from neighboring facets from reaching the CCD. Thus, ray BA, shown in Fig. 5, is allowed to reflect from facet 102a and enter the respective sensor head, but ray CA, reflected from facet 102b, is effectively prevented from entering the sensor head by baffle 120. Failure to block rays such as ray CA will result in "ghost images" being received by CCD 98, with a consequent deleterious effect on the quality of the panoramic images produced by ICD 54. Panoramic cameras based on pyramidal mirrors known in the art do not address this difficulty.

Fig. 6C is a schematic optical diagram, in a sectional plane through mirror 100 taken along line VIC-VIC in Fig. 6A, useful in understanding the function of baffle 120. A principle ray 114 reflects off facet 102a and reaches apex 108. A spurious ray 116 reflects off facet 102b, parallel to ray 114, and would pass through pupil 106 into the sensor head corresponding to facet 102a, if ray 116 were not blocked by baffle 120. The baffle must extend out far enough from the joint between the facets to block all such spurious rays.

Fig. 6D is a schematic diagram, from the same point of view as Fig. 6C, showing a detail of mirror 100 in accordance with an alternative embodiment of the present invention. In this case, mirror 100 includes baffles 118, at the center of each of facets 102, in addition to baffle 120 at the joint between the facets. The presence of baffle 118 on facet 102b, for example, blocks spurious rays that otherwise would pass outside the outer end of baffle 120 and reach the entrance pupil of the sensor head associated with facet 102a. Consequently, baffle 120 can be made shorter, so that mirror 100 is more compact and less prone to damage. Alternatively, baffles 118 may be used alone, without baffles 120, and/or additional baffles may be placed at other suitable locations along facets 102.

Fig. 7 A is a schematic representation of partial images 121 and 122 of a teleconference participant 123, as they are received by two of sensor heads 92 viewing the participant by way of respective, neighboring facets 102b and 102a of mirror 100. Preferably, lenses 96 are so designed and focused, and sensor heads 92 are so positioned relative to respective facets 102, that fields of view 110 of neighboring sensor heads substantially overlap in the azimuthal dimension, creating an overlap region 124. Partial images 121 and 122 are shown as having trapezoidal forms in the figure to reflect keystone distortion of the images, which results from the structure of multifaceted mirror 100, as is known in the art.

Fig. 7B is a graph representing intensity transfer functions corresponding to the partial images of Fig. 7A. The intensity transfer is shown as a function of azimuthal angle along an arbitrary horizontal line 127 passing through images 121 and 122, preferably a line corresponding to a common row of pixels in detectors 98 of the respective sensor heads 92. Whereas the intensity transfer is substantially constant, at a normalized value of 1, over most of the area of the partial images, in overlap region 124 there is vignetting, due inter alia to the finite aperture afforded by facets 120, which reduces the image intensity. Preferably, although not necessarily, mirror 100 is so designed that the sum of the intensity transfer functions for the two partial images in the overlap region, shown in the figure by a dashed line 125, is approximately equal to 1, as well. For example, referring to Fig. 6B, the edges of facet 102 adjacent to baffles 120 may be tinted to reduce their reflectivity. Referring again to Fig. 7 A, a method for constructing an effectively cylindrical panoramic image 126 will be described, in accordance with a preferred embodiment of the present invention. For each point along line 127, ICU 64 (Fig. 3) receives digitized image data from CCU 74. For the most part, only a single data value is received for each point, but in overlap region 124, two data values are received, from neighboring sensor heads 92, and are added together. If the sum of the intensity transfer functions, as described above, deviates significantly from 1, a multiplicative correction factor is preferably applied to the data values, so that the intensity of the image will vary smoothly over the overlap region. This process is continued over the entire azimuthal extent of line 127, preferably over 360°, and is carried out for all other horizontal lines in the image in succession, preferably corresponding to successive rows of pixels in detectors 98.

In conjunction with summing partial images 121 and 122 so that panoramic image 126 has a smooth transition in overlap region 124, the partial images are preferably mapped, either before or after summing, to a generally cylindrical image surface, indicated by the curved lines corresponding to image 126 in Fig. 7 A. Panoramic image 126 is thus approximately equivalent to an image of the scene that would be captured by a camera with a cylindrical detector array (if such a detector array and camera could be produced). The mapping may be achieved, at least in part, by producing lenses 96 with such barrel distortion as to offset the keystone distortion mentioned above. Alternatively or additionally, the rows of partial images 121 and 122 or of combined image 126 may be mapped onto the cylindrical surface using any suitable method of inteφolation known in the art.

Fig. 8 is a schematic top view of a multifaceted mirror 130 having curved facets 132, in accordance with another preferred embodiment of the present invention. Facets 132 are preferably convex, either spherical or aspheric, for example, comprising surfaces of revolution of conic sections or toroidal surfaces. As illustrated in Fig. 8, the facets typically have radii of curvature that are greater than the radius of a circle circumscribing mirror 130. The curved surfaces of facets 132 produce partial images in sensor heads 92 having reduced keystone distortion and more closely approximating a cylindrical image surface than partial images captured using planar mirror facets. The refractive power of the curved facets (in addition to that of lenses 96) also enables mirror 130 to be made smaller for a given sensor head geometry than mirror 100 could.

In any case, even with planar facets 102, in preferred embodiments of the present invention, the deviation of partial images 121 and 122 from the desired cylindrical surface of image 126 is generally small and is concentrated at the upper and lower edges of the images. Such deviation will, furthermore, be masked at least in part by the summation of the images in overlap region 124. Therefore, the inventors have found that even zero-order interpolation is sufficient to produce image 126 with adequate image quality and without obvious distortions. The method of the present invention, by virtue of using overlapping neighboring images and cylindrical image mapping, as described above, allows panoramic image 126 to be constructed rapidly and at low computational cost, relative to methods known in the art for correcting and combining contiguous partial images, as described in the above-mentioned U.S. patents 5,187,571 and 5,444,478, for example.

Additionally, in a preferred embodiment of the present invention, mapping of partial images 121 and 122 is further adjusted to compensate for the varying distances of the participants in the teleconference from ICD 54. As shown in Fig. 1, seats 46 are positioned at different distances from ICD 54 at the center of table 48, so that without image correction, the participants will appear in panoramic image 126 to be of different sizes. Therefore, a mapping function having a variable magnification as a function of azimuthal angle is preferably applied to partial images 121 and 122, so that the figures of all the participants are of approximately equal size in the panoramic image.

As noted above with reference to Fig. 3, ICU 64 and CCU 74 preferably perform additional camera control and image processing functions, so as to enhance the quality of panoramic image 126 and ensure a smooth transition and an absence of "seams" between the parts of image 126 corresponding to different partial images 121 and 122. CCU 74 provides timing signals to sensor heads 92, including mutually-synchronized pixel clocks, horizontal and vertical blanking signals and electronic shutter signals. Digitization of the video outputs of the sensor heads is preferably also synchronized to the pixel clocks.

Fig. 9A is a block diagram showing details of CCU 74, in accordance with a preferred embodiment of the present invention. CCU 74 comprises a plurality of input stages 133, preferably one stage for each of sensor heads 92 in ICD 54. A clock generator 134 and a synchronization signal generator 135 generate a common pixel clock and horizontal and vertical blanking signals for sensor heads 92 and for input stages 133. A CPU 136, which may comprise any suitable microcontroller known in the art, controls the operation of the input stages, for example, varying the gain and offset settings for each of the sensor heads, and, optionally, communicates with ICU 64 and with other elements of subsystem 40.

Fig. 9B is a block diagram showing details of one of input stages 133. Each of input stages 133 comprises a correlated-double-sampler (CDS) 137, as is known in the art, which receives a raw video signal from the respective sensor head 92 and provides a sampled-and-held output to an A/D converter 138. The digital output of converter 138 is preferably corrected, pixel by pixel, by multiplication in a multiplier 141 with a correction factor, which is stored in a memory 139. The corrected, digitized signal is then processed by a video processor 143, preferably to generate a 24-bit digital RGB video signal. All the elements of input stage 133 are commonly timed by inputs from clock generator 134 and sync generator 135. Gain, offset and correction factor data are preferably received from CPU 136, as described above.

Video processor 143 preferably comprises an iSP 2000 processor chip, manufactured by i Sight Ltd., of Tirat Hacarmel, Israel. Alternatively, other video processors known in the art may be used. Furthermore, although in the embodiment shown in Fig. 9B, the raw video signals are first digitized and then processed, other video processing configurations known in the art may also be used. For example, the video signals may be processed using analog components and methods, and only afterwards digitized, and/or the output of CCU 74 may comprise a Y/C or other type of video signal. Fig. 10A is a block diagram which schematically illustrates the functions of ICU 64, in accordance with a preferred embodiment of the present invention. ICU 64 receives multiple digital video inputs from CCU 74, preferably eight such inputs (labeled CAMO through CAJVI7), each of the inputs 24 bits wide, corresponding to the eight sensor heads 92 shown in Fig. 4B. These inputs are combined by a panoramic image composer 140 to form the panoramic image, and a close-up image composer 142 to form a close-up image of a region of interest (ROI), as described further hereinbelow. ICU 64 also includes an external video input 146, which receives video signals from VIO 66, for example, captured by camera 44, as well as timing control and synchronization signals. A background color unit 148 is used to fill in areas of the picture that are not occupied by video images. The panoramic and close-up images formed by composers 140 and 142, respectively, together with the external video input and background color are combined by a multiplexer (MUX) 144 and are output via an overlay generator 152 to VIO 66. A source selector 150 determines, for each pixel of the output image, which of the inputs to MUX 144 is to be selected.

Fig. 10B is a block diagram showing details of panoramic image composer 140, in accordance with a preferred embodiment of the present invention. Each field of digital video data received from each of the sensor head inputs from CCU 74 is stored serially in a respective memory 160. The data are then read out in proper order to a FIFO memory 162, under the control of read/write logic 161, so as to form a single, substantially continuous panoramic strip. Preferably, only one of the two video fields in each frame that is output by sensor heads 92 is stored in memory 160, for example, the odd field. In this case, the data can then be read out if memory 160 during the even field period, without the need for double buffering. The reduced vertical resolution due to operating in such a field mode, rather than full frame, is in accordance with well-known standards of video teleconferencing, such as the H.261 Common Image Format.

Logic 161 includes a RAM select FIFO 172, which indicates which of memories 160 is to be read out, and read FIFO's 168 and 170, which generate high and low address bytes from reading from the memories. A strip rotation FIFO 164 controls a viewing angle from which the panoramic image is to be rendered, as described in greater detail below. PAL's 166 and 174 receive data from FIFO's 164 and 172, respectively, and generate the control inputs to memory 160 that determine the order of memory readout to produce the desired viewing angle.

If desired, a portion of the panoramic image generated by composer 140 may be suppressed, preferably by interrupting the normal writing of camera data to an appropriate address range of memories 160. The suppressed portion of the image may be dimmed, blacked out or frozen. Such a function may be invoked, for example, if one or more of participants 46 do not wish to be seen in the image transmitted to the remote site. In this case, the dimmed or frozen portion of the image, or a blank image, is read out of the address range into FIFO 162 in place of a live image. Alternatively, a video loop corresponding to the portion of the image to be blacked out may be stored and then replayed into memories 160, in order to give the appearance of a live image.

Fig. 10C is a block diagram showing details of close-up composer 142, in accordance with a preferred embodiment of the present invention. The sensor head inputs from CCU 74 are multiplexed by multiplexers 180 in odd and even groups, so that the digital data from any one of sensor heads 92 is channeled separately from that of its immediate neighbors. A programmable delay 182 is applied to the data from each of the sensor heads, based on preprogrammed instructions stored in a ROM 184. The delay, which is preferably controlled to an accuracy of 1/4 pixel, provides linear interpolation of the input partial images to correct for image distortion. The data are then stored by FIFO's 186. Readout from the FIFO's is selected according to the view angle and boundaries of the close-up image to be generated, based on preprogrammed and/or user-controlled inputs.

The partial images, output from FIFO's 186, are merged by a programmable multiplier/adder 188, which blends neighboring partial images at their edges as illustrated in Figs. 7A and 7B. The blending is performed according to an interpolation function given generally by: Y = a - Xι + (\ - a) - X₂ (1)

wherein Y represents an output pixel value, X and X2 are the input pixel values, and a is a parameter input for each pixel from a ROM 190. Preferably, a varies between 0 and 1 over a range about 64 pixels wide, representing an area of overlap between two adjacent partial images 121 and 122, out of a typical total width of 768 pixels in each partial image. Alternatively, a may have a value greater than or equal to one for some or all of the pixels over the overlap area when the intensities of the partial images are attenuated optically by mirror facets 102, as described above. Although in the embodiment of Figs. 10A-10C, only close-up composer 142 includes the image blending function of multiplier/adder 188, it will be appreciated that a similar circuit could be used to blend the edges of the partial images in the panoramic image produced by panoramic composer 140. Due to the lower resolution of the panoramic image, however, as well as the optical blending features of mirror 100, as described above, such image blending is generally not required in the panoramic composer.

The merged image from block 188 is input to deinterlacing FIFO's 192 and from there to double-buffered line stack mirror FIFO's 194. FIFO's 192 convert each two fields of interlaced image data into a single, non-interlaced frame for subsequent processing. FIFO's 194 reverse each line of video, to compensate for the left-right reversal introduced by mirror 100. A zoom unit 196, preferably based on a GM833x3, produced by Genesis Microchip ltd. of Markham, Ontario, controls the readout from FIFO's 192 and 194 and performs interpolation operations on the non-interlaced data to provide digital zoom in the close-up image. The output of unit 196 is received by a FIFO 198, which re-interlaces the image and outputs it to MUX 144. Although ICU 64 as shown in Figs. 10A-10C and described above produces both panoramic and close-up images, and combines them together in a single video output image with external video and background color, other preferred embodiments of the present invention need not include all of these elements. For example, close-up composer 142 may be eliminated, and ICU 64 can be made to transmit only a panoramic image. If a remote site receiving the panoramic image is suitably equipped, a close-up image can be produced at the remote site by cropping a region of interest from the panoramic image and digitally zooming in on the region, as is known in the art. Alternatively, the partial images output by CCU 74 may be transmitted to the remote site, and the panoramic and close-up images may then be composed by a unit similar to ICU 64 at the remote site. Those skilled in the art will appreciate other configurational variations that may be implemented, based on the principles of the present invention. Fig. 11 is a schematic illustration useful in understanding the functions of video processing circuitry in CCU 74, in accordance with a preferred embodiment of the present invention. CCU 74 adjusts and controls the brightness, contrast, and other aspects of the video signals in the partial images. Such adjustment is preferably carried out before digitization of the signals, for example, by adjusting the gain of preamplifiers that amplify the video signals before digitization.

In the example shown in Fig. 11, the lighting in teleconferencing facility 30 is not uniform, including brighter areas near light sources 226 and dimmer areas 227. Signal processing circuitry 228 in CCU 74 determines a brightness function/, dependent on azimuthal angle φ, illustrated by a graph 225 shown in the figure. Circuitry 228 then sets the gain and/or other aspects of the video signal processing to vary as a function of azimuthal angle, normally with a higher gain in more dimly-lit areas of the image, i.e., generally inversely responsive to _ (φ). Rather than setting the gain (or other parameters) to a single fixed value for all of sensor heads 92, or to a fixed, individual value for each of the sensor heads, the gain is set to vary substantially smoothly, so that the brightness of the participants appears to be generally uniform over all of the panoramic image 126 Other image parameters, such as contrast, may similarly be controlled as a function of azimuth.

These functions of CCU 74 are preferably controlled by ICU 64, responsive to characteristics of the image and to changes therein. Additionally or alternatively, ICU 64 preferably performs image processing and enhancement functions so as to enhance the quality of the panoramic image. In a preferred embodiment of the present invention, ICU 64 controls CCU 74 and sensor heads 92 and processes image data received therefrom to enhance the dynamic range of the panoramic image, using methods of Adaptive Sensitivity, as described in U.S. Patents 5,144,442 and 5,247,366, which are incorporated herein by reference. In another preferred embodiment, ICU 64 processes the image to increase the apparent image resolution, as described, for example, in an article by I. Vitsnudel and Y.Y. Zeevi, entitled "Neural Network- Aided Design for Image Processing," SPIE Proc. (1991), which is incorporated herein by reference, or as is otherwise known in the art. It will be clear to those skilled in the art that the video processing functions of CCU and ICU described above may be carried out in a wide variety of ways, using readily-available analog and digital electronic components and signal processing methods. Furthermore, although in the preferred embodiments described above, some of the image processing, enhancement and 5 control functions have been associated with specific units within panoramic imaging subsystem 40, it will be appreciated that these functions could similarly be performed by other units in the subsystem.

Fig. 12A is a flow chart showing a method for content-responsive compression of video images for transmission by CODEC 62, in accordance with a preferred embodiment of the 0 present invention. Such compression is preferably compliant with video compression standards known in the art, such as JPEG and MPEG compression. Preferably, ICU 64 comprises a processor which compresses the digital video signals conveyed from ICU 64 to VIO 66, for transmission via the CODEC, which may then comprise a standard CODEC, known in the art. Alternatively, the CODEC may comprise a special-purpose device, adapted to work with 5 subsystem 40, whereby the compression is performed by the CODEC, with appropriate feedback to subsystem 40.

Preferably, the entire panoramic image 126 is transmitted periodically, in compressed form, as a base or reference image. To begin the compression process, panoramic image 126 is constructed, as described above. In order to optimize the efficiency of compression, i.e., to 0 reduce the volume of compressed data that must be transmitted, ICU 64 (or CODEC 62) preferably identifies areas within the image that include visual information of importance and other areas that contain only static image background. The ICU then orders CCU 74 to adjust timing signals that it provides to ICD 54 and video processing parameters applied to the signals therefrom, so as to optimize the signals for compression. The base image is then compressed 5 and transmitted.

For subsequent video frames, only certain portions of the image are transmitted and are used to update the panoramic image displayed at the remote site. Specifically, ICU 64 preferably acquires and updates only a region of interest (ROI) within the panoramic image. Preferably, the ROI comprises one or two partial images, indicated by VAU 70 as including the

-.o participant who is speaking. Alternatively, the ROI may be selected by a user at the remote site. Furthermore, within the ROI, preferably only the figure of the participant of interest is updated, while areas of background are identified, using image analysis methods known in the art, and are left unchanged.

From time to time, for example, once per second, the entire panoramic base image is updated by ICU 64 and is compressed and retransmitted, as described above. In between these updates, the ICU continues to transmit ROI image data. If the ROI changes, as indicated by VAU 70 or by user input, the new ROI is identified and becomes the basis for the frame-to- frame image updates.

Fig. 12B is a block diagram which schematically illustrates image content-responsive compression functions performed by CODEC 62, in accordance with a preferred embodiment of the present invention. Each time a new image is received from VIO 66, the image is compared with an updated preceding image, stored in memory 212, by a motion estimator 214. The estimated motion is used in controlling subsequent image compression steps. The preceding image is used to produce an image background fill 220, which is subtracted from the new image by an image subtractor 216, dependent on the estimated image motion. An image compressor 218 then compresses the subtracted image for transmission over the communication line.

The compressed output preferably carries information relating to image motion, typically motion vectors, and relating to camera parameters received from CCU 74, for use in reconstructing the image at the remote viewing site. Such camera parameters preferably include gain factors, electronic shutter speeds and color balance parameters, which are applied by CCU 74 in producing the video images. When a variation in image conditions causes any of the camera parameters to change, it is enough for CODEC 62 to transmit data corresponding to the change in the camera parameters. As long as a remote site receiving the data is similarly equipped with a CODEC capable of receiving and interpreting the camera parameter information, the image can be updated at the remote site based on the parameters, and without the need for transmitting the entire image as would by required by systems known in the art.

For each new image frame, a decompressor 222 reconstructs the complete preceding image based on the transmitted, compressed image. This preceding image is input to memory 212, as described above, and to a background extractor 223, which identifies the areas of the image that comprise only static background. The identified areas are processed by a background adapter 224, responsive to the camera parameters, and the resultant background image is input to background fill 220, for subtraction from the new image as described above. The compressed output of compressor 218 need not include any information regarding unchanged background areas of the image. As a result, substantially more efficient image compression is achieved, while the transmission of camera and motion parameters can be used to enhance image reconstruction at the remote site. In a further preferred embodiment of the present invention, not shown in the figures,

CODEC 62 also produces timing and control signals that are input to CCU 74 and/or to other elements of subsystem 40. For example, in order to reduce transmission bandwidth, the CODEC may command the CCU to convey video signals from only a single one or a few of sensor heads 92, while partial images produced by the other sensor heads are effectively frozen. However, when the CODEC is preparing to transmit an update of the full panoramic image frame, it will command the CCU to unfreeze and convey for transmission images from all of the sensor heads.

PANORAMIC IMAGE DISPLAY AND CONTROL Fig. 13 is a schematic illustration showing a panoramic teleconference image 231, as displayed on monitor 50, in accordance with a preferred embodiment of the present invention. Preferably, image 231 is acquired using an image acquisition subsystem at the remote site that is similar to subsystem 40, so that image 231 has the characteristics of panoramic image 226. Alternatively, however, the methods of displaying panoramic images represented by Fig. 13 and subsequent figures and described hereinbelow may be applied to other types of panoramic images and methods of panoramic image acquisition known in the art. Alternative apparatus and methods for displaying the panoramic image are described below with reference to Fig. 17.

As described above, panoramic image 231 is preferably mapped onto a substantially cylindrical image surface. This image surface is rendered to monitor 50 in the form of an annular strip 230, created by viewing the image surface from a virtual point of view outside the volume contained within the cylindrical surface and, preferably, substantially above its equatorial plane. As a result, both a front hemiannulus 239 and a rear hemiannulus 237 of strip 230 are visible in the rendered image. As shown in Fig. 13, when seen from the virtual view point, strip 230 assumes a generally elliptical form. Rendering of the panoramic image to annular strip 230 is preferably performed by an image construction unit such as ICU 64 at the remote site. Alternatively the image may be so rendered by ICU 64 at facility 30, and transmitted by CODEC 62 in this form. Annular strip 230 has an inner surface 235 and an outer surface 233. Since image 231 is acquired from a point of view at the center of the strip, preferably as described above, the image before rendition is mapped only onto inner surface 235. In order for the image to be visible when rendered onto front hemiannulus 239, the inner surface of the hemiannulus is first mapped onto outer surface 233 thereof. Preferably, figures on hemiannulus 239, such as figures of participants 232 and 234, and other direction-sensitive image elements, such as writing 241, are automatically identified and are each left-right reversed in the course of mapping surface 235 to surface 233, using image processing methods known in the art. Otherwise, these figures and other elements would appear backwards to a viewer of monitor 50. Figures 236 on hemiannulus 237 are shown without such reversal.

Although Fig. 13 shows a generally cylindrical image surface mapped to strip 230, it will be appreciated that the principles described above may be used to map and render images having other geometries. For example, image 231 may be mapped onto an annulus having an oblong or otherwise irregular form, corresponding to differing distances of participant figures 232, 234 and 236 from the center of the conference table. _ The cylindrical form of strip 230, taken together with the method of generating cylindrical panoramic image 226, has an advantage over other geometries, however, in that the strip can be rotated by any desired angle, as described below, without the necessity of geometrically remapping the image before it is rendered. As shown in the figure, all of the participants are shown on strip 230 at approximately the same size. They will also preferably be displayed with generally the same levels of brightness, color balance and other image parameters, based on the methods of angle- dependent image processing described above.

Another major advantage of the method of display illustrated by Fig. 13, by comparison with panoramic displays known in the art, is that all of the participants are shown together in the image in positions that are intuitively representative of their actual positions at the remote teleconference site. It will be appreciated that this type of panoramic display does not require a special, wide-aspect monitor screen, as is frequently used in displaying panoramic images, and is equally suited for different monitor sizes and types.

Preferably, a user viewing monitor 50 can control various aspects of the rendition, for example, the virtual point of view from which image 231 is rendered. Most preferably, the user can rotate strip 230 about its central axis, so as to alter which one of participant figures 232, 234 and 236 is seen at the center of the monitor. Such rotation is equivalent to panning a moving camera placed at the center of table 48. Unlike systems known in the art, however, the principles of the present invention allow the rotation to be accomplished substantially without engendering any mechanical motion. Image rotation is accomplished by giving a suitable "rotate" command, or, preferably, by selecting a region to be centered, whereupon strip 230 is rotated to the appropriate orientation. Additionally or alternatively, strip 230 may rotate automatically, responsive to which of the participants at the remote site is speaking, as indicated by VAU 70 (Fig. 3). If image 231 has been acquired and constructed at the remote site so as to have a substantially cylindrical form to begin with, like image 226, described above, then the image rotation can be accomplished without substantial geometrical remapping. Fig. 14 schematically illustrates an alternative preferred embodiment of the present invention, wherein monitor 50 displays both panoramic image 231 and an enlarged picture-in- picture (PLP) image 240 of participant 232. PIP image 240 provides a close-up view of a region of interest, for example, the region that includes the participant making a presentation in this case. Additional PLP images of other participants, or of other images of interest, such as of documents viewed by overhead camera 44 or a graphic display generated by PC 34 (both shown in Fig. 1), may also be superimposed on panoramic image 231 if desired. The region to be shown in PLP image 240 may be selected as described above, i.e., by user selection or responsive to VAU 70. Preferably, the selected region is marked by a border 242 and/or by a cursor 244, and/or has its brightness increased relative to other areas of the image. Alternatively or additionally, the user may "cut and paste" a portion of panoramic image 231 to form PLP image 240 and may further control the magnification, i.e., the zoom, and the tilt angle (elevation) at which the PLP image is rendered.

Monitor 50 may also include additional PIP images, including, for example, an image 246 of a document captured by camera 44, or an image of another participant, such as the preceding speaker. PIP images 240 and/or 246 may display local participants, too, in addition to or instead of participants at the remote site. Furthermore, system 60 may be programmed so that after a given participant has been speaking for a predetermined period of time, between 10 and 30 sec, for example, image 240 or 246 switches to another participant, such as the preceding speaker. Alternatively or additionally, the system may be programmed under certain conditions to automatically change its display mode and/or switch to display other participants. Fig. 15 schematically illustrates still another preferred embodiment of the present invention, wherein monitor 50 displays panoramic image 231 with a portion 248 of the image enlarged corresponding to the region of interest. Selection of the ROI and rotation of strip 230 are preferably performed substantially as described above. In the embodiment of Fig. 15, however, portion 248 of strip 230 is intentionally distorted, so that the ROI is shown in its proper place within the strip, rather than in a PLP window. Fig. 16 is a schematic illustration of monitor 50 showing an alternative method of displaying panoramic image 231, in accordance with another embodiment of the present invention. In this case, the generally cylindrical image surface is "unrolled," and is rendered in the form of a long strip 250, showing teleconference participants 252 and 254. A selected participant 252 is shown at the center of strip 250 and is also displayed in a PLP window 256. In other respects, this embodiment is similar to that shown in Fig. 14 and described with reference thereto.

Fig. 17 is a schematic, pictorial illustration showing a generally circular panoramic image display 257, in accordance with a preferred embodiment of the present invention. Display 257 is particularly suited for displaying images in the form of annular strip 230, constructed as described above, but it may also be adapted for displaying other types of panoramic images. In facility 30, shown in Fig. 1, display 257 would preferably be placed at the center of table 48, adjacent ICD 54, where the display could conveniently be viewed by all the participants. Although display 257 is shown in Fig. 17 as having a spherical, "crystal ball" shape, it will be understood that other shapes could equally be used, for example, a cylindrical shape.

Display 257 comprises a video projector 258, as is known in the art, which receives video signals from VIO 66 corresponding to strip 230, suitably geometrically transformed and rendered, and projects a corresponding video image upwards toward facets 259 of a multifaceted mirror 255. Facets 259 reflect the image toward an annular back-projection screen 261, whereupon the image of strip 230 is seen on the outside of the screen and extends all around the outside of display 257. Preferably, facets 259 are only partially reflecting, so that a viewer on one side of display 257 will see both sides of the panoramic image, by viewing the far side of the image through the partially-reflecting facets. It will be appreciated that many of the image control functions described above, such as rotation of the panoramic image responsive to selection of a region of interest, can similarly be performed using display 257.

Figs. 18A and 18B are schematic illustrations showing two different teleconferencing configurations in which image capture subsystem 40 and the above-described methods of image display may be used, in accordance with preferred embodiments of the present invention. In Fig. 18 A, two teleconferencing facilities 30a and 30b, both including a respective subsystem 40a and 40b, like subsystem 40, and similar in other respects to facility 30, shown in Fig. 1 and described above, are connected via a communications link 260, which may be of any suitable type known in the art. In Fig. 18B, on the other hand, facility 30 is connected via link 260 to a conventional teleconference facility 262, including a controller 264, a monitor 266 and a camera 268, preferably with pan-tilt-zoom (PTZ) control, as is known in the art. Facility 262 could be equipped, for example, with a video conferencing system produced by PictureTel Corporation, mentioned above, such as the Concorde 4500, Venue 2000 or System 4000ZX. Although there are certain aspects of the present invention that are realized only in the configuration of Fig. 18 A, both configurations allow panoramic images, such as image 231, as described above, to be displayed and controlled at both of the teleconferencing facilities that are involved.

In the configuration of Fig. 18 A, subsystems 40a and 40b include respective panoramic image capture devices (ICD) 54a and 54b and are coupled to respective monitors 50a and 50b. Thus, at both ends of the teleconference, panoramic images are captured, as described above, and are conveyed over link 260. Preferably, images captured by ICD 54a are conveyed to subsystem 40b in the form of individual partial images, such as images 121 and 122 shown in Fig. 7 A, as they are captured by individual sensor heads 92 (Fig. 4A) of the ICD. The partial images are processed and assembled by system 40b into a panoramic image, which is displayed on monitor 50b, preferably as shown in Fig. 13, 14, 15 or 16. Participants in facility 30b can directly control various aspects of the panoramic image display on monitor 50b, using image processing functions provided by subsystem 40b.

Preferably, as described above, not all of the partial images are transmitted over link 260 for each video frame. Rather, the sensor head or sensor heads whose partial images are to be transmitted are determined responsive to selection of a ROI by a user, either at the sending end (facility 30a) or the receiving end (facility 30b), or are selected automatically, based on the direction of audio signals received by subsystem 40a, for example. Further preferably, only frame-to-frame changes in the selected partial images are transmitted. The configuration of Fig. 18B also allows panoramic images, as shown in Figs. 13-16, to be viewed at both ends of the teleconference, although in this case the image processing functions are substantially all performed by subsystem 40. To acquire and display a panoramic image of the participants in facility 262 on monitor 50, subsystem 40 sends a command to controller 264, instructing camera 268 to pan, as indicated by an arrow 270. Subsystem 40 receives multiple images captured by camera 268 at various pan angles, and assembles the images into a single panoramic image, such as image 231. Subsequently, camera 268 is aimed automatically or under user control, preferably to capture images of whichever participant or participants in facility 262 are speaking. Subsystem 40 receives these images and uses them to update the panoramic image displayed on monitor 50.

To display a panoramic image of facility 30 on monitor 266, subsystem 40 acquires the image and renders it to generate a picture as shown in Fig. 13, 14, 15 or 16, in a standard video format, for example, NTSC or VGA. This picture is conveyed over link 260 to facility 262 and displayed on monitor 266 substantially without additional processing at the receiving end. Preferably, subsystem 40 also generates a graphic user interface, including on-screen menus and mouse-responsive controls, for example, as are known in the art, which is superimposed on the picture conveyed to facility 262. When a user in facility 262 interacts with the interface, the interaction invokes a predetermined command message, which is generated by processor 264 and conveyed over link 260 to subsystem 40. Thus, for example, if the user selects a particular region in the picture, for example, by pointing at the region using a mouse, arrow keys or other pointing device, this action will cause processor 264 to send a corresponding message over link 260. The message will be interpreted by subsystem 40 as a command to rotate the panoramic image, as described above, so that the selected region is centered in the picture.

Users at facility 262 may also choose to view individual partial images captured by sensor heads 92, rather than viewing the full panoramic image of facility 30.

Fig. 18C shows still another teleconferencing configuration, in accordance with an

» alternative embodiment of the present invention, in which facility 262 is in communication via link 260 with another facility 280. Neither of the facilities is equipped with an image acquisition device like ICD 54, as described above, but both have the capability of acquiring panoramic images, using methods known in the art. In the case of facility 262, the panoramic image is acquired by panning camera 268, as described above. Facility 280 may include a panoramic imaging camera 282, for example, including a fisheye lens or other suitable optics known in the art. Facility 280 also includes a processor 284 and monitor 286, as are known in the art. In the configuration of Fig. 18C, the principles of the present invention are applied in programming processors 264 and/or 284. The processors map the images captured by cameras 268 and/or 282 onto an annular strip, preferably a cylindrical strip corresponding to a 360° azimuthal view, as described above with reference to Fig. 13. In the case of camera 282, geometrical distortions due to the optics must also be corrected. The processors then render the mapped images to monitors 266 and 286, preferably in the form shown in Fig. 13, 14, 15 or 16.

STEREOSCOPIC PANORAMIC IMAGING Fig. 19 is a schematic illustration of a panoramic, stereoscopic image capture device (ICD) 290, in accordance with another preferred embodiment of the present invention. ICD 290 is based on multifaceted mirror 100, substantially as described above, comprising multiple facets 102. For each facet, there is a pair of sensor heads, comprising a left sensor head 292 and a right sensor head 294, which capture partial images having a substantially common field of view, but seen from mutually offset points of view. The points of view of the left and right sensor heads are offset by a distance D such that if the two partial images for any pair of sensor heads are input to a suitable stereoscopic display system, as is known in the art, the partial images will fuse to give a pseudo-three-dimensional image. Preferably, all of left sensor heads 292 share a common left entrance pupil, and all of right sensor heads 294 share a common right entrance pupil, as described above with reference to Fig. 4A. Other placements of the camera heads may also be used. For example, a beamsplitter opposite each of facets 102 may be used to split the image between the left and right sensor heads. Moreover, in some cases, the sensor heads corresponding to different ones of facets 102 will be positioned differently, so as to give a wide-field panoramic image when seen from a particular point of view. The respective pairs of partial images captured by the pairs of sensor heads 292 and 294 are input to an image capture subsystem, like subsystem 40 described above. The subsystem combines respective partial images captured by the plurality of left sensor heads 292 to form a left panoramic image, and similarly, combines the partial images from right sensor heads 294 to form a right panoramic image. The left and right panoramic images are then displayed, using any suitable stereoscopic image display system known in the art, whereby a pseudo-three- dimensional panoramic image is seen. Preferably, the left and right panoramic images are mapped and rendered to the display in a form similar to that shown in Fig. 13, 14, 15 or 16 above. In this case, front hemiannulus 239 and rear hemiannulus 237 are preferably stereo scopically offset, to give a suitable illusion of depth.

Although preferred embodiments are shown and described above with reference to video teleconferencing systems, it will be appreciated that the principles of the present invention may similarly be applied to panoramic image capture and display systems for other applications. While the preferred embodiments relate generally to overall systems, including various elements working in cooperation, it will be clear to those skilled in the art that many of these elements and combinations thereof may be used by themselves or in other systems, and that such use is within the scope of the present invention. It will further be appreciated that the preferred embodiments described above are cited by way of example, and the full scope of the invention is limited only by the claims.

Claims

1. Apparatus for producing a panoramic image, comprising: a plurality of video sensor heads, each sensor head generating video signals corresponding to a partial image having a respective field of view, such that at least some of the fields of view include regions of substantial overlap with respective neighboring fields of view, and a union of the fields of view substantially covers a scene having an angular extent beyond the field of view of any one of the sensor heads; and a processor, which receives and combines the partial images from the sensor heads to produce a panoramic image of the scene.

2. Apparatus according to claim 1, wherein the panoramic image covers an azimuthal angle of at least 90┬░.

3. Apparatus according to claim 2, wherein the panoramic image covers an azimuthal angle of at least 180┬░.

4. Apparatus according to claim 3, wherein the panoramic image covers an azimuthal angle substantially equal to 360┬░.

5. Apparatus according to any of the preceding claims, wherein the panoramic image has an aspect ratio of width to height of at least 2: 1.

6. Apparatus according to any of the preceding claims, wherein the sensor heads are substantially stationary.

7. Apparatus according to any of the preceding claims, wherein each of the plurality of sensor heads has a respective entrance pupil, and wherein the sensor heads are so arranged that the centers of the entrance pupils of at least some of the plurality of sensor heads substantially coincide.

8. Apparatus according to any of the preceding claims, and comprising beam-combining optics, which comprise a plurality of mutually-angled reflective surfaces.

9. Apparatus according to claim 8, wherein the beam combining optics comprise a multifaceted mirror, formed by the plurality of mutually-angled reflective surfaces.

10. Apparatus according to claim 8 or 9, wherein the plurality of mutually-angled surfaces comprise planar surfaces.

11. Apparatus according to claim 8 or 9, wherein the plurality of mutually-angled surfaces comprise curved surfaces.

12. Apparatus according to any of claims 8-11, wherein the optics comprise one or more baffles for blocking undesired reflections from the plurality of reflective surfaces.

13. Apparatus according to any of claims 8-12, wherein the optics introduce vignetting in the regions of substantial overlap.

14. Apparatus according to claim 13, wherein the optics are characterized by an intensity transfer function with respect to each of the fields of view, such that for each point in at least one of the regions of substantial overlap, the sum of the intensity transfer functions with respect to the two fields of view that overlap there is approximately equal to the value of one of the functions at a point in the center of one of the fields of view.

15. Apparatus according to any of claims 8-14, wherein the optics comprise a generally cylindrical refractive element which focuses light onto the plurality of reflective surfaces.

16. Apparatus according to claim 15, wherein the apparatus comprises a visor generally surrounding the plurality of video sensor heads, and wherein the visor comprises the generally cylindrical refractive element.

17. Apparatus according to any of claims 8-16, wherein the plurality of sensor heads comprises a plurality of pairs of sensor heads, each pair associated with a common, respective one of the plurality of respective surfaces, and wherein the panoramic image comprises a stereoscopic image.

18. Apparatus according to any of the preceding claims, wherein the panoramic image is mapped onto a generally cylindrical geometrical image surface.

19. Apparatus according to claim 18, wherein the processor renders the image surface to display the panoramic image in a generally elliptical form on a video screen.

20. Apparatus according to claim 19, wherein the processor is controlled to alter a point of view from which the image surface is rendered.

21. Apparatus according to claim 19 or 20, wherein the processor is controlled so as to geometrically rotate the image surface.

22. Apparatus according to any of claims 19-21, and comprising a user input device, wherein the processor renders the image responsive to an input from the user input device.

23. Apparatus according to claim 22, wherein the user input device comprises a pointing device, for indicating a region of interest in the image.

24. Apparatus according to any of the preceding claims, wherein the processor applies timing signals to the sensor heads so as to commonly synchronize at least some of the plurality of sensor heads.

25. Apparatus according to claim 24, wherein the processor compresses the panoramic image, and wherein the processor controls the timing signals so as to enhance the image compression.

26. Apparatus according to any of the preceding claims, wherein in combining the partial images, the processor remaps the images according to a magnification that varies as a function of azimuthal angle.

27. Apparatus according to claim 26, wherein the processor remaps the images so as to correct for perspective variations in the images.

28. Apparatus according to claim 27, wherein the processor remaps the images so that every one of a group of figures in the panoramic image appears to be at approximately the same distance from the apparatus capturing the image.

29. Apparatus according to any of the preceding claims, wherein the processor suppresses a portion of the panoramic image.

30. Apparatus according to any of the preceding claims, wherein the processor variably controls a parameter of the panoramic image as a function of azimuthal angle.

31. Apparatus according to claim 30, wherein the processor applies an azimuthally- variable gain to the video signals.

32. Apparatus according to claim 30 or 31, wherein the processor adjusts the brightness of the panoramic image by a factor that varies as a function of azimuthal angle.

33. Apparatus according to any of claims 30-32, wherein the processor adjusts the contrast of the panoramic image by applying a correction to the image that varies as a function of azimuthal angle.

34. Apparatus according to any of claims 30-33, wherein the processor corrects the video signals so that the panoramic image has a substantially uniform color balance, independent of azimuthal angle.

35. Apparatus according to any of claims 30-34, wherein the processor applies an image processing operation to the video signals, as a function of azimuthal angle, so that the panoramic image has a substantially enhanced dynamic range.

36. Apparatus according to any of the preceding claims, wherein the panoramic image comprises a reference image, and wherein the processor updates a portion of the reference image responsive to a video signal received from a selected one or more of the plurality of sensor heads.

37. Apparatus according to any of the preceding claims, and comprising a microphone array, which receives sound from the scene and generates audio signals, and an audio processor, which processes the audio signals responsive to a direction corresponding to a selected region in the scene.

38. Apparatus according to claim 37, wherein a viewer of the panoramic image selects the region by indicating a point in the panoramic image.

39. Apparatus according to claim 37 or 38, wherein the audio signals are processed to enhance the sound received from the direction.

40. Apparatus according to claim 37 or 38, wherein the audio signals are processed to mute the sound reaching the array from the direction.

41. Apparatus according to any of claims 37-40, wherein the processor receives the audio signals from the microphone array and selects one or more of the plurality of sensor heads to select based on the audio signals.

42. Apparatus for producing a panoramic image, comprising: an image capture device, which generates video signals responsive to a panoramic scene; and a processor, which receives the video signals and applies a processing operation thereto, which varies as a function of azimuthal angle, to produce a panoramic image of the scene.

43. Apparatus according to claim 42, wherein the processing operation comprises an image magnification, varying as a function of the azimuthal angle.

44. Apparatus according to claim 43, wherein the processing operation varies the magnification to adjust for differences in the distances of objects in the scene from the image capture device.

45. Apparatus according to any of claims 42-44, wherein the processing operation comprises an adjustment of the image brightness as a function of the azimuthal angle.

46. Apparatus according to any of claims 42-45, wherein the processing operation comprises an adjustment of the image contrast as a function of the azimuthal angle.

47. Apparatus according to any of claims 42-46, wherein the processing operation comprises an adjustment of the image color balance as a function of the azimuthal angle.

48. Apparatus according to any of claims 42-47, wherein the processing operation comprises an enhancement of the image dynamic range as a function of the azimuthal angle.

49. Apparatus for image compression, comprising: at least one video sensor head, which captures an image of a scene; and a controller, which receives, digitizes and compresses the image, and which exchanges control signals with the at least one video sensor head so as to reduce the data size of the compressed image.

50. Apparatus according to claim 49, wherein the controller provides timing signals to the at least one sensor head.

51. Apparatus according to claim 49-50, wherein the at least one sensor head comprises a plurality of sensor heads, which receive respective partial images, and wherein the controller controls the plurality of sensor heads and combines the partial images received therefrom to produce a panoramic image.

52. Apparatus according to claim 51, wherein the controller selects an image sector within the panoramic image and receives partial images corresponding to the image sector from one or more of the plurality of sensor heads so as to update the panoramic image.

53. Apparatus according to any of claims 49-52, wherein the controller identifies an area of the panoramic image corresponding to a generally static background region of the scene, so as to update the area less frequently than other areas of the image.

54. Apparatus according to any of claims 49-53, wherein the controller receives a video sensor parameter from the at least one sensor head and transmits the parameter to a remote site.

55. Apparatus according to claim 54, wherein the sensor parameter comprises a gain factor.

56. Apparatus according to claim 54 or 55, wherein the sensor parameter comprises a shutter speed.

57. Apparatus according to any of claims 54-56, wherein the sensor parameter comprises a color balance setting.

58. Apparatus for displaying a panoramic image, comprising: a video display; and a processor, which receives panoramic image data, geometrically maps the data onto a generally annular surface, and renders at least a portion of the surface to the display.

59. Apparatus according to claim 58, wherein the processor renders the generally annular surface in the form of two hemiannular strips.

60. Apparatus according to claim 59, wherein the processor performs a left-right reverse operation on a portion of the image on one of the hemiannular strips.

61. Apparatus according to any of claims 58-60, wherein the generally annular surface comprises a generally cylindrical surface.

62. Apparatus according to claim 61, wherein the processor renders the generally cylindrical surface to the display in a generally elliptical form.

63. Apparatus according to any of claims 58-62, wherein the processor varies a point of view from which the surface is rendered.

64. Apparatus according claim 58, wherein the video display comprises a generally annular display screen.

65. Apparatus according to claim 64, wherein the display screen comprises a back- projection screen, and comprising a video projector, which projects the image onto an inner side of the screen, so that the image is visible from the outer side thereof.

66. Apparatus according to claim 65, and comprising a partially-reflecting mirror, which directs the image projected by the projector onto the back-projection screen, so that the image on the screen can be seen through the mirror.

67. Apparatus according to any of claims 58-66, wherein the processor geometrically rotates the annular surface which is rendered.

68. Apparatus according to claim 67, and comprising a user interface, wherein a viewer of the display operates the user interface to alter the rendition of the surface.

69. Apparatus according to claim 68, wherein the user interface comprises a pointing device, which the viewer uses to indicate a desired point with respect to which the surface is to be rotated.

70. Apparatus according to claim 68 or 69, wherein the processor remaps at least a portion of the data, responsive to an input from the user interface, so as to zoom in on a region of interest indicated by the viewer.

71. Apparatus according to any of claims 58-70, wherein the processor renders a picture-in- picture image, which is inset in the panoramic image on the display.

72. A method for displaying a panoramic image on a video display, comprising: defining an annular strip and a point of view outside a volume bounded by the strip; mapping the panoramic image onto the annular strip; and rendering an image of the strip, as seen from the point of view, to the video display.

73. A method according to claim 72, wherein mapping the panoramic image onto the annular strip comprises: mapping a first segment of the image, in relative proximity to the point of view, onto a near hemiannulus; mapping a second segment of the image, relatively distant from the point of view, onto a far hemiannulus; and reversing at least a portion of the first segment.

74. A method according to claim 73, wherein reversing at least a portion of the first segment comprises reversing left-right orientation of a figure in the segment.

75. A method according to any of claims 72-74, wherein defining the annular strip comprises defining a generally cylindrical strip, and defining the point of view comprises defining a point of view above an equatorial plane of the annular strip, so that the image of the strip rendered to the display is generally elliptical in shape.

76. A method according to any of claims 72-75 and comprising rotating the annular strip about an axis thereof, relative to the point of view, so as to shift the image rendered of the strip as seen from the point of view.

77. A method according to claim 76, wherein rotating the strip comprises altering the image of the strip substantially without engendering mechanical motion.

78. A method according to claim 76 or 77, and comprising selecting a region of interest in the image, wherein rotating the strip comprises rotating the strip by such an angle that the selected region is brought to a point opposite the point of view.

79. A method according to claim 78, and comprising receiving audio signals from an area shown in the panoramic image, wherein selecting the region of interest comprises identifying a direction from which the audio signals are received and choosing a region corresponding to the direction.

80. A method according to claim 79, wherein receiving the audio signals comprises receiving signals from a acoustic signaling device.

81. A method according to any of claims 78-80, and comprising displaying an enlarged image of the selected region on the video display.

82. A method according to any of claims 72-77, and comprising displaying an enlarged image of a selected region on the video display.

83. A method according to claim 81 or 82, wherein displaying the enlarged image comprises displaying the image alongside the image of the strip.

84. A method according to claim 81 or 82, wherein displaying the enlarged image comprises mapping the panoramic image so as to enlarge a portion of the annular strip that includes the selected region.

85. A method according to any of claims 81-84, wherein displaying the enlarged image comprises enhancing the resolution of the image in the selected region.

86. A method for displaying a panoramic image, comprising: providing a generally annular back-projection screen; mapping and rendering the panoramic image to the screen; projecting the image onto the screen from inside the annulus; and viewing the image from outside the annulus.

87. A method according to claim 86, wherein projecting the image comprises projecting a video image.

88. A method according to claim 86 or 87, wherein projecting the image comprises projecting the image against a partially-reflecting mirror, which reflects the image onto the screen, and wherein viewing the image comprises viewing a portion of the screen by way of the mirror.

89. A method according to any of claims 72-77, 82, or 86-88, and comprising: receiving audio signals from an array of microphones placed adjacent to an area shown in the panoramic image; selecting a region of interest in the image; and processing the audio signals selectively, responsive to the selected region.

90. A method according to claim 89, wherein processing the signals selectively comprises reconstructing sounds originating from a vicinity of the region in the area shown in the panoramic image.

91. A method according to claim 89, wherein processing the signals selectively comprises muting sounds originating from a vicinity of the region in the area shown in the panoramic image.

92. A method according to any of claims 72-91, wherein mapping and rendering the image comprise mapping and rendering a reference image, updating a selected portion of the image, and rendering the selected, updated portion.

93. A method according to claim 92, and comprising producing the reference panoramic image by combining a plurality of images respectively captured by a plurality of video sensor heads, wherein updating the selected portion comprises updating a portion of the image captured by a selected one or more of the plurality of video sensor heads.

94. A method according to claim 93, and comprising selecting the one or more of the plurality of video sensor heads by indicating an area on the video display corresponding to the portion of the image captured by the one or more sensor heads.

95. A method according to any of claims 72-92, and comprising receiving a plurality of images from a remote location, the plurality of video images covering a panoramic field of view at the location, and generating the panoramic image by combining the video images.

96. A method for transmitting an image to a remote site, comprising: generating video signals using one or more video sensor heads; receiving and compressing the video signals for transmission; and exchanging control signals with the one or more video sensor heads so as to reduce the data size of the compressed signals.

97. A method according to claim 96, wherein exchanging the control signals comprises receiving a sensor head control parameter and transmitting the parameter to the remote site.

98. A method according to claim 97, wherein receiving the control parameter comprises receiving a gain factor.

99. A method according to claim 97 or 98, wherein receiving the control parameter comprises receiving an electronic shutter setting.

100. A method according to any of claims 97-99, wherein receiving the control parameter comprises receiving a color balance setting.

101. A method according to any of claims 97-100, wherein transmitting the parameter comprises transmitting a change in a parameter relative to an earlier value thereof.

102. A method according to any of claims 96-101, wherein generating the video signals comprises generating signals using a plurality of video sensor heads, and wherein exchanging the control signals comprises selecting one or more of the plurality of sensor heads to transmit signals for use in updating a corresponding portion of a reference image, while other portions of the image corresponding to the non-selected sensor heads are substantially frozen.

103. A method for producing a panoramic image of a scene, comprising: positioning a plurality of video sensor heads to capture a respective plurality of partial images of the scene, each partial image having an angular extent substantially less than a total angular extent of the scene whose image is to be produced and having a region of overlap with a neighboring partial image; receiving video partial images from the plurality of sensor heads; and combining the partial images to produce the panoramic image.

104. A method according to claim 103, wherein producing the panoramic image comprises producing an image covering an azimuthal field of view of at least 90┬░.

105. A method according to claim 104, wherein producing the image comprises producing an image covering an azimuthal field of view of at least 180┬░.

106. A method according to claim 105, wherein producing the image comprises producing an image covering an azimuthal field of view substantially equal to 360┬░.

107. A method according to any of claims 103-106, wherein producing the panoramic image comprises producing an image having an aspect ratio of width to height of at least 2:1.

108. A method according to any of claims 103-107, wherein positioning the sensor heads comprising positioning the sensor heads so as to be substantially stationary.

109. A method according to any of claims 103-108, wherein positioning the sensor heads comprises aligning at least some of the sensor heads and optics associated therewith, each such sensor head having a respective entrance pupil, so that the centers of the entrance pupils substantially coincide.

110. A method according to any of claims 103-109, wherein combining the partial images comprises adding each partial image to its neighboring image in the overlap region thereof.

111. A method according to any of claims 103-110, wherein combining the partial images to produce the panoramic image comprises mapping the images onto a substantially cylindrical image surface.

112. A method according to claim 111, and comprising rendering the panoramic image to a video display in a generally elliptical form.

113. A method according to claim 111, and comprising rendering and projecting the panoramic image onto a generally annular back-projection display.

114. A method according to any of claims 103-113, wherein combining the partial images comprises automatically varying a display mode of the combined images responsive to a predetermined condition in the scene.

115. A method according to any of claims 103-114, wherein positioning the plurality of sensor heads comprises positioning a plurality of pairs of sensor heads, the pair comprising left and right sensor heads, mutually offset, so as to capture two partial images of a respective, common field of view, and wherein combining the partial images comprises combining the images to produce a stereoscopic panoramic image.

116. A method according to any of claims 103-115, wherein combining the images comprises processing the images variably as a function of azimuthal angle in the panoramic image.

117. A method for producing a panoramic image comprising: receiving video images covering a panoramic field of view; and processing the images variably as a function of azimuthal angle in the panoramic image.

118. A method according to claim 116 or 117, wherein processing the images comprises applying a variable gain to video signals corresponding to the images, as a function of azimuthal angle.

119. A method according to any of claims 116-118, wherein processing the images comprises adjusting an image brightness as a function of azimuthal angle.

120. A method according to any of claims 116-119, wherein processing the images comprises adjusting an image contrast as a function of azimuthal angle.

121. A method according to any of claims 116-120, wherein processing the images comprises adjusting a color balance as a function of azimuthal angle.

122. -A method according to any of claims 116-121, wherein processing the images comprises adjusting an image magnification as a function of azimuthal angle.

123. A method according to claim 122, wherein adjusting the magnification comprises correcting the image for perspective differences among a plurality of objects seen in the panoramic image.

124. A method according to any of claims 116-123, wherein processing the images comprises suppressing a portion of the images falling within a selected range of azimuthal angles.

125. A system for video teleconferencing, comprising at least one transmitting station, which comprises: a plurality of video sensor heads, positionable at or near the center of a table around which teleconferencing participants are seated so as to capture images of the participants on opposing sides of the table; and a transmission processor, which receives and combines video signals from the sensor heads to produce and transmit a panoramic image of the scene.

126. A system according to claim 125, wherein the plurality of sensor heads capture images of the participants along generally horizontal respective optical axes.

127. A system according to claim 125 or 126, wherein the transmitting station comprises a microphone array adjacent to the plurality of sensor heads.

128. A system according to any of claims 125-127, wherein each video sensor head generates video signals corresponding to a partial image having a respective field of view, such that at least some of the fields of view include regions of substantial overlap with respective neighboring fields of view, and a union of the fields of view substantially covers a scene having an angular extent beyond the field of view of any one of the sensor heads.

129. A system according to any of claims 125-128, and comprising at least one receiving station, which comprises: a video monitor; a receiving processor, which receives the transmitted panoramic image and displays the image to the monitor; and user controls, which provide user commands to both the receiving processor and the transmission processor.

130. A system according to claim 129, wherein the panoramic image is mapped onto a generally cylindrical geometrical image surface.

131. A system according to claim 130, wherein the panoramic image is mapped by the transmission processor.

132. A system according to claim 130, wherein the panoramic image is mapped by the receiving processor.

133. A system according to any of claims 130-132, wherein the surface is rendered to the display in a generally elliptical form.

134. A system according to claim 133, wherein the surface is rendered by the transmission processor.

135. A system according to claim 133, wherein the surface is rendered by the receiving processor.

136. A system according to any of claims 130-135, wherein the user controls generate a geometrical rotation of the image surface.

137. A system according to any of claims 130-136, wherein the user controls determine a point of view from which the image surface is rendered.

138. A system according to any of claims 129-137, wherein the user controls of the receiving station select one or more of the plurality of sensor heads of the transmitting station, and wherein the video display is updated using images captured by the one or more selected sensor heads.

139. A system according to any of claims 129-138, wherein the transmitting station comprises an array of microphones, which generate respective audio signals, and wherein the receiving processor receives the audio signals and processes them to reconstruct sound originating in a vicinity of a point in the panoramic image selected using the user controls.

140. A system according to any of claims 129-139, wherein the user controls of the receiving station comprise a menu generated by the transmitting station and displayed on the video display of the receiving station.

141. A system according to any of claims 129-140, wherein the transmitting station comprises user controls capable of performing at least some of the functions of the user controls of the receiving station.

142. A system according to any of claims 129-141, wherein the transmission processor exchanges sensor control parameters with the plurality of video sensor heads and transmits at least some of the parameters to the receiving processor, for use in rendering the image.

143. A system according to claim 142, wherein the transmission processor compresses the image for transmission, and wherein the receiving processor decompresses the image using the transmitted parameters.

144. Apparatus for producing a stereoscopic panoramic image, comprising: a plurality of video sensor heads, which capture respective partial images of a scene; and a processor, which receives and combines the partial images from the sensor heads to produce a stereoscopic panoramic image of the scene.

145. Apparatus according to claim 144, wherein the plurality of video sensor heads are substantially stationary.

146. Apparatus according to claim 144 or 145, wherein the panoramic image covers a field of view having an angular extent substantially greater than that of the field of view of any one of the plurality of sensor heads.

147. Apparatus according to any of claims 144-146, wherein the panoramic image covers an azimuthal angle of at least 90┬░.

148. Apparatus according to claim 147, wherein the panoramic image covers an azimuthal angle of at least 180┬░.

149. Apparatus according to claim 148, wherein the panoramic image covers an azimuthal angle substantially equal to 360┬░.

150. Apparatus according to any of claims 144-149, wherein the panoramic image has an aspect ratio of width to height of at least 2:1.

151. Apparatus according to any of claims 144-150, wherein the plurality of video sensor heads comprises a plurality of pairs of video sensor heads, each pair comprising mutually offset left and right sensor heads, which capture respective left and right partial images of a substantially common field of view.

152. Apparatus according to claim 151, wherein the processor combines the partial images to form left and right panoramic images, which are viewed jointly to produce the stereoscopic panoramic image.

153. Apparatus according to claim 151 or 152, wherein each of the plurality of sensor heads has a respective entrance pupil, and wherein the sensor heads are so arranged that the centers of the entrance pupils of at least some of the left sensor heads substantially mutually coincide, and the centers of the entrance pupils of at least some of the right sensor heads substantially mutually coincide

154. Apparatus according to any of claims 151-153, and comprising beam-combining optics, which comprise a plurality of substantially planar, mutually-angled reflective surfaces, wherein one of the surfaces is associated with each of the plurality of pairs of sensor heads.

155. Apparatus according to claim 154, wherein the beam combining optics comprise a multifaceted mirror, formed by the plurality of mutually-angled reflective surfaces.