EP1201089A1

EP1201089A1 - 3d visualisation methods

Info

Publication number: EP1201089A1
Application number: EP00948181A
Authority: EP
Inventors: Angus Lamberton Jamieson
Original assignee: AE Patents Ltd
Current assignee: AE Patents Ltd
Priority date: 1999-07-29
Filing date: 2000-07-28
Publication date: 2002-05-02
Also published as: AU6174400A; WO2001010138A1; GB9917658D0; JP2003522444A

Abstract

A method of presenting a 3D image is described in which a scene is captured by at least two spatially separated video cameras to provide signals representing at least two viewpoints. The signals are conveyed to a viewing apparatus by standard television broadcast or video recording techniques with frames of alternate viewpoints interleaved to produce time sequential frame displays. The viewing apparatus is observed by the viewer through eyepieces which are rendered transparent and obscure in synchronism with the frame rate. A further method of presenting a 3D virtual graphics image is also described. The 3D images may be an event in which one or more participants, such as sports persons, are predefined as objects. Video data defining each object may be created and transmitted prior to the event, and at least one object may be defined as a series of salient points and offsets therefrom. During the event, sequential frames may be transmitted each of which contains data, and which may be coded and embedded digital data, defining the positional co-ordinates of said salient points, from which a complete virtual model can be reconstructed in real time at a remote site.

Description

3D VISUALISATION METHODS

This invention relates to methods of providing three- dimensional (3D) images using a restricted transmission channel such as conventional television transmission.

It is well known in principle to provide a viewer with a 3D image by presenting to the left and right eyes separate images which have been captured from slightly different viewpoints. Means must be provided to ensure that each eye sees only the appropriate image, for example red/green filters to produce an apparent black and white image, or cross-polarised lenses.

It is known to make use of spectacles incorporating electro-optical devices which are alternatively transparent and obscured in synchronism with left and right eye information. Such spectacles have been used in conjunction with computer generated images to visualise building designs, for example, three dimensionally. They have not, however, been used hitherto in conjunction with television.

The present invention seeks to provide a method which uses conventional television transmission to provide 3D images. One form of the invention provides images which can be manipulated, and is capable of providing a virtual in-the-round display.

The present invention, in its broadest aspect, provides a method of presenting a 3D image, in which a scene is captured by at least two spatially separated video cameras to provide signals representing at least two viewpoints, said signals are conveyed to a viewing apparatus by standard television broadcast or video recording techniques with frames of alternate viewpoints interleaved to produce time sequential frame displays, and the viewing device is observed by the viewer through eyepieces which are rendered transparent and obscure in synchronism with the frame rate.

From a second aspect, the invention provides a method of providing 3D images of an event in which one or more participants (which may be persons, animals, or inanimate things) are predefined as objects, wherein video data defining each object is created and transmitted prior to the event, and at least one object is defined as a series of salient points and offsets therefrom, and during the event sequential frames are transmitted each of which contains data, which may be coded and embedded digital data, defining the positional co-ordinates of said salient points. Preferably each frame is partially occupied by said data and partially by conventional video signals.

Said positional co-ordinates will typically be X, Y, Z co-ordinates obtained from video signals generated by multiple cameras.

Preferably, line of sight vectors through salient points are established by video processing software and resolved for the intersection co-ordinates.

Said offsets are preferably defined as angles or polar co-ordinates from an axis joining two salient points.

Where the object is a person, the salient points may suitably include skeletal joints such as shoulders, elbows and wrists.

In a preferred form, the 3D information is recreated at a virtual venue in the form of a floor or table surface around which viewers sit or stand. Background video scenes may be projected on walls around the viewers. Video may also be used to project faces onto facial surfaces of participants.

A further aspect of the invention provides a 3D television system in which a scene is imaged by a pair of cameras in binocular fashion to provide left and right signals, said left and right signals are interleaved and transferred to a television set by normal broadcast, cable or record/replay means to produce a picture display consisting of alternate left and right frames, and the picture display is viewed through spectacles of the kind defined above.

The broadcast signal may suitably include a code at the commencement of each frame defining it as left or right . The code may be decoded by a decoder circuit which may be in the form of a set-top box added to a conventional television set, and may include means (such as an infrared link) for transmitting a synchronising signal to the spectacles.

Embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings, in which :-

Fig. 1 is a perspective schematic view of viewers sitting around a blank floor-mounted screen, waiting for an image, generated by a 3D projection system in accordance with the first embodiment of the present invention, to be displayed on the screen; Fig. 2 is a perspective schematic view of the viewers sitting around the floor-mounted screen of Fig- 1, with the 3D projection system in accordance with the first embodiment of the present invention displaying a tennis match on the screen; Fig. 3 is a front (first direction) view of a footballer being filmed to define the player as an "object" in accordance with the first embodiment of the present invention; Fig. 4 is a perspective schematic view of a plurality (in this case two) of stereo cameras capturing the action of a football match which will be transmitted by a transmission means to the 3D projection system of Figs. 1 and 2; and Fig. 5 is a schematic diagram of a 3D projection system m accordance with a second aspect of the present invention.

Referring to Figs. 1 to 4, a first embodiment allows viewers using 3D visual techniques to view an event such as a sporting event as an instantaneous 3D reproduction m high resolution graphics. W th software, the events can be viewed or re-viewed from any viewpoint and direction or even from the point of view of any participant m the event. For example, part of a football game could be reconstructed from the viewpoint of the referee or the goalkeeper, a record breaking pole vault or high dive could be viewed from the viewpoint of the athlete. A real-time computer reconstruction will be under the control of the viewers. It would even be possible to have four stereo viewing images running simultaneously so that viewers could sit m a separate venue and view the event m full 3D from four sides on a single screen on the floor.

Virtual Venues

As viewers arrive they sit around a floor screen with the 3D projection system mounted above them m the centre. Each viewer is provided with 3D glasses which are left eye/right eye synchronised and the screen will appear blank at first (Figure 1) . The 3D projection system consists of a high speed computer system which generates/reconstructs the images and uses a downward facing vertically mounted screen projector to display the images on the floor, which can then be viewed using the synchronised left eye/right eye glasses as described above. The computer also generates an infrared synchronisation signal, which is received by and infrared sensor provided on the glasses, to keep the glasses in time with the appropriate frame. The high speed computer system is capable of producing multiple channel sets of left eye/right eye views. Each side sees the event as if they were viewing the event live at the real venue (Figure 2) . Separate left eye/right eye viewing sets will be required for all four directions of view. For example, North side would see Southward view in frames 1 and 4 while the East side would see Westward view in frames 2 and 5 and so on.

Data Transmission and Reconstruction

In order to provide this with only conventional television, it is necessary first of all to define each participant as an 'Object'. Each player in a football match would be an 'Object' as would the referee, the linesmen and the ball. The detailed dimensions and colours of each object would be transmitted before the event to the 3D visualisation receiving computers in digital video form by filming each object from at least 4 directions (Figure 3) .

Each object will be stored as a series of salient points and rotational offsets. For example the centre of a knee would be a salient point as would the centre of a heel and all points in the lower leg would be stored as offsets along the line from knee to heel and rotationally at right angles to that line. This definition allows the 'Object' to be reconstructed if only the salient points x, y, z co-ordinates are known.

It may be possible to define more salient points on a human participant but certainly the following would be required: -

Eyes, ears, head, shoulders, elbows, wrists, hands, hips, knees, heels and toes.

All other points could be defined in X, Y and Z, if the X, Y and Z of these points were known.

With some research it may be possible to greatly increase the sophistication of the Object definition. The resolution with which viewers will ultimately be able to see the objects will be the resolution created when the object filled the frame which will always be better than large scale TV shots from a distance.

The fixed background, the pitch, the goals, the corner flags etc will also be defined as 'Objects' but will not require reconstruction from salient points.

By using high resolution cameras from several angles, video recognition software can be used in real time to define the X, Y and Z co-ordinates of each salient point on each object. The attitude (vertical and horizontal angle) of each camera is entered into the processing software and the offset of any recognised salient point in the image can be resolved into a line of sight vector relative to the attitude of the camera.

For example, if a ball was seen m the top right hand quadrant of a camera's image, the vector from the camera to the ball is above and to the right of the vector describing the cameras line of sight. Hence, line of sight vectors from several cameras are established and resolved for their points of intersection.

Further redundancy could be established by including an eye safe laser measurement system to measure distances directly to all objects m that image.

If any detail is hidden from all cameras (e.g. a ball m a rugby scrum) it will equally be hidden from viewers and will not reappear until at least two cameras can spot the ball. The video recognition software will identify outlines of objects m a plane perpendicular to its line of sight and derive space line equations to the centre points. The intersection of space lines from the multiple cameras determines the 3d position of each salient point (Figure 4) .

Motion prediction software will be used to estimate the positions of hidden objects until visual contact is re- established.

A computer at the real venue will prepare a frame transmission where a relatively small percentage of the frame is used to send the co-ordinates of all the salient points previously defined and the remainder is normal video m split screen mode. Hence, the data contains coded and embedded digital data. One suggestion would be to send a 4 tile split screen image of the real venue crowd for back projection on the virtual venue walls around the viewers. Once the computer, which may for example be a Silicon Graphics (™) workstation running a MUSE operating system, at each virtual venue receives the video data, and the embedded co-ordinate data, the whole event can be reconstructed m 3D for the viewers including the images of the moving objects from their salient point co-ordinates. Live video of faces can be projected onto facial surfaces of each participant for further realism.

For action replays, analysis, close up ball out checks, training etc the stored data can be subsequently replayed on a conventional wall screen m 3D from any viewpoint or even from the point of view of any object (including the referee, the goalkeeper, the linesman or even the ball) .

Alternative Embodiment

In the embodiment illustrated m Fig. 5, a right eye camera 50 and a left eye camera 52 are physically coupled together to provide right and left video signals which are interleaved to provide alternate frames by firmware 54 and then transmitted by conventional broadcast techniques. Each frame is preceded by a code defining it as left or right. In digital transmission, the code may be a data byte at the leading end of the frame data, and in analogue transmission the code may be a suitable pulse signal or frequency burst signal adjacent in time to the normal frame sync signal .

The broadcast signal may be received and viewed by a conventional television set 56, with the signal fed in parallel to a set top box 58 which reads the left and right codes and transmits an infra-red signal to viewer spectacles 60, whereby each lens is alternated from opaque to clear in synchronism with the frame signals. Modifications and improvements may be made to the foregoing embodiment within the scope of the present invention.

Claims

CLAIMS :

1. A method of presenting a 3D image, in which a scene is captured by at least two spatially separated video cameras to provide signals representing at least two viewpoints, said signals are conveyed to a viewing apparatus by television broadcast or video recording techniques with frames of alternate viewpoints interleaved to produce time sequential frame displays, and the viewing apparatus is observed by the viewer through eyepieces which are rendered transparent and obscure in synchronism with the frame rate.

2. A method according to claim 1, wherein the 3D images are of an event in which one or more participants are predefined as objects, wherein video data defining each object is created and transmitted prior to the event, and at least one object is defined as a series of salient points and offsets therefrom, and during the event sequential frames are transmitted each of which contains data defining the positional co- ordinates of said salient points.

3. A method according to claim 2, wherein said data contains coded and embedded digital data.

4. A method according to either of claims 2 or 3 , wherein each frame is occupied partially by said data and partially by conventional video signals.

5. A method according to any of claims 2 to 4, wherein said positional co-ordinates are X, Y, Z co- ordinates obtained from video signals generated by multiple cameras.

6. A method according to any of claims 2 to 5, wherein line of sight vectors through salient points are established by video processing software and resolved for the intersection co-ordinates.

7. A method according to any of claims 2 to 6, wherein said offsets are defined as angles or polar coordinates from an axis joining two salient points.

8. A method according to any of claims 2 to 7, wherein the object is a person, the salient points includes skeletal joints, including at least one of shoulders, elbows and wrists.

9. A method according to any of claims 2 to 8 , wherein the 3D image is recreated at a virtual venue in the form of a substantially flat surface around which viewers are arranged.

10. A method according to claim 9, wherein background video scenes are projected on walls around the viewers.

11. A method of providing 3D images of an event in which one or more participants are predefined as objects, wherein video data defining each object is created and transmitted prior to the event, and at least one object is defined as a series of salient points and offsets therefrom, and during the event sequential frames are transmitted each of which contains data, which may be coded and embedded digital data, defining the positional co-ordinates of said salient points.

12. A 3D television system m which a scene is imaged by a pair of cameras in binocular fashion to provide left and right signals, said left and right signals are interleaved and transferred to a television set by broadcast, cable or record/replay means to produce a picture display consisting of alternate left and right frames, and the picture display is viewed through spectacles comprising eye pieces which are capable of being rendered transparent and obscure m synchronism with the frame rate.

13. A 3D television system as claimed m claim 12, wherein the broadcast, cable or record/replay means transmits a broadcast signal which includes a code at the commencement of each frame defining it as left or right.

14. A 3D television system according to claim 13, wherein the code is decoded by a decoder circuit.

15. A 3D television system according to claim 14, wherein the decoder circuit is m the form of a set-top box added to a conventional television set.

16. A 3D television system according to either of claims 14 or 15, wherein the decoder circuit includes means for transmitting a synchronising signal to the spectacles.