EP1486057A2 - Virtuelles studiosystem - Google Patents
Virtuelles studiosystemInfo
- Publication number
- EP1486057A2 EP1486057A2 EP03712354A EP03712354A EP1486057A2 EP 1486057 A2 EP1486057 A2 EP 1486057A2 EP 03712354 A EP03712354 A EP 03712354A EP 03712354 A EP03712354 A EP 03712354A EP 1486057 A2 EP1486057 A2 EP 1486057A2
- Authority
- EP
- European Patent Office
- Prior art keywords
- image
- participant
- camera
- background
- studio
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 claims abstract description 153
- 230000008569 process Effects 0.000 claims abstract description 27
- 238000009877 rendering Methods 0.000 claims description 35
- 210000003128 head Anatomy 0.000 claims description 21
- 230000000873 masking effect Effects 0.000 claims description 21
- 238000004519 manufacturing process Methods 0.000 claims description 14
- 230000033001 locomotion Effects 0.000 claims description 13
- 238000002310 reflectometry Methods 0.000 claims description 11
- 238000012545 processing Methods 0.000 claims description 8
- 230000003287 optical effect Effects 0.000 claims description 7
- 230000000007 visual effect Effects 0.000 claims description 7
- 230000005540 biological transmission Effects 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 4
- 230000002123 temporal effect Effects 0.000 claims description 4
- 239000011521 glass Substances 0.000 claims description 3
- 238000005286 illumination Methods 0.000 claims description 3
- 230000001360 synchronised effect Effects 0.000 claims description 3
- 230000002238 attenuated effect Effects 0.000 claims description 2
- 239000004744 fabric Substances 0.000 description 6
- 230000003068 static effect Effects 0.000 description 6
- 238000013459 approach Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- 206010065042 Immune reconstitution inflammatory syndrome Diseases 0.000 description 1
- 241001465754 Metazoa Species 0.000 description 1
- 201000009310 astigmatism Diseases 0.000 description 1
- 238000005266 casting Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000001815 facial effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000007935 neutral effect Effects 0.000 description 1
- 238000010422 painting Methods 0.000 description 1
- 230000002829 reductive effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 230000000153 supplemental effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 238000002604 ultrasonography Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/2224—Studio circuitry; Studio devices; Studio equipment related to virtual studio applications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/265—Mixing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/272—Means for inserting a foreground image in a background image, i.e. inlay, outlay
Definitions
- the present invention relates to virtual studio systems, particularly but not exclusively chroma-keying studio systems.
- Chroma-keying is a technique that allows synthetic images to be created by determining if a pixel in a camera image belongs to foreground or background based on colour.
- the foreground pixels can then be keyed into a second image, e.g. a computer-generated or static background as in virtual studios.
- the background in the studio must have the particular keying colour, typically blue. This is usually achieved by painting the studio floor and visible walls with the keying colour.
- Another option is to use a retro-reflective cloth and an active illumination ring around the camera lens as proposed in our earlier GB-A-2,321,814 the entire disclosure of which is incorporated herein by reference.
- Chroma-keying is a preferred application, the present invention is also applicable to other keying techniques (for example using temporally modulated lighting of the background to distinguish foreground and background).
- the term "participant” is intended to encompass without limitation any actor or presenter (human or animal) or indeed any form of animate entity which might benefit from a view of a virtual scene.
- One problem for a participant in a virtual studio is to interact with the objects in the virtual set that are not visible for him or her.
- a virtual object e.g. a virtual character
- eye-line problem there is often a difference between the direction in which the participant looks, and the position of the virtual character that he should be looking at. This can be quite disturbing in the final programme.
- This prior art visual cue system for virtual studios does not however deal directly with the eye-line problem, i.e. the case that a participant has to look towards a portion of a virtual object that is in space (for example the face of a virtual actor) but merely gives the participant an idea of the position of objects in the studio.
- This system tracks the position of the viewer's head and computes an image for that particular view-point of the viewer, which is then projected onto a large screen forming one wall of the environment.
- Several screens are usually used to provide a wide field-of-view, and some installations provide an all-round view using six projection screens that completely surround the viewer. Therefore it is possible to present objects that appear virtual in space.
- the head position of the viewer is usually tracked using a special device, e.g. a helmet with an electro-magnetic transmitter.
- a helmet with an electro-magnetic transmitter Such a system is of course not designed for nor suited to a virtual studio environment for a number of reasons.
- the head-tracking device would be visible in any final programme and, more fundamentally, virtual studios which may have a combination of "real" objects at arbitrary positions and require a background of a key colour do not provide the open space with large screens on which images can be projected at will as this system requires.
- the present invention is specifically concerned with a system which is suited for use in a virtual studio i.e. having a background which must be keyed out of a camera image.
- the invention provides a method of providing a visual representation of a scene to a participant in a studio, the studio being arranged to be viewed by a camera to provide a camera image and having at least a portion of background object arranged to be keyed out of the camera image by a keying process, the method comprising: obtaining an estimate of the viewpoint of the participant; rendering an image of the scene based on the viewpoint of the participant; projecting at least a portion of the rendered image onto the background object to provide a projected image portion visible to the participant, wherein the projected image portion is projected so as to be keyed out of the camera image by the keying process.
- the eye-line problem present with the static outline method of the prior art can be alleviated.
- the image is projected so as to be keyed out by the keying process.
- the estimate of the viewpoint of the participant will typically be based on a measure of position of the participant.
- the background object may comprise a static object, typically the studio background. Additionally or alternatively, the background object may comprise an object in the studio, optionally a movable object, which is to be replaced with a virtual object. For example, a card, picture or block may have a graphic superimposed on it during a production. This may be done in addition to or instead of replacing the background and/or other objects with virtual objects.
- the background object being static, it is desirable to store data concerning the geometry of the object.
- a movable background object it is desirable to receive an input of a measure of the background object position.
- a movable background object may have an active tracking device integrated therein.
- the active tracking device may be conventional for example a radio or ultrasound transmitter or receiver which in combination with one or more external complementary receivers or transmitters in the studio enable the position to be determined using one of a number of conventional techniques.
- Such an object may be independently provided in a further aspect of the invention which provides an object having an at least partially retro-reflective surface for reflecting key light to a camera to enable the object to be keyed out of the camera image, the at least partially retro-reflective surface also serving to render visible a projected image, the object having integrated therein means for co-operating with position determining apparatus for determining the position of the object in a studio.
- the means for co-operating may comprise a collection of identified markers, such as circular bar codes as described in our GB-A-2,325,807, the position determining apparatus comprising a camera and processing apparatus as described in that document.
- the means for c ⁇ -operating may comprise an active tracking device, for example including at least one transmitter or receiver.
- the image is preferably projected from at least one direction different from the direction of view of the camera; this may enable a participant facing away from the camera to see background detail which would not be visible to the camera and, particularly with a retro-reflective background, this reduces the amount of the projected image reaching the camera.
- the keying process is a chroma-keying process and the image is projected so that the colour of the background portion as seen by the camera is substan t ially a background colour.
- the keying process is not affected by the image.
- the background retro-reflects a portion of light with a first, relatively high, reflectivity and diffusely reflects light with a second, relatively low, reflectivity.
- This is an important preferred feature, as it enables a good clean key colour to be obtained by retro-reflection of the key light with any scattered light from the projected image being of significantly lower intensity.
- the background may be lit with key light from a direction substantially coincident with the camera direction so that key light is retro-reflected to the camera with a first, relatively high, intensity and the projected image is projected from a different direction.
- the direction is preferably chosen so that the projected image is not retro-reflected to the camera over a range of expected camera movement.
- the position will depend on the properties of the background; typically the background may "retro-reflect" a substantial portion of light up to a few degrees off the axis of incidence, with the reflectivity falling sharply at greater angles and becoming diffuse reflection (suitable cloth and typical properties are discussed in our GB-B-2,321,814, and GB-B-2,321,565 incorporated herein by reference - preferred features of the background discussed in those patents may be used as preferred features of the background in this application).
- the image is preferably projected so as to be visible to the participant based on diffuse reflection.
- the intensity required can simply be judged literally by eye based on ambient studio lighting conditions.
- the first reflectivity is at least double the second reflectivity, preferably at least ten times the first reflectivity.
- the key light may comprise light of a selected key colour, in a chroma-keying method.
- the key light may comprise light having a selected temporal variation.
- the method may further comprise masking a portion of the projected image based on the position of the participant to reduce projection onto the participant.
- This method may be provided independently in a second aspect which provides a method of projecting an image onto a background in a space in which a participant is positioned, the method comprising rendering an image based on the position of the participant, the method further comprising masking a portion of the projected image based on the position of the participant to reduce projection onto the participant.
- the method may comprise masking a portion of the projected image based on the position of an object to reduce projection onto the object.
- the invention provides a method of projecting an image onto a background in a space in which an object is positioned, the method comprising dynamically rendering the image, the method further comprising dynamically masking a portion of the projected image based on the position of the object to reduce projection onto the object.
- the masked portion is based on a stored or estimated outline of the participant or other object. This may facilitate masking. Additionally or alternatively, the mask may be based on one or more camera images of the participant or object.
- a mask signal is generated to mask a portion of the image; this facilitates processing.
- the rendering process may take into account any required masking so that rendering is not performed, or a reduced level of rendering is performed in regions to be masked.
- the outline is larger than the actual outline of the participant; this reduces the risk of stray projection onto the participant (or object).
- the outline may be enlarged along an axis of motion of the participant.
- the background retro-reflects a portion of light and the background is at least partially translucent. This permits back-projection of images in addition to or as an alternative to front projection.
- the background is lit with key light from a direction substantially coincident with the camera direction so that key light is retro- reflected to the camera with a first, relatively high, intensity and the projected image is projected onto the rear of the background.
- Back-projection will typically result in a lower intensity of transmitted light reaching the camera than the light retro-reflected under normal circumstances, but the intensities of each light may be managed if high intensity projection is used.
- a masking process similar to that used in the case of front projection may be used for back projection to reduce stray light entering the camera.
- the primary problem is that of light passing directly through the background, rather than reflecting off objects or participants, so the primary information to take into account is t he camera and projector positions.
- the projector position will typically be fixed and can be stored but the camera position (or field of view) may change so that is preferably provided as an input.
- the projected image may be at least partially masked or attenuated based on the position of a camera viewing the background to reduce the intensity of light from the projected image reaching the camera.
- the invention provides a method of projecting an image onto a background viewed by a camera, the method comprising masking or attenuating at least one portion of the projected image based on at least one of the position of a participant or other object and the position of the camera to reduce stray light from the projected image reaching the camera.
- the projection intensity, the key light intensity and the background properties are selected so that the intensity of the key light received at the camera is at least double, preferably at least ten times, the intensity of light received from the projected image.
- the intensity of the projected image is selected so as to be visible by the participant but so that the scattered light from the projected image does not substantially disrupt the keying process.
- the image may be projected during a period when the camera shutter is closed.
- the keying process may be based on temporal modulation of the background illumination and the projected image may be temporally modulated.
- the participant's viewpoint is preferably estimated based on the measure of position of the participant.
- the measure of position advantageously comprises an estimate of position of a body portion of a participant and wherein the viewpoint is estimated based on a physical relationship of the body portion to the eyes of the participant.
- Most preferably the position of the head of the participant is estimated.
- the methods disclosed in our GB-A-2,352,899, incorporated herein by reference, may be used as preferred methods of estimating position.
- An estimate of the position of the head of the participant may be used as an estimate of the viewpoint of the participant; this simplifies the process while remaining sufficiently accurate for most purposes.
- the method may include filtering the estimate of viewpoint to avoid sudden changes or to reject estimates which display unexpected motion or unexpected positions (for example a sudden increase in height or a height greater than the height of the participant). This may be advantageous particularly when viewpoint is estimated based on a camera image as such a method may produce occasional spurious results if, for example, an actor's arm moves above or past his head.
- the participant position may be determined based on an image from at least one position information camera.
- the at least one position information camera comprises at least one auxiliary camera. Additionally or alternatively the (studio) camera providing the studio camera image may also serve as a (main or supplementary) position information camera.
- the image is projected from a plurality of sources, preferably at at least two distinct locations. This may reduce masking of portions of the background by the participant or by objects in the set. Images from at least two sources at least partially overlap.
- the intensity of at least one image may be modulated or (additionally or alternatively) a portion of at least one image may be masked to reduce intensity variation in a region of overlap.
- the invention provides a method comprising rendering an image for projection onto a virtual studio background, the studio having a participant therein, the method comprising rendering the image from a viewpoint based on positional information concerning the participant, the method further comprising masking a portion of the image based on the position of the participant.
- rendering preferably includes receiving information concerning the position of at least one image projector.
- Rendering may include receiving information concerning the studio geometry and/or adjusting the image based on background geometry.
- Information concerning at least one of:- projector location, studio geometry, background geometry, objects within the studio, participant dimensions and camera position is preferably stored.
- the method further comprises projecting at least two images to be separately viewable by two participants.
- the images may be projected onto separate background object portions. However, more commonly the images may at least partially overlap.
- the images are multiplexed and the participants are provided with demultiplexing means so that each participant views preferentially a single image.
- This may be provided independently in a further aspect which provides a method of providing images for at least two viewers, the method comprising projecting a first image for a first viewer and a second image for a second viewer, the images at least partially overlapping and being multiplexed in at least a region of overlap and providing each viewer with respective demultiplexing means to provide each viewer with a preferential view of one image.
- the images may be colour multiplexed and each viewer or participant may be provided with demultiplexing means comprising a respective colour filter, preferably in the form of a coloured contact lens.
- the images may be polarisation multiplexed and each viewer or participant may be provided with demultiplexing means comprising a polarising filter, preferably in the form of polarising glasses or a polarised contact lens.
- demultiplexing means comprising a polarising filter, preferably in the form of polarising glasses or a polarised contact lens.
- Contact lenses used to combat astigmatism, having an asymmetric weight are known a»d such can be provided with a polarising filter.
- the invention further independently provides a contact lens having an asymmetric weight distribution and a polarising filter.
- the invention further provides use of a selectively transparent, preferably coloured and/or polarised, contact lens to demultiplex a multiplexed image, preferably for one of a plurality of viewers.
- the invention further provides a kit for optical demultiplexing of at least two multiplexed images by at least two viewers comprising a first optical apparatus (preferably a pair of (or at least one) spectacles or contact lenses) wearable by a first viewer and having a first selective optical transmission property and a second optical apparatus (preferably a pair of (or at least one) spectacles or contact lenses) wearable by a second viewer and having a second selective optical transmission property, as well as further providing use of such a kit or portion thereof to obtain at least one image from a multiplexed image.
- a first optical apparatus preferably a pair of (or at least one) spectacles or contact lenses
- a second optical apparatus preferably a pair of (or at least one) spectacles or contact lenses
- the images may be temporally multiplexed and each viewer or participant may be provided with demultiplexing means comprising an optical filter having a temporally varying light transmission.
- the temporally varying light transmission is preferably synchronised to the projectors and preferably also synchronised to the shutter of a camera viewing the scene - this may enable the filters (typically in the form of glasses) to be viewed by the camera at a time when they are transparent (or opaque) as desired.
- the aforementioned methods preferably further comprise recording a sequence of images of the participant with a studio camera.
- Implementations of the method will typically further comprise processing said sequence of images to remove or replace the background by the keying process, wherein the projected image is substantially removed.
- Another aspect of the invention provides independently a computer program or computer program product or apparatus arranged for performing any methods or sub features of methods. Preferred and optional features of the method aspects may be applied to all other aspects and vice versa.
- Another aspect of the invention further provides a projected image produced by any method.
- Another aspect of the invention further provides a virtual studio production produced by any method.
- Another aspect of the invention further provides a mask signal generated in a rendering method.
- Another aspect of the invention provides image rendering apparatus, preferably for use in an above mentioned method, the apparatus comprising: means for receiving a measure of a participant position; means for storing a measure of a participant outline; means for rendering an image based on the participant position, the rendered image having at least a portion masked based on the participant outline.
- the apparatus may include means for storing a measure of at least one projector position and/or means for storing a measure of studio geometry and/or means for determining the stored measure of participant outline based on at least one of:- a dynamic image of the participant; stored information on the participant outline; information on movement or expected movement of the participant.
- Another aspect of the invention provides a production studio arranged for use in any method, the studio including a background and means for projecting a dynamically rendered image onto the background so as to be keyed out of the image of a studio camera by a keyer.
- Another aspect of the invention provides a production studio comprising:- an at least partially retro-reflective background; means for determining a measure of position of a participant in the studio; means for rendering an image based on participant viewpoint; means for projecting the rendered image onto the background so as to be visible to the participant.
- the means for projecting may comprise at least one projector for projecting onto the front of the background.
- the means for rendering may be arranged to mask a portion of the projected image corresponding to participant position.
- the production studio may further comprise at least one camera having means for illuminating the background with a key light and/or may further comprise a keyer for keying out the background image from a camera image of the studio.
- a head-tracking system based only on visual information is preferred, so that the viewer need not wear any extra device. This can be achieved using the image from the main studio camera and ideally some auxiliary cameras. Since there is preferably a chromakey background available, the outline of the actor may advantageously be obtained in a robust way to facilitate head-tracking. The position of the actor's head may then be used to control the rendering of the scene, to give the actor the correct perspective of the virtual scene. This may allow a precise orientation of the gaze towards objects that are virtually present in the studio.
- the projection system may be combined with a robust chroma- keying system in such a way as to make the projected image clearly visible to the performer without interfering with the chroma-key process. Whilst this can be achieved to some extent by using conventional blue screens with low-level projection, a more effective method is provided by the use of retro-reflective screen material and coloured (blue) light sources on each camera as described.
- a further aspect of the invention is concerned with the projection method itself and is not limited to a virtual studio environment, but is suited thereto.
- rear projection is used which requires a lot of space behind the screens and presents particular problems with the floor. It is often more convenient to use front projection, but this presents the problem of the projected image falling on the actor, and the actor casting shadows on the screen.
- the shadows cast on the screen are not a major problem in application with which the present invention is concerned, as the projected images are used for providing visual cues rather than a fully-realistic virtual environment where they may be problematic — as noted elsewhere, the requirements of virtual studio production and a virtual environment simulation are quite distinct.
- One aspect of the invention therefore includes a method to prevent the appearance of projected images on the actor by blanking the appropriate part of each projected image, preferably using information describing the actor's position and shape (although this may be simplified to a basic outline or even a rectangle).
- Figure 1 shows the arrangement of cameras, projectors and screens in the studio;
- Figure 2 shows how a rough model of the actor is incorporated in the scene to provide a mask;
- Figure 3 shows a block diagram of the main elements of a system incorporating the invention.
- the walls of the studio area 1 are composed of a retro-reflective cyclorama cloth, as described in GB-B-2,321,814, and GB-B-2,321,565 to which specific reference should be made for details of suitable cloth and camera arrangements as required and the entire contents of each of which are herein incorporated by reference.
- One or more studio TV cameras 2, 20 view the scene. As described in the above mentioned patents, each of these is preferably fitted with a coloured light source, so that the reflective cyclorama appears brightly-lit with this colour from the viewpoint of the camera, but still appears as a neutral (typically grey) colour from other viewpoints.
- the cloth has the property that it is retro-reflective (that is it reflects a significant portion of light back to the direction of a light source) and has a generally grey appearance otherwise.
- Each studio TV camera may also be fitted with a camera tracking system to allow camera movement to be monitored.
- An example of a suitable tracking system may be found in GB-A-2,325,807, the entire disclosure of which is incorporated herein by reference, but other tracking systems may be used.
- the video signal from the studio camera, and any tracking data are passed to a renderer 26 that generates background images at broadcast TV standard.
- the camera image is keyed over these images using a chroma-keyer 27.
- These rendering and keying processes may also incorporate other conventional virtual studio practices such as generating mask signals to allow virtual objects to appear in front of performers.
- One or more auxiliary cameras 3, 21 also view the scene.
- the images from these cameras, and optionally from the main camera 2, 20 are analysed to track the position of the actor's head by a head tracker 22.
- This tracking can be carried out using one of several known tracking methods.
- a preferred example of an actor tracking method is described in GB-A-2,352,899, the entire disclosure of which is incorporated herein by reference.
- the availability of the silhouette image from the actor by using chroma- keying can be used to simplify or enhance the tracking process.
- the tracking method should ideally estimate the approximate position of the mid-point between the actor's eyes.
- This may be approximated in practice by determining a measure of the position of a part of the actor, preferably the head or a portion thereof and determining a measure of the position of the eyes based on the position of the part and a positional relationship (either predetermined or based on supplemental information). For example, the position of the top of the head may be determined and an estimate of the eye position obtained based on stored data correlating the eye position to the head either for a range of actors or a specific actor. High accuracy is not normally critical, but can be obtained if required using available conventional positioning aids or image analysis. For example an estimate that the midpoint of the eyes is 10cm below the top of the head and centrally positioned will normally suffice for most practical purposes.
- One or more projectors 4, 23 are mounted so as to project images onto the cyclorama walls.
- the projectors project onto the walls from the front, although rear projection may also be used if space permits.
- Each projector is fed from a 3D renderer 24, consisting for example of software running on a personal computer equipped with a 3D graphics accelerator.
- Each renderer calculates a view of the virtual scene with viewing geometry to match the measured head position of the actor and the geometrical arrangement of the projector and screen. Images should be computed at a sufficient rate to give the actors an acceptable rendition of the scene, ideally at least 10 frames per second, and preferably 50Hz or more.
- rendering software is well-known to one skilled in the art, and may be based on conventional software systems such OpenGL, Open Inventor or IRIS Performer.
- the rendering of the image also takes account of the keystone distortion caused by the projector not projecting at right-angles onto the projection screen.
- the constraints imposed by the physical studio arrangement will usually make the correction of such distortion impossible by using only the keystone distortion correction provided by the projector itself.
- the image from one projector may fall on two or more non-coplanar surfaces, such as two walls or one wall and the floor.
- the rendering software may be adapted to render different parts of the image using different viewing geometries so as to present the actor with a correctly-rendered image on all surfaces.
- the rendering process may take into account movement of the background object, for example as detected by tracking cameras or an active tracking system.
- a mask is calculated for each projector to mask out the light that would otherwise fall on the actor.
- One way in which such a mask signal can be generated is by computing a rough 3D model of the actor using the video signal from the auxiliary cameras 3, 21, by a processing unit 25.
- a technique such as shape-from-silhouette, as described in "W. Niem. Robust and fast modelling of 3d natural objects from multiple views. In SPIE Proceedings, Image and Video Processing II, volume 2182, pages 88-397, San Jose, February 1994" is suitable for this application.
- the actor model is passed to each projector's 3d renderer and rendered as a black object to appear in the scene at a position so as to be projected onto the real actor.
- the portion 14 of the projected image which will be projected onto the actor by the projector 15 must be masked out by placing a black object at the appropriate point in the scene.
- a black object cannot be obtained by placing the rough 3D model of the actor in the scene exactly coincident with the real actor 10, since the real actor lies outside the viewing frustum.
- the actor model is moved just inside the viewing frustum to position 16, and scaled such that the image of the actor model as viewed from the position of the real actor matches the part of the projected image that the actor's shadow would fall upon.
- the translation and scaling that must be applied to the actor model may be calculated using conventional geometrical methods, given knowledge of the position of the screen and the projector, and the viewpoint of the actor. It is advantageous to place the actor model just inside the viewing frustum, since it should appear to be the frontmost object in the scene; any scene elements between the near clipping plane and the actor model will not be blanked.
- An alternative method of blanking the appropriate portion of the projected image is to perform a separate rendering process, which renders the view of the actor model (positioned at the actual location of the actor) from the position of the projector 15, to obtain a mask signal.
- a mask signal might show the actor as a black object against an otherwise white background.
- This mask signal can then be used to mask out the portion of the screen on which the actor's shadow would fall, for example by placing the mask image in the OpenGL Stencil buffer.
- the same effect could also be achieved by first rendering the scene with a viewing frustum appropriate for the actor, and then changing the viewing frustum to correspond to the viewpoint of the projector, and rending a scene containing just the actor model, but with the OpenGL . 'depth test' control disabled. This guarantees that the actor model will be rendered on top of the previously-rendered image, thereby blanking the appropriate portion.
- a third method of masking out the actor is to generate the actor mask signal by placing an additional auxiliary camera close to the projector 15.
- the camera should be fitted with a chromakey light source if a retro-reflective screen is being used.
- a chromakeyer can then be used to obtain a key signal. It will generally be necessary to apply some scaling or other affine transformation in order to obtain good alignment between the key signal and the image projected by the projector.
- the key signal can then be used to mask out the actor, for example by placing the mask image in the OpenGL Stencil buffer as in the method described above.
- This method has the advantage that it is not necessary to compute a 3D model of the actor, since the outline of the actor from the point-of-view of the projector is captured directly.
- the mask signal is made to appear slightly larger than the actor, to account for the distance that the actor might move in the time required to generate and render the mask signal, and to compensate for any inaccuracies in the position of the generated mask from other causes such as calibration errors. It may also be desirable to estimate the speed of movement of the actor and move the mask further along the motion trajectory to the predicted position of the actor at the time that the mask will be projected. These methods could also be combined, by enlarging the actor mask only in the direction of the actor's movement. This processing could be implemented in the same processing unit 25 that generated the actor model.
- the actor mask will of course obscure some elements of the projected scene, which could potentially contain scene elements that the actor needs to see.
- more than one projector can be used to project onto each surface, projecting from different positions.
- the projectors may be specifically placed based on the scene layout to ensure adequate coverage in important areas, for example based on consideration of masked areas, or may be generically placed in a studio.
- the masked-out portion of one image will generally not coincide with the masked-out area in another, except for regions where the actor is very close to the screen. This approach will result in most areas of the screen being of double the brightness. If this is considered to be a problem in a particular embodiment, then mask signals can be derived using the above-mentioned geometric techniques to ensure that a maximum of one (or other desired number) projector illuminates any given part of the screen.
- projectors may be selectively used so that projectors that are nearly coincident with (for example within the tolerance of the scatter angle of the background cloth) the direction of view of an active camera are suppressed.
- One example of another such projection system is one that uses bright projectors and a conventional blue screen, where the projectors have a short display period, which is timed to occur during a period when the camera shutters are riot open.
- TV cameras with a 50Hz field rate can be shuttered to 1/100* of a second exposure time, and the projectors can be arranged only to project light for the l/100 th second when the camera shutters are closed.
- Different proportions can be used, for example depending on ambient lighting. For example, in dimly lit scenes, the camera may require a longer shutter period, leaving a shorter period for the projector to project each frame, but since the ambient lighting level is lower, this may suffice for the projected image to remain visible.
- the projectors have an inherent response time that is too slow to generate such an output, they can be shuttered using a mechanical shutter (or electronic shutter additional to the projector). With this arrangement, there may be no need to generate mask signals to prevent projected light from falling on the actors.
- a system incorporating this invention may also include one or more projectors that project images onto non-chromakey surfaces, such as the studio wall behind the camera. This allows the actor to see the virtual environment when looking in directions other than towards the cyclorama. The rendering of such images needs to T B 03 /0 109 4
- the actor head tracking system can be modified to track only the head of this actor. This could be implemented, for example, by using a human operator to identify the actor for whom the projected images should be optimised.
- the tracking system could be switched between actors during the scene as required, according to which actor is carrying out the most important interactions with virtual objects. More sophisticated implementations, for example that incorporate a facial recognition system into the actor head tracker, are also possible.
- An alternative method is to render the scene separately for each actor viewpoint, and project the scenes in different colours. Each actor can then be told to look at, for example, the 'red scene' or the 'green scene'. Some assistance could be provided to the actors by the use of coloured contact lenses to filter out the scene being projected for the other actors.
- the invention has been described in the context of a conventional virtual studio system, where the final image seen by the viewers is derived directly from the studio camera image, using a keyer to mix real and virtual elements.
- the images from the auxiliary cameras can be used to create a high-quality 3D model of the actors which is incorporated in the 3D model of the scene, and the complete scene may then be rendered to produce the final output.
- the studio camera 2 20, the broadcast-standard renderer 26 and the chroma-keyer 27.
- the invention can be applied for any purpose where it is necessary to present a view of a virtual scene on a screen to a person whilst simultaneously being able to use a keying system such as a chroma key system to separate that person from the screen.
- a keying system such as a chroma key system to separate that person from the screen.
- One such application is immersive telepresence. Telepresence systems currently exist in which the participant views one or more screens in front of them, and is seen by one or more cameras viewing them from the front. References to (virtual) studios in this application are intended to encompass any environment arranged so that images can be taken by a camera, and thus are intended to include applications such as telepresence or teleconferencing as well as film or broadcast studios.
- a coloured background screen will usually be placed behind them so that the participant may be separated from the background by chroma keying.
- the background scene may be assumed to be static, and the participant can then be separated from the background by the use of a keying process such as image differencing.
- a keying process such as image differencing.
- immersive telepresence or the like may be simplified.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Studio Devices (AREA)
- Studio Circuits (AREA)
- Processing Or Creating Images (AREA)
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
GB0206214 | 2002-03-15 | ||
GB0206214A GB2386489B (en) | 2002-03-15 | 2002-03-15 | Virtual studio system |
PCT/GB2003/001094 WO2003079694A2 (en) | 2002-03-15 | 2003-03-14 | Virtual studio system |
Publications (1)
Publication Number | Publication Date |
---|---|
EP1486057A2 true EP1486057A2 (de) | 2004-12-15 |
Family
ID=9933091
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP03712354A Withdrawn EP1486057A2 (de) | 2002-03-15 | 2003-03-14 | Virtuelles studiosystem |
Country Status (4)
Country | Link |
---|---|
EP (1) | EP1486057A2 (de) |
AU (1) | AU2003216819A1 (de) |
GB (1) | GB2386489B (de) |
WO (1) | WO2003079694A2 (de) |
Families Citing this family (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN100399799C (zh) * | 2004-08-24 | 2008-07-02 | 西安宏源视讯设备有限责任公司 | 虚拟演播室系统中辅助摄像机光图形接力跟踪方法 |
GB2418827B (en) | 2004-09-28 | 2010-11-10 | British Broadcasting Corp | Method and system for providing a volumetric representation of a 3-Dimensional object |
GB2437575A (en) * | 2006-04-27 | 2007-10-31 | Andrew Peter Phelan | Audience-specific image display |
DE102007041719B4 (de) * | 2006-11-06 | 2010-01-28 | Bauhaus-Universität Weimar | Verfahren zur Erzeugung erweiterter Realität in einem Raum |
GB2537826A (en) * | 2015-04-23 | 2016-11-02 | Ambx Uk Ltd | Image capture system |
DE102021106488A1 (de) | 2020-12-23 | 2022-06-23 | Arnold & Richter Cine Technik Gmbh & Co. Betriebs Kg | Hintergrund-Wiedergabeeinrichtung, Hintergrundwiedergabesystem, Aufnahmesystem, Kamerasystem, digitale Kamera und Verfahren zum Steuern einer Hintergrund-Wiedergabeeinrichtung |
CN114173020A (zh) * | 2021-12-31 | 2022-03-11 | 北京德火科技有限责任公司 | 一种应用于多虚拟演播室的前后景分离方法及系统 |
Family Cites Families (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5353074A (en) * | 1992-05-22 | 1994-10-04 | The Walt Disney Company | Computer controlled animation projection system |
FR2714741B1 (fr) * | 1993-12-30 | 1996-03-08 | Atea | Procédé, plan optique et dispositif pour l'incrustation d'images de grandes dimensions sur un décor réel à échelle humaine. |
EP0878099B1 (de) * | 1996-02-01 | 2001-07-04 | RT-Set Ltd. | Farbstanzen-studiosystem |
JP3047820B2 (ja) * | 1996-08-22 | 2000-06-05 | 凸版印刷株式会社 | 映像表示システム |
GB2321814B (en) * | 1997-01-28 | 2001-02-14 | British Broadcasting Corp | Video imaging |
GB2323733A (en) * | 1997-03-25 | 1998-09-30 | Orad Hi Tech Systems Limited | Virtual studio projection system |
GB9911935D0 (en) * | 1999-05-21 | 1999-07-21 | British Broadcasting Corp | Tracking of moving objects |
JP2002042158A (ja) * | 2000-07-28 | 2002-02-08 | Toshiba Corp | 画像合成装置および画像合成方法およびプログラムを記録した媒体 |
CA2438739C (en) * | 2001-02-16 | 2009-12-29 | Imatte, Inc. | Interactive teleconferencing display system |
-
2002
- 2002-03-15 GB GB0206214A patent/GB2386489B/en not_active Expired - Fee Related
-
2003
- 2003-03-14 WO PCT/GB2003/001094 patent/WO2003079694A2/en not_active Application Discontinuation
- 2003-03-14 AU AU2003216819A patent/AU2003216819A1/en not_active Abandoned
- 2003-03-14 EP EP03712354A patent/EP1486057A2/de not_active Withdrawn
Non-Patent Citations (1)
Title |
---|
See references of WO03079694A2 * |
Also Published As
Publication number | Publication date |
---|---|
AU2003216819A1 (en) | 2003-09-29 |
WO2003079694A2 (en) | 2003-09-25 |
GB2386489B (en) | 2006-10-04 |
WO2003079694A3 (en) | 2003-12-31 |
AU2003216819A8 (en) | 2003-09-29 |
GB0206214D0 (en) | 2002-05-01 |
GB2386489A (en) | 2003-09-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7468778B2 (en) | Virtual studio system | |
EP1138159B1 (de) | Bildkorrekturverfahren zur kompensation von blickpunktbildverzerrungen | |
US10204452B2 (en) | Apparatus and method for providing augmented reality-based realistic experience | |
Grau et al. | A combined studio production system for 3-D capturing of live action and immersive actor feedback | |
US7898504B2 (en) | Personal theater display | |
US8045060B2 (en) | Asynchronous camera/projector system for video segmentation | |
US20110109720A1 (en) | Stereoscopic editing for video production, post-production and display adaptation | |
KR20070006692A (ko) | 깊이 정보를 통하여 캡쳐 장치 설정을 최적화하는 방법 및기구 | |
EP2724542A1 (de) | Verfahren und vorrichtung zum erzeugen eines signals für eine anzeige | |
WO2011031968A1 (en) | Virtual insertions in 3d video | |
WO2009112967A1 (en) | Method and apparatus for modifying a digital image | |
US4695130A (en) | Camera flash attachment for three dimensional imaging systems | |
CN113692734A (zh) | 用于采集和投影图像的系统和方法,以及该系统的应用 | |
Hamasaki et al. | Hysar: Hybrid material rendering by an optical see-through head-mounted display with spatial augmented reality projection | |
US11443676B1 (en) | Increasing resolution and luminance of a display | |
EP1486057A2 (de) | Virtuelles studiosystem | |
JP2009141508A (ja) | テレビ会議装置、テレビ会議方法、プログラムおよび記録媒体 | |
WO2004099825A3 (en) | Phase retardance autostereoscopic display | |
Wojdala | Challenges of virtual set technology | |
GB2399248A (en) | Projection of supplementary image data onto a studio set | |
Bimber et al. | Digital illumination for augmented studios | |
Kim et al. | AR timewarping: A temporal synchronization framework for real-Time sensor fusion in head-mounted displays | |
Grau | Studio production system for dynamic 3D content | |
Someya et al. | Blending Shadows: Casting Shadows in Virtual and Real using Occlusion-Capable Augmented Reality Near-Eye Displays | |
Kuechler et al. | Imperceptible Projection Blanking for Reliable Segmentation within Mixed Reality Applications. |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
17P | Request for examination filed |
Effective date: 20040923 |
|
AK | Designated contracting states |
Kind code of ref document: A2 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LI LU MC NL PT RO SE SI SK TR |
|
AX | Request for extension of the european patent |
Extension state: AL LT LV MK |
|
17Q | First examination report despatched |
Effective date: 20100628 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN |
|
18D | Application deemed to be withdrawn |
Effective date: 20140930 |