EP1486057A2 - Systeme de studio virtuel - Google Patents

Systeme de studio virtuel

Info

Publication number
EP1486057A2
EP1486057A2 EP03712354A EP03712354A EP1486057A2 EP 1486057 A2 EP1486057 A2 EP 1486057A2 EP 03712354 A EP03712354 A EP 03712354A EP 03712354 A EP03712354 A EP 03712354A EP 1486057 A2 EP1486057 A2 EP 1486057A2
Authority
EP
European Patent Office
Prior art keywords
image
participant
camera
background
studio
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP03712354A
Other languages
German (de)
English (en)
Inventor
Graham Alexander Thomas
Oliver Grau
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
British Broadcasting Corp
Original Assignee
British Broadcasting Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by British Broadcasting Corp filed Critical British Broadcasting Corp
Publication of EP1486057A2 publication Critical patent/EP1486057A2/fr
Withdrawn legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/2224—Studio circuitry; Studio devices; Studio equipment related to virtual studio applications
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/265—Mixing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/272—Means for inserting a foreground image in a background image, i.e. inlay, outlay

Definitions

  • the present invention relates to virtual studio systems, particularly but not exclusively chroma-keying studio systems.
  • Chroma-keying is a technique that allows synthetic images to be created by determining if a pixel in a camera image belongs to foreground or background based on colour.
  • the foreground pixels can then be keyed into a second image, e.g. a computer-generated or static background as in virtual studios.
  • the background in the studio must have the particular keying colour, typically blue. This is usually achieved by painting the studio floor and visible walls with the keying colour.
  • Another option is to use a retro-reflective cloth and an active illumination ring around the camera lens as proposed in our earlier GB-A-2,321,814 the entire disclosure of which is incorporated herein by reference.
  • Chroma-keying is a preferred application, the present invention is also applicable to other keying techniques (for example using temporally modulated lighting of the background to distinguish foreground and background).
  • the term "participant” is intended to encompass without limitation any actor or presenter (human or animal) or indeed any form of animate entity which might benefit from a view of a virtual scene.
  • One problem for a participant in a virtual studio is to interact with the objects in the virtual set that are not visible for him or her.
  • a virtual object e.g. a virtual character
  • eye-line problem there is often a difference between the direction in which the participant looks, and the position of the virtual character that he should be looking at. This can be quite disturbing in the final programme.
  • This prior art visual cue system for virtual studios does not however deal directly with the eye-line problem, i.e. the case that a participant has to look towards a portion of a virtual object that is in space (for example the face of a virtual actor) but merely gives the participant an idea of the position of objects in the studio.
  • This system tracks the position of the viewer's head and computes an image for that particular view-point of the viewer, which is then projected onto a large screen forming one wall of the environment.
  • Several screens are usually used to provide a wide field-of-view, and some installations provide an all-round view using six projection screens that completely surround the viewer. Therefore it is possible to present objects that appear virtual in space.
  • the head position of the viewer is usually tracked using a special device, e.g. a helmet with an electro-magnetic transmitter.
  • a helmet with an electro-magnetic transmitter Such a system is of course not designed for nor suited to a virtual studio environment for a number of reasons.
  • the head-tracking device would be visible in any final programme and, more fundamentally, virtual studios which may have a combination of "real" objects at arbitrary positions and require a background of a key colour do not provide the open space with large screens on which images can be projected at will as this system requires.
  • the present invention is specifically concerned with a system which is suited for use in a virtual studio i.e. having a background which must be keyed out of a camera image.
  • the invention provides a method of providing a visual representation of a scene to a participant in a studio, the studio being arranged to be viewed by a camera to provide a camera image and having at least a portion of background object arranged to be keyed out of the camera image by a keying process, the method comprising: obtaining an estimate of the viewpoint of the participant; rendering an image of the scene based on the viewpoint of the participant; projecting at least a portion of the rendered image onto the background object to provide a projected image portion visible to the participant, wherein the projected image portion is projected so as to be keyed out of the camera image by the keying process.
  • the eye-line problem present with the static outline method of the prior art can be alleviated.
  • the image is projected so as to be keyed out by the keying process.
  • the estimate of the viewpoint of the participant will typically be based on a measure of position of the participant.
  • the background object may comprise a static object, typically the studio background. Additionally or alternatively, the background object may comprise an object in the studio, optionally a movable object, which is to be replaced with a virtual object. For example, a card, picture or block may have a graphic superimposed on it during a production. This may be done in addition to or instead of replacing the background and/or other objects with virtual objects.
  • the background object being static, it is desirable to store data concerning the geometry of the object.
  • a movable background object it is desirable to receive an input of a measure of the background object position.
  • a movable background object may have an active tracking device integrated therein.
  • the active tracking device may be conventional for example a radio or ultrasound transmitter or receiver which in combination with one or more external complementary receivers or transmitters in the studio enable the position to be determined using one of a number of conventional techniques.
  • Such an object may be independently provided in a further aspect of the invention which provides an object having an at least partially retro-reflective surface for reflecting key light to a camera to enable the object to be keyed out of the camera image, the at least partially retro-reflective surface also serving to render visible a projected image, the object having integrated therein means for co-operating with position determining apparatus for determining the position of the object in a studio.
  • the means for co-operating may comprise a collection of identified markers, such as circular bar codes as described in our GB-A-2,325,807, the position determining apparatus comprising a camera and processing apparatus as described in that document.
  • the means for c ⁇ -operating may comprise an active tracking device, for example including at least one transmitter or receiver.
  • the image is preferably projected from at least one direction different from the direction of view of the camera; this may enable a participant facing away from the camera to see background detail which would not be visible to the camera and, particularly with a retro-reflective background, this reduces the amount of the projected image reaching the camera.
  • the keying process is a chroma-keying process and the image is projected so that the colour of the background portion as seen by the camera is substan t ially a background colour.
  • the keying process is not affected by the image.
  • the background retro-reflects a portion of light with a first, relatively high, reflectivity and diffusely reflects light with a second, relatively low, reflectivity.
  • This is an important preferred feature, as it enables a good clean key colour to be obtained by retro-reflection of the key light with any scattered light from the projected image being of significantly lower intensity.
  • the background may be lit with key light from a direction substantially coincident with the camera direction so that key light is retro-reflected to the camera with a first, relatively high, intensity and the projected image is projected from a different direction.
  • the direction is preferably chosen so that the projected image is not retro-reflected to the camera over a range of expected camera movement.
  • the position will depend on the properties of the background; typically the background may "retro-reflect" a substantial portion of light up to a few degrees off the axis of incidence, with the reflectivity falling sharply at greater angles and becoming diffuse reflection (suitable cloth and typical properties are discussed in our GB-B-2,321,814, and GB-B-2,321,565 incorporated herein by reference - preferred features of the background discussed in those patents may be used as preferred features of the background in this application).
  • the image is preferably projected so as to be visible to the participant based on diffuse reflection.
  • the intensity required can simply be judged literally by eye based on ambient studio lighting conditions.
  • the first reflectivity is at least double the second reflectivity, preferably at least ten times the first reflectivity.
  • the key light may comprise light of a selected key colour, in a chroma-keying method.
  • the key light may comprise light having a selected temporal variation.
  • the method may further comprise masking a portion of the projected image based on the position of the participant to reduce projection onto the participant.
  • This method may be provided independently in a second aspect which provides a method of projecting an image onto a background in a space in which a participant is positioned, the method comprising rendering an image based on the position of the participant, the method further comprising masking a portion of the projected image based on the position of the participant to reduce projection onto the participant.
  • the method may comprise masking a portion of the projected image based on the position of an object to reduce projection onto the object.
  • the invention provides a method of projecting an image onto a background in a space in which an object is positioned, the method comprising dynamically rendering the image, the method further comprising dynamically masking a portion of the projected image based on the position of the object to reduce projection onto the object.
  • the masked portion is based on a stored or estimated outline of the participant or other object. This may facilitate masking. Additionally or alternatively, the mask may be based on one or more camera images of the participant or object.
  • a mask signal is generated to mask a portion of the image; this facilitates processing.
  • the rendering process may take into account any required masking so that rendering is not performed, or a reduced level of rendering is performed in regions to be masked.
  • the outline is larger than the actual outline of the participant; this reduces the risk of stray projection onto the participant (or object).
  • the outline may be enlarged along an axis of motion of the participant.
  • the background retro-reflects a portion of light and the background is at least partially translucent. This permits back-projection of images in addition to or as an alternative to front projection.
  • the background is lit with key light from a direction substantially coincident with the camera direction so that key light is retro- reflected to the camera with a first, relatively high, intensity and the projected image is projected onto the rear of the background.
  • Back-projection will typically result in a lower intensity of transmitted light reaching the camera than the light retro-reflected under normal circumstances, but the intensities of each light may be managed if high intensity projection is used.
  • a masking process similar to that used in the case of front projection may be used for back projection to reduce stray light entering the camera.
  • the primary problem is that of light passing directly through the background, rather than reflecting off objects or participants, so the primary information to take into account is t he camera and projector positions.
  • the projector position will typically be fixed and can be stored but the camera position (or field of view) may change so that is preferably provided as an input.
  • the projected image may be at least partially masked or attenuated based on the position of a camera viewing the background to reduce the intensity of light from the projected image reaching the camera.
  • the invention provides a method of projecting an image onto a background viewed by a camera, the method comprising masking or attenuating at least one portion of the projected image based on at least one of the position of a participant or other object and the position of the camera to reduce stray light from the projected image reaching the camera.
  • the projection intensity, the key light intensity and the background properties are selected so that the intensity of the key light received at the camera is at least double, preferably at least ten times, the intensity of light received from the projected image.
  • the intensity of the projected image is selected so as to be visible by the participant but so that the scattered light from the projected image does not substantially disrupt the keying process.
  • the image may be projected during a period when the camera shutter is closed.
  • the keying process may be based on temporal modulation of the background illumination and the projected image may be temporally modulated.
  • the participant's viewpoint is preferably estimated based on the measure of position of the participant.
  • the measure of position advantageously comprises an estimate of position of a body portion of a participant and wherein the viewpoint is estimated based on a physical relationship of the body portion to the eyes of the participant.
  • Most preferably the position of the head of the participant is estimated.
  • the methods disclosed in our GB-A-2,352,899, incorporated herein by reference, may be used as preferred methods of estimating position.
  • An estimate of the position of the head of the participant may be used as an estimate of the viewpoint of the participant; this simplifies the process while remaining sufficiently accurate for most purposes.
  • the method may include filtering the estimate of viewpoint to avoid sudden changes or to reject estimates which display unexpected motion or unexpected positions (for example a sudden increase in height or a height greater than the height of the participant). This may be advantageous particularly when viewpoint is estimated based on a camera image as such a method may produce occasional spurious results if, for example, an actor's arm moves above or past his head.
  • the participant position may be determined based on an image from at least one position information camera.
  • the at least one position information camera comprises at least one auxiliary camera. Additionally or alternatively the (studio) camera providing the studio camera image may also serve as a (main or supplementary) position information camera.
  • the image is projected from a plurality of sources, preferably at at least two distinct locations. This may reduce masking of portions of the background by the participant or by objects in the set. Images from at least two sources at least partially overlap.
  • the intensity of at least one image may be modulated or (additionally or alternatively) a portion of at least one image may be masked to reduce intensity variation in a region of overlap.
  • the invention provides a method comprising rendering an image for projection onto a virtual studio background, the studio having a participant therein, the method comprising rendering the image from a viewpoint based on positional information concerning the participant, the method further comprising masking a portion of the image based on the position of the participant.
  • rendering preferably includes receiving information concerning the position of at least one image projector.
  • Rendering may include receiving information concerning the studio geometry and/or adjusting the image based on background geometry.
  • Information concerning at least one of:- projector location, studio geometry, background geometry, objects within the studio, participant dimensions and camera position is preferably stored.
  • the method further comprises projecting at least two images to be separately viewable by two participants.
  • the images may be projected onto separate background object portions. However, more commonly the images may at least partially overlap.
  • the images are multiplexed and the participants are provided with demultiplexing means so that each participant views preferentially a single image.
  • This may be provided independently in a further aspect which provides a method of providing images for at least two viewers, the method comprising projecting a first image for a first viewer and a second image for a second viewer, the images at least partially overlapping and being multiplexed in at least a region of overlap and providing each viewer with respective demultiplexing means to provide each viewer with a preferential view of one image.
  • the images may be colour multiplexed and each viewer or participant may be provided with demultiplexing means comprising a respective colour filter, preferably in the form of a coloured contact lens.
  • the images may be polarisation multiplexed and each viewer or participant may be provided with demultiplexing means comprising a polarising filter, preferably in the form of polarising glasses or a polarised contact lens.
  • demultiplexing means comprising a polarising filter, preferably in the form of polarising glasses or a polarised contact lens.
  • Contact lenses used to combat astigmatism, having an asymmetric weight are known a»d such can be provided with a polarising filter.
  • the invention further independently provides a contact lens having an asymmetric weight distribution and a polarising filter.
  • the invention further provides use of a selectively transparent, preferably coloured and/or polarised, contact lens to demultiplex a multiplexed image, preferably for one of a plurality of viewers.
  • the invention further provides a kit for optical demultiplexing of at least two multiplexed images by at least two viewers comprising a first optical apparatus (preferably a pair of (or at least one) spectacles or contact lenses) wearable by a first viewer and having a first selective optical transmission property and a second optical apparatus (preferably a pair of (or at least one) spectacles or contact lenses) wearable by a second viewer and having a second selective optical transmission property, as well as further providing use of such a kit or portion thereof to obtain at least one image from a multiplexed image.
  • a first optical apparatus preferably a pair of (or at least one) spectacles or contact lenses
  • a second optical apparatus preferably a pair of (or at least one) spectacles or contact lenses
  • the images may be temporally multiplexed and each viewer or participant may be provided with demultiplexing means comprising an optical filter having a temporally varying light transmission.
  • the temporally varying light transmission is preferably synchronised to the projectors and preferably also synchronised to the shutter of a camera viewing the scene - this may enable the filters (typically in the form of glasses) to be viewed by the camera at a time when they are transparent (or opaque) as desired.
  • the aforementioned methods preferably further comprise recording a sequence of images of the participant with a studio camera.
  • Implementations of the method will typically further comprise processing said sequence of images to remove or replace the background by the keying process, wherein the projected image is substantially removed.
  • Another aspect of the invention provides independently a computer program or computer program product or apparatus arranged for performing any methods or sub features of methods. Preferred and optional features of the method aspects may be applied to all other aspects and vice versa.
  • Another aspect of the invention further provides a projected image produced by any method.
  • Another aspect of the invention further provides a virtual studio production produced by any method.
  • Another aspect of the invention further provides a mask signal generated in a rendering method.
  • Another aspect of the invention provides image rendering apparatus, preferably for use in an above mentioned method, the apparatus comprising: means for receiving a measure of a participant position; means for storing a measure of a participant outline; means for rendering an image based on the participant position, the rendered image having at least a portion masked based on the participant outline.
  • the apparatus may include means for storing a measure of at least one projector position and/or means for storing a measure of studio geometry and/or means for determining the stored measure of participant outline based on at least one of:- a dynamic image of the participant; stored information on the participant outline; information on movement or expected movement of the participant.
  • Another aspect of the invention provides a production studio arranged for use in any method, the studio including a background and means for projecting a dynamically rendered image onto the background so as to be keyed out of the image of a studio camera by a keyer.
  • Another aspect of the invention provides a production studio comprising:- an at least partially retro-reflective background; means for determining a measure of position of a participant in the studio; means for rendering an image based on participant viewpoint; means for projecting the rendered image onto the background so as to be visible to the participant.
  • the means for projecting may comprise at least one projector for projecting onto the front of the background.
  • the means for rendering may be arranged to mask a portion of the projected image corresponding to participant position.
  • the production studio may further comprise at least one camera having means for illuminating the background with a key light and/or may further comprise a keyer for keying out the background image from a camera image of the studio.
  • a head-tracking system based only on visual information is preferred, so that the viewer need not wear any extra device. This can be achieved using the image from the main studio camera and ideally some auxiliary cameras. Since there is preferably a chromakey background available, the outline of the actor may advantageously be obtained in a robust way to facilitate head-tracking. The position of the actor's head may then be used to control the rendering of the scene, to give the actor the correct perspective of the virtual scene. This may allow a precise orientation of the gaze towards objects that are virtually present in the studio.
  • the projection system may be combined with a robust chroma- keying system in such a way as to make the projected image clearly visible to the performer without interfering with the chroma-key process. Whilst this can be achieved to some extent by using conventional blue screens with low-level projection, a more effective method is provided by the use of retro-reflective screen material and coloured (blue) light sources on each camera as described.
  • a further aspect of the invention is concerned with the projection method itself and is not limited to a virtual studio environment, but is suited thereto.
  • rear projection is used which requires a lot of space behind the screens and presents particular problems with the floor. It is often more convenient to use front projection, but this presents the problem of the projected image falling on the actor, and the actor casting shadows on the screen.
  • the shadows cast on the screen are not a major problem in application with which the present invention is concerned, as the projected images are used for providing visual cues rather than a fully-realistic virtual environment where they may be problematic — as noted elsewhere, the requirements of virtual studio production and a virtual environment simulation are quite distinct.
  • One aspect of the invention therefore includes a method to prevent the appearance of projected images on the actor by blanking the appropriate part of each projected image, preferably using information describing the actor's position and shape (although this may be simplified to a basic outline or even a rectangle).
  • Figure 1 shows the arrangement of cameras, projectors and screens in the studio;
  • Figure 2 shows how a rough model of the actor is incorporated in the scene to provide a mask;
  • Figure 3 shows a block diagram of the main elements of a system incorporating the invention.
  • the walls of the studio area 1 are composed of a retro-reflective cyclorama cloth, as described in GB-B-2,321,814, and GB-B-2,321,565 to which specific reference should be made for details of suitable cloth and camera arrangements as required and the entire contents of each of which are herein incorporated by reference.
  • One or more studio TV cameras 2, 20 view the scene. As described in the above mentioned patents, each of these is preferably fitted with a coloured light source, so that the reflective cyclorama appears brightly-lit with this colour from the viewpoint of the camera, but still appears as a neutral (typically grey) colour from other viewpoints.
  • the cloth has the property that it is retro-reflective (that is it reflects a significant portion of light back to the direction of a light source) and has a generally grey appearance otherwise.
  • Each studio TV camera may also be fitted with a camera tracking system to allow camera movement to be monitored.
  • An example of a suitable tracking system may be found in GB-A-2,325,807, the entire disclosure of which is incorporated herein by reference, but other tracking systems may be used.
  • the video signal from the studio camera, and any tracking data are passed to a renderer 26 that generates background images at broadcast TV standard.
  • the camera image is keyed over these images using a chroma-keyer 27.
  • These rendering and keying processes may also incorporate other conventional virtual studio practices such as generating mask signals to allow virtual objects to appear in front of performers.
  • One or more auxiliary cameras 3, 21 also view the scene.
  • the images from these cameras, and optionally from the main camera 2, 20 are analysed to track the position of the actor's head by a head tracker 22.
  • This tracking can be carried out using one of several known tracking methods.
  • a preferred example of an actor tracking method is described in GB-A-2,352,899, the entire disclosure of which is incorporated herein by reference.
  • the availability of the silhouette image from the actor by using chroma- keying can be used to simplify or enhance the tracking process.
  • the tracking method should ideally estimate the approximate position of the mid-point between the actor's eyes.
  • This may be approximated in practice by determining a measure of the position of a part of the actor, preferably the head or a portion thereof and determining a measure of the position of the eyes based on the position of the part and a positional relationship (either predetermined or based on supplemental information). For example, the position of the top of the head may be determined and an estimate of the eye position obtained based on stored data correlating the eye position to the head either for a range of actors or a specific actor. High accuracy is not normally critical, but can be obtained if required using available conventional positioning aids or image analysis. For example an estimate that the midpoint of the eyes is 10cm below the top of the head and centrally positioned will normally suffice for most practical purposes.
  • One or more projectors 4, 23 are mounted so as to project images onto the cyclorama walls.
  • the projectors project onto the walls from the front, although rear projection may also be used if space permits.
  • Each projector is fed from a 3D renderer 24, consisting for example of software running on a personal computer equipped with a 3D graphics accelerator.
  • Each renderer calculates a view of the virtual scene with viewing geometry to match the measured head position of the actor and the geometrical arrangement of the projector and screen. Images should be computed at a sufficient rate to give the actors an acceptable rendition of the scene, ideally at least 10 frames per second, and preferably 50Hz or more.
  • rendering software is well-known to one skilled in the art, and may be based on conventional software systems such OpenGL, Open Inventor or IRIS Performer.
  • the rendering of the image also takes account of the keystone distortion caused by the projector not projecting at right-angles onto the projection screen.
  • the constraints imposed by the physical studio arrangement will usually make the correction of such distortion impossible by using only the keystone distortion correction provided by the projector itself.
  • the image from one projector may fall on two or more non-coplanar surfaces, such as two walls or one wall and the floor.
  • the rendering software may be adapted to render different parts of the image using different viewing geometries so as to present the actor with a correctly-rendered image on all surfaces.
  • the rendering process may take into account movement of the background object, for example as detected by tracking cameras or an active tracking system.
  • a mask is calculated for each projector to mask out the light that would otherwise fall on the actor.
  • One way in which such a mask signal can be generated is by computing a rough 3D model of the actor using the video signal from the auxiliary cameras 3, 21, by a processing unit 25.
  • a technique such as shape-from-silhouette, as described in "W. Niem. Robust and fast modelling of 3d natural objects from multiple views. In SPIE Proceedings, Image and Video Processing II, volume 2182, pages 88-397, San Jose, February 1994" is suitable for this application.
  • the actor model is passed to each projector's 3d renderer and rendered as a black object to appear in the scene at a position so as to be projected onto the real actor.
  • the portion 14 of the projected image which will be projected onto the actor by the projector 15 must be masked out by placing a black object at the appropriate point in the scene.
  • a black object cannot be obtained by placing the rough 3D model of the actor in the scene exactly coincident with the real actor 10, since the real actor lies outside the viewing frustum.
  • the actor model is moved just inside the viewing frustum to position 16, and scaled such that the image of the actor model as viewed from the position of the real actor matches the part of the projected image that the actor's shadow would fall upon.
  • the translation and scaling that must be applied to the actor model may be calculated using conventional geometrical methods, given knowledge of the position of the screen and the projector, and the viewpoint of the actor. It is advantageous to place the actor model just inside the viewing frustum, since it should appear to be the frontmost object in the scene; any scene elements between the near clipping plane and the actor model will not be blanked.
  • An alternative method of blanking the appropriate portion of the projected image is to perform a separate rendering process, which renders the view of the actor model (positioned at the actual location of the actor) from the position of the projector 15, to obtain a mask signal.
  • a mask signal might show the actor as a black object against an otherwise white background.
  • This mask signal can then be used to mask out the portion of the screen on which the actor's shadow would fall, for example by placing the mask image in the OpenGL Stencil buffer.
  • the same effect could also be achieved by first rendering the scene with a viewing frustum appropriate for the actor, and then changing the viewing frustum to correspond to the viewpoint of the projector, and rending a scene containing just the actor model, but with the OpenGL . 'depth test' control disabled. This guarantees that the actor model will be rendered on top of the previously-rendered image, thereby blanking the appropriate portion.
  • a third method of masking out the actor is to generate the actor mask signal by placing an additional auxiliary camera close to the projector 15.
  • the camera should be fitted with a chromakey light source if a retro-reflective screen is being used.
  • a chromakeyer can then be used to obtain a key signal. It will generally be necessary to apply some scaling or other affine transformation in order to obtain good alignment between the key signal and the image projected by the projector.
  • the key signal can then be used to mask out the actor, for example by placing the mask image in the OpenGL Stencil buffer as in the method described above.
  • This method has the advantage that it is not necessary to compute a 3D model of the actor, since the outline of the actor from the point-of-view of the projector is captured directly.
  • the mask signal is made to appear slightly larger than the actor, to account for the distance that the actor might move in the time required to generate and render the mask signal, and to compensate for any inaccuracies in the position of the generated mask from other causes such as calibration errors. It may also be desirable to estimate the speed of movement of the actor and move the mask further along the motion trajectory to the predicted position of the actor at the time that the mask will be projected. These methods could also be combined, by enlarging the actor mask only in the direction of the actor's movement. This processing could be implemented in the same processing unit 25 that generated the actor model.
  • the actor mask will of course obscure some elements of the projected scene, which could potentially contain scene elements that the actor needs to see.
  • more than one projector can be used to project onto each surface, projecting from different positions.
  • the projectors may be specifically placed based on the scene layout to ensure adequate coverage in important areas, for example based on consideration of masked areas, or may be generically placed in a studio.
  • the masked-out portion of one image will generally not coincide with the masked-out area in another, except for regions where the actor is very close to the screen. This approach will result in most areas of the screen being of double the brightness. If this is considered to be a problem in a particular embodiment, then mask signals can be derived using the above-mentioned geometric techniques to ensure that a maximum of one (or other desired number) projector illuminates any given part of the screen.
  • projectors may be selectively used so that projectors that are nearly coincident with (for example within the tolerance of the scatter angle of the background cloth) the direction of view of an active camera are suppressed.
  • One example of another such projection system is one that uses bright projectors and a conventional blue screen, where the projectors have a short display period, which is timed to occur during a period when the camera shutters are riot open.
  • TV cameras with a 50Hz field rate can be shuttered to 1/100* of a second exposure time, and the projectors can be arranged only to project light for the l/100 th second when the camera shutters are closed.
  • Different proportions can be used, for example depending on ambient lighting. For example, in dimly lit scenes, the camera may require a longer shutter period, leaving a shorter period for the projector to project each frame, but since the ambient lighting level is lower, this may suffice for the projected image to remain visible.
  • the projectors have an inherent response time that is too slow to generate such an output, they can be shuttered using a mechanical shutter (or electronic shutter additional to the projector). With this arrangement, there may be no need to generate mask signals to prevent projected light from falling on the actors.
  • a system incorporating this invention may also include one or more projectors that project images onto non-chromakey surfaces, such as the studio wall behind the camera. This allows the actor to see the virtual environment when looking in directions other than towards the cyclorama. The rendering of such images needs to T B 03 /0 109 4
  • the actor head tracking system can be modified to track only the head of this actor. This could be implemented, for example, by using a human operator to identify the actor for whom the projected images should be optimised.
  • the tracking system could be switched between actors during the scene as required, according to which actor is carrying out the most important interactions with virtual objects. More sophisticated implementations, for example that incorporate a facial recognition system into the actor head tracker, are also possible.
  • An alternative method is to render the scene separately for each actor viewpoint, and project the scenes in different colours. Each actor can then be told to look at, for example, the 'red scene' or the 'green scene'. Some assistance could be provided to the actors by the use of coloured contact lenses to filter out the scene being projected for the other actors.
  • the invention has been described in the context of a conventional virtual studio system, where the final image seen by the viewers is derived directly from the studio camera image, using a keyer to mix real and virtual elements.
  • the images from the auxiliary cameras can be used to create a high-quality 3D model of the actors which is incorporated in the 3D model of the scene, and the complete scene may then be rendered to produce the final output.
  • the studio camera 2 20, the broadcast-standard renderer 26 and the chroma-keyer 27.
  • the invention can be applied for any purpose where it is necessary to present a view of a virtual scene on a screen to a person whilst simultaneously being able to use a keying system such as a chroma key system to separate that person from the screen.
  • a keying system such as a chroma key system to separate that person from the screen.
  • One such application is immersive telepresence. Telepresence systems currently exist in which the participant views one or more screens in front of them, and is seen by one or more cameras viewing them from the front. References to (virtual) studios in this application are intended to encompass any environment arranged so that images can be taken by a camera, and thus are intended to include applications such as telepresence or teleconferencing as well as film or broadcast studios.
  • a coloured background screen will usually be placed behind them so that the participant may be separated from the background by chroma keying.
  • the background scene may be assumed to be static, and the participant can then be separated from the background by the use of a keying process such as image differencing.
  • a keying process such as image differencing.
  • immersive telepresence or the like may be simplified.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Studio Circuits (AREA)
  • Studio Devices (AREA)
  • Processing Or Creating Images (AREA)

Abstract

Le système décrit permet de donner une rétroaction à deux participants dans un studio virtuel. Une image est rendue du point de vue du participant concerné et projetée de sorte qu'elle soit visible pour le participant tout en étant découpée de l'image de la caméra du studio par un procédé d'incrustation.
EP03712354A 2002-03-15 2003-03-14 Systeme de studio virtuel Withdrawn EP1486057A2 (fr)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
GB0206214A GB2386489B (en) 2002-03-15 2002-03-15 Virtual studio system
GB0206214 2002-03-15
PCT/GB2003/001094 WO2003079694A2 (fr) 2002-03-15 2003-03-14 Systeme de studio virtuel

Publications (1)

Publication Number Publication Date
EP1486057A2 true EP1486057A2 (fr) 2004-12-15

Family

ID=9933091

Family Applications (1)

Application Number Title Priority Date Filing Date
EP03712354A Withdrawn EP1486057A2 (fr) 2002-03-15 2003-03-14 Systeme de studio virtuel

Country Status (4)

Country Link
EP (1) EP1486057A2 (fr)
AU (1) AU2003216819A1 (fr)
GB (1) GB2386489B (fr)
WO (1) WO2003079694A2 (fr)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN100399799C (zh) * 2004-08-24 2008-07-02 西安宏源视讯设备有限责任公司 虚拟演播室系统中辅助摄像机光图形接力跟踪方法
GB2418827B (en) 2004-09-28 2010-11-10 British Broadcasting Corp Method and system for providing a volumetric representation of a 3-Dimensional object
GB2437575A (en) * 2006-04-27 2007-10-31 Andrew Peter Phelan Audience-specific image display
DE102007041719B4 (de) * 2006-11-06 2010-01-28 Bauhaus-Universität Weimar Verfahren zur Erzeugung erweiterter Realität in einem Raum
GB2537826A (en) * 2015-04-23 2016-11-02 Ambx Uk Ltd Image capture system
DE102021106488A1 (de) 2020-12-23 2022-06-23 Arnold & Richter Cine Technik Gmbh & Co. Betriebs Kg Hintergrund-Wiedergabeeinrichtung, Hintergrundwiedergabesystem, Aufnahmesystem, Kamerasystem, digitale Kamera und Verfahren zum Steuern einer Hintergrund-Wiedergabeeinrichtung
CN114173020A (zh) * 2021-12-31 2022-03-11 北京德火科技有限责任公司 一种应用于多虚拟演播室的前后景分离方法及系统

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5353074A (en) * 1992-05-22 1994-10-04 The Walt Disney Company Computer controlled animation projection system
FR2714741B1 (fr) * 1993-12-30 1996-03-08 Atea Procédé, plan optique et dispositif pour l'incrustation d'images de grandes dimensions sur un décor réel à échelle humaine.
EP0993204B1 (fr) * 1996-02-01 2006-09-06 Vi[Z]Rt Ltd Système d'incrustation couleur en studio
JP3047820B2 (ja) * 1996-08-22 2000-06-05 凸版印刷株式会社 映像表示システム
GB2321814B (en) * 1997-01-28 2001-02-14 British Broadcasting Corp Video imaging
GB2323733A (en) * 1997-03-25 1998-09-30 Orad Hi Tech Systems Limited Virtual studio projection system
GB9911935D0 (en) * 1999-05-21 1999-07-21 British Broadcasting Corp Tracking of moving objects
JP2002042158A (ja) * 2000-07-28 2002-02-08 Toshiba Corp 画像合成装置および画像合成方法およびプログラムを記録した媒体
JP2004525560A (ja) * 2001-02-16 2004-08-19 アイマット・インコーポレーテッド 対話式遠隔会議表示システム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See references of WO03079694A2 *

Also Published As

Publication number Publication date
GB0206214D0 (en) 2002-05-01
GB2386489A (en) 2003-09-17
AU2003216819A1 (en) 2003-09-29
GB2386489B (en) 2006-10-04
WO2003079694A2 (fr) 2003-09-25
WO2003079694A3 (fr) 2003-12-31
AU2003216819A8 (en) 2003-09-29

Similar Documents

Publication Publication Date Title
US7468778B2 (en) Virtual studio system
EP1138159B1 (fr) Procede de correction d'images destine a compenser la distorsion de celles-ci en fonction du point de vue
US10204452B2 (en) Apparatus and method for providing augmented reality-based realistic experience
Grau et al. A combined studio production system for 3-D capturing of live action and immersive actor feedback
US7898504B2 (en) Personal theater display
US7136090B1 (en) Communications system
US8045060B2 (en) Asynchronous camera/projector system for video segmentation
US20110109720A1 (en) Stereoscopic editing for video production, post-production and display adaptation
KR20070006692A (ko) 깊이 정보를 통하여 캡쳐 장치 설정을 최적화하는 방법 및기구
WO2012176109A1 (fr) Procédé et appareil pour générer un signal pour un écran d'affichage
EP2476259A1 (fr) Insertions virtuelles dans une vidéo en 3d
EP2255339A1 (fr) Procédé et appareil permettant la modification d'une image numérique
Zhong et al. Reproducing reality with a high-dynamic-range multi-focal stereo display
Hamasaki et al. Hysar: Hybrid material rendering by an optical see-through head-mounted display with spatial augmented reality projection
US11443676B1 (en) Increasing resolution and luminance of a display
JP2022529417A (ja) 画像取得投影システム、該システムの使用及び画像取得投影挿入方法
EP1486057A2 (fr) Systeme de studio virtuel
JP2009141508A (ja) テレビ会議装置、テレビ会議方法、プログラムおよび記録媒体
WO2004099825A3 (fr) Afficheur autostereoscopique a retard de phase
JP3914728B2 (ja) プレゼンテーションシステム
Wojdala Challenges of virtual set technology
GB2399248A (en) Projection of supplementary image data onto a studio set
CN113206991A (zh) 全息显示方法、系统、计算机程序产品及存储介质
Bimber et al. Digital illumination for augmented studios
Grau Studio production system for dynamic 3D content

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20040923

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IT LI LU MC NL PT RO SE SI SK TR

AX Request for extension of the european patent

Extension state: AL LT LV MK

17Q First examination report despatched

Effective date: 20100628

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20140930