Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/222—Studio circuitry; Studio devices; Studio equipment
  • H04N5/2224—Studio circuitry; Studio devices; Studio equipment related to virtual studio applications
• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
  • G09G5/00—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators
  • G09G5/02—Control arrangements or circuits for visual indicators common to cathode-ray tube indicators and other visual indicators characterised by the way in which colour is displayed
  • G09G5/026—Control of mixing and/or overlay of colours in general
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/222—Studio circuitry; Studio devices; Studio equipment
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/222—Studio circuitry; Studio devices; Studio equipment
  • H04N5/2222—Prompting
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/222—Studio circuitry; Studio devices; Studio equipment
  • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
  • H04N5/272—Means for inserting a foreground image in a background image, i.e. inlay, outlay
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N9/00—Details of colour television systems
  • H04N9/64—Circuits for processing colour signals
  • H04N9/74—Circuits for processing colour signals for obtaining special effects
  • H04N9/75—Chroma key
• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
  • G09G2340/00—Aspects of display data processing
  • G09G2340/10—Mixing of images, i.e. displayed pixel being the result of an operation, e.g. adding, on the corresponding input pixels
• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
  • G09G2340/00—Aspects of display data processing
  • G09G2340/12—Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
  • G09G2340/00—Aspects of display data processing
  • G09G2340/12—Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
  • G09G2340/125—Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels wherein one of the images is motion video
• • G—PHYSICS
  • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
  • G09G—ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION
  • G09G2340/00—Aspects of display data processing
  • G09G2340/14—Solving problems related to the presentation of information to be displayed

Definitions

• the present invention pertains generally to video studio systems and in particular to a system that realistically integrates the background and foreground in the chroma key studio. Further, the present invention pertains to studio prompting which make chroma key studios and virtual sets more tangible, thereby making it easier for the actor to find his place within the blue screen environment.
• Chroma-keying and virtual set technology enable actors in a studio to appear as if they are in a more complex environment. Using this technology, scenes comprised of a number of sources, such as live video and computer graphics can be created and combined together.
• the actor stands in front of a plain blue screen in the studio.
• the television camera shoots both the actor (live video) and the blue screen.
• the resulting picture is then transferred to a chroma- keyer, for processing.
• a background picture from a different source (such as another camera, pre-taped video or computer graphics) is transferred to the chroma-keyer.
• Both pictures, the live and the background picture, are combined in the chroma-keyer and broadcast as one picture which shows the live video on the background.
• the final result can be a weather-forecaster standing in front of a weather map which cannot be seen at all in the physical studio.
• the chroma-keyer can differentiate between the live video and the blue background, according to the pixels. Wherever a pixel from the live video is identified, it is transferred to the combined broadcast picture. Wherever a pixel from the blue screen is identified, the appropriate pixel from the background is placed into the broadcast picture. In this way, a new picture is created and broadcast, using the background which was chosen.
• the current method to achieve a shadow with the chroma key depends upon the use of a strong light source to make a strong shadow of the foreground object (actors). Since shadows on a blue screen appear in a slightly different color than the blue screen itself, selecting a very narrow range of key color cuts the blue out, leaving a shadow as a semi-transparent foreground.
• This method forces the use of a middle course in lighting the set and finding a precise chroma key color. That is, to get an accurate background color, the entire set must be illuminated with a strong light. The use of a strong light, however, makes the shadow ofthe foreground object weak. A strong contrast between the shadow and the chroma key color is desired so that the chroma keyer will not cut it off the edges ofthe shadow as blue background. This conflict, lighting versus chroma key ends up with a compromised poor picture.
• An additional problem encountered in integrating a foreground with a virtual background is contour. The background in a virtual set is three dimensional, but the method currently used to create a shadow results in a shadow in the shape ofthe set used in the studio.
• an object filmed in a blue studio without a wall will have a shadow on the studio floor.
• the shadow 10 obtained using a keyer therefore, will be in the shape ofthe floor, see Figure 1. If the foreground object 12 is combined with a virtual background having a wall 14, the shadow ofthe object will pass under the wall and destroy a realistic impression.
• the actor finds himself in a virtual space, which he does not actually see during shooting ofthe scene.
• the actor must therefore imagine the set and practice long hours in order to point in the right direction or avoid walking into virtual walls, people and other objects which are part ofthe set.
• a monitor which displays the integrated picture for the actor.
• This solution causes the actor to focus his gaze on places which are not necessarily part of the scene.
• the weather forecaster may need to point to a map on the wall in back of him, but in order to see the feedback on the monitor regarding the location of his finger, he is required to look straight ahead.
• a flash of light which is synchronized with the cameras.
• the present invention describes a system which uses a dedicated camera positioned as the light source to capture a foreground object's contour and generate a shadow.
• the shadow follows both the foreground object's contour and conforms to the shape ofthe virtual objects that are within the set.
• a virtual shadow therefore, is generated which results in a high quality integration.
• a system is also described whereby the main objects of a set can be projected on the floor or on the studio walls in real time. This enables actors to easily find their way around the set, and to focus on the virtual objects appearing in or disappearing from the studio.
• the projected images are either projected during blanking intervals ofthe camera or projected in a chroma color.
• An additional system which uses multiple cameras to identify the location of an object in a three dimensional virtual image.
• the location ofthe object is used to determine which virtual images are located in front of the object and which virtual images are located behind the object.
• the present invention also provides a method and apparatus whereby the main objects ofthe set and special markings will be projected or screened on the floor or on the studio walls in real time. This will enable the actor to easily find his way around the set, and to focus his gaze on the virtual objects appearing in or disappearing from the studio.
• the present invention gives the actor the timings for various events taking place in the studio.
• the special markings are completely invisible to the camera and to the home viewers because they are only active during blanking intervals ofthe video camera signal such as the vertical blanking interval or the horizontal blanking interval in an NTSC, PAL, SECAM or other video signal format.
• Figure 1 is a diagram of a foreground object having a flat shadow
• Figure 2 is a system block diagram and process flow ofthe present invention
• Figure 3 is a video studio incorporating the system of Figure 1 ;
• Figure 4 is a computer and virtual background image;
• Figure 5 is a video image ofthe virtual background image of Figure 4 and a virtual shadow to create a complete image;
• Figure 6 is a system block diagram and process flow for locating a foreground object;
• Figure 7 is a video studio inco ⁇ orating the system of Figure 6;
• Figure 8 shows a prompting system block diagram and process flow; and
• Figure 9 shows use ofthe prompting system of Figure 6 in a blue-screen studio environment.
• Detailed Description ofthe Invention In the following detailed description ofthe preferred embodiments, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration specific preferred embodiments in which the inventions may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized and that logical, mechanical and electrical changes may be made without departing from the spirit and scope ofthe present inventions. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope ofthe present inventions is defined only by the appended claims.
• the term "foreground” as used herein refers to objects which are filmed in front of a blue screen and combined with another video image. Although the objects are filmed in the foreground with respect to the blue screen, the foreground objects may ultimately be a background image in a completed video.
• chroma keying requires a foreground object, such as an actor, to be filmed in front of a plain blue screen in a studio. A video camera is used to film both the foreground object and the blue screen. The resulting picture is then transferred to a chroma-keyer, for processing. The blue background is replaced, pixel-by-pixel, with a background picture from a different source.
• One such source can be a computer which generates background graphics, thereby, providing a three dimensional virtual background which is substituted for the blue screen.
• the virtual background image is generated according to the camera position and its field of view.
• a current background image is calculated according to the camera position data.
• a virtual shadow ofthe foreground object is generated and combined with the virtual background.
• the shadow shape and position must be known.
• a second dedicated camera (shadow camera) is used to capture the shape ofthe foreground object.
• the shadow camera is positioned in the studio as a light source. That is, the second camera captures the object from the light source point of view.
• the image captured by the shadow camera is used by a chroma keyer to generate the shape ofthe virtual shadow.
• a computer program By mapping each pixel ofthe shadow camera's output to the three dimensional virtual background, a computer program is able to project the shadow image on the background database graphics in the proper position. Since the shadow created is a synthesized shadow, the shadow intensity and transparency levels are parameters established by the computer program and therefore can be controlled by a director in real time.
• FIG. 2 and 3 illustrate the system used to create a virtual shadow with chroma keying.
• a main camera 100 is used to capture an image of foreground object, or talent, 102 in front of a blue screen 104.
• the resultant image is processed by a chroma keyer to isolate the foreground object from the blue screen and produce image 108.
• the foreground image is time delayed while the shadow is being generated.
• a virtual scene 1 10 is created on a computer 109 of the desired background image, see Figure 4.
• This image is referred to as a virtual background or virtual set database.
• the image may contain a three dimensional object such as a wall 101.
• a shadow camera 106 is used to capture an image of the foreground object 102 in front ofthe blue screen 104 from the point of view of a virtual light source.
• the shadow camera therefore, captures an image different than the image captured by the main camera 100.
• the output image from the shadow camera is processed by a chroma keyer to isolate the foreground object from the blue screen.
• the shadow camera's foreground image is processed with the virtual set data base to produce a completed background image 1 1 1 1 having both the virtual set and the virtual shadow 112, see Figure 5.
• the completed background image 1 1 1 is combined with the delayed foreground image 108 as captured by the main camera 100 to produce a composite image 113.
• the time delay of the main camera image is dependant upon the processing time required for combining the virtual shadow with the virtual set.
• the processing time and the time delay must be synchronized so that the shadow and the foreground image match.
• a video system using the virtual shadow technique described above can be combined with a system for locating images of foreground objects relative to images in a virtual scene. Foreground Object Location
• a three layer video may be created where the first layer (layer 1 ) consists of a computer graphics background, a second layer (layer 2) consisting of a foreground object or live video hiding parts ofthe first layer, and a third layer (layer 3) consisting of additional computer graphics, hiding parts of both layer 1 & layer 2.
• layer 1 consists of a computer graphics background
• second layer consisting of a foreground object or live video hiding parts ofthe first layer
• a third layer layer 3 consisting of additional computer graphics, hiding parts of both layer 1 & layer 2.
• the location of the foreground object in the virtual space must be known. A method and system are described below which provides the location, or depth, of a foreground object in the 3-D virtual world.
• the system includes three cameras 120 positioned in a triangle pointing to the center ofthe stage to capture the contour ofthe foreground object 102 from three different directions.
• a computer operating according to a program calculates the volumetric image ofthe foreground object and its location in the virtual space. Once the location of the foreground object in the set is known, the multiple layers of a video can be assembled in appropriate positions.
• a three dimensional background image can be created according to the main camera 100 position and its field of view, see Figure 4. That is, a virtual environment, or 3-D Set, database is created using a computer, as explained above. For each frame in the video, the current background image is calculated according to the main camera position image.
• Each of the three cameras 120 see the foreground object 102 as a two dimensional image.
• the contour image of the foreground object remains. This shape represent the physical volume from the camera's point of view to the stage surface.
• the approximate location ofthe object within the virtual background can be obtained using the cross-section ofthe overlapping volumes.
• the cross section ofthe overlapping volumes represents the object volumetric image.
• a depth location of the object on the stage can be obtained.
• the depth location allows a depth value (Z value) to be assigned to each pixel ofthe object's image.
• one of the three video cameras 120 can be the main video camera 100 and that the above described shadow camera 106 can also be used as one ofthe two remaining cameras 120.
• the video system described above including one or both ofthe virtual shadow or object location can be combined with a system for prompting actors in a blue screen environment.
• Actor Prompting The video systems, as described above, can be combined with a method and apparatus to project the main objects ofthe virtual set in a chroma keying environment for prompting actors with audio or visual cues.
• Special markings are projected or screened on to the floor 103 or on the studio walls 104 in real time. This enables the actor 102 to easily find his or her way around the virtual set, and to focus his or her gaze on the virtual objects appearing in or disappearing from the studio.
• the markings also give the actor the timings for various events taking place in the studio.
• the markings are invisible to the home viewer because they are projected in a chroma-key color.
• the special markings are completely invisible to both the first camera and to the home viewers because they are only active during blanking intervals ofthe video camera signals such as the vertical blanking interval or the horizontal blanking interval in an NTSC, PAL, SECAM or other video signal format.
• the horizontal blanking and vertical blanking intervals of a television signal correspond to the retrace period for the video or television CRT in which the electron beam is repositioned to begin tracing a new line or a new field, respectively.
• the blanking interval is also known as the retrace time.
• the method consists of first creation ofthe virtual scene on a computer.
• the prompting guide for chroma keying is used to create a virtual set on a computer using a sketch ofthe set and the important objects in it. The places where the actor must be found at various times are indicated in the computer model.
• the feedback to the actor can be screened in the studio with the help of a prompting device 122 which can be either a light projector, a projector and video, or a laser that is controlled by the computer.
• a prompting device 122 can be either a light projector, a projector and video, or a laser that is controlled by the computer.
• Table 1 shows some ofthe timing constraints for a PAL video signal in which the video prompts are projected within the blanking intervals.
• High resolution laser can draw approximately 100 dots in one frame, which is enough for either small animations at different locations on the set or much larger static drawings.
• Figure 8 shows the system block diagram A Genlock device is used to synchronize the components to a common video signal timing. All studio cameras are synchronized to the Genlock as well as the laser or light projector.
• the computer generates the prompts that the projector will display or flash during the blanking interval.
• the foreground and background video sources are combined by the chroma keyer to produce the composite video.
• the resulting video will not show the prompting from the projector since the prompting only appeared during the blanking intervals.
• the latency ofthe human eye allows the actors on the set to see the flashing prompts, but the video equipment does not contain such latency so the prompts will not be picked up.
• Figure 9 shows a typical blue-screen chroma key set typically used for reporting the weather on television.
• the actor 120 stands before a blue screen 104 in and the prompting device 122 projects prompting images onto the floor 103 and onto the blue screen to tell the actor where to stand and point.
• the main video camera 100 picks up the image 124, but the image does not contain the prompts.
• a second background video image of a map 128 is picked up by another video camera 130 to produce a background image 126 which is combined with the foreground image 124 to produce the composite video image 132 devoid of any prompting images.
• a system and method have been described which allow a foreground object to be integrated with a virtual background image through the use of a virtual shadow.
• the shadow is generated using an image captured by a shadow camera.
• the shadow camera is positioned with respect to the object so that an image is captured from the prospective of a light source.
• this image is synthesized with the virtual background image to provide a background image which includes a shadow of the foreground object. Generating this synthesized shadow into the virtual background gives a good link to the real object on the foreground and the best way to integrate these two layers into one image using chroma keying technology.
• a system and method have been described for locating an object in a three dimensional virtual environment.
• the system uses three video cameras to capture a video image of an object in front of a blue screen.
• the images are processed by a chroma keyer to isolate the object image.
• the three images and the location ofthe cameras are used to identify where the image from the main camera is relative to virtual objects.
• the video system can be combined with actor prompting techniques to provide a complete filming environment.
• Special markings are projected or screened on the floor or on the studio walls in real time. The special markings are projected in either a chroma-key color or are only active during virtual blanks ofthe camera such as the vertical blanking interval or the horizontal blanking interval in an NTSC, PAL, SECAM or other video signal format.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Physics & Mathematics (AREA)
• Computer Hardware Design (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Studio Circuits (AREA)
• Processing Of Color Television Signals (AREA)

Priority Applications (1)

Applications Claiming Priority (5)

Related Child Applications (1)

Publications (2)

Family

ID=27082227

Family Applications (2)

Family Applications Before (1)

Country Status (6)

Families Citing this family (16)

Citations (4)

Family Cites Families (5)

• 1997
  • 1997-01-29 WO PCT/IB1997/000219 patent/WO1997028654A2/en active IP Right Grant
  • 1997-01-29 EP EP99203804A patent/EP0993204B1/de not_active Expired - Lifetime
  • 1997-01-29 EP EP97903548A patent/EP0878099B1/de not_active Expired - Lifetime
  • 1997-01-29 CA CA002244467A patent/CA2244467C/en not_active Expired - Lifetime
  • 1997-01-29 DE DE69705499T patent/DE69705499T2/de not_active Expired - Lifetime
  • 1997-01-29 ES ES97903548T patent/ES2158495T3/es not_active Expired - Lifetime
  • 1997-01-29 AU AU18074/97A patent/AU1807497A/en not_active Abandoned
  • 1997-01-29 ES ES99203804T patent/ES2273457T3/es not_active Expired - Lifetime
  • 1997-01-29 DE DE69736638T patent/DE69736638T2/de not_active Expired - Lifetime

Patent Citations (5)

Non-Patent Citations (3)

Also Published As

Similar Documents

Legal Events