WO2006108279A1 - Method and apparatus for virtual presence - Google Patents

Method and apparatus for virtual presence Download PDF

Info

Publication number
WO2006108279A1
WO2006108279A1 PCT/CA2006/000551 CA2006000551W WO2006108279A1 WO 2006108279 A1 WO2006108279 A1 WO 2006108279A1 CA 2006000551 W CA2006000551 W CA 2006000551W WO 2006108279 A1 WO2006108279 A1 WO 2006108279A1
Authority
WO
WIPO (PCT)
Prior art keywords
virtual
user
computer
representation
virtual representation
Prior art date
Application number
PCT/CA2006/000551
Other languages
French (fr)
Inventor
Martin Lizee
Robert Lizee
Original Assignee
Martin Lizee
Robert Lizee
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to US66990605P priority Critical
Priority to US60/669,906 priority
Application filed by Martin Lizee, Robert Lizee filed Critical Martin Lizee
Publication of WO2006108279A1 publication Critical patent/WO2006108279A1/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/003D [Three Dimensional] image rendering
    • G06T15/10Geometric effects
    • G06T15/20Perspective computation
    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/02Accessories
    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/02Accessories
    • A63F13/06Accessories using player-operated means for controlling the position of a specific area display
    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F13/00Video games, i.e. games using an electronically generated display having two or more dimensions
    • A63F13/20Input arrangements for video game devices
    • A63F13/21Input arrangements for video game devices characterised by their sensors, purposes or types
    • A63F13/213Input arrangements for video game devices characterised by their sensors, purposes or types comprising photodetecting means, e.g. cameras, photodiodes or infrared cells
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, TV cameras, video cameras, camcorders, webcams, camera modules for embedding in other devices, e.g. mobile phones, computers or vehicles
    • H04N5/2224Studio circuitry; Studio devices; Studio equipment ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, TV cameras, video cameras, camcorders, webcams, camera modules for embedding in other devices, e.g. mobile phones, computers or vehicles related to virtual studio applications
    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/10Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals
    • A63F2300/1087Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals comprising photodetecting means, e.g. a camera
    • A63F2300/1093Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game characterized by input arrangements for converting player-generated signals into game device control signals comprising photodetecting means, e.g. a camera using visible light
    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/80Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game specially adapted for executing a specific type of game
    • A63F2300/8029Fighting without shooting
    • AHUMAN NECESSITIES
    • A63SPORTS; GAMES; AMUSEMENTS
    • A63FCARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
    • A63F2300/00Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game
    • A63F2300/80Features of games using an electronically generated display having two or more dimensions, e.g. on a television screen, showing representations related to the game specially adapted for executing a specific type of game
    • A63F2300/8082Virtual reality

Abstract

A virtual presence system as well as a method and computer-readable storage medium are disclosed for providing an immersive, responsive and physically intuitive virtual experience to a user thereof. An image capture device may be used to capture a user image from which a virtual representation of the user may be generated and displayed. In general, the virtual representation comprises a mirrored silhouette transformation of the user image such that the virtual representation and the user share a substantially same orientation. A position and motion of the user may also be calculated to position and move the virtual representation in a given virtual environment. A virtual camera may also be generated such that the virtual representation is displayed from a perspective of this virtual camera. This perspective may also be altered in response to a given user motion or displacement. Motions and actions may be imparted to the virtual camera and/or the virtual representation in the virtual environment by evaluating, for example, variations in the size and position of the user's image.

Description

TITLE OF THE INVENTION

[0001] METHOD AND APPARATUS FOR VIRTUAL PRESENCE

CROSS REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority on U.S. provisional application no. 60/669,906, filed on April 11 , 2005, which is herein incorporated by reference.

FIELD OF THE INVENTION

[0003] The present invention relates to a method and apparatus for virtual presence. In particular, the present invention relates to a method and apparatus for the interactive immersion of a silhouette image on a display screen of a user interacting therewith.

BACKGROUND OF THE INVENTION

[0004] Virtual reality systems, including some video games and other such virtual environment systems use real-time motion capture devices to translate the presence of a system user, or player, in the virtual environment of the system.

[0005] For instance, in US Patent No. 5,616,078 for a Motion-Controlled User Entertainment System by Oh for Konami Co. Ltd. and issued on April 1st, 1997, a user wears a set of detectable markers to map his/her motion on a virtual character that will exhibit similar motion in a virtual environment projected on a display screen. The markers are captured by an input device and the captured signals thereof are analyzed by a computing device to generate a skeleton used in animating the virtual character. Such a technique applies a heavy computation load on the system that affects the responsiveness of the character in the virtual environment and/or requires expensive and specialized equipment to be efficient.

[0006] In US Patent No. 6,863,609 for a Method for Controlling Movement of Viewing Point of Simulated Camera in 3D Video Game and 3D Video Game

Machine by Okuda et al. for Konami Co. Ltd. and issued on March 8th, 2005, a video game player' head is detected by an ultrasonic device or a CCD camera that approximates the position of the player's eyes and translates that position to guide the viewing point of the simulated camera in the game. In such a system, the player may only control the viewing point of a character in the game by moving his/her head and is not particularly immersed in the game.

[0007] In International Patent Application No. WO 2004/072909 A1 for a Method and Apparatus for Real Time Motion Capture by Marks for Sony Computer Entertainment Inc. and published on August 26th, 2004 [hereinafter referred to as the Sony application], a more efficient method for controlling a game character in real time is presented. Yet, to represent a human player with plural joints and moving members, a significant computation load is still applied to the system. Furthermore, though real-time motion capture is used to represent the virtual character on a screen, the disclosed method does not explore the use of a virtual camera to enhance the immersion effect.

[0008] In addition, the Sony application discloses the use of a mirror captured image to create a representation of the player in the game. As a result, the player sees a mirror representation of him/herself that feels unnatural and requires significant adaptation from the player since the character on screen and the player share a same left and right side but not a same orientation. Ultimately, the character on the screen is looking at the player, like in a mirror, but not where the player is looking. [0009] As it is known in the art, the player controlled character in a third- person 3D game (or an avatar in a virtual environment) is more intuitive to the player when the virtual camera used to generate the display is behind the controlled character because the controlled character and the player perception of the display share the same direction, i.e. left is left, right is right, forward is forward and backward is backward, both on screen and using the controls. Yet, since conventional capture systems involve a front facing capture camera, conventional techniques using captured images from these systems can't be implemented without first simulating the skeleton of the controlled character, as with the Antigrav game published by Sony on the EyeToy.

[0010] As it is also known in the art, the lag time due to processing involved in the representation of a motion controlled character (or avatar) has a crucial effect on the quality and effectiveness of the virtual experience. When the lag time increases, the responsiveness of the character decreases and the quality of the virtual experience is significantly reduced.

[0011] As a result, a system is needed to provide greater user immersivity with reduced computation needs in order to achieve a maximum level of responsiveness and detail in the virtual representation of the user in the virtual environment. The present invention, described herein and with reference to the appended illustrative drawings, provides a method and apparatus for interactive presence, and alternative embodiments thereof, that overcome the above and other drawbacks of prior art virtual presence systems.

SUMMARY OF THE INVENTION

[0012] More specifically, in accordance with the present invention, there is provided a virtual presence system for generating on a display a virtual representation of a user positioned in front of the display, the system comprising an image capture device for capturing an image of the user; and a computing device, the computing device being configured to communicate with the image capture device to generate the virtual representation using the user image and communicate the virtual representation to the display. The virtual representation comprises a mirrored silhouette transformation of the user image such that the virtual representation shares a substantially same orientation as the user.

[0013] Also in accordance with the present invention, there is provided a method for immersing a user positioned in front of a display in a virtual environment rendered on the display, the method comprising the steps of:

acquiring an image of the user;

generating a virtual representation of the user, the virtual representation comprising a mirrored silhouette transformation of the user image; and

displaying the virtual representation in the virtual environment.

[0014] Further in accordance with the present invention, there is provided a computer-readable storage medium having a computer-readable program embodied therein for directing operations of a system comprising an image capture device and a data processor in communication with the image capture device and a display, wherein the computer-readable program comprises instructions for operating the system to immerse a user of the system positioned in front of the display in a virtual environment rendered on the display in accordance with the following: acquiring an image of the user via the image capture device; generating, using the processor, a virtual representation of the user by applying a mirrored silhouette transformation to the user image; and

displaying the virtual representation in the virtual environment on the display.

[0015] Still further in accordance with the present invention, there is provided a computer-implemented method for generating on a display a virtual representation of a user positioned in front of the display, the method comprising the steps of:

acquiring an image of the user via an image capture device;

automatically generating the virtual representation of the user by implementing a mirrored silhouette transformation of the user image; and

communicating the virtual representation to the display.

BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Having thus generally described the nature of the invention, reference will now be made to the accompanying drawings, showing by way of illustration, illustrative embodiments thereof, and in which:

[0017] Figure 1 is a diagram illustrating the basic configuration of a virtual presence system in accordance with an illustrative embodiment of the present invention; [0018] Figure 1A is a diagram of an image captured by an image capture device in the system of Figure 1 ;

[0019] Figure 1 B is a representation of a silhouette image extracted and mirrored from the captured image of Figure 1A;

[0020] Figure 1C is a representation of a screen image presented on a display screen in the system of Figure 1 incorporating the silhouette image of Figure 1 B therein;

[0021] Figure 1 D is a representation of a virtual environment corresponding to the screen image of Figure 1C;

[0022] Figure 2A is a diagram illustrating a user of the system of Figure 1 at five successive positions relative to an image capture device thereof;

[0023] Figure 2B is a representation of a virtual environment incorporating successive silhouette models of the user at virtual positions respectively corresponding to the five positions of Figure 2A;

[0024] Figure 2C is a diagram illustrating a captured image of the user at each successive position of Figure 2A and their respective silhouette image.

[0025] Figure 3 is a diagram illustrating a first use of a virtual camera in the system of Figurei to frame a specific region of the successive silhouette images of Figure 2B in the virtual environment of Figure 1C;

[0026] Figure 4 is a diagram illustrating a second use of a virtual camera in the system of Figurei to frame a specific region of the successive silhouette images of Figure 2B in the virtual environment of Figure 1C; [0027] Figure 5 is a first exemplary screen display of the illustrative embodiment of Figure 1;

[0028] Figure 6 is a second exemplary screen display of the illustrative embodiment of Figure 1 and Figure 7 is a general overview of an exemplary data processing system and method to be implemented by the computing device in accordance with an illustrative embodiment of the present invention

DETAILED DESCRIPTION QF THE ILLUSTRATIVE EMBODIMENTS

[0029] Referring to Figure 1 , a virtual presence system, generally referred to using the reference numeral 10 and in accordance with an illustrative embodiment of the present invention, will now be presented. The virtual presence system 10 generally comprises an image capture device 12 for capturing images of a user, as in reference 14 of Figure 1A, a display 16 comprising a display screen 18, and a computing device 19 for interpreting the captured images 14 provided by the image capture device 12 and communicating a corresponding image, as in reference 20 of Figure 1B1 for display on the screen 18. As a result, the processed image 20, derived from the captured images 14 of the user, Is provided on the screen 18 (Figure 1 C) and virtually immersed in the virtual environment 22 of the system 10, as illustrated in Figure 1 D.

[003O] To implement the system, various image capture devices 12 may be considered. For instance, the image capture device may comprise a digital video camera coupled to a screen grabber software or hardware interacting with and/or in the computing device 19, an analog camera providing a continuous video feed to an analog-to-digital (A/D) converter used in conjunction with the computing device 19, as well as any other conventional or specialized imaging device such as infra-red (IR) devices, thermal imaging devices, sonar devices and the IiKe. Other such imaging devices will be apparent to a person of ordinary skill in the art. [0031] Furthermore, various display devices 16 can also be considered to project a combined image, as in Figure 1C. For instance, the display could comprise an LCD screen, a CRT screen, a plasma screen, or other such conventional or specialized display screens. In addition, one or more image capture devices, as in 12, may be coupled with one or more display devices, as in 16, to enhance the desired visualization effect.

[0032] Also, the computing device 19 may include a personal computer (PC) such as a desktop or laptop PC, a game console, or other such computing and interactive image processing and rendering devices. Ultimately, the image capture device(s) 12 and the display device(s) 16 work cooperatively through the computing device 19, which may implement various image processing and/or combination instructions and algorithms via a single or a series of computer-readable programs stored, maintained, embedded or loaded therein, to provide the desired visual effect of an immersive 3D virtual environment 22. To do so, the computing device 19 may comprise various components such as various static and removable ROMs and other such computer-readable storage media (e.g. CD, DVD, CD-R, CD-RW, DVD-R, DVD-RW, etc.) to store and provide the various software algorithms and instructions used to create, control and display the virtual environment 22 as well as generate, control and display the virtual representation of the user, as in 24, in this virtual environment 22. The computing device 19 may also comprise one or more processors, possibly interacting with active RAM, to implement and control the various software and hardware components, instructions and programs, and other such components that should be readily apparent to a person of skill in the art.

[0033] The computing device 19 may also further comprise device inputs and outputs for connecting thereto, for example, the image capture device 12, the display device 16, various interactive controllers and other such peripherals optionally required for a given application. Namely, a video game application may require user inputs from a game controller, such as the ubiquitous game pad, or other such controllers including, but not limited to, joysticks, interactive shooting devices, and the like. Other virtual reality applications, such as virtual training and simulation systems (e.g. military training, flight simulations, etc.) may require the use of various types of pointers, a keyboard, a scrolling device or other such user input devices. As will become apparent to a person of skill in the art, various other device configurations may be considered without departing from the general scope and nature of the present disclosure.

[0034] With particular reference now to Figures 1 C and 1 D, the image 20 projected on the screen 18 is illustratively represented by a translucent mirrored silhouette transformation (hereinafter referred to as a silhouette image 20) of the captured image 14. In other words, the captured image 14 is isolated by the computing device 19, inverted from left to right about a vertical axis, and transformed into a translucent silhouette image 20 (Figure 1 B) for display on the screen 18 (Figure 1C).

[0035] Using this approach, the user and the silhouette image 20 share a substantially same orientation. That is, the orientation of the user in the real world is substantially transferred to the silhouette image 20 on the screen 18 (Figure 1 C), and conceptually in the virtual environment 22 (Figure 1 D). Consequently, the right, left, forward and backward directions of the user in the real world (Figure 1) are substantially the same in the virtual environment 22 for the virtual representation 24 of the user therein, hereinafter also illustratively referred to as the silhouette model 24 (in general the terms virtual character, virtual player, virtual model and silhouette model may all be used herein to refer to the virtual representation 24 of the user in the virtual environment 22). Consequently, the manoeuvrability of the silhouette model 24 within the virtual environment 22, whether to move around or to selectively interact with various virtual objects 23 therein, can be more intuitive than with conventional systems. Generally, a greater feeling of immersion is attained increasing user comfort and dexterity with the system 10.

[0036] As will be apparent to a person of skill in the art, various image processing and combination software and hardware algorithms may be implemented by the computing device 19 to produce similar effects. Namely, a user image may first be extracted from the captured image 14 and then transformed into a silhouette representation thereof for subsequent reversal and combination on the screen 18. As will be discussed further hereinbelow, other image rendering and transfer techniques including, but not limited to, image outlines, graphical overlays, shadows, 3D meshes and mappings and the like, may also be used to produce a similar effect. For the sake of simplicity, the following discussion will concentrate on the use of a silhouette image, as in 20 of Figure 1 B, to represent the user via silhouette model 24 in the virtual environment 22, though the above and other user visualization techniques may be considered therein without extending the scope and general nature of the present disclosure.

[0037] To generate the silhouette image 20, various image processing and isolation techniques and methods may be considered. For example, the silhouette image 20 may be obtained from the captured image 14 using a difference matte. This technique requires that a background image be captured without the user. The silhouette image 20 is then computed once the user is in place in front of the matte by detecting color differences of the pixels respectively corresponding to the background image and the captured image 14. A detected color difference greater than a predefined threshold will identify a pixel that is to be part of the silhouette image 20, thereby isolating the image of the user from the background. A person skilled in the art will understand that any of the other conventional or proprietary difference matte algorithms nnay be considered to achieve a similar result.

[0038] In a further example, the generation of the silhouette image 20 could be obtained from the captured image 14 using chroma keying. This method requires a uniform color (usually a blue screen) to be placed behind the user. Or again, the generation of the silhouette image 20 could be obtained from the captured image 14 using luma keying that is based on the use of specific lighting to isolate the user's image. Various image processing software and/or algorithms are readily available to implement these and other such conventional and specialized methods and should be apparent to a person of ordinary skill in the art. Also, other image identification and isolation methods and techniques, optionally using various other imaging devices such as described hereinabove, can be considered without departing from the general scope and nature of the present disclosure.

[0039] With reference now to Figure 1 D, the silhouette model 24, possibly representing a virtual character in a video game or a video entertainment system, is conceptually defined by the projection 26 of the silhouette image 20 (Figure 1 B) on a virtual panel 27 in the virtual environment 22, the back of the silhouette model 24 referring to the side on which the silhouette image 20 is projected. Figuratively, the projected image 26 is viewed by a virtual camera 28 positioned behind the silhouette model 24 and providing a rear view thereof such that the user and the silhouette model 24 share a substantially same orientation.

[004O] As illustrated by Figures 2A and 2B, the projection 26 is adapted to compensate for the displacement of the user relative to the image capture device 12, which displacement is also used to move the silhouette model 24 in the virtual environment 22. For instance, as the user moves from positions A to E in the real world (Figure 2A), the position of the silhouette model 24 moves from corresponding positions A' to E' in the virtual environment 22 (Figure 2B), yet the size of the silhouette remains the same. Additionally, as the user moves from side to side, the silhouette model 24 can also move accordingly.

[0041] In Figure 2C, the captured images 14A to 14E and their respective silhouette images 2OA to 2OE are presented for the aforementioned positions A to E in the real world of Figure 2A. As the user moves backward from position A to E, the size of the captured and silhouette images 14 and 20 change within their respective frames 30 and 32. Yet, as illustrated in Figure 2 B, the size of the silhouette model 24 is substantially unchanged. Note however that the shape of the silhouette model 24 may still change if the shape of captured image 14 and silhouette image 20 are changed due to user motion and/or action.

[0042] Consequently, though the actual size of the silhouette image 20 may change, its projection on the silhouette model 24 will not. As will be discussed further hereinbelow, this will allow the position of the silhouette model 24 in the virtual environment 22 to change based on the size and/or position of the captured image 14 and/or the silhouette image 20 within their respective frames 30 and 32 without necessarily altering the size of the silhouette model 24 in the virtual environment 22 and on the display screen 18.

[0043] To position the silhouette model 24 within the virtual environment 22 and to adapt the projection 26 to compensate for the displacement of the user relative to the image capture device 12, an estimate distance of the user from the image capture device 12 is used. Illustratively, by comparing the number of pixels used to render the silhouette image 20 of a selected part or portion of the user at a given position (portions/parts including, but not limited to the user's head, arm, hand, torso or again any object carried by the user), with the number of pixels used to render a same part or portion at a known position, the distance between the two positions may be estimated.

[0044] Generally, the distance of an object from the image capture device 12 will be inversely proportional to the square root of the number of pixels required to image that object. Consequently, by taking the ratio between the number of pixels needed to render an object at a reference position where the distance between the image capture device 12 and the object is know or estimated and the number of pixels needed to render a same object at an unknown position and, by taking the square root of that ratio and multiplying the resulting value by the known distance, the distance of the unknown position may be estimated. An appropriate scaling factor may also be used to transfer position information to the system 10 and activate a customized displacement or an action of the virtual silhouette 24.

[0045] By implementing such a calculation in the system 10 in real-time for each captured image 12 and corresponding silhouette image 20, or again periodically at a predetermined imaging rate, the position of the user relative to the image capture device 12 may be calculated in real-time. This information can then be used to control the position and orientation of the silhouette model 24 in the virtual environment 22, and that again in real-time.

[0046] As will be apparent to a person of skill in the art, other such methods and algorithms may be used to calculate user distances and motion without departing from the general scope and nature of the present disclosure. For instance, the average number of pixels used in each scan line of the silhouette image 20, or of a predefined part or portion thereof, may be used to linearly interpolate the distance of the user from the image capture device 12. Alternatively, the distance between the user and the video capture device 12 may be computed using a tracking object having a known shape and size and identifying its relative distance to the image capture device 12, and thus the distance of the user carrying it, using conventional image recognition techniques. In this latter method, tracking object orientation, and thus user orientation, may also be calculated by tracking variations in the captured shape thereof. For example, a circular object may provide an elliptical image when the object is rotated, and thus provide a means to establish the object's orientation based on the eccentricity of the perceived ellipse.

[0047] Additional position tracking systems may also be coupled to the system to provide distance measurements to the system 10. For instance, one or more sonic and/or optical radar devices and techniques may be used in conjunction with the image capture device 12 to provide accurate distance measurements. Such complementary or supplementary positioning device(s) could be coupled to the system 10 to provide user distance and positioning data thereto to supplement data acquired via the image capture device 12. Ultimately, a single, or a combination of distance and positioning devices and techniques may be considered to implement the system 10 without departing from the general scope and nature of the illustrative embodiments.

[0048] Referring back to Figures 2A and 2B, the system 10 may use the estimated distances of the user at each position A to E in the real world (Figure 2A) to position the silhouette model 24 at corresponding positions A' to E' in the virtual environment 22 (Figure 2B). Yet, as illustrated in Figure 2C, as the user moves closer to the image capture device 12, the fraction of the user imaged thereby is reduced such that a representation thereof is limited by the frame 32 of the silhouette image 20. Consequently, a framing 34 of the virtual camera 28 (as illustrated in Figure 1 D) is also constrained by the existing portion of the silhouette model 24, that is the portion rendered by the silhouette image 20 at any given position. Consequently, if the silhouette model 24 does not include the silhouette of the model's legs, for instance when the user is at a position at which his/her legs are not visible to the image capture device 12, then the framing 34 (Figure 1 D) of the virtual camera 28 should not include the region in the virtual environment 22 where the legs of the silhouette model 24 would be, since visualization data of the silhouette model 24 is not available in this area. (As will be discussed further hereinbelow, estimated and/or extrapolated data could be used to enhance visualization of the silhouette model in areas where real-time data is unavailable.)

[0049] Furthermore, since a mirrored silhouette image 20 is used to immerse the user in the virtual environment 22, the position of the virtual camera 28 may also be constrained to view the silhouette model 24 from behind. Conversely, the orientation of the silhouette model 24 may be constrained to be seen from behind by the virtual camera 28. Yet, various image processing techniques and algorithms could be employed to soften these constraints and provide greater angling capabilities to the system 10.

[0050] Despite these constraints, the present embodiment still provides the ability to implement various camera effects by actively controlling the framing 34 and/or the conceptual position of the virtual camera 28.

[0051] Referring now to Figure 3, the framing 34 of the virtual camera 28 can be restrained to a selected portion of the user's silhouette image 20. Consequently, as the user moves from position A to E in the real world (Figure 2A), the virtual camera 28 maintains a constant framing 34. As will be readily understood by a person of skill in the art, this will provide the visual effect that the silhouette model 24 moves backward in the virtual environment 22 while a constant representation thereof, limited herein by the virtual camera framing 34, is maintained at every position.

[0052] For this effect, the virtual camera framing 34 may, for instance, be selected to encapsulate the portion of the silhouette image 20 available when the user's position relative to the image capture device 12 represents the closest position interpretable by the processing device 16. That is, for a given application, a range of distances will be interpretable and transposable to the virtual environment 22. A closest distance in that range will thus produce the largest silhouette image 20 transferable and thus the most limiting silhouette image framing 32. By limiting the virtual camera framing 34 to this same framing 32, or to a smaller section thereof, a constant visual representation of the silhouette model 24 will be possible for every user position supported by the application.

[0053] Alternatively, the user position range may be delimited by the selected virtual camera framing 34. That is, the sensitivity of the system to user motion may be calibrated by selecting a minimal virtual camera framing 34 defined to provide an adequate visualization of the silhouette model 24.

[0054] In another alternative, the frame 34 may be dynamically centered on the silhouette image 20, or a pre-selected part thereof, using a pixel centroid thereof. In other words, the system 10 could calculate a conceptual 'center of gravity' of the imaged object by evaluating the position of each imaged pixel in x and y and dynamically center the frame 34 thereon.

[0055] Again, a person of skill in the art will understand that other such frame calibration and alignment methods can be used to produce a similar effect.

[0056] Referring now to Figure 4, the framing 34 of the virtual camera 28 may also be altered dynamically to provide a zooming action. For instance, the user may use a backward motion to implement a zoom-out function in the application. As the distance of the user from the image capture device 12 increases, the virtual camera framing 34 may also increase, thereby providing a zoom-out effect that may optionally be dynamically coupled to a rearward motion of the silhouette model 24 in the virtual environment 22.

[0057] As mentioned hereinabove, the lateral motion of the user may also be used to control actions of the silhouette model 24 and the virtual camera 28 in the virtual environment 22. For instance, when the user moves to the left, the silhouette image 20 will be moved to the left side of the frame 32 thereof. By detecting this lateral displacement, the silhouette model 24 could also be correspondingly displaced in the virtual environment 22. Alternatively, a position or viewpoint of the virtual camera 28 could be moved to reflect a change in the viewing direction of the virtual silhouette 24.

[0058] As will be apparent to a person of skill in the art, various visualization control algorithms may be implemented to produce such lateral displacement effects. For example, the positioning of the silhouette model 24 could be determined as a function of a static virtual panel, as in reference 27 of Figure 1 D on which the silhouette image 20 is projected to construct the silhouette model 24, that moves forward and back based on the distance of the user from the image capture device 12. The lateral position of the virtual silhouette 24 could then be modified dynamically as the position of the silhouette image 20 moves laterally within the framing 32 thereof.

[0059] Alternatively, a dynamic virtual panel could follow the movement of the virtual silhouette 24. That is, based on the lateral position of the silhouette image 20 within its framing 32, a panel or model cutout could be simulated in the virtual environment 22 to reflect motion of the user in the real world. Though conceptually similar and providing a substantially same result, the implementation of these two algorithms may vary, each providing their own advantages and/or features.

[0060] Ultimately, the above and various other 3D positioning and visualization algorithm may be implemented to produce the desired effect, that is to control the actions of the silhouette model 24 through the motion of the user.

[0061] As such, a person skilled in the art will recognize that the typical camera movements used in 3D software, like zooming and panning can be available with this system 10, as well as other movements such as moving and dollying. To enhance the visualization of the latter two, additional software and hardware algorithms may be used to address various reorientations of the silhouette model 24 and various perspective inconsistencies attributed thereto.

[0062] Furthermore, other visual effects may be generated by the motion of the user relative to the image capture device 12. For example, the displacement of the user and/or his position relative to the video capture device 12 may be used in conjunction with state information in the virtual environment 22 to control the position and/or the orientation of the silhouette model 24. For instance, in a game setting, a user moving from the left to the right of the image capture device 12 could instruct a silhouette model 24 to orbit around an object rather than to strictly move from left to right. Alternatively, a user moving three (3) feet to the left could instruct a silhouette model 24 to move four times as much to give the user a feeling of power or speed.

[0063] Many other examples may also be considered. For instance, as the system 10 detects a motion of the user, the computing device 19 may be configured to implement various motions and/or displacements of the silhouette model 24 in the virtual environment 22, or again implement various modifications to a visualization thereof from the perspective of the virtual camera 28. For example, user arm, leg, hand, head and body motions may be mapped by corresponding motions of the virtual model 24 in the virtual environment 22. Also, user motions may be used to implement virtual displacements of the virtual model 24 such as a forward displacement, a backward displacement, an orbital displacement, a rotational displacement and a side displacement. Alternatively, user motions may be detected and used to control various virtual camera perspective modifications, which may include moving, rotating, panning, zooming, orbiting and/or dollying the virtual camera relative to the silhouette model 24 and/or the virtual environment. A person skilled in the art, namely in the art of video game and virtual reality system development, will understand that such controller mapping features and options used to generate and enhance user immersion, impressions, and experiences may be readily adapted to the system 10 without departing from the general scope and nature of the present disclosure.

[0064] In another example, one or more markers may be worn by the user to implement various visual features and options in a given application. For instance, various conventional algorithms may be used to compute the location of these markers from the image capture device 12 and use these locations to allow the user to further interact with the virtual environment 22, for instance for implementing collision detection algorithms with virtual objects, as in references 23 in Figure 1 D.

[0065] Furthermore, conventional or proprietary tracking algorithms may be used to compute the locations of specific user features, for instance the user's face, eyes and/or hands from the image capture device 12 (e.g. tracking on a shape, size and/or color thereof). These locations could then be used to enhance user interactions in the virtual environment 22, for instance again for implementing collision detection algorithms with virtual objects. In this configuration, the user's specific features (e.g. user's face, eyes, hands, etc.) could be used to act as markers, allowing the system 10 to track these features using conventional and/or modified tracking techniques and/or algorithms. This type of marker detection and tracking could also be used to implement various other actions in the virtual environment (e.g. virtual representation movement, virtual camera movement, etc.)

[0066] Also, in yet another example, the user may carry one or more objects that have distinctive features, like for instance a bright unusual color or a distinctive shape, to be used interactively in the virtual environment 22. For instance, conventional or proprietary motion capture algorithms could be used to compute the location and/or orientation of theses objects from the image capture device 12 to enhance user interactions in the virtual environment 22, such as for collision detection with virtual objects, for interactively displaying the captured objects, or for substituting these objects with virtual objects.

[0067] Additionally, the presentation of the silhouette model 24 may be modified or altered using conventional or proprietary visual effects to reflect user interactions in the virtual environment 22. These effects could include any type of visual add-ons and features, image distortions and enhancements, motion-activated visual cues and actions and the like.

[0068] Also, conventional gesture recognition algorithms and techniques may be used on the data provided by the video capture device 12 to implement user interactions with virtual objects 23 in the virtual environment 22. For instance, the silhouette image 20 could be directly used to compute 2D collisions with virtual objects whose positions are mapped in 2D to a plane, such as virtual plane 27 of Figure 1 D, determined by the position and orientation of the silhouette model 24. Collision information could then be used to implement various user interactions in the virtual environment 22. Detection of a velocity or direction of a user's motion could also be tracked to further govern the implementation of various user interactions with virtual objects 23.

[0069] For example, the computing device 19 could be configured to implement various predetermined collision actions in the virtual environment based on collision data calculated from a collision between the silhouette model 24 and a given virtual object 23 in the virtual environment 22. Such collisions could be generated from a user-controlled motion of the silhouette model 24 and/or a motion of the given object 23. Possible collision actions could include, but are not limited to, selecting, modifying, moving, removing and/or modifying the given object, or again moving or modifying the silhouette model 24, or a representation thereof, as a result of such collisions. Furthermore, the computing device 19 could be configured to detect and determine whether such a collision was initiated by a user motion or by a preprogrammed motion of a moving virtual object 23.

[007O] As can be seen, the system 10 provides a visually interesting virtual experience that, illustratively using virtual camera 28, augments the immersion sensation of the user in the virtual environment 22 with the added benefit of further deforming and displacing the projection of the silhouette image 20 on the silhouette model 24, which is ultimately seen on the display screen 18. Consequently, the system 10 contributes in hiding the origin of the silhouette image 20 and makes it easier for the user to accept it as an integral part of the virtual environment 22, rather than to perceive it as his own mirrored image. [0071] The system 10 thus provides the user with the illusion that his/her silhouette has been captured from behind and projected inside the virtual environment 22 and, through this silhouette image 20, he/she can interact in the virtual environment 22 in a natural way. Namely, since the user and the user's visual representation, illustratively provided by the user's silhouette model 24, share a same general orientation, the user is provided with a better sense of direction and can thus better interact within the virtual environment 22. This provides a very intuitive control of the virtual silhouette 24 as the user is provided with the illusion that his/her shadow is cast on the display screen 18 and can directly interact in the virtual environment 22.

[0072] In addition, the system 10 provides the user silhouette, as it would be observed from behind, while capturing the image for the generation thereof from the front of the user. This permits the system 10 to be installed conventionally, such that the user faces both the display 16 and the video capture device 12. Moreover, by positioning the image capture device 12 in front of the user, a frontal image of the user is obtained that provides greater visual cues usable in the implementation of the system 10, such as the position of the hands of the user and any object carried thereby. As will be apparent to a person of ordinary skill in the art, various visual cues such as tags, interactive pointers and the like may be used to enhance the visualization effect and facilitate interaction within the virtual environment 22.

[0073] Also, the use of a silhouette image 20 provides for the easy harmonization of the projected image 26 in a stylized virtual environment 22. For instance, it would be harder to respect the artistic direction of the virtual environment 22 if an actual image was used to overlay the virtual environment graphics. Furthermore, while true image overlays may be intimidating for some users, the use of silhouettes, while still providing an adequate identification of the user on the screen 18, provides a less threatening experience accessible to all.

[0074] As stated hereinabove and as will be apparent to a person of ordinary skill in the art, other user visualizations may be implemented in the system 10 without departing from the general scope and nature of the present disclosure. For instance, a user outline may be used instead of a silhouette. Alternatively, the user silhouette could be replaced by a graphical representation of a character in the virtual environment 22, such as the hero in a simulated game setting. Furthermore, additional objects and clothing may be added to the user representation, such as a hat, a gun, or other such accessories used to identify the character and various activated features and options in the virtual environment 22.

[0075] Other user visualizations, such as 3D mesh mappings, scaled graphical representations, image and graphical overlays and the like, ranging from the fully opaque to various degrees of transparency to allow better visualization of the virtual objects near the silhouette model 24, may also be considered. Furthermore, silhouette image enhancements including, but not limited to, image replacements and fillers to compensate for missing, corrupt or unavailable image data, image overlays providing uniform or patterned templates for user representation and identification, and other such features may also be considered.

[0076] Ultimately, the system 10 allows to generate a good, complex and responsive representation of the user in the virtual environment 22, that is the silhouette model 24, at low computational cost. Namely, since the silhouette model 24 can be generated partly using image-processing techniques and avoids the complex analysis of the captured image 14 to infer the posture of the user, typically needed with conventional virtual presence models, computational load is significantly reduced and system responsiveness increased. Furthermore, since calculation of the user's motion about the image capture device 12 is mainly implemented using pixel counting and interpretation techniques, though other techniques may be used cooperatively therewith or instead thereof, computational load is kept low.

[0077] Also, not only is computational load reduced by the above techniques, the position and posture of the user's virtual representation 24 is rendered with a higher precision than generally expected from conventional position and posture calculation techniques. By using a transformation of the user's image rather than a computed graphical representation thereof coarsely derived from positional and postural data, the posture and positioning of the virtual representation 24 is provided with enhanced detail.

[0078] As will be apparent to a person of skill in the art, other embodiments may include multi-player applications comprising plural silhouette models 24 interacting in a same virtual environment 22, either locally or on a shared network of computing devices and displays. For instance, plural users could be imaged by a single image capture device 12 and a user identification algorithm could be implemented to identify each user for immersion into the virtual environment 22. Alternatively, plural image capture devices 12 connected to one or more processing device 19 could extract individual user silhouette models 24 and combine them in a given virtual environment on one or more displays 16. Or again, a virtual environment 22 could be shared between two or more networked systems 10, wherein the silhouette model 24 of a first user of a first system 10 would appear as a virtual object in a second system for interaction with a second silhouette model of a second user, and vice versa.

[0079] Referring now to Figure 5, an exemplary screen display of the disclosed system 10 is presented. In this screen capture 50, a user interacts in a virtual environment 22 comprising a boxing arena and comprising interactive virtual objects 23 such as an opposing boxer 52. As can be seen, the silhouette model 24 controlled by the user can move within the virtual environment 22 and interact, such as punch, jab and push the virtual boxer 52. Also, the system 10 may be configured such that the opposing character 52 can punch, jab and push the silhouette model 24, thereby inducing modifications thereto (e.g. collision highlights, character reactions, etc.), as it would a conventional virtual character controlled by user in a conventional video game environment.

[0080] In this particular example, the system 10 is calibrated to provide an interactive boxing match between the user and the virtual boxer 52. To enhance the virtual boxing experience, the system 10 will not only track the position of the user, in this scenario by tracking the user's head, but also track the position of the user's hands. For instance, the user could carry colored objects in each hand, namely a bright pink object in a right hand and a bright green object in the left hand for the independent identification and tracking thereof using conventional or proprietary color tracking algorithms (colors should be sufficiently different from other colors in the camera frame to optimize identification and tracking). Alternatively, the skin color of the user's hands may suffice to provide sufficient color distinction to identify the hands of the user. The head of the user could then be independently identified and tracked by using the above-described pixel counting and interpretation techniques with the remaining pixels in the image.

[0081] To calibrate the system, that is to provide a known or estimated distance from which other user distances may be calculated, a calibration stage could, for example, ask the user to align his/her head with an on-screen marker predetermined to position an average user at a desired distance. For instance, the image of the head of an average user could coincide with the marker when the user stands at approximately four (4) feet from the image capture device 12. To calibrate the hands of the user, the user could be asked to put his/her hands substantially vertically above his/her head, or again lined- up with additional screen markers, such that the hands and head of the user are at a substantially same distance from the image capture device 12 during calibration. A predefined line at the top of the screen could guide this action such that the hands of an average user would exceed that line when standing at the pre-selected distance with his/her hands in the air. In one example, the game could be started once the image of the user's hands has been properly positioned above this calibration line.

[0082] Consequently, the illustrated game setting would comprise a three- body silhouette model defined by the user's head/body at a first position and distance, the user's right hand at a second position and distance, and the user's left hand at a third position. As a result, various options and features could be applied independently to each tracked body and coincide with the motion thereof in the virtual environment 22.

[0083] Yet, though the player interacts with the game in a 3D environment, the projected image 26 remains as a 2D representation of the user. As seen in Figure 6, which provides an exemplary screen display of the same setting as Figure 5 but from the vantage point of an additional virtual camera, the virtual camera projecting the silhouette image 20 remains behind the virtual user and need only project a 2D image thereof. When the player and the silhouette model 24 share a same orientation, the illusion of 3D virtual presence is obtained nonetheless, avoiding the heavy implementation of a full 3D virtual image.

[0084] As will be apparent to a person of ordinary skill in the art, various image enhancing and processing algorithms may be used to improve the visualization of the system 10, namely to adjust any distortion or perceptual inconsistencies associated with a plural-bodied silhouette model 24. For instance, various simulated image corrections could be implemented to address size inconsistencies between objects tracked at different relative distances from the image capture device 12. For example, in the boxing setting presented hereinabove, the images could be processed to compensate for potential perspective errors induced when the user moves his/her fists towards the video capture device. Without visual compensation, the imaged fist would become bigger when it should be getting smaller since the fist moves away from the virtual camera 28. Using visual compensation for movements of the fists relative to the body, these inconsistencies could be addressed. Yet, even without using any elaborate image processing and compensation algorithms, the system 10 will still provide a realistic immersive virtual experience for the user.

[0085] Referring now to Figure 7, a general overview of an exemplary data processing system and method to be implemented by the computing device 19, in accordance with the above illustrative embodiments, will now be presented. In general, as briefly described hereinabove, the computing device 19 comprises an image input 60 for communicating with the image capture device 12 and an image output 62 for communicating with the display 16. Optional user inputs, as in 63, may also be included to provide additional user interaction with the system 10.

[0086] Also, the computing device illustratively comprises an embedded and/or removable storage memory 64 (e.g. ROM, CD, DVD, CD-R, CD-RW, DVD-R, DVD-RW, etc.) for storing various processing software and/or algorithms, which may comprise a number of instructions for implementing various image processing, integration and immersion algorithms, as well as various data structures for rendering the virtual environment and interactions generated therein. These exemplary components may include, but are not limited to, virtual environment data, software and/or application program(s), image processing, isolation and rendering software, algorithms and/or application program(s), and virtual world image integration and immersion software, algorithms and/or application program(s).

[0087] Still referring to Figure 7, the computing device 19 also comprises a processor (e.g. CPU) and optional operating system 66 for implementing, illustratively via RAM 68, the various steps and instructions provided by the various components/program(s) stored in memory 64.

[0088] In particular, Figure 7 illustrates an exemplary succession of steps and instructions implemented by the CPU 66. In a first step, the CPU 66 acquires a captured image via the image input 60. The captured image is then isolated and inverted (e.g. background removal, silhouetting, scaling, framing, mirrored transformation). When positioning-dependent instructions and/or effects are available (virtual positioning, visualization, etc.), the CPU 66 may detect a position of the user (e.g. pixel count, marker detection, etc.) and implement these position-dependent instructions/effects. The processed image is then integrated into the virtual environment (e.g. integration of processed image with virtual environment data) and the integrated image communicated to the display 16 via the image output 62. Other optional steps may include generating various visual enhancements (e.g. image transparency, accessories, distortions, character overlays, add-ons, etc.) as well as computing various collisions between the virtual representation of the user and virtual objects in the virtual environment.

[0089] The person of skill in the art will understand that various other steps and instructions may be considered to implement the above system without departing from the general scope and nature of the present disclosure. Also, it will be understood that the above exemplary steps and instructions may be performed in various orders without significantly altering the end result. Namely, the various image processing and integration steps may be performed in a number of sequences to obtain a like effect.

[0090] It is to be understood that the invention is not limited in its application to the details of construction and parts illustrated in the accompanying drawings and described hereinabove. The invention is capable of other embodiments and of being practised in various ways. It is also to be understood that the phraseology or terminology used herein is for the purpose of description and not limitation. Hence, although the present invention has been described hereinabove by way of illustrative embodiments thereof, it can be modified, without departing from the spirit, scope and nature of the subject invention as defined in the appended claims.

Claims

\NHAT IS CLAIMED IS:
1. A virtual presence system for generating on a display a virtual representation of a user positioned in front of the display, the system comprising: an image capture device for capturing an image of the user; and a computing device, said computing device being configured to communicate with said image capture device to generate the virtual representation using said user image and communicate the virtual representation to the display; wherein the virtual representation comprises a mirrored silhouette transformation of said user image such that the virtual representation shares a substantially same orientation as the user.
2. The virtual presence system of Claim 1 , wherein said mirrored silhouette transformation provides that, when the user faces the display, the virtual representation appears to have been generated from a backside image of the user.
3. The virtual presence system of any one of Claims 1 and 2, the system further for incorporating the virtual representation in a virtual environment on the display, said computing device being further configured to incorporate the virtual representation in the virtual environment and communicate said incorporated virtual representation to the display.
4. The virtual presence system of Claim 3, said image capture device for capturing successive frontal images of the user and said computing device being further configured to detect a motion of the user via said successive user images and implement a predetermined motion of the virtual representation in the virtual environment based on said detected motion.
5. The virtual presence system of Claim 4, wherein said user motion comprises a user movement comprising at least one of an arm movement, a leg movement, a hand movement, a head movement and a body movement and wherein said predetermined motion comprises a virtual movement of the virtual representation corresponding to said user movement.
6. The virtual presence system of Claim 4, wherein said predetermined motion comprises a displacement of the virtual representation in the virtual environment comprising at least one of a forward displacement, a backward displacement, an orbital displacement, a rotational displacement and a side displacement.
7. The virtual presence system of any one of Claims 4 to 6, the virtual environment comprising at least one virtual object, said computing device further being configured to calculate a collision between the virtual representation and a given one of the at least one object and implement a predetermined collision action in the virtual environment based on said collision.
8. The virtual presence system of Claim 7, wherein said collision action comprises at least one of selecting said given object, modifying said given object, moving said given object, removing said given object, modifying the virtual representation and moving the virtual representation.
9. The virtual presence system of any one of Claims 7 and 8, said computing device being further configured to detect a motion of said given object and select said collision action based on whether said collision resulted from said motion of said given object or said motion of the virtual representation.
10. The virtual presence system of any one of Claims 7 to 9, wherein said collision is calculated from a 2D mapping of said given virtual object on a
2D plane of the virtual representation.
11.The virtual presence system of any one of Claims 3 to 10, said computing device being further configured to generate a virtual camera such that the virtual representation is displayed from a perspective of said virtual camera.
12. The virtual presence system of Claim 11 , said image capture device for capturing successive frontal images of the user and said computing device being further configured to detect a motion of the user via said successive user images and implement a predetermined camera action that modifies said perspective based on said detected motion.
13. The virtual presence system of Claim 12, wherein said virtual camera action comprises at least one of a camera moving action, a camera rotation action, a camera panning action, a camera zooming action, a camera orbiting action and a camera dollying action.
14. The virtual presence system of any one of Claims 12 and 13, wherein said virtual camera action is implemented relative to at least one of the virtual representation and the virtual environment.
15. The virtual presence system of any one of Claims 3 to 14, said computing device being further configured to calculate a position of the user relative to said image capture device, attribute a virtual position to the virtual representation corresponding to said user position and incorporate the virtual representation in the virtual environment at said virtual position.
16. The virtual presence system of Claim 15, wherein said user position is calculated based on a size of at least a portion of said user image.
17. The virtual presence system of Claim 16, wherein said portion comprises at least one of an imaged hand, an imaged head, an imaged torso and said user image in its entirety.
18. The virtual presence system of any one of Claims 16 and 17, wherein said size is determined by calculating at least one of a number of pixels used to render said portion and an average number of pixels used to render each scan line of said portion.
19. The virtual presence system of any one of Claims 15 to 18, wherein said computing system is further calibrated by defining a reference position for which a calibration image of the user satisfies a pre-selected calibration characteristic, said user position being calculated from a variation of said user image relative to said calibration image.
20. The virtual presence system of any one of Claims 3 to 14, the system further comprising a positioning device communicatively linked to said computing device, said computing device being further configured to acquire a position of the user, attribute a virtual position to the virtual representation corresponding to said user position and incorporate the virtual representation in the virtual environment at said virtual position.
21. The virtual presence system of any one of Claims 3 to 20, wherein the virtual environment comprises a visual user interface of at least one of a video game, an application program and a simulation program.
22. The virtual presence system of any one of Claims 1 to 21 , wherein the virtual representation is translucent.
23. The virtual presence system of any one of claims 1 to 22, the system further comprising at least one marker to be associated with a body part of the user, said computing device being further configured to detect a position of said marker and attribute a corresponding virtual position to a corresponding virtual body part of the virtual representation.
24. The virtual presence system of Claim 23, said computing device being further configured to detect a motion of said marker and attribute a corresponding virtual motion to said virtual body part.
25. The virtual presence system of any one of Claims 23 and 24, wherein said marker comprises said user body part and wherein said computing device is configured to detect said marker position via at least one of a colour, a shape and a size of said body part.
26.A method for immersing a user positioned in front of a display in a virtual environment rendered on the display, the method comprising the steps of: acquiring an image of the user; generating a virtual representation of the user, said virtual representation comprising a mirrored silhouette transformation of said user image; and displaying said virtual representation in the virtual environment.
27. The method of Claim 26, wherein said mirrored silhouette transformation provides that the user and said virtual representation share a substantially same orientation such that, when the user faces the display, said virtual representation appears to have been generated from a backside image of the user.
28. The method of any one of Claims 26 and 27, the method further comprising the step of defining a virtual camera, said displaying step further comprising displaying said virtual representation in the virtual environment from a perspective of said virtual camera.
29. The method of Claim 28, the method further comprising the steps of: detecting a motion of the user; and modifying said perspective as a function of said user motion.
30. The method of Claim 29, wherein said modifying step comprises at least one of moving said virtual camera, rotating said virtual camera, panning said virtual camera, zooming said virtual camera, orbiting said virtual camera and dollying said virtual camera relative to at least one of said virtual representation and the virtual environment.
31.The method of any one of Claims 26 to 30, further comprising the steps of: identifying a position of the user; and attributing a virtual position to said virtual representation corresponding to said user position; said displaying step further comprising displaying said virtual representation in the virtual environment at said virtual position.
32. The method of Claim any one of Claims 26 to 31 , the method further comprising the steps of: detecting a motion of the user; and implementing a predetermined motion of said virtual representation in the virtual environment corresponding to said detected motion.
33. The method of Claim 32, wherein said user motion comprises a user movement comprising at least one of an arm movement, a leg movement, a hand movement, a head movement and a body movement and wherein said predetermined motion comprises a virtual movement of said virtual representation corresponding to said user movement
34. The method of Claim 32, wherein said virtual motion comprises a virtual displacement of said virtual representation in the virtual environment comprising at least one of a forward displacement, a backward displacement, an orbital displacement, a rotational displacement and a side displacement.
35. The method of any one of Claims 32 to 34, the virtual environment comprising at least one virtual object, the method further comprising the steps of: calculating a collision between said virtual representation and a given one of the at least one virtual object; and implementing a predetermined collision action in the virtual environment based on said collision.
36. The method of Claim 35, wherein said collision action comprises at least one of selecting said given object, modifying said given object, moving said given object, removing said given object, modifying said virtual representation and moving said virtual representation.
37. The method of any one of Claims 35 and 36, the method further comprising the steps of: detecting a motion of said given object; determining whether said collision was initiated by said given object or said virtual representation; and selecting said collision action as a function of a result of said determining step.
38. The method of any one of Claims 35 to 37, wherein said collision is calculated from a 2D mapping of said given virtual object on a 2D plane of the virtual representation
39. The method of any one of Claims 26 to 38, wherein the method is implemented by a computer-readable program embodied in a computer- readable storage medium, said computer-readable program comprising instructions for automatically implementing the steps of the method.
40. A computer-readable storage medium having a computer-readable program embodied therein for directing operations of a system comprising an image capture device and a data processor in communication with the image capture device and a display, wherein the computer-readable program comprises instructions for operating the system to immerse a user of the system positioned in front of the display in a virtual environment rendered on the display in accordance with the following: acquiring an image of the user via the image capture device; generating, using the processor, a virtual representation of the user by applying a mirrored silhouette transformation to said user image; and displaying said virtual representation in the virtual environment on the display.
41. The computer-readable storage medium of Claim 40, wherein said mirrored silhouette transformation provides that the user and said virtual representation share a substantially same orientation such that, when the user faces the display, said virtual representation appears to have been generated from a backside image of the user.
42. The computer-readable storage medium of any one of Claims 40 and 41 , said acquiring instruction further comprising acquiring successive frontal images of the user via the image capture device, the computer readable program further comprising instructions for detecting a motion of the user via said successive user images and implementing a predetermined motion of said virtual representation in the virtual environment based on said detected motion.
43. The computer-readable storage medium of Claim 42, wherein said user motion comprises a user movement comprising at least one of an arm movement, a leg movement, a hand movement, a head movement and a body movement and wherein said predetermined motion comprises a virtual movement of said virtual representation corresponding to said user movement.
44. The computer-readable storage medium of Claim 42, wherein said virtual motion comprises a virtual displacement of said virtual representation in the virtual environment comprising at least one of a forward displacement, a backward displacement, an orbital displacement, a rotational displacement and a side displacement.
45. The computer-readable storage medium of any one of Claims 42 to 44 , the virtual environment comprising at least one virtual object, the computer-readable program further comprising instructions for calculating a collision between said virtual representation and a given one of the at least one virtual object and implementing a predetermined collision action in the virtual environment based on said collision.
46. The computer-readable storage medium of Claim 45, wherein said collision action comprises at least one of selecting said given object, modifying said given object, moving said given object, removing said given object, modifying said virtual representation and moving said virtual representation.
47. The computer-readable storage medium of any one of Claims 45 and 46, the computer program further comprising instructions for detecting a motion of said given object and implementing said collision action based on whether said collision resulted from said motion of said given object or said motion of said virtual representation.
48. The computer-readable storage medium of any one of Claims 45 to 47, wherein said collision is calculated from a 2D mapping of said given virtual object on a 2D plane of the virtual representation.
49. The computer-readable storage medium of any one of Claims 40 to 48, the computer program further comprising instructions for generating a virtual camera, said displaying instruction further comprising displaying said virtual representation from a perspective of said virtual camera.
50. The computer-readable storage medium of Claim 49, said acquiring instruction further comprising acquiring successive frontal images of the user via the image capture device, the computer-readable program further comprising instructions for detecting a motion of the user via said successive user images and implementing a predetermined virtual camera action that modifies said perspective based on said detected motion.
51. The computer-readable storage medium of Claim 50, wherein said virtual camera action comprises at least one of a camera moving action, a camera rotation action, a camera panning action, a camera zooming action, a camera orbiting action and a camera dollying action.
52. The computer-readable storage medium of any one of Claims 50 and 51 , wherein said virtual camera action is implemented relative to at least one of said virtual representation and the virtual environment.
53. The computer-readable storage medium of any one of Claims 40 and 52, the computer-readable program further comprising instructions for calculating a position of the user relative to the image capture device and attributing a corresponding virtual position to said virtual representation, said displaying instruction further comprising displaying said virtual representation at said virtual position in the virtual environment.
54. The computer-readable storage medium of Claim 53, wherein said user position is calculated from a size of a recognisable portion of said user image.
55. The computer-readable storage medium of Claim 54, wherein said portion comprises at least one of an imaged hand, an imaged head, an imaged torso and said user image in its entirety.
56. The computer-readable storage medium of any one of Claims 54 and 55, wherein said position is calculated using a pixel counting algorithm.
57. The computer-readable storage medium of Claim 56, wherein said pixel counting algorithm comprises at least one of counting a number of pixels used to render said portion and counting an average number of pixels used to render each scan line of said portion.
58. The computer-readable storage medium of any one of Claims 53 to 57, the computer-readable program further comprising instructions for establishing a reference position for which a calibration image of the user satisfies a pre-selected calibration characteristic, said calculating instruction further comprising calculating said user position relative to said reference position as a function of a variation of said user image relative to said calibration image.
59. The computer-readable storage medium of any one of Claims 40 to 52, the system further comprising a positioning device communicatively linked to said data processor, the computer-readable program further comprising instructions for acquiring a position of the user, attributing a virtual position to the virtual representation corresponding to said user position and incorporating the virtual representation in the virtual environment at said virtual position.
60. The computer-readable storage medium of any one of Claims 40 to 59, wherein the virtual environment comprises a visual user interface of at least one of a video game, an application program and a simulation program.
61. The computer-readable storage medium of any one of Claims 40 to 60, wherein said virtual representation is translucent.
62. The computer-readable storage medium of any one of claims 40 to 61 , the system further comprising at least one marker to be associated with a body part of the user, the computer-readable program further comprising instructions for detecting a position of said marker, attributing a corresponding virtual position to a corresponding virtual body part of the virtual representation, said displaying instruction further comprising displaying said virtual body part at said corresponding virtual position.
63. The computer-readable storage medium of Claim 62, said detecting instructions further comprising detecting a motion of said marker and said attributing instructions further comprising attributing a corresponding virtual motion to said virtual body part.
64. The computer-readable storage medium on any one of Claims 62 and 63, wherein said marker comprises said user body part and wherein said marker detecting instruction comprises detecting said marker via at least one of a colour, a shape and a size of said user body part.
65.A computer-implemented method for generating on a display a virtual representation of a user positioned in front of the display, the method comprising the steps of: acquiring an image of the user via an image capture device; automatically generating the virtual representation of the user by implementing a mirrored silhouette transformation of said user image; and communicating the virtual representation to the display.
66. The computer-implemented method of Claim 65, the method further for integrating the virtual representation in a virtual environment, the method further comprising the step after said generating step of integrating the virtual representation with graphical data representative of the virtual environment, said communicating step further comprising communicating said integrated virtual representation to the display.
67. The computer-implemented method of Claim 66, the method further comprising the step before said communicating step of defining a virtual camera, said integrating step further comprising integrating the virtual representation in the virtual environment such that a visualization thereof is provided from a perspective of said virtual camera.
68. The computer-implemented method of Claim 67, said acquiring step further comprising acquiring successive frontal images of the user via said image capture device, the method further comprising the steps of: detecting a motion of the user from said successive user images; and automatically modifying said perspective as a function of said user motion.
69. The computer-implemented method of any one of Claims 66 to 68, further comprising the steps of: identifying a position of the user via said image capture device; and automatically attributing a virtual position to the virtual representation corresponding to said user position; said integrating step further comprising integrating the virtual representation in the virtual environment at said virtual position.
70. The computer-implemented method of any one of Claims 66 to 68, further comprising the steps of: identifying a position of the user via a positioning device; and automatically attributing a virtual position to the virtual representation corresponding to said user position; said integrating step further comprising integrating the virtual representation in the virtual environment at said virtual position.
71. The computer-implemented method of Claim any one of Claims 66 to 70, said acquiring step further comprising acquiring successive frontal images of the user via said image capture device, the method further comprising the steps of detecting a motion of the user from said successive user images and attributing a predetermined virtual motion to the virtual representation corresponding to said detected motion, said integrating step further comprising integrating said virtual motion of the virtual representation in the virtual environment.
72. The computer-implemented method of Claim 71 , the virtual environment comprising at least one virtual object, the method further comprising the steps of: calculating a collision between the virtual representation and a given one of the at least one virtual object; and implementing a predetermined collision action in the virtual environment based on said collision.
73. The computer-implemented method of Claim 72, wherein said collision action comprises at least one of selecting said given object, modifying said given object, moving said given object, removing said given object, modifying the virtual representation and moving the virtual representation.
74. The computer-implemented method of any one of Claims 72 and 73, the method further comprising the steps of: detecting a motion of said given object; automatically determining whether said collision was initiated by said given object or the virtual representation; and automatically selecting said collision action as a function of a result of said determining step.
75. The computer-implemented method of any one of Claims 72 to 74, wherein said collision is calculated from a 2D mapping of said given virtual object on a 2D plane of the virtual representation.
76. The computer-implemented method of any one of Claims 65 to 75, wherein the method is implemented by a computer-readable program embodied in a computer-readable storage medium, said computer- readable program comprising instructions for automatically implementing the steps of the computer-implemented method.
PCT/CA2006/000551 2005-04-11 2006-04-11 Method and apparatus for virtual presence WO2006108279A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US66990605P true 2005-04-11 2005-04-11
US60/669,906 2005-04-11

Publications (1)

Publication Number Publication Date
WO2006108279A1 true WO2006108279A1 (en) 2006-10-19

Family

ID=37086568

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CA2006/000551 WO2006108279A1 (en) 2005-04-11 2006-04-11 Method and apparatus for virtual presence

Country Status (1)

Country Link
WO (1) WO2006108279A1 (en)

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
NL2004273A (en) * 2010-02-22 2010-05-19 Valeri Mischenko Embedding humans and objects in virtual reality environments.
WO2011014467A2 (en) 2009-07-29 2011-02-03 Microsoft Corporation Auto-generating a visual representation
WO2012049643A1 (en) * 2010-10-12 2012-04-19 Nokia Corporation Method and apparatus for determining motion with scalar representations of movement
EP2613281A1 (en) * 2006-12-29 2013-07-10 Qualcomm Incorporated Manipulation of virtual objects using enhanced interactive system
US8520024B2 (en) 2007-05-18 2013-08-27 Uab Research Foundation Virtual interactive presence systems and methods
US8610831B2 (en) 2010-10-12 2013-12-17 Nokia Corporation Method and apparatus for determining motion
US8818883B2 (en) 2009-07-23 2014-08-26 Apple Inc. Personalized shopping avatar
US9710968B2 (en) 2012-12-26 2017-07-18 Help Lightning, Inc. System and method for role-switching in multi-reality environments
US9886552B2 (en) 2011-08-12 2018-02-06 Help Lighting, Inc. System and method for image registration of multiple video streams
US9940750B2 (en) 2013-06-27 2018-04-10 Help Lighting, Inc. System and method for role negotiation in multi-reality environments
US9959629B2 (en) 2012-05-21 2018-05-01 Help Lighting, Inc. System and method for managing spatiotemporal uncertainty
US10150034B2 (en) 2016-04-11 2018-12-11 Charles Chungyohl Lee Methods and systems for merging real world media within a virtual world

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1249260A2 (en) * 2001-03-29 2002-10-16 Konami Corporation Video game device, video game method, video game program, and video game system
US20040085451A1 (en) * 2002-10-31 2004-05-06 Chang Nelson Liang An Image capture and viewing system and method for generating a synthesized image
WO2004072909A1 (en) * 2003-02-11 2004-08-26 Sony Computer Entertainment Inc. Method and apparatus for real time motion capture

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1249260A2 (en) * 2001-03-29 2002-10-16 Konami Corporation Video game device, video game method, video game program, and video game system
US20040085451A1 (en) * 2002-10-31 2004-05-06 Chang Nelson Liang An Image capture and viewing system and method for generating a synthesized image
WO2004072909A1 (en) * 2003-02-11 2004-08-26 Sony Computer Entertainment Inc. Method and apparatus for real time motion capture

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8559676B2 (en) 2006-12-29 2013-10-15 Qualcomm Incorporated Manipulation of virtual objects using enhanced interactive system
EP2613281A1 (en) * 2006-12-29 2013-07-10 Qualcomm Incorporated Manipulation of virtual objects using enhanced interactive system
US8520024B2 (en) 2007-05-18 2013-08-27 Uab Research Foundation Virtual interactive presence systems and methods
US8818883B2 (en) 2009-07-23 2014-08-26 Apple Inc. Personalized shopping avatar
WO2011014467A2 (en) 2009-07-29 2011-02-03 Microsoft Corporation Auto-generating a visual representation
EP2459289A2 (en) * 2009-07-29 2012-06-06 Microsoft Corporation Auto-generating a visual representation
EP2459289A4 (en) * 2009-07-29 2013-11-13 Microsoft Corp Auto-generating a visual representation
NL2004273A (en) * 2010-02-22 2010-05-19 Valeri Mischenko Embedding humans and objects in virtual reality environments.
US8610831B2 (en) 2010-10-12 2013-12-17 Nokia Corporation Method and apparatus for determining motion
WO2012049643A1 (en) * 2010-10-12 2012-04-19 Nokia Corporation Method and apparatus for determining motion with scalar representations of movement
US9952671B2 (en) 2010-10-12 2018-04-24 Nokia Technologies Oy Method and apparatus for determining motion
US10181361B2 (en) 2011-08-12 2019-01-15 Help Lightning, Inc. System and method for image registration of multiple video streams
US9886552B2 (en) 2011-08-12 2018-02-06 Help Lighting, Inc. System and method for image registration of multiple video streams
US9959629B2 (en) 2012-05-21 2018-05-01 Help Lighting, Inc. System and method for managing spatiotemporal uncertainty
US9710968B2 (en) 2012-12-26 2017-07-18 Help Lightning, Inc. System and method for role-switching in multi-reality environments
US9940750B2 (en) 2013-06-27 2018-04-10 Help Lighting, Inc. System and method for role negotiation in multi-reality environments
US10482673B2 (en) 2013-06-27 2019-11-19 Help Lightning, Inc. System and method for role negotiation in multi-reality environments
US10150034B2 (en) 2016-04-11 2018-12-11 Charles Chungyohl Lee Methods and systems for merging real world media within a virtual world

Similar Documents

Publication Publication Date Title
Maes et al. The ALIVE system: Full-body interaction with autonomous agents
Vallino et al. Interactive augmented reality
Chua et al. Training for physical tasks in virtual environments: Tai Chi
Zheng et al. Virtual reality
JP2552427B2 (en) TV game system
AU2017204738B2 (en) Massive simultaneous remote digital presence world
US4884219A (en) Method and apparatus for the perception of computer-generated imagery
CA2680462C (en) Method for real time interactive visualization of muscle forces and joint torques in the human body
US9776083B2 (en) Spatially-correlated multi-display human-machine interface
CN102411783B (en) Move from motion tracking user in Video chat is applied
CN102222347B (en) Creating range image through wave front coding
JP5976019B2 (en) Theme-based expansion of photorealistic views
US8487871B2 (en) Virtual desktop coordinate transformation
JP3558104B2 (en) Three-dimensional virtual object display apparatus and method
CN105452935B (en) The perception based on predicting tracing for head mounted display
JP5547968B2 (en) Feedback device for instructing and supervising physical movement and method of operating
Thompson et al. Does the quality of the computer graphics matter when judging distances in visually immersive environments?
US20100039380A1 (en) Movable Audio/Video Communication Interface System
US20100128112A1 (en) Immersive display system for interacting with three-dimensional content
US8175374B2 (en) Volume recognition method and system
CN102184009B (en) Hand position post processing refinement in tracking system
US20080267450A1 (en) Position Tracking Device, Position Tracking Method, Position Tracking Program and Mixed Reality Providing System
DE69832119T2 (en) Method and apparatus for the visual detection of people for active public interfaces
CN102448566B (en) Gestures beyond skeletal
CN102356373B (en) Virtual object manipulation

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application
WWW Wipo information: withdrawn in national office

Country of ref document: DE

NENP Non-entry into the national phase in:

Ref country code: DE

NENP Non-entry into the national phase in:

Ref country code: RU

WWW Wipo information: withdrawn in national office

Country of ref document: RU

122 Ep: pct application non-entry in european phase

Ref document number: 06721800

Country of ref document: EP

Kind code of ref document: A1