JP2010533006A - System and method for communicating with a virtual world - Google Patents

Abstract

Systems and methods for communicating with a virtual world are disclosed. The virtual world may include a simulated public space and one or more simulated private spaces. Each private space may be associated with a particular user. The user's interaction with the virtual world may be represented by one or more actions of the avatar. A communication device may be simulated in the virtual world. Communication may be performed between the simulated communication device and the actual communication device. The virtual world may be generated by one or more processors. The processor simulates a communication device in the virtual world and allows communication between the simulated communication device and a real communication device. Systems and methods for virtual world communication using avatars, systems and methods for custom avatar gestures, systems and methods for mapping user emotional states to avatars, and communication between virtual and real communication devices Systems and methods for routing are also disclosed.
[Selection] Figure 2E

Description

(Refer to related applications)
This application enjoys the benefit of the priority of co-pending UK patent application No. 0703974.6 with the same assignee, the name “ENTERTAINMENT DEVICE”, filed on March 1, 2007. This application is disclosed herein by reference.

  This application enjoys the benefit of the priority of co-pending UK patent application No. 070425.2, named “ENTERTAINMENT DEVICE AND METHOD”, filed Mar. 5, 2007, with the same assignee. This application is disclosed herein by reference.

  This application enjoys the benefit of the priority of co-pending UK patent application No. 070425.1 with the same assignee, name “ENTERTAINMENT DEVICE AND METHOD”, filed Mar. 5, 2007. This application is disclosed herein by reference.

  This application enjoys the benefit of the priority of co-pending UK patent application number 070427.8 with the same assignee, name “ENTERTAINMENT DEVICE AND METHOD”, filed Mar. 5, 2007. This application is disclosed herein by reference.

  This application enjoys the benefit of the priority of co-pending UK patent application No. 070426.8 with the same assignee, entitled “ENTERTAINMENT DEVICE AND METHOD”, filed Mar. 5, 2007. This application is disclosed herein by reference.

  This application benefits from the priority of co-pending US Provisional Patent Application No. 60 / 889,397, entitled “VIRTUAL WORLD COMMUNICATION SYSTEMS AND METHODS”, filed Mar. 1, 2007. This application is disclosed herein by reference.

  This application is related to co-pending U.S. Patent Application No. 11 / 682,281 with the same assignee, inventor Thomas Gillo et al., “SYSTEM AND METHOD FOR COMMUNICATION WITH AVIRTUAL WORD”, filed March 5, 2007, and related. To do. This application is disclosed herein by reference.

  This application is co-pending U.S. patent application Ser. No. 11 / 682,284 with the same assignee, inventor Thomas Gillo et al., “SYSTEM AND METHOD FOR ROUTING COMMUNICATIONS AMONG REAL AND VIRTUAL COMMUNICATION DEVICES, 7th of March, 200”. Related to the application. This application is disclosed herein by reference.

  This application is related to co-pending U.S. Patent Application No. 11 / 682,287 with the same assignee, inventor Thomas Gillo et al., The name “SYSTEM AND METHOD FOR COMMUNICING WITH AN AVATAR”, filed March 5, 2007, To do. This application is disclosed herein by reference.

  This application is co-pending U.S. Patent Application No. 11 / 682,292, with the same assignee, inventor Thomas Gillo et al. Related to. This application is disclosed herein by reference.

  This application is related to co-pending US Patent Application No. 11 / 682,298 with the same assignee, inventor Thomas Gillo et al., Name “AVATAR CUSTOMIZATION”, agent case number SCEA0701JDI05, filed March 5, 2007, To do. This application is disclosed herein by reference.

  This application is related to co-pending U.S. patent application Ser. No. 11 / 682,299, entitled “AVATAR EMAIL AND METHODS FOR COMMUNICATING BETWEEN REAL AND VIRTUAL WORLDS”, filed Mar. 5, 2007. This application is disclosed herein by reference.

(Technical field)
The present invention relates to interactive computer entertainment, and more particularly to communication between users in a virtual world.

  A virtual world is a simulated environment in which users can interact with each other via one or more computer processors. The user appears on the video screen as a so-called avatar. The degree of interaction between the simulated environment and the avatar is realized by one or more computer applications that govern the interaction, such as physical simulation or information exchange between users.

The nature of interactions between users in a virtual world is often limited by the constraints of the system that implements the virtual world.
The embodiment of the present invention has been created in view of these points.

  The teachings of the present invention can be readily understood by considering the following detailed description in conjunction with the accompanying drawings, in which:

FIG. 6 is a screenshot showing an example of a world map representing a virtual world that may be used in conjunction with an embodiment of the present invention. FIG. 6 is a screenshot showing an example of a public space in a virtual world that may be used in conjunction with an embodiment of the present invention. FIG. 6 is a screenshot showing an example of a private space in a virtual world that may be used in conjunction with an embodiment of the present invention. It is a screenshot which shows the example of the virtual communication apparatus which concerns on embodiment of this invention. It is a figure which shows typically the virtual world system which concerns on embodiment of this invention. It is a functional block diagram which shows the embodiment of the multimedia processing apparatus which concerns on embodiment of this invention. This multimedia processing device allows the user to recognize and interact with the virtual world. FIG. 2 is a functional block diagram illustrating an implementation of a multimedia processing apparatus that may be used in conjunction with an embodiment of the present invention. 1 illustrates an implementation of a multimedia processing system that may be used in conjunction with an embodiment of the present invention. 2C-2D illustrate an imaging device that includes an array of microphones used in conjunction with embodiments of the present invention. It is a block diagram which shows the example of the call routing between the actual communication apparatus and virtual communication apparatus based on embodiment of this invention. FIG. 2 illustrates an example of communication between a real communication device and a virtual communication device according to an embodiment of the present invention. 1 is a block diagram illustrating a video game device that can be used to interact with a virtual world, in accordance with an embodiment of the present invention. FIG. It is a block diagram which shows the embodiment by the cell processor of the video game device which concerns on embodiment of this invention.

  While the following description includes many specific details for purposes of illustration, those skilled in the art will appreciate that many variations and modifications may be made to the following details within the scope of the present invention. By the way. Accordingly, the embodiments of the present invention described below are not intended to detract from the generality of the claimed invention or impose limitations.

  According to an embodiment of the present invention, the user may interact with the virtual world. Here, the term virtual world means something that represents a real or fictional environment. This environment has an interaction law that is simulated by one or more processors. Real users recognize this environment via one or more display devices and / or real users interact with this environment via one or more user interfaces. As used herein, the term user interface refers to a real device used by a user to send input to the virtual world or to receive output from the virtual world. The virtual world may be simulated by one or more processor modules. The plurality of processor modules may be linked to each other via a network. A user may interact with the virtual world via a user interface device. The user interface device can communicate with a plurality of processor modules and other user interface devices via a network. Certain aspects of the virtual world may be presented to the user in the form of graphics on a graphics display such as a computer monitor, television monitor, or similar display. Other aspects of the virtual world may be shown to the user in the form of audio from a speaker, where the speaker may be provided with a graphics display.

  As an example, the virtual world may comprise a simulated public space and one or more simulated private spaces. In some embodiments, such public and private spaces may be shown to the user via a graphic display showing a schematic representation or map of the virtual world. By way of example, as shown in FIG. 1A, the world map 10 may show a “home” position 11. The home location 11 may be a private space in a virtual world limited to a specific user. Other users can only “visit” the home location 11 if there is an invitation from a user associated with that home location 11. The world map 10 may also show various other locations 12. The user can visit those locations 12 by, for example, selecting with a cursor or by a similar graphical user interface. Vendors may sponsor these locations, and these locations may be represented on the map by corresponding company logos or other well-known symbols. Such a location may be visited by a virtual world user. The virtual world may or may not have a fixed amount of virtual “real estate”. In the preferred embodiment, the amount of virtual real estate is not fixed.

  In one embodiment of the invention, the virtual world may have multiple public spaces, referred to herein as “lobby”. Each lobby may have a separate chat channel associated with it. This allows users in the lobby to interact with each other. Each lobby may have the appearance of a lobby in a public building, such as a hotel, office building, apartment house, movie theater, or other public building. FIG. 1B shows a screenshot of such a lobby. The lobby may include things that the user can interact with. Such a thing is a game, for example. As seen in FIG. 1B, a portion of the virtual world is shown to the user in the form of three-dimensional (3D) graphics. Here, the term three-dimensional (3D) shape refers to a representation having length, width, and depth (or at least an illusion of depth). The lobby may include a “screen” 13. This screen 13 is an area in the space that can be used to show photos, recorded video and streaming video.

  In the virtual world, the user may be displayed by an avatar. Each avatar in the virtual world may be uniquely associated with an individual user. The user's name or pseudonym may be displayed next to the avatar so that the users can easily recognize each other. A user's interaction with the virtual world may be represented by one or more corresponding avatar actions. A plurality of different users may interact with each other through an avatar in a public space. An avatar representing a user may have an appearance similar to a human, animal or object. A human-shaped avatar may have the same or different gender as the user. The avatar may be displayed on the display so that the user can see the avatar along with other objects in the virtual world. Alternatively, the display may display the world from the viewpoint of the avatar, and the avatar itself may not be displayed. The view of the virtual world from the user (or from the avatar) may be considered as the field of view of the virtual camera. Here, the virtual camera refers to a viewpoint in the virtual world that can be used to draw a three-dimensional scene in the virtual world into a two-dimensional image. Users may interact with each other through their own avatar using chat channels associated with individual lobbies. Users may enter text via their user interface to chat with other users. The text may then appear on or next to the user's avatar. The appearance is, for example, in the form of a comic balloon, often called a chat bubble. Such chats may be simplified by using a chat system that uses pre-prepared phrases, often referred to as quick chat. In quick chat, the user may select one or more chat phrases from a menu.

  In the embodiment of the present invention, the public space is public in the sense that it is not associated with a particular user or user group and that no user or user group can exclude other users from the public space. In comparison, each private space is associated with a particular user among multiple users. A private space is private in the sense that a particular user associated with the private space can restrict access to that private space by other users. A private space may have the appearance of a private property that is often seen. For example, as seen in FIG. 1C, the private space may be provided to resemble an apartment house or a private house. Private space may include virtual objects. Examples of virtual things include, but are not limited to, furniture 15, decorations 16 and virtual communication devices 17 such as virtual radios and virtual video screens.

  In some embodiments of the invention, users in the virtual world may communicate using a virtual communication device. Here, the term virtual communication device generally refers to a simulation in the virtual world of a real-world device that is generated using the assets of the systems that make up the virtual world. As an example, as shown in FIG. 1D, the virtual communication device 18 is displayed on the display in a manner that makes it easier for the user to operate the device. In the example shown in FIG. 1D, the virtual communication device has the appearance of a portable game console, such as a Sony PlayStation Portable (PSP). The actual controller button used by the user to interact with the virtual world may be mapped to the corresponding button 19 or other control of the virtual communication device, in which case the interaction between the user and the virtual communication device is It becomes easy.

  The virtual communication device may have a position in a virtual world associated therewith, and the position may be fixed or movable. The communication device may be simulated by simulating the interface of the simulated communication device in the virtual world and presenting the simulated interface to the user for interaction therethrough. By way of example, the virtual device may have a shape or appearance that can be perceived by a user in a virtual world. This shape or appearance may mimic that of a corresponding real-world device so as to facilitate user interaction. For example, a virtual telephone may be displayed as having a button that a user operates using a controller. The virtual phone may further be displayed as having a speaker, mouthpiece, and possibly a graphics display screen. The simulated communication device may be a simulated handheld communication device, such as a simulated telephone, a simulated mobile phone (eg, a cellular phone) Phone or cordless phone), simulated voice over internet protocol (VoIP) phone, simulated portable text messaging device, simulated portable email device, simulated portable gaming device, simulated Two-way radio, or other simulated handheld device.

  According to the embodiment of the present invention, a virtual communication device may be simulated in a virtual world, and communication may be performed between the simulated communication device and an actual communication device. The actual communication device may be a real handheld communication device, such as a telephone, a mobile phone (eg, a cellular phone or a cordless phone), a voice over internet protocol (VoIP), etc. A telephone, a portable text message device, a portable electronic mail device, a portable game device, a two-way radio, or other handheld device. The actual communication device preferably communicates with one another through one or more communication channels independent of the virtual world. Here, the term “communication channel independent of the virtual world” means a communication channel in which the existence of the virtual world is not required because communication is performed through the channel. For example, a virtual phone can be used to call a real cellular phone (or vice versa) via communication assets provided by the virtual world. However, that real cellular telephone can make calls to other real cellular telephones or telephones even in the absence of a virtual world. In some embodiments, a real phone may generate a unique ringing tone when receiving a call from a simulated phone. In alternative embodiments, the simulated communication device and the actual communication device may communicate with each other using text messages and / or video images.

  FIG. 1E is a block diagram illustrating an example of a system 20 that can be used to simulate a virtual world. The system 20 includes a simulation server 22 and a view server 24. Each simulation server 22 may include one or more processor modules that execute coded instructions that simulate a portion of the virtual world. As an example, each simulation server may include one or more multi-core processors such as dual core, quad core, or cell processors. Although a limited number of simulation servers 22 and one view server 24 are shown in FIG. 1E, this setting can be freely expanded for any number of servers. Both the number of simulation servers 22 and the number of view servers 24 may be increased or decreased. For example, one simulation server 22 may accept many view servers 24, and many simulation servers 22 may accept one view server 24. By adding more simulation servers 24, the virtual world can be simulated larger and / or better. By adding more view servers 24, the system 20 can handle more users. Of course, the system 20 can achieve both larger and better simulation and more user acceptance by adding more of both the simulation server 22 and the view server 24. Theoretically, the number of simulation servers 22 can be increased to infinity. However, considering that the network bandwidth is limited, it is reasonable to assume that the number of view servers 14 will reach a finite upper limit if the number of view servers exceeds a certain number of users due to limitations due to computation and network bandwidth. Will.

  For purposes of illustration and not limitation to the embodiments of the present invention, examples relating to cell processors are described below. Cell processors include, for example, Cell Broadband Engine Architecture, Copyright Business International Business Machines Corporation, Sony Computer Entertainment Corporation, Sony Computer Entertainment Corporation, and Sony Computer Entertainment Corporation. ), Issued August 8, 2005, for details. A copy of this book can be found at http: // cell. scei. co. It can be downloaded from JP /. This book is disclosed herein by reference. A typical cell processor has a power processor unit (PPU) and up to eight additional processors called synergistic processing units (SPUs). Each SPU is typically a chip or a part of a chip that includes a main processor and an auxiliary processor. All SPUs and PPUs can access the main memory through, for example, a memory flow controller (MFC). The SPU can perform parallel operation processing in cooperation with a program running on the main processor. The SPU has a small local memory (typically around 256 kilobytes), which must be managed by software. That is, code and data must be manually transferred to and from the local SPU memory. In order to obtain high performance, this code and data must be managed by the SPU software (PPU software involvement must be minimized). There are many techniques for managing code and data from SPUs. Examples of such techniques are described in US patent application Ser. No. 11 / 238,077, inventor John P. et al. Bates, Payton White, Attila Vass, name “CELL PROCESSOR APPARATUS AND METHODS”, filed Sep. 27, 2005, and US Patent Application No. 11 / 238,095, inventor Richard B. Stenson and John P. Bates, name “CELL PROCESSOR TASK AND DATA MANAGEMENT”, filed on September 27, 2005, and US patent application 11 / 238,086, inventor Tatsuya Iwamoto, name “CELL PROCESSOR TASK AND DATA MANAGEN 9” Filed on the 27th, and US patent application no. Bates, Payton R.D. White, Richard B. Stenson, Howard Berkey, Attila Vass, Mark Cerny, John Morgan, name "SPU TASK MANAGER FOR CELL PROCESSOR", US Patent Application No. 11 / 257,761, m “SECURE OPERATION OF CELL PROCESSORS”, filed October 24, 2005, and US patent application Ser. No. 11 / 461,390, inventor John P. Bates, Keisuke Inoue, Mark Cerny, name “CELL PROCESSOR METHODS AND APPARATUS”, filed July 31, 2006. All these applications are disclosed herein by reference.

  The simulation servers 22 may communicate with each other and the view server 24 via a high speed data transport link 26. As an example, the data transport link may be a 10 gigabit per second Ethernet connection. The simulation servers 22 may be installed apart from each other or may be installed close to each other. In order to optimize data transfer, it is desirable to place the simulation servers 22 physically close to each other, for example in the same room or in the same server rack. The view server 24 receives the simulation data from the simulation server 22 and sends the view data to the client devices 28 that are scattered apart through the wide area network 30. This wide area network is, for example, the Internet or another wide area network. Client device 28 may be any suitable device that can communicate over network 30. Communication over network 30 may be slower than communication over high speed data link 26.

  As an example, the client device 28 may be a video game console device such as Sony PlayStation 3. Alternatively, the client device 28 may be any computer device from a handheld to a workstation or the like. A handheld video game device such as a PlayStation Portable manufactured by Sony Computer Entertainment Inc. in Tokyo, Japan is an example that can be used particularly as the client device 28 in the embodiment of the present invention among other handheld devices. Client device 28 may send instructions to view server 24 regarding the desired interaction with other clients' avatars and simulated environments. For example, a client user may wish to move his or her avatar to another location in the simulated environment. Each client device 28 sends instructions to one of the view servers 24. This instruction is relayed by the view server and delivered to a simulation server that performs the calculations necessary to simulate the exchange.

  Other devices 29 may also communicate with each other over the network 30. Examples of such other devices include telephones, cellular telephones, voice over internet protocol (VoIP) telephones, personal computers, portable web browsers, portable electronic mail devices, text message devices, portable game devices, and the like. Such communication between the other devices 29 may be independent of the simulation server 22 and the view server 26 that generate the virtual world. Although other devices 29 are not considered part of the system 20, they may interact with the system 20 via the network 30.

  The user of the client device 28 is usually interested in things around it. View server 24 ensures that each client 28 receives relevant data about its surroundings in the proper order. The view server 24 determines what the client needs based on the position, orientation, movement, etc. of the client's avatar. As an example, each view server may generate code and / or data used by a client device to display public or private space.

  In order to achieve a complex simulated world, it is desirable for peer-to-peer communication to occur between a client and a server or between a client device and another client device. For example, voice / video (A / V) chat between users in the same public space may be achieved by direct peer-to-peer communication between those users. Such peer-to-peer communication can reduce the load on the server. In an embodiment of the present invention, peer ribs are used to traverse network address translators (NATs) to establish peer-to-peer connections between users in the same public space. Crossing NAT, for example, U.S. Patent Application No. 11 / 245,853, Inventor Yutaka Takeda, Name "METHOD FOR PEER- TO-PEER COMMUNICATION TRAVERSING NETWORK ADDRESS TRANSLATORS OF TYPE 5 CY It is described in. This application is disclosed herein by reference.

  FIG. 1F shows one embodiment of a multimedia processing system 100 that can be used as the client device 28 and a user interface with the virtual world generated by the system 20. The processing system 100 may include a composite device that can process a plurality of contents across a plurality of media such as still images, moving images, music, broadcasts, and games. Processing multiple content includes presentation, recording, and other related tasks performed by the multimedia processing system 100. By way of example, the multimedia processing system 100 includes a multimedia processing device 102, a display 104 (eg, a monitor or television), and a controller 114. The buttons of the controller 114 may be mapped to the corresponding buttons 19 of the virtual controller 18 described above in FIG. 1D and above.

  The multimedia processing device 102 may receive multimedia content from various media sources, such as broadcast media, Internet (or other network) media, optical disc 110, memory card 112, and the like. Content from broadcast media may be received through the broadcast data channel 106. On the other hand, content from media on the Internet can be received through the network data channel 108. Broadcast and network data channels 106, 108 may be either wireless channels or wired channels. Content from broadcast media and Internet media may be recorded and stored by the multimedia processing device 102. The received content may be used not only for interaction with the virtual world, but also for various functions (eg, games) of the multimedia processing device 102.

  The received multimedia content may be displayed on the display 104. The display may include a video monitor such as a cathode ray tube (CRT) or a flat screen for displaying still images and moving images. Display 104 may further include one or more audio speakers to provide audio to the user. The controller 114 allows the user to input various instructions relating to multimedia processing, and allows the user to control the functions of the multimedia processing apparatus 102.

  System 100 may include audio and video inputs to facilitate user interaction with visible images and / or audible audio provided by display 104. Such inputs may include a video imaging device 116 such as a camera and an audio signal acquisition device 118 such as a microphone. Video imaging device 116 may be mounted on display 104 or may be integrated with display 104. Also, the video imaging device 116 is connected to the multimedia processing device 102 by a cable, or by an over-the-air connection such as an optical data link (for example, infrared) or a radio frequency data link (for example, Bluetooth). It may be connected. It should be understood that the imaging device 116 may be placed in any other nearby location where an image near the front of the display 104 can be acquired. There are various techniques for capturing these movements and exchanges, and examples of such techniques are described in British patent applications GB0304024.3 (PCT / GB2004 / 000693) and GB 03024022.7 (PCT / GB2004 / 000703). Yes. Each of which was filed on February 21, 2003 and is disclosed herein by reference. The imaging device 116 may be a digital camera such as a USB 2.0 type camera, for example. Such a camera may have a field of view of about 75 degrees, an F-number on the order of 1.5, and can acquire images at a frame rate of up to about 120 frames per second. As an example, the video imaging device may be an EyeToy Camera that can be purchased from Logitech, Inc., located in Fremont, California. Media processing device 102 includes a game console, a television, a digital video recorder (DVR), a cable set-top-box, a home media server, or a device that can render itself according to control from the user. It may be a household electrical device. In an alternative embodiment, the imaging device may be a three-dimensional (3D) camera. Here, a 3D camera (or a zed camera) refers to an imaging device that facilitates determining the depth of an object in an image. The term “depth” in this context refers to the position of the object in a direction perpendicular to the image plane.

  FIG. 2A is a functional block diagram illustrating an embodiment of the multimedia processing apparatus 102. In the embodiment shown, the multimedia processing device 102 includes a controller 114, a video imaging device 116, an audio signal acquisition device 118, a data input / output (I / O) unit 200, a display output unit 202, A display control unit 204, a holding unit 208, and a game / virtual world processor 206 are included. By way of example, the game / virtual world processor 206 may be and include a parallel processor such as a cell processor. The parallel processor has a power processing unit (PPU) coupled with one or more synergistic processing units (SPU). Cell processors are described, for example, in US patent application Ser. No. 11 / 238,077, which is hereby incorporated by reference. The multimedia processing device 102 further includes programs and instructions for implementing various functions such as a data input function, a data maintenance function, an image processing function, a drawing function, and other related functions.

  The controller 114 includes an orientation determination unit 222 that determines one or a combination of four directions (that is, upward, downward, leftward, and rightward) from a user input, an instruction determination unit 224 that determines an instruction from the user input, May be included. The instructions may include a command for indicating multimedia content, a command for terminating its presentation, a command for invoking a menu screen, and a command for issuing other related commands and / or instructions. Outputs from the controller 114, the video imaging device 116, and the audio signal acquisition device 118 are directed to the display output unit 202, the display control unit 204, and the game / virtual world processor 206.

  In the embodiment shown in FIGS. 1B and 2A, orientation determiner 222 and instruction determiner 224 may be comprised of a combination of buttons, circuits, and programs for activating, sensing, and determining direction and instructions. . The button can include a cross key or a joystick. The button related to the instruction for calling the menu screen may be toggled, and in this case, each time the button is pressed, the menu screen is switched between the display mode and the non-display mode.

  In one embodiment, the orientation determining unit 222 may determine the diagonal movement of the button as a binary command that defines the direction of movement as one of the two orientations. Here, the diagonal movement between upward and rightward can be defined as either upward or rightward. In another embodiment, the orientation determination unit 222 may determine the diagonal movement of the button as an analog command that determines the direction of movement to a certain direction within the measurement accuracy. Here, the diagonal movement between upward and rightward can be defined as northwest. As described below, directional movement may be determined through interactions between the user, the video imaging device 116 and the display control 204.

  The data I / O unit 200 is stored in the disk 110, a broadcast input unit 212 that inputs broadcast content via the broadcast channel 106, a network communication unit 214 that inputs and outputs data such as web content via the network channel 108, and the disk 110. A disk reading unit 216 that inputs data to be read and a memory card reading unit 218 that inputs data to the memory card 112 and / or outputs data from the memory card 112 may be included. The output of the data I / O unit 200 may be directed to the display output unit 202, the display control unit 204, the game processor 206, and the holding unit 208.

  The display output unit 202 may include a decoding unit 232, a synthesis unit 234, an output buffer 236, and an on-screen buffer 238. The decoding unit 232 decodes input data received from the data I / O unit 200 or the holding unit 208. Here, the input data may include broadcast content, movies, and music. The synthesizer 234 processes the input data decoded based on the user policy / instruction received from the controller 114. The output of the combining unit 234 is held in the output buffer 236. The on-screen buffer 238 may hold menu screen image data generated by the display control unit 204. The output of the display output unit 202 is transmitted to the display 104.

  The display control unit 204 may include a menu manager 242, an effect processor 244, a content control unit 246, and an image generation unit 248. The menu manager 242 receives media from the holding unit 208 and the data I / O unit 200, and manages the media body and multimedia content shown on the menu screen. The effect processor 244 processes the operation of icons and icon arrays on the menu screen. The effect processor 244 manages various operations and effects to be displayed on the menu screen. The content control unit 246 controls the processing of media bodies and multimedia content, and the handling of data from the data I / O unit, the holding unit 208, and the game / virtual world processor 206. The image generation unit 248 generates a menu screen including a media icon array and a content icon array.

  The game / virtual world processor 206 uses the data read from the data I / O unit 200 or the holding unit 208 to execute a game and / or a virtual world program. The game / virtual world processor 206 executes a game program or facilitates user interaction with the virtual world based on user instructions received from the controller 114. Display data exchanged with the executed game program or virtual world is transmitted to the display output unit 202.

In an embodiment of the present invention, signals from video imaging device 116 and audio signal acquisition device 118 allow a user to interact with and manipulate the image shown on display 104. In particular, embodiments of the present invention may allow a user to “grab” and “drag” an object from one location on display 104 to another. As shown in FIG. 2B, the video imaging device 116 is directed to the user I image I _U and images it. The image I _U may then be displayed on a background of other images on the display 104 using a technique known as alpha blending.

The term “alpha blending” generally refers to the convex combination of two colors and is for transparency effects in the field of computer graphics. The value alpha in the color code may be in the range of 0.0 to 1.0, with 0.0 indicating a sufficiently transparent color and 1.0 indicating a sufficiently opaque color. As an example, when the color Value1 is drawn on the background of the color Value0, the resulting color value (Value) may be given by the following equation.
Value = Value0 (1.0-alpha) + Value1 (alpha)

  As seen in 32-bit RGBA, the alpha component is used to evenly mix the three components red, green and blue. Alternatively, three alpha values are designated corresponding to each of the primary colors in spectral color filtering.

  When the user's hand h is recognized, the effects processor may correlate the orientation of the user's hand displacement with an orientation input that would normally be received from the controller 114. The speed input may be controlled by the magnitude of the displacement.

In some embodiments, the image I _U may include the user's head H and hand h. To facilitate user interaction with the image I _U, the image I _U of users it is to be noted that the may be displayed on the screen as a mirror image of the user U. Here, when the user U moves the hand h to the left side of the user, the hand image I _h also moves to the left side of the user. The effects processor 244 may be configured to recognize the user's hand h and to recognize changes in the aspect ratio (height to width ratio) of the hand image Ih. This change in aspect ratio may be used to inform the controller 114 that the user has “grabbed” or “clicked” on the object 140 on the display. Then, the effects processor 244 can move the selected object in accordance with the movement of the image I _h of the user's hand h. In some embodiments, the user may grip a deformable “C” shaped object 142. The deformable “C” -shaped object 142 is given a color that can be more easily recognized by the effects processor 244 when interpreting the image from the video imaging device 116. The deformation of the object 142, referred to herein as “clam”, is a change in aspect ratio that is recognized as a command to “grab” or “click” the object on the display 104. Can be provided.

It is often desirable for the effects processor 244 to be able to recognize whether the user U is using the left or right hand to manipulate the object 140 on the display 104. For example, when an object on the display 104 is operated with the left hand, it is often desirable that the object appears on the left side of the user's head H. In such a case, the controller may include software that recognizes the user's hand h, head H, arm A, and waist C by corresponding images I _h , I _H , I _A , and I _C , respectively. Based on this information, the controller 114 can determine whether the user U is using the left hand or the right hand. For example, if the user's hand h is on the left side of the head H and the arm A does not cross the waist, the user U can determine that he is using the left hand. Similarly, if the user's hand h is on the left side of the head and the arm crosses the waist, it can be determined that the user U is using the right hand.

  In an embodiment of the present invention, the imaging device 116 and the audio signal acquisition device 118 may be integrated as one device. For example, FIGS. 2C and 2D illustrate an imaging device 120 that may be used in the multimedia processing system 100. The device 120 includes an optical imaging device 122 which is a digital camera (or 3D camera), for example, and one or more microphones 124. The microphones 124 may be arranged in an array and spaced apart by a known distance. To the extent that they are exemplary and do not lose generality, the microphones 124 may be spaced apart in a linear array and adjacent microphones may be separated by a distance of about 2 centimeters between centers. Individual microphones may have a resonant frequency of about 16 kilohertz. Such a microphone array may be used to locate and track the location of one or more sound sources in conjunction with the operation of the device 102 and interaction with the virtual world. The use of such microphone arrays for sound source location and tracking is described, for example, in US patent applications 11 / 381,724, 11 / 381,725, and 11 / 381,729, filed May 4, 2006. Are listed. These are disclosed herein by reference.

  In certain embodiments of the invention, it is desirable for the microphone 124 to move with the imaging device 122. For example, the microphone 124 may be attached to the frame 126. The frame 126 is assumed to have a fixed positional relationship between the microphone and the imaging device, for example, the lens 128. Although multiple microphones are shown arranged in a horizontal straight line, they may alternatively be oriented vertically, may be oriented diagonally, and are arranged in two dimensions. May be.

  In certain embodiments, the device 120 may include a visible LED 130 and an infrared LED 132. These can be used to illuminate objects in the field of view of the imaging device 122. To facilitate acquisition of infrared images, the lens 128 includes a so-called “day-night” coating that allows visible light and selected infrared frequencies to pass (eg, frequencies around 940 nm). But you can.

  By way of example, elements of system 20 and device 102 may be configured to cause his or her avatar to pick up a virtual mobile phone, dial a number, and make a real or virtual phone call. . If the call recipient is another user in the virtual world, the system 20 and device 102 may be suitably programmed to connect to the user's virtual phone. Here, this connection is made via VoIP when the user is online and interacting with the virtual world, for example, at the time the call is made. Elements of system 20 and device 102 may be configured by default to route the telephone to the receiving virtual telephone (if any). If the recipient is not online, the phone may be re-routed to the recipient's actual communication device. Examples of real-world communication devices are telephones (eg, fixed telephone lines, cell phones, or VoIP phones), or voicemail (which can accompany real or virtual phones), or portable game devices and the like Any network device that supports VoIP, including, but not limited to, may be included. Alternatively, the phone may be routed by default to the user's real communication device. In such an embodiment, elements of system 20 and device 102 may be used to allow for intelligent bi-directional routing between the virtual world and real communication devices.

  For purposes of illustration and without loss of generality, communication between a real device and a virtual device can be understood with reference to FIGS. 2E and 2F. As shown in FIG. 2E, two or more users 251, 252 may interact with the virtual world, eg, via network 30 via system 20 as described above with respect to FIG. 1E. Individual users may be associated with the system 20 through the network 30 via client devices 253, 254. These client devices 253, 254 may be appropriately configured, for example, as described above with respect to FIGS. 1F and 2A. Individual client devices 253, 254 may include hardware and / or software suitable for creating virtual communication devices 255, 256. The device avatar may represent a virtual communication device in the virtual world. The device avatar may have the appearance of a real device as described above, for example. Alternatively, the user may customize the device avatar to have a totally arbitrary and / or fictitious appearance. Individual users 251, 252 may be able to use real-world communication devices such as landlines 257, 258 and mobile phones 259, 260. Those skilled in the art will appreciate that embodiments of the present invention may be implemented with any number of users and any number of real and / or virtual communication devices.

  Individual client devices 253, 254 may include configurable routers 261, 262 to facilitate the routing of telephone calls between real and virtual devices. The routers 261 and 262 may exist in the form of software or hardware in servers, peer-to-peer networks, combinations thereof, and the like. In the example shown in FIG. 2E, routers 261 and 262 are provided on client devices 253 and 254, but this is not the only possible arrangement. Alternatively, the routers 261 and 262 may be provided at any location on the simulation server 22, the view server 24, or other devices connected to the network 30. Routers 261 and 262 may be accessed from various devices and by multiple means. This includes, but is not limited to, virtual phones or communication devices, real communication devices, network web pages, and the like. Each router 261, 262 may have one or more routing priorities to control the routing function. The routers 261 and 262 may execute the routing priority process in the communication source (source) or the communication target (target). In the sense used in this embodiment, a communication “source” generally refers to the party that originates the communication (eg, a “caller” of a telephone or a “sender” of a text message or email). ]). In the sense used in the present embodiment, a communication “target” generally refers to a communication receiving side. It is shown that the source and target need not be virtual world users.

  As an example, consider the case where the first user 251 desires to communicate with the second user 252 using, for example, the virtual communication device 255. In this case, the first user 251 is the telephone source and the second user 252 is the telephone target. The first user router 261 may be set to preferentially attempt to connect to the virtual communication device 256 of the second user 252. If the user is not online and not using the virtual world, the first user's router 261 may attempt to connect to the second user's landline 258. If that fails, the router 261 may attempt to connect to the mobile phone 260 of the second user 252. In an alternative example, it is shown that the second user's router 260 may perform its own routing priority process when it receives communication from the first user 251. For example, the second user's router 262 preferentially routes calls from the first user 251 to the second user's mobile phone 260, then routes to the second user's landline phone 258, and then You may route to the virtual user 256 of the second user.

  In a preferred embodiment, each individual user 251, 252 may have one telephone number corresponding to each of the user's real and virtual communication devices. The routers 261, 262 may automatically route the phone to one user number among the different devices that the user has. Such routing may be based on user priority, user behavior, or some combination thereof. It is shown that the routers 261, 262 may be programmed to have different priorities depending on the identity of the communication source and / or the identity of the communication target.

  For example, in one mode, the first user's router 261 may receive a call from a source calling the first user's number. In that mode, the first user 251 may provide the router 261 with information indicating that the first user 251 is online. Such information may be programmed into the multimedia processing device 102 using, for example, the controller 114. Alternatively, the router 261 may check whether the first user 251 is online. If so, the router 261 may route the “phone” to the first user's virtual communication device 255. This virtual communication device 255 may be set to “ring” even if the first user 251 is online via the second user's client device 254.

  In other modes, the router 261 may obtain information about whether the first user 251 is online and whether the target (eg, the second user 252) is offline, and so on. A check may be made to determine if there is. In this case, the first user's router 261 may route the “telephone” to a second user's real communication device, such as a landline phone 258 or a mobile phone 260.

  In other modes, the first user's router 261 may or may not obtain information about whether the first user 251 is online and whether the second user 252 is online. You may decide whether there is. In this case, the “text message” may be routed in the virtual world to, for example, the second user's avatar or the target second user's virtual device 256.

  In other modes, the router 261 may obtain information about whether the second user 252 is online or may check to determine if it is. If the second user 252 is offline, a “text message” may be routed to a real-world device associated with the second user, such as a landline phone 258 or a mobile phone 260.

  Many other permutations are also possible with respect to the above example. For example, in the above example, the source may make a call from within the virtual world, may make a call in the virtual world through an avatar virtual device, through any VoIP device or service, or any reality. You may make a call through any telephone line and source.

  In other settings, the smart routing described above may be performed based on the user's priority, such as when the user wants a real mobile phone to ring instead of an avatar phone when online. In other settings, the above-described clever routing may be performed based on state control (STATE CONTROLS). With this state control, the phone is routed to an avatar phone or a real phone only under certain circumstances, depending on the application settings. For example, if the target is participating in an online game and does not want to be disturbed, the call may be routed to the target's real or virtual voicemail. In yet another setting, the phone is routed to a virtual device, and if that device does not ring, the phone may be rerouted to a real device, such as a real phone, instead of going to virtual voicemail.

  As shown in FIG. 2F, the embodiment of the present invention allows a situation where the first user 251 makes a call to the second user 252 using a real phone such as a landline phone 257. The first user 251 talks to the phone and the first user's avatar 263 appears in the second user's virtual communication device 256. The avatar 263 is displayed on the display 104 connected to the multimedia processing apparatus 102 owned by the second user 252. On the display 104, the first user's name 266 may also be displayed near the first user's avatar 263. The first user's utterance 265 may be converted to text using speech recognition software and / or hardware that may be implemented on device 102, simulation server 22, view server 24, or other device. Good. The resulting text may be displayed on the display 104 as text bubbles 264 near the first user's avatar 263. The audio speaker 267 may play an audible sound of the first user's speech 265 during communication between the first user and the second user.

  It is shown that the same routing process can be used for other types of messaging such as text messaging and email. One advantage of this system is that a real phone and / or text message can be routed from one user to another in a way that avoids long distance charges and other phone charges generated by real communication devices. That's what it means. The recipient's (user's) real phone or text message device may include middleware to facilitate interaction with the virtual world maintained by the system 20.

  In some embodiments, a user may be able to access virtual world content using real-world communication devices. For example, a mobile phone or a portable internet device may be used to make changes to the user's avatar, public space, or private space. Alternatively, a real communication device may be used to access the content of the virtual communication device from a remote location. In particular, a real communication device may be used as an interface between a simulated communication device and a user. For example, suppose that the virtual communication device is a virtual digital video recorder (DVR) placed in the user's private space. A user may access a virtual DVR to record real or virtual events using a real mobile phone and an electronic programming guide.

  As described above, performing communication between a real communication device and a virtual communication device may include video communication. According to an embodiment, an avatar image may be displayed on a real communication device during video communication. A system that generates a virtual world may lip-sync the avatar image to real or synthesized speech generated by a user associated with the avatar. For example, a user may record an audio message to be sent to a real device as part of a video message. The system may generate a video message with an avatar speaking a voice message. In this case, the movement of the lips of the avatar is synchronized with the user's statement in the message. Alternatively, the user may enter the text of the message on the virtual device. The system may synthesize speech for the avatar from the text and generate a video image of the avatar. In this case, the movement of the lips of the avatar is synchronized with the synthesized voice. In other embodiments, the user may record audio and video messages using, for example, video imaging device 116 and audio signal acquisition device 118.

  In some embodiments, the avatar 14 may express emotions through animations, facial color changes, sounds, particles, or chat bubble changes to convey certain emotions. Such an avatar expression of emotion (often referred to as “emotes”) may be pre-programmed and triggered by a user command. In some embodiments of the present invention, emotions shown by the user in the interaction with the virtual world may be mapped to emotions represented by the user's avatar. In some embodiments, the user may select an emotional state that can be reflected in the avatar. As an example, Avatar Emotes may be selected from a menu provided to the user by device 102. For example, if the user selects “happy”, the user's avatar may be represented with a smile. If the user selects “sad”, the avatar may represent frowning. This menu driven feeling may be awkward if the user wants to use it quickly. Thus, in one embodiment, the device 102 may detect the state of the user's emotion in real time and appropriately change the characteristics of the user's avatar to reflect that state. Such tracking of the user's emotional state in real time is particularly useful when mapping the user's emotional state to the avatar, for example during video communication in which an image of the user's avatar is displayed on a real device. .

  By way of non-limiting example, the device 102 can track one or more images of the user U using the imaging device 116 and analyze one or more facial colors of the user in real time to track the emotional state of the user. Also good. The game / virtual world processor 206 analyzes these images using the user's lips, eyes, eyelids, eyebrows, cheeks, cheeks, teeth, or nose and other body language such as posture, arm and hand placement. And may be programmed to determine the state of emotion of the user. Such complexion and / or body language analysis can be enhanced by generating an image using a 3D camera.

  Alternatively, the user's emotional state may be tracked in real time by analyzing the user's voice inflection shown in the user's utterance or other utterance detected by the audio signal acquisition device 118. If the user communicates via text, the emotion may be tracked by analyzing the text for a word, phrase, or language pattern that indicates the emotional state. In addition, other biometric data such as electrocardiograph (EKG) data, electroencephalogram (EEG) data, electrodermal reflex data, or thermographic data may be used to track the user's emotional state. Such data may be acquired through appropriate sensors provided in the controller 114 and analyzed by appropriately configured software, hardware, or firmware installed in the processor 206. The thermographic data is also obtained when the imaging device 116 can acquire an infrared image.

  Once the emotional state of the user is determined, various combinations of body language and complexion indicating the emotional state may be reflected in the emoticons represented by the animation of the avatar (e.g. Angry shows anger).

  In some embodiments, a user may wish to use a customized gesture or “emotes” for his avatar. To facilitate this, one or more custom gestures may be generated for the avatar. These custom gestures may be associated with one or more user interface signals so that the user's avatar can make a gesture based on the command. As an example, custom gestures may be generated using motion capture or performance capture techniques to record and digitize the user's body movements or user facial expression mappings as the user makes gestures. In certain embodiments, the imaging device 116 may be used for this purpose. Alternatively, a commercial motion capture studio or performance capture studio may be used for this purpose.

  In motion capture, a user or other performer may attach a marker to each joint and recognize the movement by the position of the marker or the angle between the markers. A voice marker, inertia marker, LED marker, magnetic marker, or reflective marker, or any combination thereof, is optimally at least twice as fast as the desired speed of motion, with sub-millimeter position accuracy, Be tracked. Motion capture computer software records position, angle, speed, acceleration, and impulse, and provides an accurate digital representation of movement. As an example, an optical motion capture system may triangulate the 3D position of a marker between one or more cameras calibrated to provide overlapping images. Passive optical systems may use markers covered with a retroreflective material that reflects light generated near the camera lens. Camera sensitivity can be adjusted using the narrow range of light sensitivity that most cameras have so that only bright markers are sampled and skin and fabric can be ignored. Alternatively, an active optical system where the marker itself has a power source and emits light from itself may be used. Power may be provided to individual markers in turn, in which case it may be synchronized with the capture system. This allows individual markers to be recognized independently for a given capture frame at the expense of frame rate.

  Performance capture differs from normal motion capture due to the interactive nature of performance. Performance capture captures all of your body, hands, and facial expressions at once, rather than acquiring the motion data you want to reference and later editing the motion together.

  When the user's body movements and / or facial expressions for a gesture are digitized, the digitized gesture is a coded instruction or other user interface regarding the animation of the avatar so that the avatar performs the gesture. It may be used to generate a signal. The code and other user interface signals may be distributed to one or more other users. One or more other users can customize their avatars to make custom gestures. Customized avatar gestures, combined with customized avatar clothes, footwear, hairstyles, ethnic characteristics, and other custom avatar characteristics, indicate that they belong to a particular group in society It may be a means. In certain embodiments, it may be desirable to moderate the use of custom gestures, for example, to avoid unnecessarily stimulating other users or committing law. “Adjusting” or “modifying or moderating” here means imposing a certain degree of regulation on actions that can be accepted in the virtual world. Such adjustment can be implemented by the view server 24. This view server 24 may analyze whether the custom gesture is not rude or otherwise inadequate.

  Adjusting the display of custom gestures restricts avatars from making custom gestures for certain users according to predetermined criteria, or makes certain users unable to perceive avatars that perform custom gestures You may include that. This predetermined criteria may include the sensitivity of the viewer to feel insult based on the age of the user, the viewer of the gesture, religious, ethnic or other attribution of the viewer.

  According to alternative embodiments of the present invention, the systems and methods described above may be modified to implement communication using a virtual world. In particular, an avatar and an email sender may be associated with each other. The user may generate an email in the virtual world and associate one or more images of his / her avatar with the email. The email may be sent from the virtual world to a real device. The avatar image may be presented at the e-mail delivery destination, for example, by self-decompression of the attachment of the e-mail. The email may be generated using a virtual communication device in a virtual world, for example. By way of example without limitation, the e-mail recipient may be a real communication device, for example, any real device configured to receive e-mail messages. Real communication devices may communicate with one another through one or more communication channels independent of the virtual world.

  To the extent that it is exemplary and does not lose generality, the virtual world may include a simulated public space and one or more simulated private spaces that facilitate interaction between multiple users. Each private space is associated with an individual user of the plurality of users, for example as described above.

  The recorded or synthesized voice and email may be associated and the voice may be provided with one or more images at the destination. The avatar image may include an animation of an avatar generated specifically for email. The animation may be presented at the destination, for example by self-extracting an email attachment. In addition, one or more gestures may be mapped to the animation of the avatar, for example as described above. Gestures can be mapped by recording the voice and / or video from which the email message originated and mapping one or more features of the voice and / or video to one or more features of the animated avatar. Good.

  In some embodiments, the theme and the movement of the virtual camera in the animation may be associated. By way of example without limitation, the theme may include virtual camera angle selection, tracking, panning, tilting, zooming, close-up, simulated light, and the like. The position of the virtual camera may be fixed or moved. In addition, the theme may include selecting a background depiction for the avatar.

  In some embodiments, creating the email may include tracking the emotional state of the sender, eg, as described above, and mapping that emotional state to the theme. For example, a calm or quiet emotional state may be mapped to a theme characterized by a fixed camera position or a relatively slow movement of the virtual camera. The emotional state of upset or excitement may be mapped to a theme characterized by rattling camera movements, extreme close-ups, harsh camera angles, and the like.

  Avatar email communication of the type described above may be implemented, for example, by appropriately configuring the system 20 of FIG. 1E and / or the multimedia device 102 of FIGS. 1F and 2A.

  According to embodiments of the present invention, a virtual world system and method of the type described above uses a console video game device as a user interface for interacting with the client device 28 and the virtual world generated by, for example, elements of the system 20. Can be implemented. As shown in FIG. 3, the console video game apparatus 300 includes a processor 301 and a memory 302 (for example, RAM, DRAM, ROM, etc.). In addition, the video game apparatus 300 may include a plurality of processors 301 as long as parallel processing is implemented. The memory 302 includes data and game program code 304, and includes a portion that facilitates user interaction with the virtual world as described above. Specifically, the memory 302 may include inertial signal data 306 including stored controller path information as described above. The memory 302 may include stored gesture data 308, such as data representing one or more gestures associated with the game program 304, for example. Coded instructions executed on processor 302 may use multi-input mixer 305 configured and functioning as described above.

  The system 300 may also include known support functions 310 such as input / output (I / O) elements 311, a power supply (P / S) 312, a clock (CLK) 313, and a cache 314. The device 300 may optionally include a mass storage device 315 such as a disk drive, CD-ROM drive, tape drive, etc. for storing programs and / or data. The controller may optionally include a display unit 316 and an input unit 318 to facilitate communication between the device 300 and the user. The display unit 316 may be in the form of a cathode ray tube (CRT) that displays text, numerical values, graph symbols and images, or a flat panel screen. User interface 318 may include a keyboard, mouse, joystick, light pen, and other devices. User input 318 may also include a microphone, video camera, or other signal converter to directly capture the signal to be analyzed. The device 300 may include a network interface 319 so that it can communicate with a virtual world server and other similarly configured devices via a network such as the Internet. The processor 301, the memory 302, the user input 318, the network interface 319 and other components of the device 300 exchange signals (eg, coded instructions and data) with each other via the system bus 320 as shown in FIG. May be.

  The microphone array 322 may be connected to the system 300 via the input / output device 311. The microphone array may comprise about 2 to about 8 microphones, preferably about 4, spaced from adjacent microphones by a distance of less than about 4 cm, preferably about 1-2 cm. Preferably, the microphones in array 322 are omnidirectional microphones. An optional imaging device 323 (for example, a digital camera) may be connected to the device 300 via the input / output device 311. One or more positioning actuators 325 may be mechanically connected to the camera to control the positioning of the imaging device. These actuators 325 may exchange signals with the processor 301 via the input / output device 311.

  As used herein, the term “input / output device” generally refers to any program, operation or device that sends and receives data to and from device 300 and peripheral devices. All data transmission and reception may be considered as output from one device and input to another device. Peripheral devices may include devices such as a writable CD-ROM that can operate as both an input device and an output device, in addition to an input-only device such as a keyboard and mouse, and an output-only device such as a printer. The term “peripheral device” includes an external device such as a mouse, keyboard, printer, monitor, microphone, game controller, camera, external Zip driver or scanner, CD-ROM drive, CD-R drive or internal modem, etc. Internal devices, or other peripheral devices such as flash memory reader / writers and hard drives.

  In some embodiments of the present invention, the device 300 includes a controller 330 connected to the processor via the input / output device 311 through the use of a wired (eg USB cable) or wireless, eg infrared or radio frequency (such as Bluetooth) connection. You may have. The controller 330 may include an analog joystick control 331 and a known button 333 that provide control signals normally used during video game play. Such a video game may be implemented as processor readable data and / or instructions from a program 304 stored in the memory 302 or other processor readable media associated with the mass storage device 315.

  The joystick control 331 may normally be configured to transmit a signal along the X axis when the control stick is moved to the left and right, and to transmit a signal along the Y axis when moved back and forth (up and down). In a joystick configured for three-dimensional movement, a signal of movement along the Z axis may be transmitted when the stick is twisted to the left (counterclockwise) or right (clockwise). These three axes of X, Y, and Z may be referred to as roll, pitch, and yaw, respectively, particularly when related to aircraft.

  In addition to conventional features, the controller 330 may include one or more inertial sensors 332. Inertial sensor 332 can provide position information and / or direction information to processor 301 via an inertial signal. The direction information may include angle information such as tilt, roll or yaw of the controller 330. By way of illustration, the inertial sensor 332 may include any number of accelerometers, gyroscopes or tilt sensors, or any combination thereof. In a preferred embodiment, the inertial sensor 332 includes a tilt sensor adapted to sense the direction of the joystick controller relative to the tilt and roll axis, a first accelerometer adapted to sense acceleration along the yaw axis. And a second accelerometer adapted to sense angular acceleration relative to the yaw axis. The accelerometer may be implemented, for example, as a MEMS device that includes a mass attached with one or more springs and a sensor that detects displacement of the mass in one or more directions. A signal from a sensor that depends on the displacement of the mass may be used to determine the acceleration of the joystick controller 330. Such a technique is realized by instructions from the game program 304 stored in the memory 302 and executed by the processor 301.

  By way of example, an accelerometer suitable as inertial sensor 332 may be a single mass that is telescopically connected to the frame at 3-4 points, for example, with a spring or the like. The pitch axis and the roll axis are present on a plane that intersects the frame provided in the joystick controller 330. As the frame (and joystick controller 330) rotates about the pitch and roll axes, the mass is displaced under the influence of gravity and the springs expand and contract in a manner that depends on the pitch and / or roll angle. The displacement of the mass is detected and converted into a signal depending on the pitch amount and / or the roll amount. Angular acceleration around the yaw axis or linear acceleration along the yaw axis generates characteristic patterns of spring expansion and contraction or mass movement, which are detected and signaled according to the amount of angular acceleration or linear acceleration. Converted. Such an accelerometer device can measure tilt, roll angular acceleration about the yaw axis, and linear acceleration along the yaw axis by tracking mass movement and spring stretching force. There are many different ways to track the position of the mass and / or the forces acting on it, including resistive strain gauges, acoustic sensors, magnetic sensors, Hall effect elements, piezoelectric elements, capacitive sensors, etc. included.

  The joystick controller 330 may include one or more light sources 334 such as light emitting diodes (LEDs). The light source 334 is used to distinguish the controller from other controllers. For example, if there are one or more LEDs, they can be identified by flashing or holding the LED pattern code. For example, five LEDs may be provided on the joystick controller 330 in a straight line or a two-dimensional pattern. Although it is preferable to arrange the LEDs in a straight line, as another method, the LEDs are arranged in a square shape or an arch shape to facilitate determination of the LED arrangement screen when analyzing the LED pattern image obtained by the imaging device 323. It may be made like this. Furthermore, the LED pattern code can also be used to determine the position of the joystick controller 330 during game play. For example, LEDs can help identify controller tilt, yaw and roll. With this detection pattern, it is possible to provide better emotions to the user in games such as airplane games. The imaging device 323 may capture an image including the joystick controller 330 and the light source 334. When such an image is analyzed, the position and / or orientation of the joystick controller can be determined. Such analysis may be implemented by program code instructions 304 stored in memory 302 and executed by processor 301. In order to facilitate imaging of the light source 334 by the imaging device 323, the light source 334 may be disposed at a plurality of different ends of the joystick controller 330, for example, forward and backward (shown as shadows). With such an arrangement, the imaging device 323 can obtain images of the light source 334 in various directions of the joystick controller 330 depending on how the user holds the joystick controller 330.

  Further, the light source 334 may provide a telemetry signal to the processor 301 in, for example, a pulse code, amplitude modulation, or frequency modulation format. Such a telemetry signal may represent which joystick button is pressed and / or how hard the button is pressed. The telemetry signal may be encoded into an optical signal by, for example, pulse encoding, pulse width modulation, frequency modulation, or light intensity (amplitude) modulation. The processor 301 may decode the telemetry signal from the optical signal and execute the game command according to the decoded telemetry signal. The telemetry signal may be decoded from the analysis of the image of the joystick controller 330 obtained by the imaging device 323. Alternatively, the device 301 may comprise an independent optical sensor dedicated to receiving telemetry signals from the light source 334. The use of LEDs in conjunction with intensity determinations in an interface with a computer program is described, for example, in US patent application Ser. "AMOUNT WHEN INTERFACING WITH A COMPUTER PROGRAM" (agent case number SONYP052), inventor Richard L. Marks et al., The disclosure of which is incorporated herein by reference in its entirety. The analysis of the image including the light source 334 can also be used to determine the position and / or orientation of the telemetry and joystick controller 330. Such a technique may be realized by instructions of a program 304 stored in the memory 302 and executed by the processor 301.

  The processor 301 uses the inertial signal from the inertial sensor 332 together with the optical signal from the light source 334 detected by the imaging device 323 and / or the sound source location and sound source characteristic information from the acoustic signal detected by the microphone array 322. Information regarding the location and / or orientation of the controller 330 and / or the user may be inferred. For example, the source location of an “acoustic radar” and its characteristics can be used with a microphone array 322 to track a moving source while the movement of the joystick controller is tracked independently (through inertial sensor 332 and / or light source 334). To do. In an acoustic radar, a calibrated listening area may be selected during run time. Sounds generated from sound sources outside the calibrated listening area may be filtered out. The calibrated listening area may include a listening area corresponding to the volume at the focus or field of view of the imaging device 323. An example of an acoustic radar is disclosed in US patent application Ser. No. 11 / 381,724, inventor Xiadong Mao, name “METHODS AND APPARATUS FOR TARGETED SOUND DETECTION AND CHARACTERIZATION”, filed May 4, 2006, and disclosed therein. The contents are incorporated herein by reference.

  Various combinations of various forms for supplying control signals to the processor 301 may also be used in conjunction with embodiments of the present invention. Such techniques may be implemented by program code instructions 304 stored in memory 302 and executed by processor 301, and optionally select a calibrated listening area during execution time, other than the calibrated listening area. The sound generated from the sound source may include one or more instructions that direct one or more processors to be filtered out. The calibrated listening area may include a listening area corresponding to the volume at the focus or field of view of the imaging device 323.

The program 304 optionally generates a discrete time domain input signal Xm (t) from the microphones M ₀ ... M _M in the microphone array 322, determines the listening sector, and selects semi-blind to select finite impulse response filter coefficients. ) One or more instructions may be included that direct one or more processors to separate different sound sources from the input signal Xm (t) using a listening sector in sound source separation. The program 304 may further include instructions for applying one or more fractional delays to a selected input signal Xm (t) other than the input signal X ₀ (t) from the reference microphone M ₀ . Each non-integer delay may be selected to optimize the signal to noise ratio of the discrete time domain output signal y (t) from the microphone array. Fractional delays the signal from the reference microphone M ₀ may be chosen to be the first to signals from other microphones of the array. The program 304 sets the non-integer delay Δ to y (t + Δ) = X (t + Δ) * b ₀ + X (t−1 + Δ) * b ₁ + X (t−2 + δ) * b ₂ +... + X (t−N + Δ) b _N Instructions to be introduced into the output signal y (t) of the microphone array may be included. Here, Δ is 0 to ± 1. Examples of such techniques are disclosed in detail in US patent application Ser. No. 11 / 381,729, entitled “ULTRA SMALL MICROPHONE ARRAY”, inventor Xiadong Mao, filed May 4, 2006. Is incorporated herein by reference.

  The program 304, when executed, may include one or more instructions that cause the system 300 to select a pre-calibrated listening area that includes a sound source. Such instructions may cause the device to determine whether the sound source is in the initial region or on a particular side of the region. If the sound source is not in the default area, the instruction may select another area on a particular side of the default area at runtime. Another region may be characterized by the attenuation of the input signal closest to the optimal value. These instructions, when executed, may calculate the attenuation of the input signal from the microphone array 322 and the attenuation to the optimum value. The instructions, when executed, may be such that the apparatus 300 determines the attenuation value of the input signal for one or more regions and selects the region where the attenuation is closest to the optimal value. Examples of such techniques are disclosed in US patent application Ser. No. 11 / 381,725, filed May 4, 2006, entitled “METHOD AND APPARATUS FOR TARGETED SOUND DETECTION”, inventor Xiadong Mao. Is incorporated herein by reference.

  The signal from inertial sensor 332 provides part of the tracking information input, and the signal generated from tracking one or more light sources 334 by imaging device 323 provides another part of the tracking information input. By way of example and without limitation, such a “mixed mode” signal is used in a football-type video game where the quarterback is applied to the right after the feint moving the head to the left. Throw into. Specifically, the game player holding the controller 330 makes a voice while turning his head to the right and throwing it like a football with his head turned to the left. Along with the “acoustic radar” program code, the microphone array 320 can track the user's voice. The imaging device 323 can track the movement of the user's head or other commands that do not require sound or a controller. Sensor 332 may track the movement of a joystick controller (representing football). The imaging device 323 may also track the light source 334 of the controller 330. The user can throw a “ball” when the acceleration of the joystick controller 330 reaches a certain amount and / or in a certain direction, or based on a key command triggered by pressing a button on the controller 330.

  In some embodiments of the present invention, inertial signals such as from accelerometers or gyroscopes may be used to position controller 330. Specifically, the acceleration signal from the accelerometer may be temporally integrated once to determine the speed change, and the speed may be time integrated to determine the position change. If the initial position and speed at a certain time are known, the absolute position can be determined using these values, speed and position change. Although the position determination using the inertial sensor can be performed faster than the method using the imaging device 323 or the light source 334, the inertial sensor 332 is likely to cause a type of error known as “drift”. This is a phenomenon in which a discrepancy D may occur between the position of the joystick 330 (shown in a perspective view) calculated from the inertia signal and the actual position of the joystick controller due to errors accumulated over time. Embodiments of the present invention can handle such errors in various ways.

  For example, the drift can be manually canceled by setting the initial position of the controller 330 to be equal to the current calculated position. The user can trigger a command to reset the initial position using one or more buttons on the controller 330. Alternatively, an image-based drift can be canceled by resetting the current position to a position determined from a reference image obtained from the imaging device 323. Compensation for such image-based drift can be done manually, for example, when the user triggers one or more buttons on the joystick controller 330. Alternatively, compensation for image-based drift can be performed automatically, for example, at regular intervals or in response to game play. Such a technique is implemented by program code instructions 304 stored in the memory 302 and executed by the processor 301.

  In some embodiments, it may be desirable to correct error data in the inertial sensor signal. For example, the signal from the inertial sensor 332 can be oversampled, and a moving average can be calculated from the oversampled signal to remove error data from the inertial sensor signal. In certain situations, it may be desirable to oversample the signal, remove high and / or low values from some subset of data points, and calculate a moving average from the remaining data points. Also, other data sampling techniques and data processing techniques that adjust the signal from the inertial sensor to remove or reduce the superiority from the error data can be used. Which technique is selected depends on the nature of the signal, the computation on the signal, the nature of the game play, or a combination of these. Such a technique is realized by instructions of a program 304 stored in the memory 302 and executed by the processor 301.

  As described above, the processor 301 performs the analysis of the inertia signal data 306 according to the data 306 and the program code instructions of the program 304 stored and retrieved in the memory 302 and executed by the processor module 301. In addition, the processor may implement the virtual world simulation function described above as part of the program 304. In particular, the program 304 implements all or part of the various methods for communicating with the virtual world described above and / or the methods for interacting with the three-dimensional virtual world described above and / or the avatar email communication described above. May be. The code portion of program 304 may be compliant with any of a number of different programming languages such as assembly, C ++, JAVA, or many other languages. The processor module 301 forms a general-purpose computer that becomes a dedicated computer when executing a program such as the program code 304. Here, the program code 304 has been described as being embedded in software and executed by a general-purpose computer. However, for those skilled in the art, a task management method is another method, such as an application specific integrated circuit (ASIC) or other. It will be understood that it is implemented using hardware such as a hardware circuit. Thus, it should be understood that embodiments of the present invention, whether in whole or in part, are implemented in software, hardware, and a combination of both.

  In some embodiments, among other things, the program code 304 causes one or more processors to analyze signals from the inertial sensor 332 to generate position and / or direction information that can be used during video game play or with a virtual world. A processor-readable instruction set may be provided for use during communication with the 3D virtual world during communication. When executed, the program code 304 causes the imaging device 323 to monitor the front visual field, identify one or more light sources 334 within the visual field, detect changes in light from the light sources 334, and detect changes. A processor-executable instruction including one or more instructions that trigger an input command to the processor 301 may be optionally provided. Using an LED with an imaging device to trigger the action of a game controller is described in US Patent Application No. 10 / 759,782, filed January 16, 2004, “METHOD AND APPARATUS FOR LIGHT INPUT DEVICE”, inventor Richard. L. Marks, the disclosure of which is hereby incorporated by reference.

  When executed, program code 304 includes one or more instructions that, as described above, use signals from inertial sensors and signals from an imaging device that tracks one or more light sources as inputs to the game system. Processor-executable instructions may optionally be included. Program code 304 may optionally comprise processor-executable instructions that, when executed, include one or more instructions that compensate for drift of inertial sensor 332.

  Although embodiments of the present invention have been described as examples relating to video game controller 330, embodiments of the present invention including system 300 have inertial sensing capabilities and may be inertial sensor signals, whether wireless or otherwise. It can be used for any of a main body, a molded product, a knob, or a structure operated by a user having transmission capability.

  By way of illustration, embodiments of the invention can be implemented in a parallel processing system. Such parallel processing systems typically include multiple processors configured to execute portions of a program in parallel using separate processors. By way of example and without limitation, FIG. 4 illustrates one type of cell processor 400 according to an embodiment of the present invention. The cell processor 400 can be used as the processor 301 in FIG. 3 or in the simulation server 22 or the view server 24 in FIG. 1E. In the example of FIG. 4, the cell processor 400 includes a main memory 402, a power processor element (PPE) 404, and a number of synergistic processor elements (SPE) 406. In the example of FIG. 4, the cell processor 400 includes one PPE 404 and eight SPEs 406. In such a configuration, seven SPEs 406 are used for parallel processing, and one is reserved as a backup when one of the other seven fails. Alternatively, the cell processor may include a plurality of groups (PPE groups) composed of a plurality of PPEs and a plurality of groups (SPE groups) composed of a plurality of SPEs. In such a case, hardware resources are shared by each unit in one group. However, SPE and PPE need to appear to be independent elements to the software. Thus, the embodiment of the present invention is not limited to use in the configuration shown in FIG.

  Main memory 402 is typically a dedicated hardware register or array used for functions such as system configuration, data synchronous transfer, memory mapped I / O, and I / O subsystems in addition to general purpose non-volatile storage. Also equipped. In an embodiment of the present invention, video game program 403 may reside in main memory 402. The video program 403 may include an inertial, image and acoustic analyzer as described above for FIGS. 4, 5A, 5B and 5C, or a combination thereof. Program 403 may run on the PPE. The program 403 may be divided into a plurality of signal processing tasks executed at the SPE and / or PPE.

  By way of example, PPE 404 may be a 64-bit power PC processor unit (PPU) with associated caches L1 and L2. The PPE 404 is a general-purpose processing device that can access system management resources (for example, a memory protection table). Hardware resources may be explicitly mapped to the real address space as seen by the PPE. Therefore, the PPE can address any of these resources directly using the appropriate effective address value. The main functions of the PPE 404 are task management and task assignment for the SPE 406 of the cell processor 400.

  Although only one PPE is shown in FIG. 4, in a cell processor implementation such as the Cell Broadband Engine Architecture (CBEA), the cell processor 400 is divided into a plurality of PPE groups organized into a plurality of PPE groups. You may have PPE. These PPE groups may share access to the main memory 402. Further, the cell processor 400 may include two or more SPEs. SPE groups may also share access to main memory 402. Such a configuration is also within the scope of the present invention.

  Each SPE 406 includes an SPU and a unique local storage area LS. The local storage area LS includes one or more separation areas in the memory storage, and each area is associated with a dedicated SPU. Each SPU may be configured to only execute instructions (including data loading and data saving operations) from within its own associated storage area. In such a configuration, data transfer between the local storage area LS and the rest of the system 400 is performed by issuing a direct memory access (DMA) command from the memory flow controller (MFC) (for each individual SPE). ) Data is transferred to and from the local storage area. An SPU that does not perform any system management function is a computer device having a simpler structure than the PPE 404. SPUs generally have SIMD (single instruction, multiple data) functions, usually perform data processing and initiate requested data transfer to perform assigned tasks (according to access characteristics set by the PPE) To do. The purpose of the SPU is to enable applications that require higher computer device densities so that the provided instruction set can be used effectively. A significant number of SPEs in systems managed by PPE 404 allow for cost-effective processing in a wide range of applications.

  Each SPE 406 includes a dedicated memory flow controller (MFC) with an associated memory management unit that can hold and process memory protection information and access permission information. The MFC provides the main method for data transfer, protection and synchronization between the main storage of the cell processor and the local storage of the SPE. The MFC command describes the transfer to be performed. The data transfer command may be referred to as an MFC direct memory access (DMA) command (or MFCDMA instruction).

  Each MFC may support multiple DMA transfers simultaneously and maintain and process multiple MFC commands. Each MFCDMA data transfer command request may be accompanied by both a local storage address (LSA) and an effective address (EA). A local storage address can be addressed directly only to the local storage address of the associated SPE. The effective address may comprise a more general application, for example, if it can be renamed to the real address space, it can refer to the main memory including all SPE local storage areas.

  To facilitate communication between SPEs 406 and / or between SPE 406 and PPE 404, SPE 406 and PPE 404 may include a signal notification register for signaling events. PPE 404 and SPE 406 may be connected by a star topology that acts like a router where PPE 404 sends messages to SPE 406. Alternatively, each SPE 406 and PPE 404 may have a one-way signal notification register called a mailbox. The mailbox is used by the SPE 406 to host OS synchronization operations.

  Cell processor 400 may include input / output (I / O) functions that can interface with peripheral devices such as microphone array 412, optional imaging device 413, and game / virtual world controller 730. The controller unit 730 may include an inertial sensor 732 and a light source 734. Furthermore, the element interconnection bus 410 can connect the various parts listed above. Each SPE and PPE can access the bus 410 via a bus interface unit (BIU). The cell processor 400 controls the data flow between the I / O 408 and the bus 410, the two controllers normally found in the processor, the memory interface controller (MIC) that controls the data flow between the bus 410 and the main memory 402. A bus interface controller (BIC). Although the requirements for MIC, BIC, BIU and bus 410 vary greatly from implementation to implementation, those skilled in the art will be familiar with their functionality and implementation circuitry.

  The cell processor 400 may include an internal interrupt controller (IIC). The IIC device manages the priority of interrupts presented to the PPE. The IIC can handle interrupts from other elements of the cell processor 400 without using the main system interrupt controller. The IIC can also be thought of as a second level controller. The interrupt controller of the main system may process an interrupt generated outside the cell processor.

  In embodiments of the present invention, calculations to facilitate interaction with the virtual world may be performed in parallel using the PPE 404 and / or one or more SPEs 406. Each calculation may be performed as one or more separate tasks undertaken by different SPEs 406 as they become available.

  While the above is a complete description of the preferred embodiment of the present invention, it is possible to use various modifications, changes and equivalents. Accordingly, the scope of the invention should not be determined with reference to the above content, but instead should be determined in conjunction with the appended claims and their full scope of equivalents. . All suitable or otherwise described features herein may be combined with any other suitable or otherwise described features herein. In the claims, the indefinite article "A" or "An" refers to one or more quantities of items that follow the article, unless expressly stated otherwise. The appended claims should not be construed to include means plus function limitations, unless such limitations are expressly made using the phrase “means for”.

Claims (169)

A method for communicating with a virtual world, the virtual world including a simulated public space and one or more simulated private spaces that facilitate communication between multiple users, Each of the spaces is associated with an individual user of the plurality of users, the method comprising:
Simulating a communication device in the virtual world;
Communicating between the simulated communication device and a real communication device.   The method of claim 1, wherein the real communication device communicates with another real communication device through one or more communication channels independent of the virtual world. Simulating the communication device
Simulating the interface of the simulated communication device in the virtual world;
The method of claim 1, comprising presenting the simulated interface to a user for interaction therewith.   The method of claim 1, wherein the public space is provided to resemble a lobby.   The method of claim 1, wherein the one or more private spaces include one or more private spaces provided to resemble a room.   The method of claim 1, wherein the simulated communication device is a simulated handheld communication device.   The simulated handheld communication device is a simulated phone, a simulated mobile phone, a simulated text message device, a simulated email device, or a simulated two-way radio. Item 7. The method according to Item 6.   The method of claim 1, wherein the real communication device is a real handheld communication device.   9. The method of claim 8, wherein the real handheld communication device is a real phone, a real mobile phone, a real text message device, a real email device, or a real two-way radio.   The method of claim 1, wherein communicating between the simulated communication device and the real communication device includes placing a call between the real phone and the simulated phone. .   Communicating between the simulated communication device and the real communication device means sending one or more text messages between the simulated communication device and the real communication device. The method of claim 1 comprising. Associating an avatar with an individual user of the virtual world;
The method of claim 1, further comprising: expressing each individual user's interaction with the virtual world by one or more corresponding avatar actions.   The method of claim 1, wherein communicating between the simulated communication device and the real communication device includes accessing virtual communication device content using the real communication device.   The method of claim 1, wherein the virtual world is a three-dimensional virtual world.   Communicating between the simulated communication device and a real communication device includes using the real communication device as an interface between the simulated communication device and a user. Item 2. The method according to Item 1. Communicating between the simulated communication device and a real communication device is
Determining whether the recipient of the communication is a user of the virtual world;
Determining whether the recipient is interacting with the virtual world during communication;
When the receiver is communicating with the virtual world during communication, the receiver via a simulated communication device on a real device used by the receiver to interact with the virtual world The method of claim 1, comprising: contacting. Communicating between the simulated communication device and a real communication device is
Determining whether the recipient of the communication is a user of the virtual world;
Determining whether the recipient is interacting with the virtual world during communication;
When the receiver does not communicate with the virtual world during communication, a real communication device or a communication device simulated on the real device used by the receiver to interact with the virtual world. Leaving a message to the recipient. Communicating between the simulated communication device and a real communication device is
Determining whether the recipient of the communication is a user of the virtual world;
Determining whether the recipient is interacting with the virtual world during communication;
Contacting the receiver via an actual communication device when the receiver is not communicating with the virtual world at the time of communication,
The method of claim 1, wherein the real communication device communicates with other real communication devices through one or more communication channels independent of the virtual world. A method for communicating with a virtual world, the virtual world including a simulated public space and one or more simulated private spaces that facilitate communication between multiple users, Each of the spaces is associated with an individual user of the plurality of users, the method comprising:
Associating an avatar with an individual user of the virtual world;
Expressing each user's interaction with the virtual world by one or more corresponding avatar actions;
Communicating between the virtual world and a real communication device,
A method in which the real communication device communicates with another real communication device through one or more communication channels independent of the virtual world.   Communicating between the real world and the virtual world includes changing one or more characteristics associated with the avatar, the public space, or the private space via the real device. Item 20. The method according to Item 19.   The communication between the virtual world and a real communication device includes performing communication between the real communication device and a communication device simulated in the virtual world. The method described in 1.   Communicating between the simulated communication device and the real communication device means sending one or more text messages between the simulated communication device and the real communication device. The method of claim 21 comprising.   The communication between the virtual world and the real communication device includes establishing voice communication between the simulated communication device and the real communication device. the method of.   24. Establishing voice communication between the simulated communication device and the real communication device includes placing a call between the simulated phone and the real phone. the method of. The real phone generates a unique ring when receiving a call from the simulated communication device,
25. The method of claim 24, wherein the distinctive ringing tone is different from a normal ringing tone generated by the real telephone when receiving a call from another.   The method of claim 21, wherein the simulated communication device is a simulated video recorder.   Communicating between the real communication device and the simulated communication device includes using the real communication device as an interface between the simulated video recorder and a user, 27. The method of claim 26.   Using the real communication device as an interface includes configuring the simulated video recorder to record real or simulated video events using the real communication device. Item 27. The method according to Item 26.   The method according to claim 28, wherein the actual communication device is a telephone, a mobile phone, a portable game machine, a portable information terminal, or a portable electronic mail device.   The method of claim 21, wherein communicating between the real communication device and the virtual communication device includes video communication.   32. The method of claim 30, wherein video communication includes displaying the avatar image on the real communication device during the video communication.   32. The method of claim 31, further comprising lip-syncing the avatar to real or synthesized speech generated by a user associated with the avatar.   The method of claim 32, further comprising mapping emotions represented by the user to emotions represented by the avatar.   34. The method of claim 33, wherein mapping emotions represented by the user comprises obtaining one or more images of the user and analyzing one or more facial colors of the user.   35. The method of claim 34, wherein acquiring one or more images of the user and analyzing one or more facial colors of the user during the video communication includes using a three-dimensional camera.   35. The method of claim 33, wherein mapping emotions represented by the user comprises analyzing voice inflections of the user.   34. The method of claim 33, wherein mapping emotions represented by the user comprises analyzing biometric data indicative of the state of emotions of the user during a user portion of the video communication.   38. The method of claim 37, wherein the biometric data comprises electrocardiograph (EKG) data, electroencephalogram (EEG) data, skin electrical reflex data, or thermographic data.   35. The method of claim 33, wherein mapping emotions represented by the user comprises tracking the emotional state of the user in real time during a user portion of the video communication. A system for communicating with a virtual world, the system comprising one or more processors that generate the virtual world,
The virtual world includes a simulated public space and one or more simulated private spaces that facilitate interaction between multiple users, each of the private spaces being an individual of the multiple users. Associated with the user,
The one or more processors further includes
Simulate a communication device in the virtual world,
A system for communicating between the simulated communication device and a real communication device.   41. The system of claim 40, wherein the real communication device communicates with another real communication device through one or more communication channels independent of the virtual world.   41. The system of claim 40, wherein the public space is provided to resemble a lobby.   41. The system of claim 40, wherein the one or more private spaces include one or more private spaces provided to resemble a room.   41. The system of claim 40, wherein the simulated communication device is a simulated handheld communication device.   The simulated handheld communication device is a simulated phone, a simulated mobile phone, a simulated text message device, a simulated email device, or a simulated two-way radio. Item 45. The system according to Item 44.   41. The system of claim 40, wherein the real communication device is a real handheld communication device.   47. The system of claim 46, wherein the real handheld communication device is a real phone, a real mobile phone, a real text message device, a real email device, or a real two-way radio.   41. The system of claim 40, wherein the simulated communication device is a simulated telephone that communicates with a real telephone.   41. The system of claim 40, wherein the simulated communication device is a simulated text message device that communicates with a real text message device via one or more text messages. The simulated communication device is:
Determine whether the recipient of the communication is a user of the virtual world,
Determine whether the recipient is communicating with the virtual world during communication;
When the receiver is communicating with the virtual world during communication, the receiver via a simulated communication device on a real device used by the receiver to interact with the virtual world 41. The system of claim 40, wherein the system contacts. The simulated communication device is:
Determine whether the recipient of the communication is a user of the virtual world,
Determine whether the recipient is communicating with the virtual world during communication;
When the receiver does not communicate with the virtual world during communication, a real communication device or a communication device simulated on the real device used by the receiver to interact with the virtual world. 41. The system of claim 40, leaving a message to the recipient. The simulated communication device is:
Determine whether the recipient of the communication is a user of the virtual world,
Determine whether the recipient is communicating with the virtual world during communication;
If the receiver does not communicate with the virtual world during communication, contact the receiver via an actual communication device,
41. The system of claim 40, wherein the real communication device communicates with another real communication device through one or more communication channels independent of the virtual world. The one or more processors further includes
Associate avatars with individual users in the virtual world,
41. The system of claim 40, wherein each individual user's interaction with the virtual world is represented by one or more corresponding avatar actions.   41. The system of claim 40, wherein the one or more processors are configured to allow the real communication to access virtual communication device content.   41. The system of claim 40, wherein the virtual world is a three-dimensional virtual world.   41. The system of claim 40, wherein the one or more processors are configured such that the real communication device can be used as an interface between the simulated communication device and a user.   41. The system of claim 40, wherein the one or more processors include a cell processor.   41. The real communication device according to claim 40, wherein the actual communication device is a telephone, a cellular phone, a portable information terminal, a camera, a voice recorder, a portable electronic mail device, a portable Internet access device, a portable game machine, or a personal computer. system. A user interface connected to the one or more processors;
41. The system of claim 40, wherein the user interface facilitates interaction between the user and the virtual world. A system for communicating with a virtual world, the system comprising one or more processors that generate the virtual world,
The virtual world includes a simulated public space and one or more simulated private spaces that facilitate interaction between multiple users, each of the private spaces being an individual of the multiple users. Associated with the user,
The one or more processors further includes
Associate avatars with individual users in the virtual world,
Expressing each user's interaction with the virtual world by one or more corresponding avatar actions,
Perform communication between the virtual world and the actual communication device,
The real communication device communicates with other real communication devices through one or more communication channels independent of the one or more processors generating a virtual world.   61. The system of claim 60, wherein the one or more processors change one or more characteristics associated with the avatar, the public space or the private space in response to a signal generated by the real device.   64. The system of claim 60, wherein the one or more processors facilitate communication between the real communication device and a communication device simulated in the virtual world.   64. The system of claim 62, wherein the one or more processors facilitate communication between the real communication device and the simulated communication with one or more text messages.   The one or more processors facilitate communication between the real communication device and the simulated communication by voice communication between the simulated communication device and the real communication device; 64. The system of claim 62.   68. The system of claim 64, wherein voice communication between the simulated communication device and the real communication device includes a phone call between a simulated telephone and a real telephone. The real phone generates a unique ring when receiving a call from the simulated communication device,
66. The system of claim 65, wherein the unique ringing tone is different from a normal ringing tone generated by the real telephone when receiving a call from another.   64. The system of claim 62, wherein the simulated communication device is a simulated video recorder.   68. The system of claim 67, wherein the one or more processors are configured such that the real communication device can be used as an interface between the simulated video recorder and a user.   68. The one or more processors are configured such that the real communication device can be used to configure the simulated video recorder to record real or simulated video events. The system described in.   70. The real communication device according to claim 69, wherein the actual communication device is a telephone, a cellular phone, a portable information terminal, a camera, a voice recorder, a portable electronic mail device, a portable Internet access device, a portable game machine, or a personal computer. system.   The system of claim 70, wherein communicating between the real communication device and the virtual communication device includes video communication.   72. The system of claim 71, wherein video communication includes displaying the avatar image on the real device during the video communication.   73. The system of claim 72, wherein the one or more processors cause the avatar to lip sync to real or synthesized speech generated by the user.   The system according to claim 73, further comprising means for mapping an emotion represented by the user to an emotion represented by the avatar.   75. The system of claim 74, wherein the means for mapping emotions indicated by the user includes means for obtaining one or more images of the user and means for analyzing one or more facial colors of the user.   The system of claim 75, wherein the means for acquiring one or more images of the user during the video communication includes a three-dimensional camera.   77. The system of claim 76, wherein mapping emotions represented by the user comprises analyzing voice inflections of the user.   77. The system of claim 76, wherein the means for mapping emotions represented by the user comprises means for analyzing biometric data indicative of the state of emotion of the user during a user portion of the video communication.   79. The system of claim 78, wherein the biometric data comprises electrocardiograph (EKG) data, electroencephalogram (EEG) data, skin electrical reflex data, or thermographic data.   75. The system of claim 74, wherein means for mapping emotions represented by the user comprises means for tracking the state of emotions of the user in real time during a user portion of the video communication.   61. The system of claim 60, wherein the one or more processors are configured to allow the real communication to access virtual communication device content.   61. The system of claim 60, wherein the virtual world is a three dimensional virtual world.   61. The system of claim 60, wherein the one or more processors are configured such that the real communication device can be used as an interface between the simulated communication device and a user.   61. The system of claim 60, wherein the one or more processors include a cell processor.   61. The real communication device according to claim 60, wherein the actual communication device is a telephone, a mobile phone, a portable information terminal, a camera, a voice recorder, a portable electronic mail device, a portable Internet access device, a portable game machine, or a personal computer. system. A user interface connected to the one or more processors;
61. The system of claim 60, wherein the user interface facilitates interaction between the user and the virtual world. A method of interaction between a three-dimensional virtual world and a user, wherein the virtual world comprises one or more public spaces and one or more private spaces, wherein the user is represented by an avatar in the virtual world, and the user Can operate the avatar via a user interface, and the method includes:
a) detecting the state of emotion of the user via the interface;
b) mapping the emotional state of the user to one or more features that can be perceived by other users of the avatar;
The method in which the other user can perceive the emotional state of the user by perceiving the avatar.   90. The method of claim 87, wherein the one or more features include one or more complexions of the avatar.   90. The method of claim 87, wherein the one or more features include one or more body languages of the avatar.   90. The method of claim 87, wherein the one or more features include one or more complexions and body language of the avatar.   88. The method of claim 87, wherein detecting the state of emotion of the user comprises tracking one or more complexions of the user.   92. The method of claim 91, wherein tracking one or more facial colors includes tracking the one or more facial colors using a three-dimensional camera.   90. The method of claim 87, wherein detecting the user's emotional state comprises tracking one or more body languages of the user.   88. The method of claim 87, wherein detecting the user's emotional state comprises analyzing the user's voice inflection.   88. The method of claim 87, wherein detecting the user's emotional state comprises analyzing text generated by the user.   88. The method of claim 87, wherein detecting the user's emotional state comprises analyzing the user's biometrics.   99. The method of claim 96, wherein the biometric comprises electrocardiograph (EKG) data, electroencephalogram (EEG) data, electrodermal reflex data, or thermographic data. A method of interaction between a three-dimensional virtual world and a user, wherein the virtual world comprises one or more public spaces and one or more private spaces, wherein the user is represented by an avatar in the virtual world, and the user Can operate the avatar via a user interface, and the method includes:
Generating a custom gesture for the avatar;
Associating one or more user interface signals with the gesture. Further distributing the user interface signal to one or more other users;
99. The method of claim 98, wherein the other user can customize his avatar to perform the custom gesture.   99. The method of claim 98, wherein generating the custom gesture includes motion capture or performance capture of an actor performing the gesture.   101. The method of claim 100, wherein the actor is the user.   99. The method of claim 98, further comprising making the expression of the custom gesture by the avatar moderate in the virtual space.   Making the expression of the custom gesture moderate means limiting a range in which a specific user can make an avatar perform the custom gesture based on a predetermined criterion, or based on a predetermined criterion. 105. The method of claim 102, comprising limiting a range in which the particular user can perceive an avatar performing the custom gesture.   104. The predetermined criteria, wherein the predetermined criteria include a sensitivity that a viewer may feel insulting based on the age of the user, the age of the viewer of the gesture, or religion, ethnicity or other attribution of the viewer. Method.   99. The method of claim 98, further comprising customizing one or more other characteristics of the avatar to associate the user and / or avatar with a social group.   106. The method of claim 105, wherein customizing one or more other characteristics of the avatar comprises customizing clothing, footwear, hairstyle or ethnic characteristics of the avatar. A method for routing communications between a real communication device and a virtual communication device, comprising:
Associating the virtual world user with a virtual communication device and one or more real communication devices;
Performing routing priority processing between the virtual communication device and the one or more real communication devices upon receiving communication addressed to the user.   108. The method of claim 107, wherein the user performs the routing priority process. 108. The method of claim 107, wherein a source of the communication performs the routing priority process.   108. The method of claim 107, wherein the routing priority process is based on the user preference.   108. The method of claim 107, wherein the routing priority process is based on the user behavior.   108. The method of claim 107, wherein the routing priority process depends on the identity of the communication source.   108. The method of claim 107, wherein the routing priority process depends on the identity of the communication target. The routing priority process is based on one or more state controls,
108. The method of claim 107, wherein the state control routes the communication to the real or virtual communication device only in predetermined situations.   108. The method of claim 107, further comprising routing communications addressed to the user according to the routing priority process. Routing the communication is
Determining whether the user is “online” in the virtual world;
116. The method of claim 115, comprising routing the communication to the virtual communication device when the user is online. Routing the communication is
Determining whether the user is “online” in the virtual world;
116. The method of claim 115, comprising routing the communication to one or more real communication devices when the user is not online. Routing the communication is
Determining whether the user is “online” in the virtual world;
116. The method of claim 115, comprising routing a text message to either the virtual communication device or an avatar associated with the user when the user is online.   116. The method of claim 115, wherein the communication destined for the user originates from within the virtual world. Communicating between the simulated communication device and a real communication device includes communicating communication from a source to a target;
An avatar is associated with the source,
108. The method of claim 107, further comprising displaying the avatar on the target in the virtual world during the communication.   121. The method of claim 120, wherein the communication originates from a real communication device.   121. The method of claim 120, further comprising presenting audio from the source to the target during the communication.   121. The method of claim 120, further comprising presenting text from the source to the target during the communication.   124. The method of claim 123, wherein presenting text from the source includes translating speech spoken from the source into text and presenting the resulting text to the target.   129. The method of claim 124, wherein presenting the resulting text avatar includes displaying the resulting text in one or more speech bubbles near the avatar. A system for routing communication between a real communication device and a virtual communication device,
With one or more processors to create a virtual world,
The one or more processors generate virtual communication devices and implement configurable routers;
The configurable router facilitates routing in communication between one or more real communication devices and the virtual communication device.   127. The system of claim 126, wherein the router is accessible from the virtual communication device or the one or more real communication devices.   128. The system of claim 127, wherein the one or more real communication devices include one or more network web pages.   127. The system of claim 126, wherein the router routes communications based on a user associated with the virtual communication device.   127. The system of claim 126, wherein the router routes communications based on a source of the communications.   129. The system of claim 126, wherein the router routes communications based on one or more preferences of users associated with the virtual communication device.   127. The system of claim 126, wherein the router routes communications based on one or more user actions associated with the virtual communication device.   127. The system of claim 126, wherein the router routes communications based on the identity of the communication source. 127. The system of claim 126, wherein the router routes communications based on the identity of the communications target. The router routes communications based on one or more state controls,
127. The system of claim 126, wherein the state control routes the communication to the real or virtual communication device only in predetermined situations. The router
Determine if the user is "online" in the virtual world;
127. The system of claim 126, wherein the communication is routed to the virtual communication device when the user is online. The router
Determine if the user is "online" in the virtual world;
129. The system of claim 126, wherein the communication is routed to one or more real communication devices when the user is not online. The router
Determine if the user is "online" in the virtual world;
127. The system of claim 126, wherein text messages are routed to either the virtual communication device or an avatar associated with the user when the user is online. The router communicates communication from a source to a target, and displays the avatar on the target in the virtual world during the communication;
127. The system of claim 126, wherein the avatar is associated with the source.   140. The system of claim 139, wherein the one or more processors present audio from the source to the target during the communication.   140. The system of claim 139, wherein the one or more processors present text from the source to the target during the communication.   142. The system of claim 141, wherein the one or more processors translate speech spoken from the source into text and present the resulting text to the target.   143. The method of claim 142, wherein presenting the resulting text avatar includes displaying the resulting text in one or more speech bubbles near the avatar. A method for communicating using a virtual world,
Associating an avatar with the email source,
Generating an email in the virtual world;
Associating one or more images of the avatar with the email;
Sending the email to a real device;
Presenting the one or more images at a destination of the e-mail. Associating recorded or synthesized speech with the email;
145. The method of claim 144, further comprising presenting the audio with the one or more images at the destination.   145. The method of claim 144, wherein associating one or more images includes generating an animation of the avatar.   147. The method of claim 146, wherein presenting the one or more images includes presenting the animation at the destination.   147. The method of claim 146, wherein associating one or more images further comprises mapping one or more gestures to the animation of the avatar. Mapping one or more gestures
Recording the voice and / or video of the originator of the email message;
148. The method of claim 148, comprising mapping one or more features of the audio and / or video to one or more features of the avatar in the animation.   147. The method of claim 146, further comprising associating a theme with the movement of the virtual camera during the animation.   The method of claim 150, wherein the theme includes a fixed position of a virtual camera.   161. The method of claim 150, wherein the theme includes virtual camera movement.   161. The method of claim 150, wherein generating the email comprises tracking the source emotional state and mapping the theme to an emotional state.   145. The method of claim 144, wherein generating the email includes using a virtual communication device in the virtual world. The virtual world includes a simulated public space and one or more simulated private spaces that facilitate communication between multiple users;
145. The method of claim 144, wherein each private space is associated with an individual user of the plurality of users.   145. The method of claim 144, wherein the destination is an actual communication device.   157. The method of claim 156, wherein the real communication device communicates with other real communication devices through one or more communication channels independent of the virtual world. A system for communicating using a virtual world, comprising one or more processors, wherein the one or more processors are:
Create a virtual world,
Associate your avatar with the email source,
Create an email in the virtual world,
Associate one or more images of the avatar with the email,
Send the email to a real device,
A system that presents the one or more images at a destination of the e-mail. The virtual world includes a simulated public space and one or more simulated private spaces that facilitate communication between multiple users;
159. The system of claim 158, wherein each private space is associated with an individual user of the plurality of users. The one or more processors are:
Associating recorded and / or synthesized speech with the email;
159. The system of claim 158, wherein the audio is presented with the one or more images at the destination.   159. The system of claim 158, wherein the one or more processors generate an animation of the avatar.   164. The system of claim 161, wherein the one or more processors present the animation at the destination.   163. The system of claim 162, wherein the one or more processors map one or more gestures to the animation of the avatar. The one or more processors are:
Record the voice and / or video of the originator of the email message;
166. The system of claim 163, mapping one or more features of the audio and / or video to one or more features of the avatar in the animation.   164. The system of claim 161, wherein the one or more processors associate a theme with a virtual camera movement during the animation.   166. The system of claim 165, wherein the theme includes a fixed location of a virtual camera.   166. The system of claim 165, wherein the theme includes virtual camera movement.   166. The system of claim 165, wherein the one or more processors track the source emotional state and map the theme to an emotional state.   159. The system of claim 158, wherein the one or more processors generate a virtual communication device in the virtual world and allow a user to generate the email using the virtual communication device.

Images