KR20220160699A - Web-based video conferencing virtual environment with navigable avatars and its applications - Google Patents

Abstract

A web-based video conferencing system that allows video avatars to navigate within a virtual environment is disclosed herein. The system has a presentation mode that allows the presentation stream to be texture mapped to a presenter's screen located within the virtual environment. The relative left and right sounds are adjusted to provide a sense of the avatar's position in virtual space. The sound is further adjusted based on the area where the avatar is located and the area where the virtual camera is located. Video stream quality is adjusted based on relative position in virtual space. Three-dimensional modeling is available inside the virtual videoconferencing environment.

Description

Web-based video conferencing virtual environment with navigable avatars and its applications

[Cross References to Related Applications]

This application is filed as US Patent No. 10,979,672, issued on April 13, 2021, and US Practical Patent Application Serial No. 17/075,338, filed on October 20, 2020, currently filed on March 11, 2021 U.S. Utility Patent Application No. 17/198,323 filed on August 17, 2021, U.S. Utility Patent Application No. 17/075,362, filed on October 20, 2020, currently registered as U.S. Patent No. 11,095,857, filed on August 17, 2021; US Patent Application Serial No. 17/075,390, filed on October 20, 2020, currently registered as US Patent No. 10,952,006, issued on March 16, 2021, US Patent issued on July 20, 2021 U.S. Utility Patent Application No. 17/075,408, filed on October 20, 2020, currently registered as No. 11,070,768; Priority is claimed to U.S. Utility Patent Application Serial No. 17/075,428, filed October 20, and U.S. Utility Patent Application Serial No. 17/075,454, filed October 20, 2020. The contents of each of these applications are incorporated herein by reference in their entirety.

[Technical field]

This technical field generally relates to video conferencing.

[Related technology]

Videoconferencing involves the reception and transmission of audio-video signals by users in different locations for interpersonal communication in real time. Video conferencing is widely available on many computing devices from a variety of different services, including the ZOOM service available from Zoom Communications Inc. of San Jose, California. Some video conferencing software, such as the FaceTime application available from Apple Inc. of Cupertino, California, comes standard with mobile devices.

Generally, these applications work by displaying video of other conference participants and outputting audio. When there are multiple participants, the screen may be divided into multiple rectangular frames, each displaying a participant's video. Sometimes these services work by having a larger frame presenting video of the person speaking. When different individuals speak, the frame will switch between the speakers. The application captures video from a camera integrated with the user's device and audio from a microphone integrated with the user's device. The application then transmits that audio and video to other applications running on other users' devices.

Many of these video conferencing applications have screen sharing capabilities. When a user decides to share their screen (or a portion of their screen), a stream is sent to other users' devices along with the contents of their screen. In some cases, other users may even control what is on the user's screen. In this way, users can collaborate on a project or present a presentation to other meeting participants.

Recently, the importance of video conferencing technology has increased. Many workplaces, trade shows, conferences, conferences, schools, and places of worship have been closed or people have been encouraged not to attend for fear of spreading disease, particularly COVID-19. Virtual meetings using video conferencing technology are increasingly replacing physical meetings. Additionally, this technology offers advantages over physically meeting to avoid travel and commuting.

Often, though, the use of this video conferencing technology causes a loss of sense of place. There is an experiential aspect of meeting physically in person while being in the same location that is lost when a meeting is conducted virtually. There is a social aspect to being able to pose and look up to your peers. The feeling of this experience is important in creating relationships and social connections. However, this feeling is lacking as far as conventional video conferences are concerned.

Furthermore, additional problems arise with these video conferencing technologies when multiple participants in a conference begin to gather. In a meeting that meets physically, people can have side conversations. You can project your voice so that only those close to you can hear what you are saying. In some cases, you may even have a private conversation in connection with a larger meeting. However, in a virtual conference, when multiple people are speaking at the same time, the software mixes the two audio streams substantially equally, allowing the participants to speak on top of each other. Thus, when a large number of people are involved in a virtual meeting, private conversation is not possible, and conversation tends to be more in the form of one-to-many utterances. Here, too, virtual meetings lose opportunities for participants to make social connections, communicate more effectively, and network.

Moreover, due to limitations in network bandwidth and computing hardware, the performance of many video conferencing systems starts to slow down when many streams are deployed in a conference. Many computing devices are equipped to process video streams from a small number of participants, but are insufficiently equipped to process video streams from dozens of participants. Because many schools operate completely virtual, a class of 25 can seriously slow down school-issued computing devices.

Massively multiplayer online games (MMOGs or MMOs) can generally handle significantly more than 25 players. These games often have hundreds or thousands of players on a single server. MMOs often allow players to navigate avatars through virtual worlds. Sometimes these MMOs allow users to talk to each other or send messages to each other. Examples include the ROBLOX game available from Roblox Corporation of San Mateo, Calif. and the MINECRAFT game available from Mojang Studios of Stockholm, Sweden.

Having bare avatars interact with each other also has limitations in terms of social interaction. These avatars are generally unable to convey the facial expressions that people often make inadvertently. In a video conference, these facial expressions are observable. Some publications may describe placing a video on an avatar in a virtual world. However, these systems typically require special software and have other limitations that limit their usefulness.

Improved methods for video conferencing are needed.

In one embodiment, the device enables video conferences between a first user and a second user. The device includes a processor coupled to memory, a display screen, a network interface, and a web browser. The network interface includes: (i) data specifying a three-dimensional virtual space, (ii) a position and orientation in the three-dimensional virtual space, the position and orientation being input by the first user, and (iii) the first user's It is configured to receive a video stream captured from a camera on the device. The camera of the first user is arranged to capture photographic images of the first user. A web browser implemented in the processor is configured to download a web application from a server and execute the web application. A web application includes a texture mapper and renderer. The texture mapper is configured to texture map the video stream onto the three-dimensional model of the avatar. The renderer is configured to render, from the point of view of the second user's virtual camera, a three-dimensional virtual space comprising a texture-mapped three-dimensional model of the avatar located at that location and oriented in that direction for display to the second user. By managing texture mapping within the web application, the embodiments eliminate the need to install special software.

In one embodiment, a computer implemented method enables presentation in a virtual conference involving a plurality of participants. In this method, data specifying a three-dimensional virtual space is received. A position and orientation in three-dimensional virtual space are also received. The location and orientation were entered by a first participant of the plurality of participants to the conference. Finally, a video stream captured from a camera on the first participant's device is received. A camera was positioned to capture photographic images of the first participant. The video stream is texture mapped onto the avatar's three-dimensional model. Additionally, a presentation stream is received from the first participant's device. The presentation stream is texture mapped onto a 3D model of the presentation screen. Finally, from the viewpoint of the virtual camera of a second participant among the plurality of participants, a three-dimensional virtual space having a texture-mapped avatar and a texture-mapped presentation screen is rendered for display to the second participant. In this way, embodiments enable presentation in a social conference environment.

In one embodiment, a computer implemented method provides audio for a virtual conference involving a plurality of participants. In this method, a three-dimensional virtual space containing an avatar texture-mapped from the viewpoint of a first user's virtual camera to a video of a second user is rendered for display to the first user. The virtual camera is at a first position in the 3D virtual space and the avatar is at a second position in the 3D virtual space. An audio stream from a microphone of a second user's device is received. A microphone was positioned to capture the second user's utterance. The volume of the received audio stream is adjusted to determine the left and right audio streams to provide a sense of where the second position is relative to the first position in three-dimensional virtual space. The left audio stream and the right audio stream are output to be reproduced to the first user in stereo.

In one embodiment, a computer implemented method provides audio for a virtual conference. In this method, a three-dimensional virtual space containing an avatar texture-mapped from the viewpoint of a first user's virtual camera to a video of a second user is rendered for display to the first user. The virtual camera is at a first position in the 3D virtual space and the avatar is at a second position in the 3D virtual space. An audio stream from a microphone of a second user's device is received. It is determined whether the virtual camera and the avatar are located in the same area among the plurality of areas. When it is determined that the virtual camera and avatar are not located in the same area, the audio stream is attenuated. The attenuated audio stream is output to be reproduced to the first user. In this way, embodiments enable private and secondary conversations in a virtual videoconferencing environment.

In one embodiment, a computer implemented method efficiently streams video for a virtual conference. In this method, a distance between a first user and a second user in a virtual meeting space is determined. A video stream captured from a camera on a device of a first user is received. The camera is positioned to capture photographic images of the first user. The resolution or bitrate of the video stream is reduced based on the determined distance such that closer distances result in greater resolution than greater distances. The video stream is transmitted to the second user's device at a reduced resolution or bitrate for display to the second user in the virtual meeting space. The video stream is texture mapped onto the first user's avatar for display to the second user in the virtual meeting space. In this way, embodiments efficiently allocate bandwidth and computing resources even when there are a large number of conference participants.

In one embodiment, a computer implemented method enables modeling in virtual videoconferencing. In this method, a three-dimensional model of a virtual environment, a mesh representing a three-dimensional model of an object, and video streams from participants in a virtual videoconference are received. The video stream is texture mapped to an avatar that can be navigated by participants. Meshes representing three-dimensional models of texture-mapped avatars and objects within the virtual environment are rendered for display.

System, device, and computer program product embodiments are also disclosed.

Additional embodiments, features, and advantages of the present invention, as well as the structure and operation of various embodiments, are described in detail below with reference to the accompanying drawings.

The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate the present disclosure and, together with the description, further explain the principles of the present disclosure and enable those skilled in the relevant art to make and use the present disclosure. play a role in
1 is a diagram illustrating an exemplary interface for providing video conferencing in a virtual environment in which video streams are mapped onto avatars.
2 is a diagram illustrating a three-dimensional model used to render a virtual environment with avatars for videoconferencing.
3 is a diagram illustrating a system for providing video conferencing in a virtual environment.
4A-4C illustrate how data is transferred between various components of the system in FIG. 3 to provide video conferencing.
5 is a flowchart illustrating a method for adjusting relative left and right volumes to provide a sense of position in a virtual environment during a video conference.
6 is a chart illustrating how the volume rolls off as the distance between avatars increases.
7 is a flowchart illustrating a method for adjusting relative volume to present different volume areas in a virtual environment during a video conference.
8A and 8B are diagrams illustrating different volume areas in a virtual environment during a videoconference.
9A-9C are diagrams illustrating traversing a hierarchy of volume areas in a virtual environment during a videoconference.
10 illustrates an interface with a 3D model in a 3D virtual environment.
11 illustrates presentation screen sharing in a 3D virtual environment used for video conferences.
12 is a flowchart illustrating a method for apportioning available bandwidth based on the relative positions of avatars within a three-dimensional virtual environment.
13 is a chart illustrating how priority values may drop as the distance between avatars increases.
14 is a chart illustrating how allocated bandwidth can vary based on relative priority.
15 is a diagram illustrating components of devices used to provide video conferencing within a virtual environment.
The drawing in which an element first appears is typically indicated by the leftmost digit or digits in the corresponding reference number. In the drawings, like reference numbers may indicate identical or functionally similar elements.

Video conference using avatars in a virtual environment

1 is a diagram illustrating an example of an interface 100 for providing video conferencing in a virtual environment where video streams are mapped onto avatars.

Interface 100 may be displayed to participants in a video conference. For example, interface 100 can be rendered for display to participants and continuously updated as the video conference progresses. A user can control the orientation of his virtual camera using, for example, keyboard inputs. In this way, the user can navigate through the virtual environment. In one embodiment, the different inputs may change the X and Y positions and pan and tilt angles of the virtual camera in the virtual environment. In further embodiments, the user may use the inputs to change the height (Z coordinate) or yaw of the virtual camera. In yet further embodiments, the user can enter inputs that cause the virtual camera to “hop” upward and return to its original position simulating gravity. Inputs available for navigating the virtual camera include, for example, WASD keyboard keys to move the virtual camera back and forth and left and right in the X-Y plane, the space bar key to "hop" the virtual camera, and the pan and tilt angles. It may include keyboard and mouse inputs, such as mouse movements specifying changes.

Interface 100 includes avatars 102A and 102B, each representing a different participant in a video conference. Avatars 102A and 102B are texture mapped into video streams 104A and 104B from the devices of the first participant and the second participant, respectively. A texture map is an image that is applied (mapped) to the surface of a shape or polygon. Here, images are respective frames of video. The camera devices that capture the video streams 104A and 104B are positioned to capture the faces of the respective participants. In this way, avatars are texture mapped into moving images of faces as participants in the conference speak and listen.

Similar to the way a virtual camera is controlled by user viewing interface 100, the position and orientation of avatars 102A and 102B are controlled by the respective participants they represent. Avatars 102A and 102B are three-dimensional models represented by a mesh. Each avatar 102A and 102B may have the participant's name below the avatar.

Respective avatars 102A and 102B are controlled by various users. Each of these may be placed at a point corresponding to where its virtual cameras are located within the virtual environment. Just as user viewing interface 100 can move a virtual camera around, various users can move their respective avatars 102A and 102B around.

The virtual environment rendered on the interface 100 includes a background image 120 and a three-dimensional model 118 of an arena. An arena may be a place or building where a video conference is to be held. An arena may include a floor area bounded by walls. The 3D model 118 may include meshes and textures. Other ways of mathematically representing the surface of the three-dimensional model 118 may be possible. For example, polygon modeling, curve modeling, and digital sculpting may be possible. For example, the three-dimensional model 118 may be represented by voxels, splines, geometric primitives, polygons, or any other possible representation in three-dimensional space. The three-dimensional model 118 may also include specifications of light sources. Light sources may include, for example, point light sources, directional light sources, spotlight sources, and ambient light sources. Objects can also have certain properties that describe how objects reflect light. In examples, properties may include diffuse lighting, ambient lighting, and spectral lighting interactions.

In addition to the arena, the virtual environment may include a variety of other three-dimensional models illustrating the different components of the environment. For example, the 3D environment may include a decoration model 114 , a speaker model 116 , and a presentation screen model 122 . Like models 118, these models can be represented using any mathematical way of representing a geometric surface in three-dimensional space. These models may be separate from model 118 or combined into a single representation of the virtual environment.

Decorative models, such as model 114, serve to enhance realism and increase the aesthetic appeal of the arena. As will be described in more detail below with respect to FIGS. 5 and 7 , the speaker model 116 can virtually emit sounds such as presentations and background music. The presentation screen model 122 may serve to provide an exit for presenting a presentation. The presenter's video or presentation screen share may be texture mapped onto the presentation screen model 122 .

Button 108 may present a list of participants to the user. In one example, after the user selects button 108, the user may chat with other participants either individually or as a group by sending text messages.

Button 110 may enable a user to change attributes of the virtual camera used to render interface 100 . For example, a virtual camera may have a field of view that specifies an angle at which data is rendered for display. Modeling data within the camera's field of view may be rendered, whereas modeling data outside the camera's field of view may not be rendered. By default, the virtual camera's field of view can be set to somewhere between 60° and 110°, corresponding to a wide-angle lens and human vision. However, selecting button 110 may cause the virtual camera to increase its field of view beyond 170°, which corresponds to a fisheye lens. This may enable a user to have a wider ambient awareness of their surroundings in a virtual environment.

Finally, button 112 allows the user to exit the virtual environment. Selecting button 112 may cause a notification to be sent to devices belonging to the other participants signaling the devices of the other participants to stop displaying the avatar corresponding to the user who was previously viewing interface 100 . have.

In this way, the interface virtual 3D space is used to conduct video conferences. Every user controls an avatar, and a user can control an avatar to move, look around, jump, or do other things that change position or direction. The virtual camera shows the user a virtual 3D environment and other avatars. Other users' avatars have as an integral part a virtual display showing the user's webcam image.

By providing users with a sense of space and allowing users to see each other's faces, the embodiments provide a more social experience than conventional web conferencing or conventional MMO gaming. A more social experience has a variety of applications. For example, it can be used in online shopping. For example, interface 100 may be used for virtual grocery stores, places of worship, trade shows, B2B sales, B2C sales, schooling, restaurants or cafeterias, product launches, construction sites (eg, for architects, engineers, contractors). Visits, office spaces (e.g. people virtually work "at their desks"), remotely controlling machines (ships, vehicles, airplanes, submarines, drones, drilling rigs, etc.), plants/plants Control rooms, medical procedures, garden design, guided virtual bus tours, music events (e.g. concerts), lectures (e.g. TED Talks), political party meetings, board meetings, underwater research, inaccessible locations. Research, emergency (eg fire) training, cooking, shopping (including checkout and delivery), virtual arts and crafts (eg painting and pottery), weddings, funerals, baptisms, distance sports training, counseling , fear therapy (eg, face-to-face therapy), fashion shows, amusement parks, home décor, watching sports, watching esports, watching performances captured using 3-D cameras, playing board and role-playing games, medical imagery review, viewing geologic data, language learning, meetings in spaces for the blind, meetings in spaces for the hearing impaired, participation in events for people who cannot normally walk or stand, presenting news or weather, talk shows, book signings, voting ; administration, fencing/swordfighting/martial arts, reenactment (eg crime scene and/or accident reenactment), rehearsal of real events (eg weddings, presentations, shows, spacewalks), real events captured by 3D cameras evaluation or viewing of animals, livestock shows, zoos, experiencing tall/short/blind/deaf/white/black life (e.g. (e.g. modified video streams or still images of virtual worlds to simulate the perspective the user wants to experience a reaction to), interviews, game shows, interactive novels (e.g. murder mysteries), virtual fishing, virtual voyages. , psychological research, behavioral analysis, virtual sports (e.g., mountaineering/bouldering), control of lights at home or other locations (domotics), memory palaces, archeology, gift shops, Virtual visits to provide comfort, virtual medical procedures to explain procedures and make people feel more comfortable, and virtual exchanges/financial markets/stock markets (e.g., incorporating real-time data and video feeds into the virtual world, real-time trading and analysis ), virtual locations that people must go to as part of their work so that they can actually meet each other organically (for example, if you want to create an invoice, this can only be done inside the virtual location), and AR headsets so that you can see people's facial expressions ( or helmet), augmented reality (e.g. useful for military, law enforcement, firefighters, special operations), and providing reservations (e.g. for certain vacation homes/cars, etc.) has applications.

2 is a diagram 200 illustrating a three-dimensional model used to render a virtual environment with avatars for video conferencing. As illustrated in FIG. 1 , the virtual environment here includes a three-dimensional arena 118 and various three-dimensional models including three-dimensional models 114 and 122 . As also illustrated in FIG. 1 , diagram 200 includes avatars 102A and 102B navigating around the virtual environment.

As described above, the interface 100 in FIG. 1 is rendered from the perspective of a virtual camera. That virtual camera is illustrated in diagram 200 as virtual camera 204 . As mentioned above, the user viewing interface 100 in FIG. 1 can control the virtual camera 204 and navigate the virtual camera in three-dimensional space. The interface 100 is continuously updated according to the new location of the virtual camera 204 and any changes to the models within the field of view of the virtual camera 204 . As described above, the field of view of the virtual camera 204 may be a frustum defined, at least in part, by a horizontal field of view and a vertical field of view.

As described above with respect to FIG. 1 , a background image or texture may define at least a portion of the virtual environment. The background image may capture aspects of the virtual environment that are intended to appear at a distance. The background image may be a texture mapped onto the sphere 202 . Virtual camera 204 may be at the origin of sphere 202 . In this way, distant features of the virtual environment can be efficiently rendered.

In other embodiments, other shapes may be used to texture map the background image instead of sphere 202 . In various alternative embodiments, the shape may be a cylinder, a cube, a rectangular prism, or any other three-dimensional geometry.

3 is a diagram illustrating a system 300 for providing video conferencing in a virtual environment. System 300 includes a server 302 coupled to devices 306A and 306B via one or more networks 304 .

Server 302 provides services for establishing a video conference session between device 306A and device 306B. As will be described in more detail below, server 302 is responsible for monitoring conference participants' devices (e.g., devices 306A and 306B) as new participants join the conference and as existing participants leave the conference. ) to communicate notifications. The server 302 communicates messages describing the position and orientation in the 3-D virtual space to each participant's virtual cameras within the 3-D virtual space. Server 302 also communicates video and audio streams between the participants' respective devices (eg, devices 306A and 306B). Finally, server 302 stores and transmits data describing data specifying a three-dimensional virtual space to respective devices 306A and 306B.

In addition to the data needed for the virtual conference, server 302 may provide actionable information that tells devices 306A and 306B how to render the data to provide an interactive conference.

Server 302 responds to requests with a response. Server 302 may be a web server. A web server is software and hardware that uses the Hypertext Transfer Protocol (HTTP) and other protocols to respond to client requests made over the World Wide Web. A web server's main job is to display website content by storing, processing and delivering web pages to users.

In an alternative embodiment, communication between devices 306A and 306B occurs on a peer-to-peer basis and not through server 302 . In that embodiment, one or more of data describing the location and orientation of respective participants, notifications regarding new and existing participants, and video and audio streams of respective participants are sent to devices without going through server 302. There is direct communication between 306A and 306B.

Network 304 enables communication between various devices 306A and 306B and server 302 . The network 304 includes an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless wide area network (WWAN), a MAN ( metropolitan area network), a portion of the Internet, a portion of a Public Switched Telephone Network (PSTN), a cellular telephone network, a wireless network, a WiFi network, a WiMax network, any other type of network, or any combination of two or more such networks. can be a combination.

Devices 306A and 306B are respective devices of respective participants of the virtual conference. Devices 306A and 306B each receive data necessary to conduct the virtual conference and render data necessary to provide the virtual conference. As will be described in more detail below, devices 306A and 306B include a display for presenting rendered conferencing information, inputs allowing the user to control a virtual camera, and providing audio to the user for the conferencing. a speaker (eg, headset) for listening, a microphone for capturing the user's voice input, and a camera positioned to capture video of the user's face.

Devices 306A and 306B may be any type of computing device, including a laptop, desktop, smartphone, or tablet computer, or wearable computer (eg, smartwatch or augmented reality or virtual reality headset).

Web browsers 308A and 308B may retrieve and present network resources (eg, web pages) addressed by link identifiers (such as Uniform Resource Locators or URLs) and present network resources for display. . Specifically, web browsers 308A and 308B are software applications for accessing information on the World Wide Web. Typically, web browsers 308A and 308B make this request using a hypertext transfer protocol (HTTP or HTTPS). When a user requests a web page from a particular website, the web browser retrieves the necessary content from the web server, interprets and executes the content, and then sends the page shown as the client/party conferencing application 308A and 308B to the device 306A. and 306B) on the display on the display. In examples, the content may have HTML and client-side scripting, such as JavaScript. Once displayed, the user can enter information and make selections on the page, which can cause web browsers 308A and 308B to make further requests.

Conferencing applications 310A and 310B may be web applications downloaded from server 302 and configured to be executed by respective web browsers 308A and 308B. In one embodiment, conferencing applications 310A and 310B may be JavaScript applications. In one example, conferencing applications 310A and 310B may be written in a higher level language, such as the Typescript language, and translated or compiled into JavaScript. Conferencing applications 310A and 310B are configured to interact with the WebGL JavaScript application programming interface. It can have control code specified in JavaScript and shader code written in GLSL ES (OpenGL ES Shading Language). Using the WebGL API, conferencing applications 310A and 310B may utilize graphics processing units (not shown) of devices 306A and 306B. Moreover, OpenGL rendering of interactive 2D and 3D graphics without the use of plugins.

Conferencing applications 310A and 310B receive data describing the positions and orientations of other avatars and three-dimensional modeling information describing the virtual environment from server 302 . Additionally, conferencing applications 310A and 310B receive video and audio streams of other conference participants from server 302 .

Conferencing applications 310A and 310B render three-dimensional modeling data, including data describing the three-dimensional environment and data representing respective participant avatars. This rendering may involve rasterization, texture mapping, ray tracing, shading, or other rendering techniques. In one embodiment, rendering may involve ray tracing based on the properties of the virtual camera. Ray tracing involves creating an image by tracing the path of light as pixels in the image plane and simulating the effects of encounters with virtual objects. In some embodiments, ray tracing can simulate optical effects such as reflection, refraction, scattering, and dispersion to enhance realism.

In this way, users enter the virtual space using web browsers 308A and 308B. A scene is displayed on the user's screen. The user's webcam video stream and microphone audio stream are sent to server 302 . When other users enter the virtual space, avatar models for them are created. The location of this avatar is transmitted to the server and received by other users. Other users also receive notification from server 302 that an audio/video stream is available. A user's video stream is placed on an avatar created for that user. The audio stream is played as if coming from the avatar's position.

4A-4C illustrate how data is transferred between various components of the system in FIG. 3 to provide video conferencing. Like FIG. 3 , each of FIGS. 4A-4C depicts a connection between server 302 and devices 306A and 306B. In particular, FIGS. 4A-4C illustrate example data flows between the devices.

4A illustrates a diagram 400 illustrating how server 302 transmits data describing a virtual environment to devices 306A and 306B. Specifically, devices 306A and 306B both send the 3D arena 404, background texture 402, spatial hierarchy 408 and any other 3D modeling information 406 from server 302. receive

As described above, background texture 402 is an image representing distant features of the virtual environment. The image may be regular (eg brick wall) or irregular. Background texture 402 may be encoded in any common image file format, such as bitmap, JPEG, GIF, or other file image format. This describes, for example, a background image to be rendered for a sphere far away.

The 3D arena 404 is a 3D model of the space where the conference will be held. As described above, this may include, for example, its own texture information to be mapped onto the mesh and possibly the three-dimensional primitives it describes. This may define a space in which the virtual camera and respective avatars may navigate within the virtual environment. As such, it may be bounded by edges (eg, walls or fences) that indicate to users the perimeter of the navigable virtual environment.

Spatial hierarchy 408 is data specifying partitions in the virtual environment. These partitions are used to determine how the sound is processed before being transmitted between participants. As will be explained below, this partition data can be hierarchical and can describe sound processing to enable areas where participants in a virtual conference can have private or side conversations.

The 3D model 406 is any other 3D modeling information needed to conduct the meeting. In one embodiment, this may include information describing the respective avatars. Alternatively or additionally, this information may include product demonstrations.

Once the information needed to conduct the conference is transmitted to the participants, FIGS. 4B and 4C illustrate how the server 302 transfers the information from one device to another. 4B illustrates a diagram 420 showing how server 302 receives information from respective devices 306A and 306B, and FIG. 4C illustrates server 302 sending information to respective devices 306B and 306A. ) illustrates a diagram 420 showing how to transmit to. Specifically, device 306A sends position and direction 422A, video stream 424A, and audio stream 426A to server 302, and server 302 sends position and direction 422A, video stream 424A, and audio stream 426A to device 306B. And device 306B sends position and direction 422B, video stream 424B, and audio stream 426B to server 302, which server 302 sends position and direction 422B, video stream 424B. ), and audio stream 426B to device 306A.

Position and orientation 422A and 422B describe the position and orientation of the virtual camera relative to the user using device 306A. As described above, a position can be a coordinate in 3-dimensional space (eg, x, y, z coordinates), and a direction can be a direction in 3-dimensional space (eg, pan, tilt, roll). )) can be. In some embodiments, the user may not be able to control the roll of the virtual camera, so the direction can only specify the pan and tilt angles. Similarly, in some embodiments, the user may not be able to change the avatar's z-coordinate (because the avatar is constrained by virtual gravity), and the z-coordinate may be unnecessary. In this way, each of position and orientation 422A and 422B may include coordinates and pan and tilt values on a horizontal plane in at least a three-dimensional virtual space. Alternatively or additionally, the user may "jump" his or her avatar, so the Z position may be specified only by an indication of whether or not the user jumps their avatar.

In different examples, location and direction 422A and 422B may be sent and received using HTTP request responses or using socket messaging.

Video streams 424A and 424B are video data captured from the cameras of respective devices 306A and 306B. Video can be compressed. For example, the video may use any commonly known video codecs, including MPEG-4, VP8, or H.264. Video can be captured and transmitted in real time.

Similarly, audio streams 426A and 426B are audio data captured from the microphones of the respective devices. Audio can be compressed. For example, audio may use any commonly known audio codecs, including MPEG-4 or vorbis. Audio can be captured and transmitted in real time. Video stream 424A and audio stream 426A are captured, transmitted, and presented synchronously with each other. Similarly, video stream 424B and audio stream 426B are captured, transmitted, and presented synchronously with each other.

Video streams 424A and 424B and audio streams 426A and 426B may be transmitted using a WebRTC application programming interface. WebRTC is an API available in JavaScript. As described above, devices 306A and 306B download and run web applications, as conferencing applications 310A and 310B, which may be implemented in JavaScript. Conferencing applications 310A and 310B may use WebRTC to receive and transmit video streams 424A and 424B and audio streams 426A and 426B by making API calls from their JavaScript.

As mentioned above, when a user leaves a virtual meeting, this exit is communicated to all other users. For example, if device 306A leaves the virtual meeting, server 302 will communicate that exit to device 306B. As a result, device 306B will stop rendering the avatar corresponding to device 306A, removing the avatar from the virtual space. Additionally, device 306B will cease receiving video stream 424A and audio stream 426A.

As described above, conferencing applications 310A and 310B periodically navigate the virtual space based on new information about respective video streams 424A and 424B, position and orientation 422A and 422B, and the three-dimensional environment. or intermittently re-render. For simplicity, each of these updates is now described from the perspective of device 306A. However, the skilled person will understand that device 306B will behave similarly given similar changes.

When device 306A receives video stream 424B, device 306A texture maps frames from video stream 424A onto an avatar corresponding to device 306B. The corresponding texture mapped avatar is re-rendered within the three-dimensional virtual space and presented to the user of device 306A.

When device 306A receives the new location and orientation 422B, device 306A creates an avatar corresponding to device 306B placed in the new location and oriented in the new orientation. The created avatar is re-rendered within the three-dimensional virtual space and presented to the user of device 306A.

In some embodiments, server 302 may transmit updated model information describing the three-dimensional virtual environment. For example, server 302 may transmit updated information 402 , 404 , 406 , or 408 . When that happens, device 306A will re-render the virtual environment based on the updated information. This can be useful when circumstances change over time. For example, an outdoor event may change from daytime to dusk as the event progresses.

Again, when device 306B exits the virtual conference, server 302 sends a notification to device 306A indicating that device 306B is no longer participating in the conference. In that case, device 306A will re-render the virtual environment without an avatar for device 306B.

Although FIGS. 3 and 4A-4C are illustrated with two devices for simplicity, one skilled in the art will appreciate that the techniques described herein may be extended to any number of devices. Further, while FIGS. 3 and 4A-4C illustrate a single server 302 , those skilled in the art will understand that the functionality of server 302 may be distributed among multiple computing devices. In one embodiment, the data transmitted in FIG. 4A may come from one network address for server 302, while the data transmitted in FIGS. 4B and 4C are to/from another network address for server 302. can be transmitted

In one embodiment, participants may set their webcam, microphone, speakers and graphics settings prior to entering the virtual meeting. In an alternative embodiment, after launching the application, users may enter a virtual lobby, where they are greeted by an avatar controlled by a real person. This person can view and modify the user's webcam, microphone, speakers and graphics settings. The guide may also inform the user how to use the virtual environment, for example by teaching the user about viewing, moving and interacting. When ready, the user automatically exits the virtual waiting room and joins the real virtual environment.

Adjustment of volume for video conference in virtual environment

Embodiments also adjust the volume to provide a sense of position and space within the virtual meeting. This is illustrated, for example, in FIGS. 5-7 , 8A and 8B and 9A-9C , each of which is described below.

5 is a flowchart illustrating a method 500 for adjusting relative left and right volumes to provide a sense of position in a virtual environment during a video conference.

In step 502, the volume is adjusted based on the distance between the avatars. As described above, an audio stream from the microphone of another user's device is received. Volumes of both the first audio stream and the second audio stream are adjusted based on the distance between the second location and the first location. This is illustrated in FIG. 6 .

6 shows a chart 600 illustrating how the volume rolls off as the distance between avatars increases. Chart 600 illustrates volume 602 on the x-axis and y-axis. As the distance between users increases, the volume remains constant until a reference distance 602 is reached. At that point, the volume starts to drop. In this way, all other things being equal, a closer user will often be heard louder than a more distant user.

How fast the sound descends depends on the roll off factor. This may be a coefficient built into the settings of the video conferencing system or client device. As illustrated by lines 608 and 610, a larger roll off rate will drop the volume faster than a smaller roll off rate.

Returning to Figure 5, in step 504, the relative left and right audio is adjusted based on the direction in which the avatar is located. That is, the volume of the audio output from the user's speaker (eg headset) will vary to provide a sense of where the speaking user's avatar is located. Based on the direction of the location where the user receiving the audio is located (for example, the location of the virtual camera) and the location where the user generating the audio stream is located (for example, the location of the speaking user's avatar) The relative volume of the audio streams is adjusted. The locations may be on a horizontal plane within a three-dimensional virtual space. The relative volumes of the left and right audio streams are adjusted to provide a sense of where the second position is relative to the first position in three-dimensional virtual space.

For example, at step 504, the audio corresponding to the avatar to the left of the virtual camera will be adjusted so that the audio is output at a higher volume in the receiving user's left ear than in the right ear. Similarly, the audio corresponding to the avatar to the right of the virtual camera will be adjusted so that the audio is output at a higher volume in the receiving user's right ear than in the left ear.

In step 506, the relative left-right audio is adjusted based on the direction in which one avatar is oriented relative to the other avatar. Based on the angle between the direction the virtual camera is facing and the direction the avatar is facing, such that the more perpendicular this angle tends to have a larger volume difference between the left and right audio streams, the left audio stream and the right audio stream The relative volume of the audio stream is adjusted.

For example, when the avatar is directly facing the virtual camera, the relative left and right volumes of the avatar's corresponding audio streams at step 506 may not be adjusted at all. When the avatar is facing the left side of the virtual camera, the relative left and right volumes of the avatar's corresponding audio streams can be adjusted such that the left side is greater than the right side. And, when the avatar is facing the right side of the virtual camera, the relative left and right volumes of the corresponding audio streams of the avatar may be adjusted such that the right side is greater than the left side.

In one example, the calculation at step 506 may involve taking the cross product of the angle at which the virtual camera is facing and the angle at which the avatar is facing. The angles can be the direction they are facing in the horizontal plane.

In one embodiment, a check may be performed to determine the audio output device the user is using. Adjustments in steps 504 and 506 may not occur if the audio output device is not a set of headphones or other type of speaker that provides a stereo effect.

Steps 502 to 506 are repeated for every audio stream received from every other participant. Based on the calculations in steps 502 to 506, left and right audio gains are calculated for every other participant.

In this way, the audio streams for each participant are tailored to provide a sense of where the participant's avatar is located in the three-dimensional virtual environment.

Not only is the audio stream adjusted to provide a sense of where the avatars are located, but in certain embodiments, the audio stream can be adjusted to provide private or semi-private volume regions. In this way, virtual environments allow users to have private conversations. It also allows users to mingle with each other and have individual, side conversations that are not possible with conventional videoconferencing software. This is illustrated, for example, with respect to FIG. 7 .

7 is a flowchart illustrating a method 700 for adjusting relative volume to present different volume areas in a virtual environment during a video conference.

As described above, the server may provide the client devices with a specification of sound or volume fields. A virtual environment can be partitioned into different volume areas. In step 702, the device determines in which sound areas the respective avatars and virtual camera are located.

For example, FIGS. 8A and 8B are diagrams illustrating different volume areas in a virtual environment during a videoconference. 8A illustrates a diagram 800 with a volume area 802 enabling quasi- or sub-conversation between a user controlling an avatar 806 and a user controlling a virtual camera. In this way, users around the conference table 810 can converse without disturbing others in the room. The sound from the user controlling the avatar 806 in the virtual camera may drop out as the virtual camera exits the volume area 802, but not completely. This allows passers-by to join the conversation if they wish.

Interface 800 also includes buttons 804, 806, and 808, which will be described below.

8B illustrates a diagram 800 with a volume area 804 enabling private conversation between a user controlling an avatar 808 and a user controlling a virtual camera. Once inside the volume area 804 , audio from the user controlling the avatar 808 and the user controlling the virtual camera may only be output to those inside the volume area 804 . Because audio from those users is never played to other users in the conference, their audio streams may not be transmitted to other user devices at all.

Volume spaces may be hierarchical as illustrated in FIGS. 9A and 9B. FIG. 9B is a diagram 930 illustrating a layout with different volume areas arranged in a hierarchical structure. Volume areas 934 and 935 are within volume area 933 and volume areas 933 and 932 are within volume area 931 . These volume areas are represented in a hierarchical tree, as illustrated in diagram 900 and FIG. 9A.

In diagram 900, node 901 represents volume area 931 and is the root of the tree. Nodes 902 and 903 are children of node 901 and represent volume areas 932 and 933 . Nodes 904 and 906 are children of node 903 and represent volume areas 934 and 935 .

If the user located in area 934 wants to hear the user located in area 932, the audio stream must pass through a number of different virtual "walls", each attenuating the audio stream. Specifically, the sound must pass through the walls of area 932, the walls of area 933, and the walls of area 934. Each wall is attenuated by a specific factor. This calculation is explained with respect to steps 704 and 706 in FIG. 7 .

At step 704, the hierarchy is traversed to determine what various sound regions are among the avatars. This is illustrated, for example, in FIG. 9C. Starting from the node corresponding to the virtual region of the speaking voice (node 904 in this case), a route is determined to the receiving user's node (node 902 in this case). To determine the path, links 952 from node to node are determined. In this way, a subset of the area between the area containing the avatar and the area containing the virtual camera is determined.

In step 706, the audio stream from the speaking user is attenuated based on the respective wall transmittances of the region subset. Each respective wall transmittance specifies how much the audio stream is attenuated.

Additionally or alternatively, in that case different areas have different roll off rates, and the distance-based calculation shown in method 600 may be applied to the individual areas based on the respective roll off rates. In this way, different areas of the virtual environment project sound at different rates. The audio gains determined in the method as described above with respect to FIG. 5 may be applied to the audio stream to determine the left and right audio accordingly. In this way, wall transmittance, roll off rate, and side-to-side adjustment to provide a sense of direction of sound can all be applied together to provide a comprehensive audio experience.

Different audio regions may have different functions. For example, the volume area may be a podium area. When the user is located in the podium area, some or all of the attenuation described with respect to FIG. 5 or FIG. 7 may not occur. Attenuation may not occur, for example, due to roll off ratio or wall permeability. In some embodiments, the relative left and right audio can still be adjusted to provide a sense of direction.

For illustrative purposes, the methods described with respect to FIGS. 5 and 7 describe audio streams from a user with a corresponding avatar. However, the same methods can be applied to other sound sources than avatars. For example, a virtual environment may have three-dimensional models of speakers. Sound may be emitted from the speakers in the same way as the avatar models described above, either due to a presentation or simply to provide background music.

As mentioned above, wall transmittance can be used to completely isolate the audio. In one embodiment, this may be used to create a virtual office. In one example, each user may have a monitor in their physical (possibly home) office that displays a conferencing application that is constantly on in the virtual office and logged into the virtual office. There may be a feature that allows the user to indicate whether the user is in the office or should not be disturbed. When the do not disturb indicator is off, co-workers or managers can enter the virtual space and knock or walk in, just like in a real office. Visitors can leave notes when staff are not in the office. When the staff returns, they can read the notes left by the visitor. The virtual office may have a whiteboard and/or interface displaying messages for the user. The messages may be email and/or may be from a messaging application such as the SLACK application available from Slack Technologies, Inc. of San Francisco, CA.

Users can customize or personalize their virtual offices. For example, users can erect models of posters or other wall ornaments. Users can change the models or orientation of a desk or decorative ornament, such as a planter. Users can change the lighting or look out the window.

Returning to FIG. 8A , interface 800 includes various buttons 804 , 806 , and 808 . When the user presses the button 804, the attenuation described above with respect to the methods in FIGS. 5 and 7 may not occur, or may only occur in a lesser amount. In that situation, the user's voice is output uniformly to the other users, allowing the user to address all participants in the conference. As will be explained below, the user video can also be output to the presentation screen within the virtual environment. When the user presses button 806, speaker mode is enabled. In that case, audio is output from sound sources in the virtual environment, such as playing background music. When the user presses the button 808, a screen sharing mode may be enabled, allowing the user to share the content of a screen or window on their device with other users. Content can be presented on a presentation model. This will also be explained below.

Presenting in a 3D Environment

10 illustrates an interface 1000 with a three-dimensional model 1004 in a three-dimensional virtual environment. As described above with respect to FIG. 1 , interface 1000 may be displayed to a user capable of navigating around a virtual environment. As illustrated in interface 1000 , the virtual environment includes an avatar 1004 and a three-dimensional model 1002 .

The 3D model 1002 is a 3D model of a product placed inside the virtual space. People can join this virtual space to observe the model and walk around it. Products can have localized sounds to enhance the experience.

More specifically, when a presenter in the virtual space wants to show a 3D model, the presenter selects the desired model from the interface. This sends a message to the server to update the details (including the model's name and path). This will be automatically communicated to clients. In this way, the three-dimensional model can be rendered for display concurrently with presenting the video stream. Users can navigate a virtual camera around a three-dimensional model of a product.

In different examples, the object may be a product demonstration or may be an advertisement for a product.

11 illustrates an interface 1100 with presentation screen sharing in a three-dimensional virtual environment used for video conferencing. As described above with respect to FIG. 1 , interface 1100 may be displayed to a user capable of navigating around a virtual environment. As illustrated in interface 1100 , the virtual environment includes an avatar 1104 and a presentation screen 1106 .

In this embodiment, a presentation stream from a device of a participant in a conference is received. The presentation stream is texture mapped onto the 3D model of the presentation screen 1106. In one embodiment, the presentation stream may be a video stream from a camera on the user's device. In another embodiment, the presentation stream may be a screen share from a user's device, where a monitor or window is shared. Via screen sharing or otherwise, the presentation video and audio streams can also be from an external source, eg a live stream of an event. When the user enables presenter mode, the user's presentation stream (and audio stream) tagged with the name of the screen the user wants to use is posted to the server. Other clients are notified that a new stream is available.

The presenter can also control the position and orientation of audience members. For example, the presenter may have the option of choosing to rearrange all other participants in the conference so that they are positioned and oriented towards the presentation screen.

The audio stream is captured from the microphone of the first participant's device in synchronization with the presentation stream. The audio stream from the user's microphone can be heard by other users as coming from the presentation screen 1106 . In this way, the presentation screen 1106 may be a sound source as described above. Since the user's audio stream is projected from the presentation screen 1106, it can be suppressed from coming from the user's avatar. In this way, the audio stream is output to be reproduced in synchronization with the display of the presentation stream on the screen 1106 in the three-dimensional virtual space.

Bandwidth allocation based on distance between users

12 is a flowchart illustrating a method 1200 for apportioning available bandwidth based on relative positions of avatars within a three-dimensional virtual environment.

In step 1202, a distance between a first user and a second user in the virtual meeting space is determined. The distance may be the distance between them on a horizontal plane in a three-dimensional space.

In step 1204, the received video streams are prioritized such that video streams of closer users are given priority over video streams from more distant users. Priority values may be determined as illustrated in FIG. 13 .

13 shows a chart 1300 showing priority 1306 and distance 1302 on the y-axis. As illustrated by line 1306, the priority state remains at a constant level until the reference distance 1304 is reached. After reaching the criterion distance, the priority starts to drop.

At step 1206, the bandwidth available to the user device is distributed among the various video streams. This may be done based on the priority values determined in step 1204. For example, the priorities can be scaled proportionally so that the sum of all equals one. For any videos for which insufficient bandwidth is available, the relative priority may be zero. The priorities are then adjusted again for the remaining video streams. Bandwidth is allocated based on these relative priority values. Additionally, bandwidth for audio streams may be reserved. This is illustrated in FIG. 14 .

14 illustrates a chart 1400 with the y-axis representing bandwidth 1406 and the x-axis representing relative priority. After a video has been allocated the minimum valid bandwidth 1406, the bandwidth 1406 allocated to a video stream is increased proportionally to its relative priority.

Once the allocated bandwidth is determined, the client can request video from the server at the selected and allocated bandwidth/bitrate/frame rate/resolution for that video. This may initiate a negotiation process between the client and server to start streaming video with the specified bandwidth. In this way, the available video and audio bandwidth is divided fairly among all users, where users with twice the priority will receive twice the bandwidth.

In one possible implementation, all clients transmit multiple video streams to the server, at different bitrates and resolutions, using simulcast. Other clients can then tell the server which of these streams they are interested in and want to receive.

In step 1208, it is determined whether the available bandwidth between the first user and the second user in the virtual meeting space makes display of the video at that distance ineffective. This determination can be made by either the client or the server. If by the client, the client sends a message to the server to stop sending the video to the client. If ineffective, transmission of the video stream to the second user's device is stopped, and the second user's device is notified to replace the video stream with a still image. The still image may simply be the last video frames received (or one of the last video frames received).

In one embodiment, a similar process can be performed for audio, reducing the quality given the size of the portion reserved for audio. In another embodiment, each audio stream is given a consistent bandwidth.

In this way, the embodiments increase performance for all users, and video and audio stream quality may be reduced for users who are more distant and/or less important than in the case of the server. This is not done when sufficient bandwidth budget is available. Reduction is done in both bitrate and resolution. This improves video quality because the bandwidth available to that user can be utilized more efficiently by the encoder.

Separately, video resolution is scaled down based on distance, and users twice as far away have half the resolution. In this way, given the limitations of screen resolution, unnecessary resolutions may not be downloaded. Thus, bandwidth is conserved.

15 is a diagram of a system 1500 illustrating components of devices used to provide video conferencing within a virtual environment. In various embodiments, system 1500 may operate according to the methods described above.

Device 306A is a user computing device. Device 306A may be a desktop or laptop computer, smartphone, tablet, or wearable (eg, watch or head mounted device). Device 306A includes a microphone 1502 , a camera 1504 , a stereo speaker 1506 , and an input device 1512 . Although not shown, device 306A also includes a processor and permanent, non-transitory and volatile memory. Processors may include one or more central processing units, graphics processing units, or any combination thereof.

A microphone 1502 converts sound into an electrical signal. A microphone 1502 is positioned to capture the voice of the user of device 306A. In different examples, the microphone 1502 can be a condenser microphone, an electret microphone, a moving coil microphone, a ribbon microphone, a carbon microphone, a piezoelectric microphone, a fiber optic microphone, a laser microphone, a water microphone, or a MEMS microphone.

Camera 1504 captures image data, typically by capturing light through one or more lenses. Camera 1504 is positioned to capture photographic images of a user of device 306A. Camera 1504 includes an image sensor (not shown). The image sensor may be, for example, a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor. An image sensor may include one or more photodetectors that detect light and convert it into electrical signals. These electrical signals captured together in a similar time frame constitute a still photographic image. A series of still photographic images captured at regular intervals together constitute a video. In this way, camera 1504 captures images and videos.

The stereo speaker 1506 is a device that converts electrical audio signals into corresponding left and right sounds. The stereo speaker 1506 outputs the left audio stream and the right audio stream generated by the audio processor 1520 (below) to be reproduced in stereo to the user of the device 306A. Stereo speakers 1506 include both peripheral speakers and headphones designed to reproduce sound directly to the user's left and right ears. Exemplary loudspeakers include moving-iron loudspeakers, piezoelectric loudspeakers, static magnetic loudspeakers, electrostatic loudspeakers, ribbon and planar magnetic loudspeakers, flexural wave loudspeakers, flat panel loudspeakers, heil air motion transducers. air motion transducer), transparent ion conduction speaker, plasma arc speaker, thermoacoustic speaker, rotary woofer, moving coil, electrostatic, electret, planar magnetic, and balanced armature.

Network interface 1508 is a software or hardware interface between two equipment or protocol layers in a computer network. Network interface 1508 receives video streams from server 302 for the respective participants in the conference. The video stream is captured from cameras on the devices of other participants in the video conference. The network interface 1508 also receives data specifying the three-dimensional virtual space and arbitrary models therein from the server 302 . For each of the other participants, the network interface 1508 receives a position and orientation in a three-dimensional virtual space. Position and orientation are entered by each of the respective other participants.

Network interface 1508 also transmits data to server 302 . It transmits the position of the user's virtual camera of device 306A used by renderer 1518 and transmits video and audio streams from camera 1504 and microphone 1502 .

Display 1510 is an output device for presenting electronic information in a visual or tactile form (the latter used, for example, in tactile electronic displays for the visually impaired). The display 1510 may be a television set, a computer monitor, a head mounted display, a heads-up display, the output of an augmented reality or virtual reality headset, a broadcast reference monitor, a medical monitor, a mobile display (for mobile devices), ( It may be a smartphone display (for a smartphone). To present information, the display 1510 may be an electroluminescent (ELD) display, a liquid crystal display (LCD), a light emitting diode (LED) backlit LCD, a thin film transistor (TFT) LCD, a light emitting diode (LED) display, an OLED display, or an AMOLED. It may include a display, a plasma (PDP) display, and a quantum dot (QLED) display.

Input device 1512 is equipment used to provide data and control signals to an information processing system, such as a computer or information appliance. The input device 1512 enables the user to enter a new desired position of the virtual camera used by the renderer 1518, thereby enabling navigation in a three-dimensional environment. Examples of input devices include keyboards, mice, scanners, joysticks, and touch screens.

Web browser 308A and web application 310A have been described above with respect to FIG. 3 . The web application 310A includes a screen capturer 1514 , a texture mapper 1516 , a renderer 1518 , and an audio processor 1520 .

The screen capturer 1514 captures a presentation stream, specifically a screen share. Screen capturer 1514 can interact with APIs made available by web browser 308A. By calling a function available from the API, screen capturer 1514 can cause web browser 308A to ask the user which window or screen the user would like to share. Based on the answer to that query, web browser 308A may return a video stream corresponding to the screen share to screen capturer 1514, which may then return to server 302 and ultimately other Pass it to the network interface 1508 for transmission to the participants' devices.

A texture mapper 1516 texture maps the video stream onto a 3D model corresponding to the avatar. Texture mapper 1516 may texture map respective frames from the video to the avatar. Additionally, the texture mapper 1516 may texture map the presentation stream to the 3D model of the presentation screen.

Renderer 1518 renders, from the point of view of the user's virtual camera of device 306A, a texture-mapped image of the avatars for the respective participants positioned and oriented in the corresponding location received for output to display 1510. A 3D virtual space containing 3D models is rendered. Renderer 1518 also renders any other three-dimensional models, including, for example, presentation screens.

Audio processor 1520 adjusts the volume of the received audio stream to determine the left audio stream and the right audio stream to provide a sense of where the second location is relative to the first location in three-dimensional virtual space. In one embodiment, the audio processor 1520 adjusts the volume based on the distance between the second location and the first location. In another embodiment, the audio processor 1520 adjusts the volume based on the direction from the second location to the first location. In another embodiment, the audio processor 1520 adjusts the volume based on the direction of the second position relative to the first position on a horizontal plane in a three-dimensional virtual space. In another embodiment, the audio processor 1520 determines that based on the direction the virtual camera is facing in 3D virtual space, when the avatar is positioned to the left of the virtual camera, the left audio stream tends to have a higher volume and the avatar When is positioned to the right of the virtual camera, adjust the volume so that the audio stream on the right tends to have a higher volume. Finally, in another embodiment, the audio processor 1520 determines, based on the angle between the direction the virtual camera is facing and the direction the avatar is facing, that the angle more perpendicular to where the avatar is facing corresponds to the left audio stream. Adjust the volume so that it tends to have a larger volume difference between the right audio streams.

The audio processor 1520 may also adjust the volume of the audio stream based on the area where the speaker is located relative to the area where the virtual camera is located. In this embodiment, a three-dimensional virtual space is divided into a plurality of regions. These areas can be hierarchical. Wall transmittance can be applied to attenuate the volume of the talking audio stream when the speaker and the virtual camera are located in different areas.

The server 302 includes an attendance notifier 1522 , a stream coordinator 1524 , and a stream forwarder 1526 .

Attendance notifier 1522 notifies conference participants when they join and leave the conference. When a new participant joins the conference, the attendance notifier 1522 sends a message to the devices of the other participants in the conference indicating that the new participant has joined. Attendance notifier 1522 signals stream forwarder 1526 to begin passing video, audio, and location/direction information to other participants.

Stream coordinator 1524 receives a video stream captured from a camera on a first user's device. Stream coordinator 1524 determines available bandwidth for transmitting data for the virtual conference to the second user. This determines the distance between the first user and the second user in the virtual meeting space. And, it distributes the available bandwidth between the first video stream and the second video stream based on the relative distance. In this way, the stream coordinator 1524 prioritizes video streams of closer users over video streams from more distant users. Additionally or alternatively, stream coordinator 1524 may be located on device 306A, perhaps as part of web application 310A.

Stream forwarder 1526 broadcasts (with adjustments made by stream coordinator 1524) the received position/orientation information, video, audio, and screen sharing screens. Stream forwarder 1526 may transmit information to device 306A in response to a request from conferencing application 310A. Conferencing application 310A may send the request in response to the notification from attendance notifier 1522 .

Network interface 1528 is a software or hardware interface between two equipment or protocol layers in a computer network. Network interface 1528 transmits the model information to the devices of the various participants. Network interface 1528 receives video, audio, and screen sharing screens from various participants.

Screen Capturer 1514, Texture Mapper 1516, Renderer 1518, Audio Processor 1520, Attendance Notifier 1522, Stream Coordinator 1524, and Stream Forwarder 1526 are respectively hardware, software, firmware, or any combination thereof.

Identifiers, such as "(a)," "(b)," "(i)," "(ii)", etc., are sometimes used for different elements or steps. These identifiers are used for clarity and do not necessarily specify a sequence of elements or steps.

The invention has been described above with the aid of functional building blocks that illustrate the implementation of specified functions and their relationships. The boundaries of these functional building blocks have been arbitrarily defined herein for convenience of explanation. Alternate boundaries may be defined, as long as the specified functions and their relationships are properly performed.

that other embodiments may be readily modified and/or adapted for various applications, such as specific embodiments, by applying knowledge in the art, without undue experimentation and without departing from the general concept of the invention; The foregoing description of specific embodiments will thus fully reveal the general nature of the present invention. Accordingly, such adaptations and modifications are intended to be within the meaning and scope of equivalents of the disclosed embodiments based on the teaching and guidance presented herein. It should be understood that the terminology or phraseology in this specification is for descriptive purposes and not for limitation, so that terminology or phraseology in this specification can be interpreted by those skilled in the art based on the teachings and guidelines.

The breadth and scope of this invention should not be limited by any of the exemplary embodiments described above, but only in accordance with the appended claims and their equivalents.

Claims (146)

A system for enabling videoconferencing between a first user and a second user, comprising:
a processor coupled to the memory;
display screen;
(i) data specifying a three-dimensional virtual space, (ii) a position and direction in the three-dimensional virtual space, the position and direction being input by the first user, and a camera on the first user's device. a network interface configured to receive a video stream captured from the camera, wherein the camera is arranged to capture photographic images of the first user;
and a web browser implemented in the processor and configured to download a web application from a server and execute the web application, wherein the web application:
a texture mapper configured to texture map the video stream onto a three-dimensional model of an avatar; and
A renderer, wherein the renderer:
(i) from the point of view of the second user's virtual camera, the three-dimensional virtual space comprising the texture-mapped three-dimensional model of the avatar located at the location and oriented in the direction for display to the second user; render,
(ii) change the viewpoint of the virtual camera of the second user when input is received from the second user indicating a desire to change the viewpoint of the virtual camera;
(iii) re-rendering, from the changed viewpoint of the virtual camera, the three-dimensional virtual space including the texture-mapped three-dimensional model of the avatar located at the location and oriented in the direction for display to the second user. A system configured to do so. 2. The method of claim 1 , wherein the device further comprises a graphics processing unit, wherein the texture mapper and the renderer include WebGL application calls that enable the web application to texture map or render using the graphics processing unit. system. A computer implemented method for enabling a video conference between a first user and a second user, comprising:
transmitting a web application to a first client device of the first user and a second client device of the second user;
(i) a position and orientation in a three-dimensional virtual space, wherein the position and orientation are input by the first user, and (ii) a video stream captured from a camera on the first client device, wherein the camera is arranged to capture photo images of a user; receiving, from the first client device running the web application; and
transmitting the location and direction and the video stream to the second client device of the second user, wherein the web application, when executed on a web browser, causes the second client device to:
(i) cause the video stream to be texture mapped onto a three-dimensional model of an avatar, and (ii) located at the location and oriented in the direction for display to the second user, from the viewpoint of the second user's virtual camera. render the 3D virtual space including the texture-mapped 3D model of the avatar;
(ii) when input is received from the second user indicating a desire to change the viewpoint of the virtual camera, cause the second user to change the viewpoint of the virtual camera;
(iii) re-rendering, from the changed viewpoint of the virtual camera, the three-dimensional virtual space including the texture-mapped three-dimensional model of the avatar located at the location and oriented in the direction for display to the second user. A method comprising executable instructions that cause 4. The method of claim 3, wherein the web application includes WebGL application calls that enable the web application to texture map or render using a graphics processing unit of the second client device. A computer implemented method for enabling a video conference between a first user and a second user, comprising:
Receiving data specifying a 3D virtual space;
receiving a position and direction in the three-dimensional virtual space, wherein the position and direction are input by the first user;
receiving a video stream captured from a camera on the device of the first user, the camera arranged to capture photographic images of the first user;
texture mapping the video stream onto a three-dimensional model of an avatar by a web application implemented in a web browser; and
The texture-mapped three-dimensional model of the avatar positioned at the location and oriented in the direction for display to the second user, from the viewpoint of the second user's virtual camera, by the web application implemented in the web browser. Rendering the 3D virtual space including;
When input from the second user indicating a desire to change the viewpoint of the virtual camera is received:
changing the viewpoint of the virtual camera of the second user; and
re-rendering, from the changed viewpoint of the virtual camera, the three-dimensional virtual space including the texture-mapped three-dimensional model of the avatar located at the location and oriented in the direction for display to the second user; Including, how. According to claim 5,
receiving an audio stream captured in synchronization with the video stream from a microphone of the device of the first user, the microphone being arranged to capture speech of the first user; and
outputting the audio stream for playback to the second user in synchronism with the display of the video stream in the three-dimensional virtual space;
Further comprising a method. The method of claim 5, wherein the viewpoint of the virtual camera is defined by at least coordinates on a horizontal plane in the 3D virtual space and pan and tilt values. 6. The method of claim 5, when the new position and orientation of the first user in the three-dimensional virtual space are received:
re-rendering the three-dimensional virtual space comprising a texture-mapped three-dimensional model of the avatar positioned in the new location and oriented in the new direction for display to the second user. 6. The method of claim 5, wherein the texture mapping step comprises, for each frame of the video stream, iteratively mapping pixels onto the three-dimensional model of the avatar. 6. The method of claim 5, wherein the data, the position and direction, and the video stream are received from a server to a web browser, and wherein the texture mapping and rendering are performed by the web browser. According to claim 10,
receiving, from the server, a notification indicating that the first user is no longer present; and
re-rendering the three-dimensional virtual space without the texture-mapped three-dimensional model of the avatar for display to the second user on the web browser. According to claim 11,
receiving, from the server, a notification indicating that a third user has entered the three-dimensional virtual space;
receiving a second location and a second direction of the third user in the 3D virtual space;
receiving a second video stream captured from a camera on the device of the third user, the camera arranged to capture photographic images of the third user;
texture mapping the second video stream onto a second three-dimensional model of a second avatar; and
the 3D virtual model comprising the second texture-mapped 3D model located at the second position and oriented in the second direction for display to the second user, from the viewpoint of the virtual camera of the second user. The method further comprising rendering the space. 6. The method of claim 5, wherein the receiving of the data specifying the three-dimensional virtual space comprises receiving a mesh specifying a conference space and receiving a background image, and the rendering comprises constructing the background image. and mapping the texture onto the image. A non-transitory tangible computer readable device having stored thereon instructions that, when executed by at least one computing device, cause the at least one computing device to perform operations for enabling a video conference between a first user and a second user, , the operations are:
receiving data designating a 3-dimensional virtual space;
receiving a position and direction in the 3D virtual space, the position and direction being input by the first user;
receiving a video stream captured from a camera on the device of the first user, the camera arranged to capture photographic images of the first user;
texture mapping the video stream onto a 3D model of an avatar;
rendering the 3D virtual space including the texture-mapped 3D model of the avatar located at the location and oriented in the direction, from the viewpoint of the second user's virtual camera, for display to the second user; ;
When input from the second user indicating a desire to change the viewpoint of the virtual camera is received:
changing the viewpoint of the virtual camera of the second user; and
re-rendering, from the changed viewpoint of the virtual camera, the 3D virtual space including the texture-mapped 3D model of the avatar located at the location and oriented in the direction for display to the second user; contains,
wherein the data, the position and direction, and the video stream are received from a server to a web browser, and the texture mapping and rendering are performed by the web browser. 15. The method of claim 14, wherein the operations are:
receiving an audio stream captured in synchronization with the video stream from a microphone of the device of the first user, the microphone being positioned to capture utterances of the first user; and
and outputting the audio stream for playback to the second user in synchronism with the display of the video stream in the 3D virtual space. The device of claim 14, wherein the viewpoint of the virtual camera is defined by at least coordinates on a horizontal plane in the 3D virtual space and pan and tilt values. 15. The method of claim 14, wherein the actions are: when a new position and orientation of the first user in the three-dimensional virtual space are received:
and re-rendering the three-dimensional virtual space including the texture-mapped three-dimensional model of the avatar located in the new location and oriented in the new direction for display to the second user. 15. The device of claim 14, wherein the texture mapping comprises, for each frame of the video stream, iteratively mapping pixels onto the 3D model of the avatar. 15. The method of claim 14, wherein the operations are:
receiving, from the server, a notification indicating that the first user is no longer present; and
and re-rendering the three-dimensional virtual space without the texture-mapped three-dimensional model of the avatar for display to the second user on the web browser. 20. The method of claim 19, wherein the operations are:
receiving, from the server, a notification indicating that a third user has entered the three-dimensional virtual space;
receiving a second position and a second direction of the third user in the 3D virtual space;
receiving a second video stream captured from a camera on the device of the third user, the camera arranged to capture photographic images of the third user;
texture mapping the second video stream onto a second 3-dimensional model of a second avatar; and
the 3D virtual model comprising the second texture-mapped 3D model located at the second position and oriented in the second direction for display to the second user, from the viewpoint of the virtual camera of the second user. A device further comprising an operation of rendering a space. 15. The method of claim 14, wherein the receiving of the data specifying the 3D virtual space comprises receiving a mesh specifying a conference space and receiving a background image, and the rendering includes constructing the background image. A device that includes an operation of mapping a texture onto an image. A computer implemented method for enabling a video conference between a first user and a second user, comprising:
Receiving data specifying a 3D virtual space;
receiving a position and direction in the three-dimensional virtual space, wherein the position and direction are input by the first user;
receiving a video stream captured from a camera on the device of the first user, the camera arranged to capture photographic images of the first user;
texture mapping the video stream onto a three-dimensional model of an avatar by a web application implemented in a web browser;
The texture-mapped three-dimensional model of the avatar positioned at the location and oriented in the direction for display to the second user, from the viewpoint of the second user's virtual camera, by the web application implemented in the web browser. Rendering the 3D virtual space including;
receiving, from a server, a notification indicating that the first user is no longer present; and
re-rendering the three-dimensional virtual space without the texture-mapped three-dimensional model of the avatar for display to the second user on the web browser. The method of claim 22,
receiving, from the server, a notification indicating that a third user has entered the three-dimensional virtual space;
receiving a second location and a second direction of the third user in the 3D virtual space;
receiving a second video stream captured from a camera on the device of the third user, the camera arranged to capture photographic images of the third user;
texture mapping the second video stream onto a second three-dimensional model of a second avatar; and
the 3D virtual model comprising the second texture-mapped 3D model located at the second position and oriented in the second direction for display to the second user, from the viewpoint of the virtual camera of the second user. The method further comprising rendering the space. A system for enabling videoconferencing between a first user and a second user, comprising:
a processor coupled to the memory;
display screen;
(i) data specifying a three-dimensional virtual space, (ii) a position and direction in the three-dimensional virtual space, the position and direction being input by the first user, and a camera on the first user's device. a network interface configured to receive a video stream captured from the camera, wherein the camera is arranged to capture photographic images of the first user;
and a web browser implemented in the processor and configured to download a web application from a server and execute the web application, wherein the web application:
a mapper configured to map the video stream onto a three-dimensional model of an avatar; and
the three-dimensional virtual space comprising the three-dimensional model of the avatar with the mapped video stream located at the location and oriented in the direction, from the viewpoint of the second user's virtual camera, for display to the second user. A system, including a renderer configured to render. 25. The system of claim 24, wherein the device further comprises a graphics processing unit, the mapper and the renderer including WebGL application calls that enable the web application to map or render using the graphics processing unit. A computer implemented method for enabling a video conference between a first user and a second user, comprising:
transmitting a web application to a first client device of the first user and a second client device of the second user;
(i) a position and orientation in a three-dimensional virtual space, wherein the position and orientation are input by the first user, and (ii) a video stream captured from a camera on the first client device, wherein the camera is arranged to capture photo images of a user; receiving, from the first client device running the web application; and
and transmitting the location and direction and the video stream to the second client device of the second user, wherein the web application, when executed on a web browser, transmits the video stream to a three-dimensional model of an avatar. and the three-dimensional model of the avatar mapped from the viewpoint of the second user's virtual camera to the video stream located at the location and oriented in the direction for display to the second user. A system comprising executable instructions that render a dimensional virtual space. 27. The method of claim 26, wherein the web application includes WebGL application calls that enable the web application to map or render using a graphics processing unit of the second client device. A computer implemented method for enabling a video conference between a first user and a second user, comprising:
Receiving data specifying a 3D virtual space;
receiving a position and direction in the three-dimensional virtual space, wherein the position and direction are input by the first user;
receiving a video stream captured from a camera on the device of the first user, the camera arranged to capture photographic images of the first user;
mapping, by a web application implemented in a web browser, the video stream onto a three-dimensional model of an avatar; and
comprising the three-dimensional model of the avatar located at the location and oriented in the direction for display to the second user, from the viewpoint of the second user's virtual camera, by the web application implemented in the web browser; and rendering the three-dimensional virtual space. According to claim 28,
receiving an audio stream captured in synchronization with the video stream from a microphone of the device of the first user, the microphone being arranged to capture utterances of the first user; and
outputting the audio stream for playback to the second user in synchronization with the display of the video stream within the three-dimensional virtual space. 29. The method of claim 28, wherein when input from the second user indicating a desire to change the viewpoint of the virtual camera is received:
changing the viewpoint of the virtual camera of the second user; and
Re-rendering, from the changed viewpoint of the virtual camera, the three-dimensional virtual space including the three-dimensional model of the avatar located at the location and oriented in the direction for display to the second user. , Way. 31. The method of claim 30, wherein the viewpoint of the virtual camera is defined by at least coordinates on a horizontal plane in the three-dimensional virtual space and pan and tilt values. 29. The method of claim 28, wherein when a new position and orientation of the first user in the three-dimensional virtual space is received:
re-rendering the three-dimensional virtual space including the three-dimensional model of the avatar positioned in the new location and oriented in the new direction for display to the second user. 29. The method of claim 28, wherein the mapping comprises, for respective frames of the video stream, iteratively mapping pixels onto the three-dimensional model of the avatar. 29. The method of claim 28, wherein the data, the position and direction, and the video stream are received from a server to a web browser, and wherein the mapping and rendering are performed by the web browser. 35. The method of claim 34,
receiving, from the server, a notification indicating that the first user is no longer present; and
re-rendering the three-dimensional virtual space without the three-dimensional model of the avatar for display to the second user on the web browser. The method of claim 35,
receiving, from the server, a notification indicating that a third user has entered the three-dimensional virtual space;
receiving a second location and a second direction of the third user in the 3D virtual space;
receiving a second video stream captured from a camera on the device of the third user, the camera arranged to capture photographic images of the third user;
mapping the second video stream onto a second three-dimensional model of a second avatar; and
Rendering of the three-dimensional virtual space including the second three-dimensional model located at the second position and oriented in the second direction for display to the second user, from the viewpoint of the virtual camera of the second user A method further comprising the step of doing. 29. The method of claim 28, wherein the step of receiving data specifying the three-dimensional virtual space includes receiving a mesh specifying a conference space and receiving a background image, and wherein the rendering step constructs the background image. A method comprising mapping onto a phase. A non-transitory tangible computer readable device having stored thereon instructions that, when executed by at least one computing device, cause the at least one computing device to perform operations for enabling a video conference between a first user and a second user, , the operations are:
receiving data designating a 3-dimensional virtual space;
receiving a position and direction in the 3D virtual space, the position and direction being input by the first user;
receiving a video stream captured from a camera on the device of the first user, the camera arranged to capture photographic images of the first user;
mapping the video stream onto a 3D model of an avatar; and
Rendering, from the viewpoint of the second user's virtual camera, the three-dimensional virtual space including the three-dimensional model of the avatar located at the position and oriented in the direction for display to the second user. , device. 39. The method of claim 38, wherein the operations are:
receiving an audio stream captured in synchronization with the video stream from a microphone of the device of the first user, the microphone being positioned to capture utterances of the first user; and
and outputting the audio stream for playback to the second user in synchronism with the display of the video stream in the 3D virtual space. 39. The method of claim 38, wherein the operations are: when input from the second user indicating a desire to change the viewpoint of the virtual camera is received:
changing the viewpoint of the virtual camera of the second user; and
Re-rendering, from the changed viewpoint of the virtual camera, the 3D virtual space including the 3D model of the avatar located at the location and oriented in the direction for display to the second user. , device. 41. The device of claim 40, wherein the viewpoint of the virtual camera is defined by at least coordinates on a horizontal plane in the three-dimensional virtual space and pan and tilt values. 39. The method of claim 38, wherein the operations are: when a new position and orientation of the first user in the three-dimensional virtual space is received:
and re-rendering the three-dimensional virtual space including the three-dimensional model of the avatar located in the new location and oriented in the new direction for display to the second user. 39. The device of claim 38, wherein the mapping comprises, for respective frames of the video stream, iteratively mapping pixels onto the three-dimensional model of the avatar. 39. The device of claim 38, wherein the data, the position and direction, and the video stream are received from a server to a web browser, and wherein the mapping and rendering are performed by the web browser. 45. The method of claim 44, wherein the operations are:
receiving, from the server, a notification indicating that the first user is no longer present; and
and re-rendering the three-dimensional virtual space without the three-dimensional model of the avatar for display to the second user on the web browser. 46. The method of claim 45, wherein the operations are:
receiving, from the server, a notification indicating that a third user has entered the three-dimensional virtual space;
receiving a second position and a second direction of the third user in the 3D virtual space;
receiving a second video stream captured from a camera on the device of the third user, the camera arranged to capture photographic images of the third user;
mapping the second video stream onto a second 3-dimensional model of a second avatar; and
Rendering of the three-dimensional virtual space including the second three-dimensional model located at the second position and oriented in the second direction for display to the second user, from the viewpoint of the virtual camera of the second user A device further comprising an operation of doing. 39. The method of claim 38, wherein the receiving of the data specifying the three-dimensional virtual space includes receiving a mesh specifying a conference space and receiving a background image, and the rendering includes constructing the background image. A device comprising an operation of mapping onto an image. A computer implemented method for presenting in a virtual conference involving a plurality of participants, comprising:
Receiving data specifying a 3D virtual space;
receiving a position and direction in the three-dimensional virtual space, the position and direction being input by a first participant of the plurality of participants of the virtual conference;
receiving a video stream captured from a camera on the device of the first participant, the camera arranged to capture photographic images of the first participant;
mapping the video stream onto a three-dimensional model of an avatar;
receiving a presentation stream from the device of the first participant;
mapping the presentation stream onto a 3D model of a presentation screen; and
rendering, from the viewpoint of a virtual camera of a second participant of the plurality of participants, the three-dimensional virtual space having the mapped avatar and the mapped presentation screen for display to the second participant; . The method of claim 48,
receiving an audio stream captured in synchronization with the presentation stream from a microphone of the device of the first participant, the microphone being arranged to capture utterances of the first participant; and
outputting the audio stream for playback to the second participant in synchronization with the display of the presentation stream within the three-dimensional virtual space. The method of claim 48,
receiving a position of a third participant among the plurality of participants in the 3D virtual space;
receiving an audio stream from a microphone of the third participant's device, the microphone being arranged to capture the third participant's utterance; and
adjusting the audio stream to provide a sense of the received position of the third participant in the three-dimensional virtual space relative to the position of the virtual camera;
wherein the rendering comprises rendering the three-dimensional virtual space with an avatar for the third participant at the received location for display to the second participant. The method of claim 48,
receiving a position of the first participant in the 3-dimensional virtual space;
receiving an audio stream from a microphone of the first participant's device, the microphone being arranged to capture the first participant's utterance;
adjusting the audio stream to provide a sense of the received position of the first participant in the three-dimensional virtual space relative to the position of the virtual camera;
rendering the three-dimensional virtual space with an avatar for the third participant at the received location for display to the second participant; and
adjusting the audio stream to provide a sense of the location of the mapped presentation screen relative to the location of the virtual camera when entering presentation mode. 49. The method of claim 48, wherein the presentation stream is a video of the first participant. 49. The method of claim 48, wherein the presentation stream is the first participant's screen share. 49. The method of claim 48, wherein the step of mapping the video stream maps respective frames of the video stream onto the three-dimensional model of the avatar to present moving images of the face of the first participant on the avatar. A method comprising the steps of: 55. The method of claim 54, wherein the avatar comprises a surface, and wherein the mapping step comprises mapping the respective frames onto the surface. 56. The method of claim 55, wherein the rendering step comprises rendering the mapped avatar at the position and orientation in the three-dimensional virtual space, wherein the first participant determines the location and orientation input by the first participant. wherein the location and orientation of the mapped avatar within the rendered three-dimensional virtual space can be changed based on a change in orientation. 56. The method of claim 55, wherein the rendering step comprises rendering the avatar positioned at the position in the three-dimensional virtual space and the surface oriented in the direction in the three-dimensional virtual space;
receiving a new direction in the three-dimensional virtual space, the new direction being input by the first participant;
re-rendering the three-dimensional virtual space from the viewpoint of the virtual camera of the second participant such that the surface of the texture-mapped avatar is oriented in the new orientation for display to the second participant. , Way. 58. The method of claim 57, wherein when the new direction is input by the first participant, the virtual camera of the first participant changes according to the new direction, and the virtual camera of the first participant changes to the three-dimensional virtual space. specify how the first participant is rendered for display to the first participant. A non-transitory tangible computer readable device having stored thereon instructions that, when executed by at least one computing device, cause the at least one computing device to perform operations for a presentation in a virtual conference including a plurality of participants, the operations comprising: :
receiving data designating a 3-dimensional virtual space;
receiving a position and direction in the three-dimensional virtual space, the position and direction being input by a first participant among the plurality of participants of the virtual conference;
receiving a video stream captured from a camera on the first participant's device, the camera arranged to capture photographic images of the first participant;
mapping the video stream onto a 3D model of an avatar;
receiving a presentation stream from the device of the first participant;
mapping the presentation stream onto a 3D model of a presentation screen; and
and rendering the three-dimensional virtual space having the mapped avatar and the mapped presentation screen for display to the second participant from a viewpoint of a virtual camera of a second participant among the plurality of participants. . 60. The method of claim 59, wherein the operations are:
receiving an audio stream captured in synchronization with the presentation stream from a microphone of the device of the first participant, the microphone being positioned to capture utterances of the first participant; and
and outputting the audio stream for playback to the second participant in synchronism with the display of the presentation stream within the 3D virtual space. 60. The method of claim 59, wherein the operations are:
receiving a location of a third participant among the plurality of participants in the 3D virtual space;
receiving an audio stream from a microphone of the third participant's device, the microphone being positioned to capture the third participant's utterance; and
adjusting the audio stream to provide a sense of the received position of the third participant in the three-dimensional virtual space relative to the position of the virtual camera;
wherein the act of rendering comprises rendering the three-dimensional virtual space with an avatar for the third participant at the received location for display to the second participant. 60. The method of claim 59, wherein the operations are:
receiving a location of the first participant in the 3D virtual space;
receiving an audio stream from a microphone of the first participant's device, the microphone being positioned to capture the first participant's utterance;
adjusting the audio stream to provide a sense of the received position of the first participant in the three-dimensional virtual space relative to the position of the virtual camera;
rendering the three-dimensional virtual space with an avatar for the third participant at the received location for display to the second participant; and
and adjusting the audio stream to provide a sense of the location of the mapped presentation screen relative to the location of the virtual camera when entering a presentation mode. 60. The device of claim 59, wherein the presentation stream is a video of the first participant. 60. The device of claim 59, wherein the presentation stream is the screen sharing of the first participant. A system for presenting in a virtual conference involving a plurality of participants, comprising:
a processor coupled to the memory;
(i) data designating a three-dimensional virtual space, (ii) a position and direction in the three-dimensional virtual space, wherein the position and direction are input by a first participant of the plurality of participants of the virtual conference; (iii) a video stream captured from a camera on the first participant's device, the camera being arranged to capture photographic images of the first participant; and (iv) receiving a presentation stream from the first participant's device. configured network interface;
a mapper implemented in the processor and configured to map the video stream onto a 3-dimensional model of an avatar and map the presentation stream onto a 3-dimensional model of a presentation screen; and
implemented on the processor, to render, from a viewpoint of a virtual camera of a second participant of the plurality of participants, the three-dimensional virtual space having the mapped avatar and the mapped presentation screen for display to the second participant; A system, containing a configured renderer. 66. The system of claim 65, wherein the presentation stream is a video of the first participant. 66. The system of claim 65, wherein the presentation stream is the first participant's screen share. A computer implemented method for presenting in a virtual conference involving a plurality of participants, comprising:
From a first device of a first participant of the plurality of participants of the virtual conference, (i) a position and direction in the three-dimensional virtual space, the position and direction being input by the first participant, (ii) ) a video stream captured from a camera on the first device, the camera arranged to capture photographic images of the first participant, and (iii) receiving a presentation stream; and
transmitting the presentation stream to a second device of a second participant of the plurality of participants, wherein the second device (i) maps the presentation stream onto a three-dimensional model of a presentation screen; (ii) maps the video stream to an avatar; (iii) from the viewpoint of a virtual camera of the second participant of the plurality of participants, located at the location and oriented in the direction for display to the second participant; and render a three-dimensional virtual space having the mapped three-dimensional model of the avatar and the mapped presentation screen space. 69. The method of claim 68, wherein the presentation stream is a video of the first participant. 69. The method of claim 68, wherein the presentation stream is the first participant's screen share. 69. The method of claim 68,
The method further comprising transmitting, to the second device, a web application having executable code specifying how the second device maps and renders the presentation screen. A computer implemented method for providing a virtual conference involving a plurality of participants, comprising:
rendering a three-dimensional virtual space from the point of view of a virtual camera of a first user for display to the first user, the three-dimensional virtual space including an avatar having texture-mapped video of a second user, wherein the virtual space includes an avatar of a second user; the camera is at a first position in the 3D virtual space and the avatar is at a second position in the 3D virtual space;
receiving an audio stream from a microphone of the second user's device, the microphone being arranged to capture the second user's utterance;
adjusting the volume of the received audio stream to determine a left audio stream and a right audio stream to provide a sense of where the second location is relative to the first location in the three-dimensional virtual space; and
and outputting the left audio stream and the right audio stream in stereo to be played back to the first user. 73. The method of claim 72, wherein the adjusting comprises adjusting the relative volumes of the left audio stream and the right audio stream based on a direction from the second location to the first location. 74. The method of claim 73, wherein the adjusting comprises adjusting relative volumes of the left audio stream and the right audio stream based on a direction from the second location to the first location on a horizontal plane in the three-dimensional virtual space. A method comprising steps. 73. The method of claim 72, wherein the adjusting step adjusts the relative volume of the left audio stream and the right audio stream based on a direction of the second position relative to the first position on a horizontal plane in the three-dimensional virtual space. A method comprising the steps of: 73. The method of claim 72, wherein the adjusting step based on the direction the virtual camera is facing in the three-dimensional virtual space, wherein the left audio stream has a higher volume when the avatar is positioned to the left of the virtual camera. and adjusting the relative volumes of the left and right audio streams such that the right audio stream tends to have a higher volume when the avatar is positioned to the right of the virtual camera. . 77. The method of claim 76, wherein the adjusting step is based on an angle between the direction the virtual camera is facing and the direction the avatar is facing, such that the angle is more perpendicular to where the avatar is facing the left audio stream. adjusting the relative volumes of the left audio stream and the right audio stream to tend to have a larger volume difference between the right audio stream and the right audio stream. 73. The method of claim 72, wherein the adjusting step comprises adjusting the volume of both the first audio stream and the second audio stream based on a distance between the second location and the first location. . A non-transitory tangible computer readable device having stored thereon instructions that, when executed by at least one computing device, cause the at least one computing device to perform operations for a presentation in a virtual conference including a plurality of participants, the operations comprising: :
rendering a 3D virtual space from the viewpoint of a first user's virtual camera for display to the first user, the 3D virtual space including an avatar having a texture mapped video of a second user, the virtual space the camera is at a first position in the 3D virtual space and the avatar is at a second position in the 3D virtual space;
receiving an audio stream from a microphone of the second user's device, the microphone being positioned to capture the second user's utterance;
adjusting the volume of the received audio stream to determine a left audio stream and a right audio stream to provide a sense of where the second location is relative to the first location in the three-dimensional virtual space; and
and outputting the left audio stream and the right audio stream in stereo to be reproduced by the first user. 80. The device of claim 79, wherein the adjusting comprises adjusting relative volumes of the left audio stream and the right audio stream based on a direction from the second location to the first location. 80. The method of claim 79, wherein the adjusting comprises adjusting relative volumes of the left audio stream and the right audio stream based on a direction from the second position to the first position on a horizontal plane in the three-dimensional virtual space. A device containing an action. 80. The method of claim 79, wherein the adjusting operation adjusts the relative volume of the left audio stream and the right audio stream based on a direction of the second position relative to the first position on a horizontal plane in the three-dimensional virtual space. A device that includes an operation to do. 83. The method of claim 82, wherein the adjusting operation causes the left audio stream to have a higher volume when the avatar is positioned to the left of the virtual camera based on the direction the virtual camera is facing in the three-dimensional virtual space. and adjusting relative volumes of the left audio stream and the right audio stream such that the right audio stream tends to have a higher volume when the avatar is positioned to the right of the virtual camera. . 84. The method of claim 83, wherein the adjusting operation is based on an angle between the direction the virtual camera is facing and the direction the avatar is facing, such that the angle is more perpendicular to the direction the avatar is facing the left audio stream. and adjusting the relative volumes of the left audio stream and the right audio stream to tend to have a larger volume difference between the right audio stream and the right audio stream. 80. The device of claim 79, wherein the adjusting comprises adjusting the volume of both the first audio stream and the second audio stream based on a distance between the second location and the first location. . A system for providing a virtual conference involving a plurality of participants, comprising:
a processor coupled to the memory;
a renderer embodied in the processor and configured to render a three-dimensional virtual space for display to the first user from the viewpoint of a first user's virtual camera, the three-dimensional virtual space avatar having texture-mapped video of a second user; including, wherein the virtual camera is at a first position in the 3D virtual space and the avatar is at a second position in the 3D virtual space;
a network interface configured to receive an audio stream from a microphone of the second user's device, the microphone being arranged to capture speech of the second user;
an audio processor configured to adjust a volume of the received audio stream to determine a left audio stream and a right audio stream to provide a sense of where the second location is relative to the first location in the three-dimensional virtual space; and
and a stereo speaker outputting the left audio stream and the right audio stream in stereo to be reproduced to the first user. 87. The system of claim 86, wherein the audio processor is configured to adjust relative volumes of the left audio stream and the right audio stream based on a direction from the second location to the first location. 88. The apparatus of claim 87 , wherein the audio processor is configured to adjust relative volumes of the left audio stream and the right audio stream based on a direction from the second location to the first location on a horizontal plane within the three-dimensional virtual space. being, the system. 87. The method of claim 86, wherein the audio processor is configured to adjust relative volumes of the left audio stream and the right audio stream based on a direction of the second location relative to the first location on a horizontal plane in the three-dimensional virtual space. configured system. 90. The audio processor of claim 89, wherein the audio processor tends to have a higher volume when the avatar is positioned to the left of the virtual camera, based on the direction the virtual camera is facing in the three-dimensional virtual space. and adjust the relative volumes of the left audio stream and the right audio stream such that the right audio stream tends to have a higher volume when the avatar is positioned to the right of the virtual camera. 87. The method of claim 86, wherein the audio processor determines, based on an angle between the direction the virtual camera is facing and the direction the avatar is facing, that the angle is more perpendicular to where the avatar is facing the left audio stream. and adjust the relative volumes of the left audio stream and the right audio stream to tend to have a larger volume difference between the right audio stream. 87. The system of claim 86, wherein the audio processor is configured to adjust the volume of both the first audio stream and the second audio stream based on a distance between the second location and the first location. A computer implemented method for providing audio for a virtual conference, comprising:
(a) rendering, from the viewpoint of a first user's virtual camera, at least a portion of a three-dimensional virtual space for display to the first user, wherein the three-dimensional virtual space includes an avatar representing a second user; , the virtual camera is at a first position in the 3D virtual space and the avatar is at a second position in the 3D virtual space, the 3D virtual space is divided into a plurality of regions;
(b) receiving an audio stream from a microphone of the second user's device, the microphone being arranged to capture the second user's utterance;
(c) determining whether the virtual camera and the avatar are located in the same area among the plurality of areas;
(d) determining whether the avatar is in a podium area of the plurality of areas;
(e) attenuating the audio stream when it is determined that the virtual camera and the avatar are not located in the same area and the avatar is not located in the podium area; and
(f) outputting the audio stream to be played back to the first user. 94. The method of claim 93, wherein the audio stream is a first audio stream, the three-dimensional virtual space includes a second avatar representing a third user, and the determining step (c) includes the virtual camera and the avatar Determining that they are located in the same area;
(g) receiving a second audio stream from a microphone of the first user's device, the microphone being arranged to capture speech of the first user;
(h) determining that the second avatar and the virtual camera are located in an area of the 3D virtual space different from the same area in which the virtual camera is located; and
(i) attenuating the first audio stream and the second audio stream to prevent the third user from hearing the first audio stream and the second audio stream, and between the first user and the second user; A computer implemented method further comprising enabling a pseudo-private conversation. 94. The method of claim 93, wherein each of the plurality of regions has a wall transmission factor specifying how much the audio stream is attenuated in (e). 94. The method of claim 93, wherein each region of the plurality of regions has a distance transmission factor;
(g) determining a distance between the virtual camera and the avatar in the 3D virtual space;
(h) determining at least one region between the virtual camera and the avatar; and
(i) attenuating the audio stream based on the distance determined in (g) and the distance transmittance corresponding to the at least one region determined in (h). 94. The method of claim 93, wherein the plurality of regions are organized as a hierarchy. 98. The method of claim 97, wherein respective areas of the plurality of areas have a wall transmittance;
(g) traversing the hierarchy to determine a region subset of the plurality of regions between the region containing the avatar and the region containing the virtual camera; and
(h) attenuating the audio stream based on the respective wall transmittance corresponding to the region subset determined in (g). A non-transitory tangible computer readable device having stored thereon instructions that, when executed by at least one computing device, cause the at least one computing device to perform operations to provide audio for a virtual meeting, the operations comprising:
(a) rendering at least a portion of a 3D virtual space for display to the first user, from the viewpoint of a first user's virtual camera, wherein the 3D virtual space includes an avatar representing a second user; , the virtual camera is at a first position in the 3D virtual space and the avatar is at a second position in the 3D virtual space, the 3D virtual space is divided into a plurality of regions;
(b) receiving an audio stream from a microphone of the second user's device, the microphone being positioned to capture the second user's utterance;
(c) determining whether the virtual camera and the avatar are located in the same area among the plurality of areas;
(d) determining whether the avatar is in a podium area among the plurality of areas;
(e) attenuating the audio stream when it is determined that the virtual camera and the avatar are not located in the same area and the avatar is not located in the podium area; and
(f) outputting the audio stream to be reproduced to the first user. 101. The method of claim 99, wherein the audio stream is a first audio stream, the three-dimensional virtual space includes a second avatar representing a third user, and the determining operation (c) includes the virtual camera and the avatar and determining that they are located in the same area, wherein the operations are:
(g) receiving a second audio stream from a microphone of the first user's device, the microphone being positioned to capture speech of the first user;
(h) determining that the second avatar and the virtual camera are located in an area of the 3D virtual space different from the same area in which the virtual camera is located; and
(i) attenuating the first audio stream and the second audio stream to prevent the third user from hearing the first audio stream and the second audio stream, and between the first user and the second user; The device further comprising an operation enabling a pseudo-private conversation. 101. The device of claim 99, wherein each area of the plurality of areas has a wall transmittance specifying how much the audio stream is attenuated in (e). 100. The method of claim 99, wherein each area of the plurality of areas has a distance transmittance, and the operations are:
(g) determining a distance between the virtual camera and the avatar in the 3D virtual space;
(h) determining at least one area between the virtual camera and the avatar; and
(i) attenuating the audio stream based on the distance determined in (g) and the distance transmittance corresponding to the at least one area determined in (h). 101. The device of claim 99, wherein the plurality of areas are organized as a hierarchy. 104. The method of claim 103, wherein respective areas of the plurality of areas have a wall transmittance, and the operations are to:
(g) traversing the hierarchy to determine a region subset of the plurality of regions between a region containing the avatar and a region containing the virtual camera; and
(h) attenuating the audio stream based on the respective wall transmittance corresponding to the region subset determined in (g). A system for providing audio for a virtual conference, comprising:
a processor coupled to the memory;
a renderer implemented on a processor and configured to render, from the viewpoint of a virtual camera of a first user, at least a portion of a three-dimensional virtual space for display to the first user, wherein the three-dimensional virtual space represents an avatar representing a second user; wherein the virtual camera is at a first position in the 3D virtual space and the avatar is at a second position in the 3D virtual space, and the 3D virtual space is divided into a plurality of regions configured as a hierarchical structure. become -;
a network interface configured to receive an audio stream from a microphone of the second user's device, the microphone being arranged to capture speech of the second user; and
determine whether the virtual camera and the avatar are in the same area of the plurality of areas and whether the avatar is in a podium area of the plurality of areas, and the virtual camera and the avatar are not located in the same area and an audio processor configured to attenuate the audio stream and output the audio stream to be played back to the first user when it is determined that the avatar is not in the podium area. 106. The system of claim 105, wherein each area of the plurality of areas has a wall transmittance specifying how much the audio stream is attenuated. 106. The method of claim 105, wherein each region of the plurality of regions has a distance transmittance, and the audio processor: (i) determines a distance between the virtual camera and the avatar in the 3D virtual space; ( ii) determine at least one area between the virtual camera and the avatar, and (iii) attenuate the audio stream based on the determined distance and the distance transmittance corresponding to the determined at least one area. , system. 106. The method of claim 105, wherein each region of the plurality of regions has a wall transmittance, and the audio processor determines a region sub of the plurality of regions between a region including the avatar and a region including the virtual camera. traverse the hierarchy to determine a set, and attenuate the audio stream based on a respective wall transmittance corresponding to the determined subset of regions. A computer implemented method for streaming video for a virtual conference, comprising:
(a) determining a distance between a first user and a second user in the virtual meeting space;
(b) receiving a video stream captured from a camera on the device of the first user, the camera arranged to capture photographic images of the first user;
(c) selecting a reduced resolution or bitrate of the video stream based on the determined distance such that a closer distance results in a higher resolution or bitrate than a greater distance; and
(d) requesting transmission of the video stream at the reduced resolution or bitrate to the device of the second user for display to the second user within the virtual meeting space, wherein the video stream is transmitted within the virtual meeting space. mapped onto the first user's avatar for display to the second user. 109. The method of claim 109,
(e) receiving a second video stream captured from a camera on a device of a third user, the camera arranged to capture photographic images of the third user;
(f) determining available bandwidth for transmitting data for the virtual conference to the second user;
(g) determining a second distance between the first user and the second user in the virtual meeting space; and
(h) the use between the video stream received in (b) and the second video stream received in (e) based on the second distance determined in (g) for the distance determined in (a). The method further comprises apportioning the available bandwidth. 111. The method of claim 110, wherein the allocating (h) comprises prioritizing video streams of closer users over video streams from more distant users. 111. The method of claim 110,
(i) receiving a first audio stream from the device of the first user;
(j) further comprising receiving a second audio stream from the device of the third user, wherein the distributing (h) comprises reserving a portion of the first and second audio streams. Way. 112. The method of claim 112,
(k) reducing the quality of the first and second audio streams according to the size of the reserved portion. 114. The method of claim 113, wherein the reducing step (k) comprises reducing the quality independently of the second distance determined in (g) relative to the distance determined in (a). 109. The method of claim 109,
(e) determining that the distance between the first user and the second user at the virtual meeting space makes display of video at the distance ineffective;
In response to the determination in (e):
(f) stopping transmission of the video stream to the device of the second user; and
(g) notifying the device of the second user to replace the video stream with a still image. 110. The method of claim 109, wherein the video stream at the reduced resolution is onto an avatar to be rendered at the second user's location within the virtual meeting space for display by the second user's device to the second user. Mapped, how. A non-transitory tangible computer readable device having stored thereon instructions that, when executed by at least one computing device, cause the at least one computing device to perform operations for streaming video for a virtual meeting, the operations comprising:
(a) determining a distance between a first user and a second user in the virtual meeting space;
(b) receiving a video stream captured from a camera on the device of the first user, the camera arranged to capture photographic images of the first user;
(c) selecting a reduced resolution or bitrate of the video stream based on the determined distance such that a closer distance results in a higher resolution or bitrate than a greater distance; and
(d) requesting transmission of the video stream at the reduced resolution or bitrate to the device of the second user for display to the second user within the virtual meeting space, wherein the video stream is transmitted within the virtual meeting space mapped onto the first user's avatar for display to the second user. 118. The method of claim 117, wherein the operations are:
(e) receiving a second video stream captured from a camera on a third user's device, the camera arranged to capture photographic images of the third user;
(f) determining an available bandwidth for transmitting data for the virtual conference to the second user;
(g) determining a second distance between the first user and the second user in the virtual meeting space; and
(h) the use between the video stream received in (b) and the second video stream received in (e) based on the second distance determined in (g) relative to the distance determined in (a). The device further comprising allocating available bandwidth. 119. The device of claim 118, wherein the distributing (h) comprises giving priority to video streams of closer users over video streams from more distant users. 119. The method of claim 118, wherein the operations are:
(i) receiving a first audio stream from the device of the first user;
(j) further comprising receiving a second audio stream from the device of the third user, wherein the distributing (h) includes reserving a portion of the first and second audio streams; device. 121. The method of claim 120, wherein the actions are:
(k) reducing the quality of the first and second audio streams according to the size of the reserved portion. 122. The device of claim 121, wherein the reducing operation (k) comprises reducing the quality independently of the second distance determined in (g) relative to the distance determined in (a). 118. The method of claim 117, wherein the operations are:
(e) determining that the distance between the first user and the second user at the virtual meeting space makes display of video at the distance ineffective;
In response to the determination in (e):
(f) stopping transmission of the video stream to the device of the second user; and
(g) notifying the device of the second user to replace the video stream with a still image. 118. The method of claim 117, wherein the video stream at the reduced resolution is onto an avatar to be rendered at the second user's location in the virtual meeting space for display by the second user's device to the second user. The device being mapped. A system for streaming video for a virtual conference, comprising:
a processor coupled to the memory;
a network interface that receives a video stream captured from a camera on a device of a first user, the camera arranged to capture photographic images of the first user; and
determining a distance between a first user and a second user in a virtual conference space, and reducing the resolution of the video stream based on the determined distance such that a closer distance results in a greater resolution or bitrate than a greater distance A stream conditioner configured to
wherein the network interface is configured to transmit the video stream of the reduced resolution to a device of the second user for display to the second user in the virtual meeting space, wherein the video stream is configured to transmit the video stream to the second user in the virtual meeting space. mapped onto the first user's avatar for display to the user -, system. 126. The method of claim 125, wherein the network interface receives a second video stream captured from a camera on a third user's device, the camera arranged to capture photographic images of the third user;
The stream coordinator (i) determines an available bandwidth for transmitting data for the virtual meeting to the second user, and (ii) determines a second distance between the first user and the second user in the virtual meeting space. and (iii) allocate the available bandwidth between the video stream and the second video stream based on the second distance relative to the distance. 127. The system of claim 126, wherein the stream coordinator is configured to prioritize video streams of closer users over video streams from more distant users. 127. The method of claim 126, wherein the network interface is configured to receive a first audio stream from the first user's device and to receive a second audio stream from the third user's device, the stream coordinator comprising the first and second audio streams. A system configured to reserve a portion of the second audio streams. A computer implemented method for streaming video for a virtual video conference, comprising:
Receiving a 3D model of a virtual environment;
Receiving a mesh representing a 3D model of an object;
receiving a video stream of a first participant of the virtual video conference, the video stream comprising a plurality of frames;
mapping respective frames of the plurality of frames of the video stream onto a three-dimensional model represented by a mesh to create an avatar navigable by the first participant, the mesh being created independent of the video stream. -; and
rendering, from the viewpoint of a second participant's virtual camera, the mesh representing the mapped avatar and the three-dimensional model of the object in the virtual environment for display for the second participant. 130. The method of claim 129, wherein the object is a product and the second participant can navigate the virtual camera around the three-dimensional model of the product. 130. The method of claim 129, wherein the object is an advertisement. 129. The method of claim 129,
and demonstrating the product at the virtual meeting space. 130. The method of claim 129, wherein the virtual environment is a building. 130. The method of claim 129, wherein the mesh is a first mesh,
and sending a request for the first mesh from the second participant, wherein the step of receiving the first mesh occurs in response to the request. 129. The method of claim 129,
receiving the location and direction of the first user in the 3D virtual space;
receiving a video stream captured from a camera on the device of the first user, the camera arranged to capture photographic images of the first user; and
further comprising mapping the video stream onto a three-dimensional model of an avatar; wherein the rendering comprises rendering the virtual meeting to include a mapped three-dimensional model of the avatar positioned at the location and oriented in the orientation. A non-transitory tangible computer readable device having stored thereon instructions that, when executed by at least one computing device, cause the at least one computing device to perform operations for streaming video for a virtual meeting, the operations comprising:
receiving a 3D model of the virtual environment;
receiving a mesh representing a 3D model of an object;
receiving a video stream of a first participant of the virtual video conference, the video stream comprising a plurality of frames;
mapping respective frames of the plurality of frames of the video stream onto a three-dimensional model represented by a mesh to create an avatar navigable by the first participant, the mesh being created independent of the video stream. -; and
rendering, from the viewpoint of a second participant's virtual camera, the mesh representing the mapped avatar and the three-dimensional model of the object in the virtual environment for display for the second participant. A tangible computer readable device. 137. The non-transitory tangible computer readable device of claim 136, wherein the object is a product and the second participant can navigate the virtual camera around the three-dimensional model of the product. 137. The non-transitory tangible computer readable device of claim 136, wherein the object is an advertisement. 138. The method of claim 137, wherein the operations are:
A non-transitory tangible computer-readable device further comprising demonstrating the product in the virtual meeting space. 137. The non-transitory tangible computer readable device of claim 136, wherein the virtual environment is a building. 137. The method of claim 136, wherein the mesh is a first mesh, and the operations are:
and sending a request for the first mesh from the second participant, wherein the step of receiving the first mesh occurs in response to the request. 137. The method of claim 136, wherein the operations are:
receiving a location and direction of the first user in the 3D virtual space;
receiving a video stream captured from a camera on the device of the first user, the camera arranged to capture photographic images of the first user; and
further comprising mapping the video stream onto a 3D model of an avatar; Wherein the act of rendering comprises an act of rendering the virtual meeting to include a mapped three-dimensional model of the avatar disposed at the location and oriented in the direction. A system for streaming video for virtual videoconferencing, comprising:
a processor coupled to the memory;
(i) a three-dimensional model of a virtual environment, (ii) a mesh representing a three-dimensional model of an object, (iii) a network interface configured to receive a video stream of a first participant of the virtual video conference, the video stream comprising a plurality of contains frames -;
A mapper that maps respective frames of the plurality of frames of the video stream onto a three-dimensional model represented by a mesh to create an avatar navigable by the first participant, the mesh being created independent of the video stream. -; and
a renderer configured to render, from the viewpoint of a second participant's virtual camera, the mesh representing the mapped avatar and the three-dimensional model of the object in the virtual environment for display for the second participant. . 144. The system of claim 143, wherein the object is a product and the second participant can navigate the virtual camera around the three-dimensional model of the product. 144. The system of claim 143, wherein the object is an advertisement. 144. The system of claim 143, wherein the virtual environment is a building.

Images