JP7318139B1 - Web-based videoconferencing virtual environment with steerable avatars and its application - Google Patents

Abstract

Disclosed herein is a web-based videoconferencing system that allows video avatars (102A, 102B) to navigate within a virtual environment. The system has a presentation mode that allows the presentation stream to be texture-mapped onto the presenter's screen (104A, 104B) placed in the virtual environment. Relative left and right audio are adjusted to provide a sense of the avatar's position in the virtual space. The audio is further adjusted based on the area in which the avatar is located and the area in which the virtual camera is located. The quality of the video stream is adjusted based on relative position in virtual space. Three-dimensional modeling is available within the virtual videoconferencing environment.

Description

Cross-references to related applications
[0001] This application is derived from U.S. utility patent application no. 17/075,338, U.S. Utility Patent Application No. 17/198,323 filed Mar. 11, 2021, now U.S. Patent No. 11,095 issued Aug. 17, 2021 U.S. Utility Patent Application No. 17/075,362 filed Oct. 20, 2020, issued as No. 857, now U.S. Patent No. 10, issued Mar. 16, 2021. U.S. Utility Patent Application No. 17/075,390 filed Oct. 20, 2020, issued as No. 952,006, now U.S. Patent No. 11, issued Jul. 20, 2021; U.S. Utility Patent Application No. 17/075,408 filed Oct. 20, 2020, issued as No. 070,768, now U.S. Patent No. U.S. Utility Patent Application No. 17/075,428 filed October 20, 2020, published as No. 11,076,128 and U.S. Utility Patent Application filed October 20, 2020 No. 17/075,454 is claimed. The contents of each of these applications are hereby incorporated by reference in their entirety.

background field
[0002] This field relates generally to videoconferencing.

Related technology
[0003] Videoconferencing involves the transmission and reception of audiovisual signals by multiple users at different locations for real-time communication between people. Videoconferencing is widely available on many computing devices from a variety of different services, including the ZOOM service available from Zoom Communications Inc. of San Jose, CA. Some video conferencing software, such as the FaceTime® application available from Apple Inc. of Cupertino, CA, comes standard with mobile devices.

[0004] In general, these applications operate by displaying video and outputting audio of other conference participants. If there are multiple participants, the screen may be split into several rectangular frames, each displaying one participant's video. These services may also work by having a larger frame presenting the video of the person speaking. As different individuals speak, the frames switch between speakers. The application captures video from a camera integrated into the user's device and audio from a microphone integrated into the user's device. The application then sends its audio and video to other applications running on other users' devices.

[0005] Many of these video conferencing applications have screen sharing capabilities. When a user specifies sharing their screen (or part of their screen), a stream is sent to the other users' devices along with the contents of the screen. In some cases, other users can even control what is on that user's screen. In this way, users can collaborate on projects or give presentations to other meeting participants.

[0006] In recent years, videoconferencing technology has gained in importance. Fearing the spread of disease, especially COVID-19, many workplaces, trade fairs, meetings, conferences, schools and places of worship have been closed or discouraged from attending. Virtual meetings using videoconferencing technology are increasingly replacing physical meetings. In addition, this technology offers advantages over physical fit because it avoids business travel and commuting.

[0007] In many cases, however, the use of this videoconferencing technology results in a loss of sense of place. There is an experiential aspect of meeting physically face-to-face and being in the same place, which is lost when meetings are held virtually. There is a social aspect where users can pose and see their peers. This sense of experience is important in making relationships and social connections. When it comes to traditional video conferencing, this feeling is still lacking so far.

[0008] Further problems arise with these videoconferencing techniques when a conference is started with several participants. In a physical meeting conference, people can chat. You can speak in a way that only people who are close to you can hear you. In some cases, you can even have a private conversation in the context of a larger meeting. However, when a virtual conference is used and multiple people are speaking at the same time, the software mixes the two audio streams approximately equally and forces the participants to interrupt each other to speak. Therefore, when multiple people are involved in a virtual meeting, private conversation is not possible and interaction tends to be in the form of utterances from one to many. Again, virtual meetings miss the opportunity for participants to make social connections and communicate more effectively to network.

[0009] Furthermore, due to network bandwidth and computational hardware limitations, the performance of many video conferences begins to slow when many streams occur in the conference. Many computing devices are equipped to handle video streams from 2-3 participants and are not equipped to handle video streams from 12 or more participants. In many schools that operate entirely virtually, a class of 25 can slow school-issued computing devices significantly.

[0010] Massively multiplayer online games (MMOGs or MMOs) can generally handle significantly more than 25 participants. These games often have hundreds or thousands of players on a single server. MMOs often allow a player to navigate an avatar around a virtual world. These MMOs may allow users to talk to each other or send messages to each other. Examples are ROBLOX games available from Roblox Corporation of San Mateo, CA and MINECRAFT games available from Mojang Studios of Stockholm, Sweden.

[0011] Having naked avatars interact with each other also has limitations with respect to social interaction. These avatars typically cannot convey the facial expressions that people often make unintentionally. These facial expressions are observable in a teleconference. Several publications may describe videos placed on avatars in virtual worlds. However, these systems typically require specialized software and have other limitations that limit their usefulness.

[0012] There is a need for improved methods of videoconferencing.

overview
[0013] In one embodiment, a device enables a video conference between a first user and a second user. The device includes a processor coupled to memory, a display screen, a network interface, and a web browser. The network interface comprises (i) data specifying a three-dimensional virtual space, (ii) a position and orientation in the three-dimensional virtual space that are input by the first user, and (iii) the It is configured to receive a video stream captured from a camera of the first user's device. The first user's camera is positioned to capture a photographic image of the first user. A web browser is implemented on the processor and configured to download a web application from a server and execute the web application. Web applications include texture mappers and renderers. A texture mapper is configured to texture map the video stream onto the three-dimensional model of the avatar. a renderer comprising the texture-mapped three-dimensional model of the avatar positioned at the location and oriented in the direction for display to the second user from the perspective of the second user's virtual camera; Configured to render a three-dimensional virtual space. By managing texture mapping within a web application, embodiments avoid the need to install specialized software.

[0014] In one embodiment, a computer-implemented method enables presentation in a virtual conference including multiple participants. In the method, data specifying a three-dimensional virtual space is received. A position and orientation in the three-dimensional virtual space are also received. The position and orientation were entered by the first of the multiple participants to the conference. Finally, a video stream captured from the camera of the first participant's device is received. A camera was positioned to capture a photographic image of the first participant. The video stream is texture-mapped onto a three-dimensional model of the avatar. Additionally, a presentation stream from the first participant's device is received. The presentation stream is texture mapped onto a 3D model of the presentation image. Finally, the three-dimensional virtual space with the texture-mapped avatar and the texture-mapped presentation screen is displayed to a second of the multiple participants from the perspective of the second participant's virtual camera. is rendered for In this manner, embodiments enable presentations in a social conference environment.

[0015] In one embodiment, a computer-implemented method provides audio for a virtual conference including multiple participants. In the method, a three-dimensional virtual space containing an avatar with texture-mapped video of the second user is rendered for display to the first user from the viewpoint of the first user's virtual camera. The virtual camera is at a first position in the three-dimensional virtual space and the avatar is at a second position in the three-dimensional virtual space. An audio stream is received from a microphone of a second user's device. A microphone was positioned to pick up the second user's speech. Volumes of the received audio streams are adjusted to specify the left audio stream and the right audio stream to provide a sense of where the second location is relative to the first location in the three-dimensional virtual space. The left and right audio streams are output in stereo and played to the first user.

[0016] In one embodiment, a computer-implemented method provides audio for a virtual conference. In the method, a three-dimensional virtual space containing an avatar with texture-mapped video of the second user is rendered for display to the first user from the perspective of the first user's virtual camera. The virtual camera is at a first position in the three-dimensional virtual space and the avatar is at a second position in the three-dimensional virtual space. An audio stream is received from a microphone of a second user's device. It is specified whether the virtual camera and avatar are located in the same area of the multiple areas. If the virtual camera and avatar are specified not to be located in the same area, the audio stream will be attenuated. An attenuated audio stream is output and played to the first user. In this manner, embodiments enable private conversations and chit-chat in a virtual videoconferencing environment.

[0017] In one embodiment, a computer-implemented method efficiently streams video for a virtual conference. In the method, a distance is identified between a first user and a second user in the virtual conference space. A video stream captured from a camera of a first user's device is received. The camera was positioned to capture a photographic image of the first user. The resolution or bitrate of the video stream is reduced based on the specified distance such that closer distances have higher resolution than longer distances. The video stream is transmitted at a reduced resolution or bitrate to the second user's data for display to the second user within the virtual conference space. The video stream should be texture mapped to the first user's avatar for display to the second user in the virtual conference space. In this manner, embodiments efficiently allocate bandwidth and computational resources, even when there are a large number of conference participants.

[0018] In one embodiment, a computer-implemented method enables modeling in a virtual video conference. In the method, a three-dimensional model of a virtual environment, a mesh representing a three-dimensional model of an object and a video stream from participants of a virtual videoconference are received. The video stream is texture-mapped onto avatars that can be steered by the participants. Texture-mapped avatars and meshes representing three-dimensional models of objects in the virtual environment are rendered for display.

[0019] Systems, devices, and computer program product embodiments are also disclosed.

[0020] Further embodiments, features, and advantages of the present inventions, as well as the structure and operation of the various embodiments, are described in detail below with reference to the accompanying drawings.

Brief description of the drawing
[0021] The accompanying drawings, which are incorporated in and constitute a part of the present invention, illustrate the disclosure and, together with the description, explain the principles of the disclosure and enable a person skilled in the art to make and use the disclosure. It also functions as if

[0022] FIG. 3 illustrates an example interface for providing video conferencing in a virtual environment with video streams mapped to avatars. [0023] Fig. 3 depicts a three-dimensional model used to render a virtual environment with avatars for a video conference; [0024] Figure 1 illustrates a system for providing video conferencing in a virtual environment; [0025] FIG. 4 illustrates how data is transferred between the various components of the system of FIG. 3 to provide a video conference. [0025] FIG. 4 illustrates how data is transferred between the various components of the system of FIG. 3 to provide a video conference. [0025] FIG. 4 illustrates how data is transferred between the various components of the system of FIG. 3 to provide a video conference. [0026] Fig. 6 is a flow chart illustrating a method for adjusting relative left and right volume to provide a sense of position in a virtual environment during a video conference. [0027] Fig. 5 is a chart showing how the volume decreases as the distance between avatars increases. [0028] FIG. 7 is a flowchart illustrating a method of adjusting relative volume to provide different volume areas in a virtual environment during a video conference. [0029] Fig. 4 illustrates different volume areas in a virtual environment during a video conference; [0029] Fig. 4 illustrates different volume areas in a virtual environment during a video conference; [0030] Fig. 10 illustrates traversing the volume area hierarchy in a virtual environment during a video conference; [0030] Fig. 10 illustrates traversing the volume area hierarchy in a virtual environment during a video conference; [0030] Fig. 10 illustrates traversing the volume area hierarchy in a virtual environment during a video conference; [0031] Fig. 3 illustrates interaction with a three-dimensional model in a three-dimensional virtual environment; [0032] Fig. 3 depicts presentation screen sharing in a three-dimensional virtual environment used for video conferencing. [0033] Fig. 4 is a flow chart illustrating a method of allocating available bandwidth based on relative positions of avatars within a three-dimensional virtual environment. [0034] Fig. 7 is a chart showing how priority values can decrease as the distance between avatars increases. [0035] Fig. 6 is a chart showing how the allocated bands can be changed based on relative priority. [0036] Fig. 3 illustrates components of a device used to provide video conferencing within a virtual environment;

[0037] The drawing in which an element first appears is typically indicated by the leftmost digit or digits in the corresponding reference number. In the drawings, like reference numbers may indicate identical or functionally identical elements.

Detailed description Teleconferencing with avatars in a virtual environment
[0038] FIG. 1 illustrates an example interface 100 for providing video conferencing in a virtual environment in which video streams are mapped to avatars.

[0039] Interface 100 may be displayed to participants in a video conference. For example, interface 100 may be rendered for display to participants and may be constantly updated as the videoconference progresses. The user may control the orientation of the user's virtual camera using keyboard input, for example. In this manner, the user can navigate around the virtual environment. In one embodiment, different inputs may change the X and Y position and pan and tilt angles of the virtual camera in the virtual environment. In further embodiments, the user may use input to change the height (Z coordinate) or yaw of the virtual camera. In a further embodiment, the user may enter an input to cause the virtual camera to "hop" up to simulate gravity while the virtual camera returns to its original position. Inputs available for navigating the virtual camera include, for example, the WASD keyboard keys to move the virtual camera forward, backward, left and right in the XY plane, the spacebar key to "hop" the virtual camera, and change the pan and tilt angles. can include keyboard input and mouse input, such as mouse movements that specify

[0040] Interface 100 includes avatars 102A and 102B, each representing a different participant in the videoconference. Avatars 102A and 102B have texture-mapped video streams 104A and 104B from the devices of the first and second participants, respectively. A texture map is an image that is applied (mapped) to the surface of a shape or polygon. Here, an image is each frame of the video. Camera devices capturing video streams 104A and 104B are positioned to capture each participant's face. In this way, the avatar is texture-mapped, moving the image of the face as participants in the meeting listen.

[0041] Similar to the way virtual cameras are controlled by user viewing interface 100, the location and orientation of avatars 102A and 102B are controlled by each participant they represent. Avatars 102A and 102B are three-dimensional models represented by meshes. Each avatar 102A and 102B may have the participant's full name under the avatar.

[0042] Each avatar 102A and 102B is controlled by a different user. Each avatar may be positioned at a point corresponding to where the avatar's own virtual camera is positioned within the virtual environment. Different users can move around their respective avatars 102A and 102B, just as the user viewing interface 100 can move around a virtual camera.

[0043] The virtual environment rendered in the interface 100 includes a background image 120 and a three-dimensional model 118 of the arena. An arena can be a venue or building where a video conference is to take place. The arena may include floor areas separated by walls. Three-dimensional model 118 may include meshes and textures. Other methods of mathematically representing the surface of three-dimensional model 118 may be possible as well. For example, polygon modeling, curve modeling, and digital sculpting may be possible. For example, three-dimensional model 118 may be represented by voxels, splines, geometric primitives, polygons, or any other possible representation in three-dimensional space. The three-dimensional model 118 can also include specifications for the light sources. Light sources can include, for example, point light sources, directional light sources, spotlight light sources, and ambient light sources. Objects can also have certain properties that describe how they reflect light. In examples, properties may include diffuse lighting interactions, ambient lighting interactions, and spectral lighting interactions.

[0044] In addition to the arena, the virtual environment may include various other three-dimensional models that depict different components of the environment. For example, the three-dimensional environment can include decoration model 114 , speaker model 116 , and presentation screen model 122 . Just like models 118, these models can be represented using any mathematical method for representing geometric surfaces in three-dimensional space. These models may be separate from model 118 or may be combined into a single representation of the virtual environment.

[0045] Decorative models, such as model 114, serve to enhance realism and increase the aesthetic appeal of the arena. Speaker model 116 may virtually emit sounds such as presentations and background music, as described in more detail below with respect to FIGS. The presentation screen model 122 can function to provide outlets for demonstrating presentations. A presenter or presentation screen sharing video may be texture mapped onto the presentation screen model 122 .

[0046] Button 108 may provide the user with a list of participants. In one example, after the user selects button 108, the user can chat with other participants by sending text messages individually or as a group.

Button 110 may allow a user to change attributes of the virtual camera used to render interface 100 . For example, a virtual camera may have a field of view that specifies the angle at which data is rendered for display. Modeling of data within the camera's field of view may be rendered, while modeling of data outside the camera's field of view may not be rendered. By default, the field of view of the virtual camera can be set anywhere between 60° and 110°, commensurate with wide-angle lenses and human vision. However, when button 110 is selected, the virtual camera can increase the field of view beyond 170° to match the fisheye lens. This may allow the user to have a broader awareness of their surroundings in the virtual environment.

[0048] Finally, button 112 causes the user to exit the virtual environment. Selection of button 112 may notify devices belonging to other participants with a notification signaling the device to stop displaying the avatar corresponding to the user previously viewing interface 100 .

[0049] In this way, the interface virtual 3D space is used to hold a video conference. Any user controls an avatar, and a user can control an avatar to move around, look around, jump, or do other things that change position or orientation. A virtual camera presents the user with a virtual 3D environment and other avatars. The other user's avatar has as an integral part a virtual display showing the user's webcam image.

[0050] By giving users a sense of space and allowing them to see each other's faces, embodiments provide a more social experience than traditional web conferencing or traditional MMO games. A more social experience has multiple uses. For example, it can be used in online shopping. For example, the interface 100 can be used for virtual supermarkets, houses of worship, trade fairs, B2B sales, B2C sales, schooling, restaurants or canteens, product releases, building site visits (e.g. for architects, engineers, contractors), office spaces (e.g. people work virtually "at their desks"), remote control of machines (ships, vehicles, planes, submarines, drones, drilling equipment, etc.), factory/facility control rooms, medical procedures, garden design, guided virtual Bus tours, musical events (e.g. concerts), lectures (e.g. TED talks), political group meetings, board meetings, underwater surveys, surveys of hard-to-reach locations, emergency drills (e.g. fire), cooking, shopping (payment and delivery), virtual arts and crafts (e.g. painting and pottery), weddings, funerals, baptisms, remote sports training, counseling, coping with fear (e.g. face-to-face therapy), fashion shows, amusement parks, home Decoration, watching sports, watching e-sports, viewing performances captured using 3D cameras, playing board games and rule-playing games, walking on/through medical images, viewing geological data, language learning , meetings in spaces for the visually impaired, meetings in spaces for the deaf, participation in events by people who normally cannot walk or stand, news or weather presentations, talk shows, book signings, Voting, MMOs, buying/selling virtual locations (such as those available in some MMOs such as the SECOND LIFE game available from Linden Research, Inc. of San Francisco, CA), flea markets, garage sales, travel Agencies, banks, archives, computer process control, fencing/sword fighting/martial arts, reenactment videos (e.g. crime scenes and/or accident reenactments), real event rehearsals (e.g. weddings, presentations, shows, space walks), 3D cameras. Livestock shows, zoos, experiencing life as a tall/short/blind/deaf/white/black person (e.g. user wants to experience reactions) changing video streams or still images of virtual worlds to simulate viewpoints), job interviews, game shows, interactive fiction (e.g. murder mysteries), virtual fishing, virtual sailing, psychological research, behavioral analysis, virtual sports (e.g. climbing/bouldering), lighting control, etc. in the home or elsewhere (home automation), memory palaces, archeology, gift shops, virtual visits, procedures to make the customer more comfortable when visiting in person virtual medical procedures and virtual exchanges/financial markets/stock markets (e.g. integrating real-time data and video feeds into real-time trading and analysis of virtual worlds) to explain and make people feel more comfortable, actually interacting with each other virtual places people need to go to as part of their work (e.g. if you want to create an invoice, you can only do so from within the virtual place), and facial expressions ( Augmented reality that projects a person's face onto an AR headset (or helmet) so that it can be seen (e.g. useful for military, law enforcement, firefighters, special operations), as well as reservations (e.g. specific villas/cars/etc.) ) has a use in providing

[0051] Figure 2 is a diagram 200 showing a three-dimensional model used to render a virtual environment with avatars for a video conference. Much like that shown in FIG. 1, the virtual environment here includes a 3D arena 118 and various 3D models, including 3D models 114 and 122 . Also as shown in FIG. 1, diagram 200 includes avatars 102A and 102B navigating around a virtual environment.

[0052] As mentioned above, the interface 100 in FIG. 1 is rendered from the perspective of a virtual camera. That virtual camera is shown as virtual camera 204 in diagram 200 . As previously mentioned, the user viewing interface 100 in FIG. 1 can control the virtual camera 204 and navigate the virtual camera in three-dimensional space. The interface 100 is consistently updated according to the new position of the virtual camera 204 and any changes in the model within the shadow of the virtual camera 204 . As noted above, the field of view of virtual camera 204 may be a frustum defined at least in part by the horizontal field of view and the vertical field of view of the angle of view.

[0053] As described above with respect to Figure 1, a background image or texture may define at least part of the virtual environment. A background image may capture aspects of the virtual environment that are intended to be seen in the distance. A background image may be texture mapped onto the sphere 202 . Virtual camera 204 may be the origin of sphere 202 . In this way, distant features of the virtual environment can be rendered efficiently.

[0054] In other embodiments, other shapes may be used in place of the sphere 202 to texture map to the background image. In various alternative embodiments, the shape may be cylindrical, cubic, rectangular prism, or any other three-dimensional geometry.

[0055] FIG. 3 illustrates a system 300 for providing videoconferencing in a virtual environment. System 300 includes a server 302 coupled via one or more networks 304 to devices 306A and 306B.

[0056] Server 302 provides a service to connect a videoconference session between devices 306A and 306B. As described in more detail below, server 302 communicates notifications to conference participant devices (e.g., devices 306A and 306B) when new participants join the conference and when existing participants leave the conference. do. The server 302 communicates messages describing the position and orientation in the three-dimensional virtual space of each participant's virtual camera in the three-dimensional virtual space. Server 302 also communicates video and audio streams between each of the participant's devices (eg, devices 306A and 306B). Finally, server 302 stores and transmits data describing data specifying the three-dimensional virtual space to each of devices 306A and 306B.

[0057] In addition to the data necessary for a virtual conference, server 302 may provide executable information that instructs devices 306A and 306B on how to render the data to provide an interactive conference.

[0058] The server 302 responds to the request with a response. Server 302 may be a web server. A web server is software and hardware that responds to client requests made over the World Wide Web using HTTP (Hypertext Transfer Protocol) and other protocols. A web server's primary job is to display web site content through storing, processing, and serving web pages to users.

[0059] In an alternative embodiment, communication between devices 306A and 306B does not occur through server 302, but on a peer-to-peer basis. In that embodiment, data describing the location and direction of each participant, notifications about new and existing participants, and one or more of each participant's video and audio streams are transmitted through server 302 rather than through server 302. , are communicated directly between devices 306A and 306B.

Network 304 enables communication between various devices 306 A and 306 B and server 302 . Network 304 can include ad hoc networks, intranets, extranets, virtual facility networks (VPNs), local area networks (LANs), wireless LANs (WLANs), wide area networks (WANs), wireless wide area networks (WWANs), metropolitan area networks. (MAN), part of the Internet, part of the Public Switched Telephone Network (PSTN), cellular telephone network, wireless network, WiFi network, WiMax network, any other type of network, or two or more of such. any combination of suitable networks.

[0061] Devices 306A and 306B are each device of each participant in the virtual conference. Devices 306A and 306B each receive the data necessary to conduct the virtual conference and render the data necessary to provide the virtual conference. As described in more detail below, devices 306A and 306B include displays for presenting rendered conference information, inputs for allowing users to control virtual cameras, and speakers for providing audio to users for conferences. (such as a headset), a microphone for capturing the user's voice input, and a camera positioned to capture video of the user's face.

[0062] Devices 306A and 306B may be any type of computing device, including laptop, desktop, smartphone or tablet computers, or wearable computers (such as smartwatches or augmented reality or virtual reality headsets). can be done.

[0063] Web browsers 308A and 308B can retrieve network resources (such as web pages) addressed by link identifiers (eg, uniform resource locators or URLs) and present the network resources for display. In particular, web browsers 308A and 308B are software applications for accessing information on the World Wide Web. Web browsers 308A and 308B typically make this request using the Hypertext Transfer Protocol (HTTP or HTTPS). When a user requests a web page from a particular website, the web browser retrieves the required content from the web server, interprets and executes the content, and then the device shown as client/partner conferencing applications 308A and 308B. The page is displayed on the displays at 306A and 306B. In an example, the content can include client-side script such as HTML and JavaScript. Once displayed, the user can enter information and make selections on the page, which causes web browsers 308A and 308B to make further requests.

[0064] Conferencing applications 310A and 310B may be web applications downloaded from server 302 and configured to be executed by respective web browsers 308A and 308B. In one embodiment, conferencing applications 310A and 310B may be JavaScript applications. In one example, conferencing applications 310A and 310B can be written in a high level language such as the Typescript language and translated or compiled into JavaScript. Conferencing applications 310A and 310B are configured to interact with the WebGL JavaScript application programming interface. It may have control code specified in JavaScript and shader code written in OpenGL ES Shading Language (GLSL ES). Using the WebGL API, conferencing applications 310A and 310B may be able to take advantage of the graphics processing units (not shown) of devices 306A and 306B. Additionally, OpenGL rendering of interactive 2D and 3D graphics without plugins.

[0065] Conferencing applications 310A and 310B receive from server 302 data describing the positions and orientations of other avatars and three-dimensional modeling information describing the virtual environment. Additionally, conferencing applications 310A and 310B receive video and audio streams of other conference participants from server 302 .

[0066] Conferencing applications 310A and 310B render three three-dimensional modeling data, including data describing the three-dimensional environment and data representing each participant avatar. This rendering may involve rasterization techniques, texture mapping techniques, ray tracing techniques, shading techniques, or other rendering techniques. In one embodiment, rendering may involve ray tracing based on virtual camera properties. Ray tracing involves generating an image by tracing a light path as pixels in an image plane, simulating the effect of encountering a virtual object. In some embodiments, ray tracing may simulate optical effects such as reflection, refraction, scattering, and diffusion to enhance realism.

[0067] In this manner, a user may enter a virtual space using web browsers 308A and 308B. The scene is displayed on the user's screen. The user's webcam video stream and Micron audio stream are sent to server 302 . As other users enter the virtual space, avatar models of those users are created. This avatar position is sent to the server and received by other users. Other users also get notifications from server 302 that audio/video streams are available. A user's video stream is placed in an avatar created for that user. The audio stream is played as coming from the avatar's position.

[0068] Figures 4A-4C illustrate how data is transferred between the various components of the system in Figure 3 to provide a video conference. Like FIG. 3, each of FIGS. 4A-4C show connections between server 302 and devices 306A and 306B. In particular, Figures 4A-4C show example data flow between these devices.

[0069] Figure 4A shows a diagram 400 illustrating how server 302 sends data describing a virtual environment to devices 306A and 306B. In particular, both devices 306A and 306B receive 3D arena 404, background textures 402, spatial hierarchy 408, and any other 3D modeling information 406 from server 302. FIG.

[0070] As mentioned above, the background texture 402 is an image that represents a distinct feature of the virtual environment. Images can be regular (such as a brick wall) or irregular. Background texture 402 may be encoded in any common image file format such as bitmap, JPEG, GIF, or other file image format. For example, describe a background image rendered against a distant sphere.

[0071] Three-dimensional arena 404 is a three-dimensional model of the space in which the conference takes place. As noted above, the 3D arena 404 may include, for example, a mesh and possibly its own texture information that maps to the 3D primitives it describes. The virtual camera and each avatar may define a space that can be navigated within the virtual environment. Thus, it may be delimited by edges (such as walls or fences) that indicate to the user the perimeter of the navigable virtual environment.

[0072] Spatial hierarchy 408 is data that specifies a parcel in a virtual environment. These participants are used to specify how the audio is processed before being transferred between participants. As described below, this segment data may have a hierarchy and may describe the audio treatments that are allowed in areas where participants in the virtual conference may have private conversations or chit-chat.

[0073] Three-dimensional model 406 is any other three-dimensional modeling information needed to conduct the conference. In one embodiment, this may include information describing each avatar. Alternatively or additionally, this information may include product demos.

[0074] Once the information necessary to conduct the conference has been sent to the participants, Figures 4B and 4C illustrate how the server 302 transfers the information between devices. 4B shows a diagram 420 showing how server 302 receives information from each device 306A and 306B, and FIG. 4C shows a diagram 420 showing how server 302 sends information to each device 306B and 306A. show. In particular, device 306A sends position and orientation 422A, video stream 424A, and audio stream 426A to server 302, and server 302 sends position and orientation 422A, video stream 424A, and audio stream 426A to device 306B. Device 306B then sends position and orientation 422B, video stream 424B, and audio stream 426B to server 302, and server 302 sends position and orientation 422B, video stream 424B, and audio stream 426B to device 306A.

[0075] Position and orientation 422A and 422B describe the position and orientation of the virtual camera of the user using device 306A. As mentioned above, the position can be a coordinate in three-dimensional space (eg, x, y, z coordinates) and the direction can be a direction in three-dimensional space (eg, pan, tilt, roll). In some embodiments, the user may not be able to control the roll of the virtual camera, so the directions may specify pan and tilt angles only. Similarly, in some embodiments, the user may not be able to control the avatar's z-coordinate (because the avatar is constrained by virtual gravity), so the z-coordinate may be unnecessary. Thus, position and orientation 422A and 422B may each include at least horizontal coordinates and pan and tilt values in three-dimensional virtual space. Alternatively or additionally, the user may be able to "jump" the avatar, so the Z position may only be specified by an indication of whether the user is making the avatar jump.

[0076] In different examples, location and orientation 422A and 422B may be sent and received using HTTP request responses or using socket messaging.

[0077] Video streams 424A and 424B are video data captured from the camera of each device 306A and 306B. Video can be compressed. For example, video may be MPEG-4, VP8, or H.264. Any commonly known video codec may be used, including H.264. Video can be captured and transmitted in real time.

[0078] Similarly, audio streams 426A and 426B are audio data captured from the microphones of each device. Audio can be compressed. For example, video can use any commonly known audio codec, including MPEG-4 or vorbis. Audio can be captured and transmitted in real time. Video stream 424A and audio stream 426A are captured, transmitted and presented synchronously with each other. Similarly, video stream 424B and audio stream 426B are also captured, transmitted and presented synchronously with each other.

[0079] Video streams 424A and 424B and audio streams 426A and 426B may be transmitted using the WebRTC application programming interface. WebRTC is an API that can be used with JavaScript. As described above, devices 306A and 306B download and execute web applications as conference applications 310A and 310B, which may be implemented in JavaScript. Conferencing applications 310A and 310B may use WebRTC to receive and send video streams 424A and 424B and audio streams 426A and 426B by making API calls from JavaScript.

[0080] As alluded to earlier, when a user leaves a virtual conference, this departure is communicated to all other users. For example, if device 306A leaves the virtual conference, server 302 communicates its departure to device 306B. As a result, device 306B stops rendering the avatar corresponding to device 306A and removes the avatar from the virtual space. Additionally, device 306B stops receiving video stream 424A and audio stream 426A.

[0081] As described above, conferencing applications 310A and 310B periodically navigate the virtual space based on new information from each video stream 424A and 424B, position and orientation 422A and 422B, and new information related to the three-dimensional environment. Or it can re-render intermittently. For brevity, each of these updates is described here from the perspective of device 306A. However, those skilled in the art will appreciate that device 306B behaves similarly, given similar modifications.

[0082] When device 306A receives video stream 424B, it texture maps frames from video stream 424A onto an avatar corresponding to device 306B. The texture-mapped avatar is re-rendered in the three-dimensional virtual space and presented to the user of device 306A.

[0083] When device 306A receives the new location and orientation 422B, it generates an avatar corresponding to device 306B located at the new location and facing the new orientation. The generated avatar is re-rendered within the three-dimensional virtual space and presented to the user of device 306A.

[0084] In some embodiments, the server 302 may transmit updated model information describing the three-dimensional virtual environment. For example, server 302 may send updated information 402 , 404 , 406 , or 408 . When that happens, device 306A re-renders the virtual environment based on the updated information. This can be useful if the environment changes over time. For example, an outdoor event may change from day to dusk as the event progresses.

[0085] Again, when device 306B leaves the virtual meeting, server 302 sends a notification to device 306A indicating that device 306B is no longer in the meeting. Device 306A then re-renders the virtual environment without the avatar of device 306B.

[0086] Although FIG. 3 in FIGS. 4A-4C is shown with two devices for simplicity, those skilled in the art will appreciate that the techniques described herein can be extended to any number of devices. will understand. Also, while FIG. 3 in FIGS. 4A-4C shows a single server 302, those skilled in the art will appreciate that the functionality of server 302 can be distributed across multiple computing devices. In one embodiment, the data transferred in FIG. 4A may be from one network address of server 302, while the data transferred in FIGS. Can be forwarded from a network address.

[0087] In one embodiment, participants may configure webcams, microphones, speakers, and graphical settings prior to entering a virtual meeting. In an alternative embodiment, after starting the application, the user may enter a virtual lobby, where they are greeted by an avatar controlled by a real person. This person can view and change the user's webcam, microphone, speakers, and graphical settings. Attendants can also instruct users on how to use the virtual environment, for example, by teaching them to see, move around, and interact. When the user is ready, the user automatically exits the virtual waiting room and joins the real virtual environment.

Volume control in video conferencing in virtual environment
[0088] Embodiments also adjust the volume to provide a sense of position and space within the virtual meeting. This is illustrated, for example, in FIGS. 5-7, 8A, 8B, and 9A-9C, each of which is described below.

[0089] FIG. 5 is a flowchart illustrating a method 500 for adjusting relative left and right volumes to provide a sense of position in a virtual environment during a video conference.

[0090] At step 502, the volume is adjusted based on the distance between the avatars. As described above, an audio stream is received from the microphone of another user's device. Volumes of both the first and second audio streams are adjusted based on the distance between the first position and the second position. This is shown in FIG.

[0091] Figure 6 shows a chart 600 showing how the volume decreases as the distance between the avatars increases. Chart 600 shows volume 602 on the x-axis and y-axis. As the distance between users increases, the volume remains constant until a reference distance 602 is reached. When the reference distance 602 is reached, the volume begins to fall. In this way, all other things being equal, closer users are louder than distant users.

[0092] The rate at which the speech drops depends on the drop factor. This may be a factor built into the settings of the videoconferencing system or client device. As shown by lines 608 and 610, a larger drop factor will drop the volume more quickly than a smaller one.

[0093] Returning to Figure 5, at step 504, the relative left and right audio is adjusted based on the direction in which the avatar is positioned. That is, the volume of the audio output by the user's speakers (eg, headset) is altered to provide a sense of where the speaking user's avatar is located. The relative volume of the left and right audio streams is determined by the location of the user generating the audio stream (e.g. speaking) relative to the location of the user receiving the audio (e.g. the location of the virtual camera). user's avatar's location) orientation. The position can be on a horizontal plane in the three-dimensional virtual space. The relative volume of the left and right audio is streamed to provide a sense of where the second location is in the three-dimensional virtual space relative to the first location.

[0094] For example, in step 504, the audio corresponding to the left avatar of the virtual camera is adjusted such that the audio is output louder in the left ear than in the right ear of the receiving user. Similarly, the audio corresponding to the avatar on the right side of the virtual camera is adjusted such that the audio is output louder in the receiving user's right ear than in the left ear.

[0095] At step 506, the relative left and right audio is adjusted based on the direction one avatar is facing relative to the other avatar. The relative volume of the left and right audio streams is based on the angle between the direction the virtual camera is facing and the direction the avatar is facing: the more vertical the angle, the greater the volume difference between the left and right audio streams. adjusted to tend to

[0096] For example, if the avatar is directly facing the virtual camera, the relative left and right volume of the avatar corresponding to the audio stream may not be adjusted at step 506 at all. If the avatar faces the left side of the virtual camera, the relative left and right volume of the avatar corresponding to the audio stream may be adjusted so that the left is louder than the right. And if the avatar faces the right side of the virtual camera, the relative left and right volume of the avatar corresponding to the audio stream can be adjusted so that the right is louder than the left.

[0097] In one example, the calculation in step 506 may include taking the cross product of the angle the virtual camera is facing and the angle the avatar is facing. The angle can be the direction facing on the horizontal plane.

[0098] In one embodiment, a check may be made to identify the audio output device that the user is using. If the audio output device is not a set of headphones or another type of speaker that provides a stereo effect, the adjustments in steps 504 and 506 may not be made.

[0099] Steps 502-506 are repeated for every audio stream received from every other participant. Left and right audio gains at every other participant are calculated based on the calculations in steps 502-506.

[0100] In this way, each participant's audio stream is adjusted to provide a sense of where the participant's avatar was placed in the three-dimensional virtual environment.

[0101] Not only are the audio streams tailored to provide a sense of where the avatar is located, but in certain embodiments the audio streams are also adjusted to provide a private or semi-private volume area. can be adjusted. In this manner, the virtual environment allows users to conduct private conversations. The virtual environment also allows users to interact with each other and have personalized chats not possible with traditional videoconferencing software. This is illustrated, for example, with respect to FIG.

[0102] Figure 7 is a flowchart illustrating a method 700 for adjusting relative volume to provide different volume areas in a virtual environment during a video conference.

[0103] As noted above, the server may provide specifications of the audio or volume area to the client device. The virtual environment can be partitioned into different volume areas. At step 702, the device identifies in which audio area each avatar and virtual camera is located.

[0104] For example, Figures 8A and 8B are diagrams illustrating different volume areas in a virtual environment during a video conference. FIG. 8A shows a diagram 800 with a volume area 802 that allows for semi-private conversations or chats between the user controlling the avatar 806 and the user controlling the virtual camera. In this way, users around the conference table 810 can converse without disturbing others in the room. Audio from the user controlling the virtual camera avatar 806 may be attenuated when exiting the volume area 802, but not completely. Passers-by can then join the conference if they wish to participate.

[0105] Interface 800 also includes buttons 804, 806, and 808, which are described below.

[0106] Figure 8B shows a diagram 800 with a volume area 804 that allows private conversation between the user controlling the avatar 808 and the user controlling the virtual camera. Once within the volume area 804 , only audio from the user controlling the avatar 808 and the user controlling the virtual camera may be output to the user within the volume area 804 . If no audio is played from those users to other users in the conference, their audio streams may not even be sent to other user devices.

[0107] The volume space may be layered as shown in Figures 9A and 9B. FIG. 9B shows a layout with different volume areas arranged in a hierarchy. Volume areas 934 and 935 are within volume area 933 and volume areas 933 and 932 are within volume area 931 . These volume areas are represented in a hierarchical tree, as shown in diagrams 900 and 9A.

[0108] In diagram 900, node 901 represents volume area 931 and is the root of the tree. Nodes 902 and 903 are children of node 901 and represent volume areas 932 and 933 . Nodes 904 and 906 are children of node 903 and represent volume areas 934 and 935 .

[0109] If a user located in area 934 is trying to hear a speaking user located in area 932, the audio stream passes through several different virtual "walls" that each attenuate the audio stream. have to pass. Specifically, the sound needs to pass through the walls in area 932, the walls in area 933, and the walls in area 934. Each wall attenuates by a specific factor. This calculation is described with respect to steps 704 and 706 of FIG.

[0110] At step 704, the hierarchy is traversed to identify various audio areas between the avatars. This is shown, for example, in FIG. 9C. Starting from the node corresponding to the virtual area of speech (in this case node 904), a path is identified to the receiving user's node (in this case node 902). To identify the path, links 952 between nodes are identified. In this way a subset of areas between the area containing the avatar and the area containing the virtual camera is identified.

[0111] At step 706, the audio stream from the speaking user is attenuated based on each wall transmission coefficient of the subset of areas. Each wall transfer coefficient specifies the amount by which the audio stream is attenuated.

[0112] Additionally or alternatively, different areas may then have different reduction factors, and the distance-based calculations shown in method 600 may be applied to individual areas based on each reduction factor. In this way, different areas of the virtual environment emit sounds at different rates. The audio gain specified in the method described above with respect to FIG. 5 may be applied to the audio stream to specify left and right audio accordingly. In this way, both the wall transfer factor, the drop factor, and the left/right adjustment for providing directional spacing to the sound can be applied together to provide a comprehensive audio experience.

[0113] Different audio areas may have different functions. For example, the volume area can be the podium area. If the user is located in the podium area, some or all of the attenuation described with respect to Figures 5 or 7 may not occur. For example, no damping may occur due to a drop factor or wall transmission factor. In some embodiments, relative left and right audio may still be adjusted to provide a sense of direction.

[0114] For purposes of illustration, the methods described with respect to Figures 5 and 7 describe audio streams from users with corresponding avatars. However, it is also possible to apply the same method to other audio sources than avatars. For example, the virtual environment may have a 3D model of a loudspeaker. Sound can be emitted from speakers similar to the avatar model described above, either through presentation or simply to provide background music.

[0115] As alluded to earlier, the wall transmission coefficient may be used to globally isolate the audio. In one embodiment, this can be used to create a virtual office. In one example, each user may have a monitor in their physical (perhaps home) office that is always on and displays a conferencing application logged into the virtual office. There may be features that allow users to indicate if they are in the office and are not allowed to enter. If the no entry indicator is off, a colleague or manager can wander into the virtual space and knock or enter just like in a physical office. A visitor may be able to leave a note if the worker is not in the office. When the worker returns, he can read the notes left by the visitor. A virtual office may have a whiteboard and/or an interface that displays messages to the user. The message may be email and/or from a messaging application such as the SLACK application available from Slack Technologies, Inc. of San Francisco, CA.

[0116] Users may be able to customize or personalize their virtual office. For example, the user may be able to display models of posters or other wall decorations. A user may be able to change the model or orientation of an ornament, such as a desk or a plant. The user may be able to change the lighting or the view from the window.

[0117] Returning to FIG. When the user presses button 804, the attenuation described above with respect to the methods of FIGS. 5 and 7 can occur, or can occur only to a lesser extent. In that situation, the user's voice is output uniformly to the other users, allowing the user to speak to all participants in the meeting. User videos can similarly be output to presentation screens within the virtual environment, as described below. When the user presses button 806, speaker mode is activated. In that case, audio is output from an audio source within the virtual environment, such as to play background music. When the user presses button 808, screen sharing may be enabled, allowing the user to share the contents of the screen or window on their device with other users. Content can be presented in a presentation model. This is also discussed below.

Presentation in a 3D environment
[0118] Figure 10 shows an interface 1000 with a three-dimensional model 1004 in a three-dimensional virtual environment. As described above with respect to FIG. 1, interface 1000 may be displayed to a user who can navigate around the virtual environment. As shown in interface 1000 , the virtual environment includes avatar 1004 and three-dimensional model 1002 .

[0119] The three-dimensional model 1002 is a 3D model of the product placed inside the virtual space. People can join this virtual space, observe the model, and walk around it. Products may have localized audio to enhance the experience.

[0120] More specifically, if the presenter in the virtual space wishes to show a 3D model, the user selects the desired model from the interface. It sends a message to the server updating the details (including the name and path of the model). This is automatically communicated to the client. In this way, the three-dimensional model can be rendered for display concurrently with the presentation of the video stream. A user can navigate a virtual camera around the three-dimensional model of the product.

[0121] In different examples, the object may be a product demo or an advertisement for the product.

[0122] Figure 11 shows an interface 1100 with presentation screen sharing in a three-dimensional virtual environment used in a video conference. As described above with respect to FIG. 1, interface 1100 may be displayed to a user who can navigate around the virtual environment. As shown in interface 1100 , the virtual environment includes avatar 1104 and presentation screen 1106 .

[0123] In this embodiment, a presentation stream is received from the devices of the participants in the conference. The presentation stream is texture mapped onto a 3D model of the presentation screen 1106 . In one embodiment, the presentation stream may be a video stream from the camera of the user's device. In another embodiment, the presentation stream may be a screen share from the user's device, where a monitor or window is shared. The presentation video and audio stream can also be from an external source, such as a live stream of an event, through screen sharing or other methods. When a user activates presentation mode, that user's presentation stream (and audio stream) is published to the server tagged with the name of the screen the user wishes to use. Other clients are notified that new streams are available.

[0124] The presenter may also be able to control the location and orientation of the audience members. For example, the presenter may have the option selected to rearrange all other participants in the meeting so that they are positioned and oriented to face the presentation screen.

[0125] An audio stream is captured from the microphone of the first participant's device in sync with the presentation stream. The audio stream from the user's microphone can be heard by other users as from presentation screen 1106 . In this way, presentation screen 1106 can be a sound source as described above. Since the user's audio stream is projected from the presentation screen 1106, that from the user's avatar may be suppressed. In this way, the audio stream is output and played in synchronism with the display of the presentation stream on screen 1106 in the three-dimensional virtual space.

Bandwidth allocation based on distance between users
[0126] Figure 12 is a flowchart illustrating a method 1200 for allocating available bandwidth based on the relative position of avatars within a three-dimensional virtual environment.

[0127] At step 1202, the distance between the first user and the second user in the virtual conference space is determined. The distance may be the distance between users on a horizontal plane in three-dimensional space.

[0128] At step 1204, the received video streams are prioritized such that those from closer users are given higher priority than video streams from farther users. Priority values may be specified as shown in FIG.

[0129] Figure 13 shows a chart 1300 showing priority 1306 and distance 1302 on the y-axis. A priority state that remains constant until a reference distance 1304 is reached, as indicated by line 1306 . After reaching the reference distance, the priority starts to drop.

[0130] At step 1206, the available bandwidth to the user device is divided among the various video streams. This may be done based on the priority value identified in step 1204 . For example, the priorities may be adjusted proportionally so that they all add up to one. Any video with insufficient available bandwidth may have a relative priority of zero. The priorities are then adjusted again for the remaining video streams. Bandwidth is allocated based on these relative priority values. Additionally, bandwidth may be reserved for audio streams. This is shown in FIG.

[0131] Figure 14 shows a chart 1400 with a y-axis representing bandwidth 1406 and an x-axis representing relative priority. After the minimum bandwidth 1406 has been allocated to a video to be effective, the bandwidth 1406 allocated to a video stream increases in proportion to its relative priority.

[0132] Once the allocated bandwidth is identified, the client may request the video from the server at the selected and allocated bandwidth/bitrate/framerate/resolution for that video. This may initiate a negotiation process between the client and server to begin streaming video over the specified bandwidth. In this way, the available video and audio bandwidth is split fairly among all users, with users with twice the priority getting twice as much bandwidth.

[0133] In one possible implementation, using simulcast, all clients send multiple video streams at different bitrates and resolutions to the server. Other clients can then indicate to the server one of these streams that the client is interested in and would like to receive.

[0134] At step 1208, whether the available bandwidth between the first user and the second user in the virtual conference space is such that displaying video at that distance is inefficient. is specified. This identification can be done by either the client or the server. When done by the client, the client sends a message for the server to stop sending video to the client. If so, transmission of the video stream to the second user's device is discontinued and the second user's device is notified to replace the video stream with a still image. The still image may simply be the last video frame (or one of the last video frames) received.

[0135] In one embodiment, a similar process may be performed for audio to reduce the quality given the size of the portion reserved for the audio. In another embodiment, each audio stream is given a fixed bandwidth.

[0136] In this manner, embodiments can increase performance for all users and servers, and reduce the quality of video and audio streams for remote and/or less important users. This is not done if sufficient bandwidth budget is available. The reduction is done in both bitrate and resolution. This improves the quality of the video as the available bandwidth for that user can be utilized more efficiently by the encoder.

[0137] Independently, the video resolution is reduced based on distance, with users twice as far away having half the resolution. In this way, unnecessary resolutions may not be downloaded given the limitations of screen resolution. Bandwidth is thus conserved.

[0138] Figure 15 is a diagram of a system 1500 showing components of a device used to provide videoconferencing within a virtual environment. In various embodiments, system 1500 can operate according to the methods described above.

[0139] Device 306A is a user computing device. Device 306A can be a desktop or laptop computer, smart phone, tablet, or wearable (eg, watch or head-mounted display). Device 306 A includes microphone 1502 , camera 1504 , stereo speakers 1506 and input device 1512 . Although not shown, device 306A also includes a processor and permanent temporary nonvolatile memory. A processor may include one or more central processing units, graphics processing units, or any combination thereof.

[0140] The microphone 1502 converts sound into an electrical signal. Microphone 1502 is positioned to capture speech of the user of device 306A. In different examples, the microphone 1502 can be a condenser microphone, an electret microphone, a moving coil microphone, a ribbon microphone, a carbon microphone, a piezoelectric microphone, a fiber optic microphone, a laser microphone, a water microphone, or a MEMS microphone.

[0141] Camera 1504 typically captures image data by capturing light through one or more lenses. Camera 1504 is positioned to capture a photographic image of the user of device 306A. Camera 1504 includes an image sensor (not shown). The image sensor can be, for example, a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor. An image sensor may include one or more photodetectors that detect light and convert it into electrical signals. These electrical signals captured together within a similar time frame constitute a still photographic image. A series of still photographic images captured together at regular intervals constitutes a video. In this manner, camera 1504 captures images and video.

[0142] Stereo speakers 1506 are devices that convert electrical audio signals into corresponding left and right sounds. Stereo speakers 1506 output left and right audio streams generated by audio processor 1520 (below) and played in stereo to a user of device 306A. Stereo speakers 1506 include both ambient speakers and headphones designed to play sound directly to the left and right ears of the user. Examples of loudspeakers include moving iron loudspeakers, piezoelectric loudspeakers, magnetostatic loudspeakers, electrostatic loudspeakers, ribbon and planar magnetic loudspeakers, bending wave loudspeakers, flat panel loudspeakers, Heil air motion transducers, transparent ion conduction loudspeakers, There are plasma arc speakers, thermoacoustic speakers, rotary woofers, moving coils, electrostatics, electrets, planar magnetics, and balanced armchairs.

[0143] Network interface 1508 is a software or hardware interface between two devices or between two protocol layers in a computer network. Network interface 1508 receives the video stream of each meeting participant from server 302 . A video stream is captured from the camera of another participant's device to the videoconference. Network interface 1508 also received data from server 302 specifying the three-dimensional virtual space and any models therein. For each other participant, network interface 1508 receives the position and orientation in the three-dimensional virtual space. Position and orientation are entered by each of the other participants.

Network interface 1508 also transmits data to server 302 . Network interface 1508 transmits the position of the user's virtual camera of device 306 A used by renderer 1518 and transmits video and audio streams from camera 1504 and microphone 1502 .

[0145] Display 1510 is an output device for presenting electronic information in visual or tactile form (tactile forms are used, for example, in tactile electronic displays for visually impaired people). The display 1510 can be a television set, computer monitor, head-mounted display, heads-up display, augmented or virtual reality headset output, broadcast reference monitor, medical monitor, mobile display (for mobile devices), smart phone display (for smart phones). ). For presenting information, the display 1510 may be an electroluminescent (ELD) display, a liquid crystal display (LCD), a light emitting diode (LED) backlight LCD, a thin film transistor (TFT) LCD, a light emitting diode (LED) display, an OLED display, an AMOLED display. Display may include plasma (PDP) display, quantum dot (QLED) display.

[0146] Input devices 1512 are equipment used to provide data and control signals to an information processing system, such as a computer or information appliance. Input device 1512 allows a user to enter a new desired position for the virtual camera used by renderer 1518, thereby allowing navigation in a three-dimensional environment. Examples of input devices include keyboards, mice, scanners, joysticks, and touch screens.

[0147] Web browser 308A and web application 310A are described above with respect to FIG. Web application 310 A includes screen capture 1514 , texture mapper 1516 , renderer 1518 and audio processor 1520 .

[0148] Screen capture 1514 captures a presentation stream, specifically a screen share. Screen capture 1514 may interact with APIs provided by web browser 308A. By calling a function available from the API, screen capture 1514 may cause web browser 308A to ask the user which window or screen they would like to share. Based on the answer to that query, web browser 308A may return a video stream corresponding to screen sharing to screen capture 1514, which is sent to server 302 and ultimately to other participants' devices. Pass it to network interface 1508 for transmission.

[0149] Texture mapper 1516 texture maps the video stream onto a three-dimensional model corresponding to the avatar. A texture mapper 1516 may texture map each frame from the video to the avatar. Additionally, texture mapper 1516 may texture map the presentation stream to a three-dimensional model of the presentation screen.

[0150] Renderer 1518 renders the texture-mapped 3D image of each participant's avatar positioned and oriented at the corresponding position received for output to display 1510, from the perspective of the virtual camera of the user of device 306A. Render a 3D virtual space containing the original model. Renderer 1518 also renders any other three-dimensional model, including presentation screens, for example.

[0151] Audio processor 1520 adjusts the volume of the received audio streams to identify the left audio stream and the right audio stream of where the second position is in the three-dimensional virtual space relative to the first position. provide sensation. In one embodiment, audio processor 1520 adjusts the volume based on the distance between the second location relative to the first location. In another embodiment, audio processor 1520 adjusts the volume based on the orientation of the second position relative to the first position. In yet another embodiment, the audio processor 1520 adjusts the volume based on the orientation of the second position relative to the first position on a horizontal plane within the three-dimensional virtual space. In yet another embodiment, the audio processor 1520 tends to have the left audio stream have a louder volume when the avatar is positioned to the left of the virtual camera, and when the avatar is positioned to the right of the virtual camera. Adjust the volume based on the direction the virtual camera is facing in the 3D virtual space so that the right audio stream tends to have a higher volume. Finally, in yet another embodiment, the audio processor 1520 uses the angle between the direction the virtual camera faces and the direction the avatar faces, the more perpendicular the angle is the direction the avatar faces, left and right. Adjust the volume so that the warmth between the audio streams tends to be greater.

[0152] Audio processor 1520 may also adjust the volume of the audio stream based on the area where the speakers are located relative to the area where the virtual camera is located. In this embodiment, the three-dimensional virtual space is segmented into multiple areas. These areas may have a hierarchy. A wall transmission factor may be applied to attenuate the volume of the speech audio stream if the speaker and virtual camera are placed in different areas.

Server 302 includes attendee notifier 1522 , stream coordinator 1524 , and stream forwarder 1526 .

[0154] Attendee notifier 1522 notifies meeting participants when they join and leave the meeting. When a new participant joins the meeting, attendee notifier 1522 sends a message to the devices of other participants in the meeting indicating that the new participant has joined. Attendee notifier 1522 signals stream forwarder 1526 to begin forwarding video, audio, and position/orientation information to other participants.

[0155] Stream conditioner 1524 receives the video stream captured from the camera of the first user's device. Stream adjuster 1524 identifies available bandwidth for transmitting virtual conference data to the second user. Stream adjuster 1524 identifies the distance between the first user and the second user in the virtual conference space. Stream adjuster 1524 then distributes the available bandwidth between the first video stream and the second video stream based on relative distance. In this way, stream adjuster 1524 gives higher priority to video streams of near users over video streams from farther users. Additionally or alternatively, stream conditioner 1524 may be located on device 306A, perhaps as part of web application 310A.

[0156] Stream forwarder 1526 broadcasts the received position/orientation information, video, audio, and screen sharing screens (with adjustments made by stream adjuster 1524). Stream forwarder 1526 may send information to device 306A in response to a request from conferencing application 310A. Conferencing application 310A may send the request in response to notification from attendee notifier 1522 .

[0157] Network interface 1528 is a software or hardware interface between two devices or between two protocol layers in a computer network. Network interface 1528 transmits the model information to various participant devices. Network interface 1528 receives video, audio, and screen sharing screens from various participants.

[0158] Screen Capture 1514, Texture Mapper 1516, Renderer 1518, Audio Processor 1520, Attendee Notifier 1522, Stream Coordinator 1524, and Stream Forwarder 1526 are each implemented in hardware, software, firmware, or any combination thereof. can be implemented in

[0159] Identifiers such as "(a)", "(b)", "(i)", "(ii)", etc. may be used for different elements or steps. These identifiers are used for clarity and do not necessarily indicate the ordering of the elements or steps.

[0160] The present invention has been described above in terms of functional building blocks that illustrate the implementation of specified functions and their relationships. The boundaries of these functional building blocks have been arbitrarily defined herein for convenience of explanation. Alternate boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed.

[0161] The above description of specific embodiments is intended to enable others, by applying knowledge within the skill in the art, to implement specific implementations without undue experimentation and without departing from the general concepts of the invention. The general nature of the invention will be sufficiently clear that it can be readily modified and/or adapted for various uses, such as configuration. Therefore, such adaptations and modifications are intended to be within the meaning and range of equivalents of the disclosed embodiments, based on the teaching and guidance presented herein. It is to be understood that the phraseology and terminology used herein are for the purpose of description rather than limitation, and are, therefore, to be interpreted by those skilled in the art in light of the teachings and guidance.

[0162] The breadth and scope of the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

Claims (24)

A system for enabling video conferencing between a first user and a second user, comprising:
a processor coupled to the memory;
a display screen;
a network interface, comprising: (i) data specifying a three-dimensional virtual space; (ii) a position and orientation in said three-dimensional virtual space, said position and orientation being input by said first user; A network interface configured to receive a video stream captured from a camera of a first user's device, said camera positioned to capture a photographic image of said first user. and,
a web browser implemented on the processor, the web browser configured to download a web application from a server and execute the web application;
with
The web application is
a mapper configured to map the video stream onto a three-dimensional model of an avatar;
a renderer and
including
The renderer comprises the avatar to which the video stream is mapped, positioned at the location and oriented for display to the second user, from the perspective of the second user's virtual camera. A system configured to render said three-dimensional virtual space including a three-dimensional model. said device further comprising a graphics processing unit, said mapper and said renderer comprising WebGL application calls that enable said web application to map or render using said graphics processing unit;
The system of claim 1. A computer-implemented method for enabling a video conference between a first user and a second user, comprising:
transmitting a web application to a first client device of the first user and a second client device of the second user;
From the first client device running the web application, (i) a position and orientation in a three-dimensional virtual space as input by the first user, and (ii) the first receiving a video stream captured from a camera of a client device of said camera positioned to capture a photographic image of said first user;
transmitting the position, the orientation, and the video stream to the second client device of the second user, wherein the web application includes executable instructions;
including
The executable instructions, when executed in a web browser, map the video stream to a three-dimensional model of an avatar and display it to the second user from the perspective of the second user's virtual camera. and rendering the three-dimensional virtual space including the three-dimensional model of the avatar positioned at the location and to which the video stream oriented in the direction is mapped. 4. The method of claim 3, wherein the web application includes WebGL application calls that enable the web application to be mapped or rendered using a graphics processing unit of the second client device. A computer-implemented method for enabling a video conference between a first user and a second user, comprising:
receiving data specifying a three-dimensional virtual space;
receiving a position and orientation in said three-dimensional virtual space, said position and said orientation being input by said first user;
receiving a captured video stream from a camera of the first user's device, the camera positioned to capture a photographic image of the first user;
mapping the video stream onto a three-dimensional model of an avatar by a web application running in a web browser;
of the avatar positioned at the position and facing the direction for display to the second user by the web application running on the web browser, from the perspective of the second user's virtual camera; rendering the three-dimensional virtual space containing a three-dimensional model;
A method, including receiving an audio stream captured synchronously with the video stream from a microphone of the device of the first user, the microphone positioned to capture speech of the first user. , receiving and
outputting the audio stream for playback to the second user in synchronization with display of the video stream in the three-dimensional virtual space;
6. The method of claim 5, further comprising: if input is received from the second user indicating a desire to change the viewpoint of the virtual camera;
changing the viewpoint of the virtual camera of the second user;
re-rendering the three-dimensional virtual space including the three-dimensional model of the avatar positioned at the location and facing the direction for display to the second user from the altered viewpoint of the virtual camera; and
6. The method of claim 5, further comprising: 8. The method according to claim 7, wherein said viewpoint of said virtual camera is defined by at least coordinates on a horizontal plane and pan and tilt values in said three-dimensional virtual space. when a new position and orientation of the first user in the three-dimensional virtual space is received,
6. The method of claim 5, further comprising re-rendering the three-dimensional virtual space including a three-dimensional model of the avatar positioned at the new location and oriented at the new orientation for display to the second user. the method of. 6. The method of claim 5, wherein said mapping comprises repeatedly mapping pixels to said three-dimensional model of said avatar for each frame of said video stream. 6. The method of claim 5, wherein the data, the position and orientation, and the video stream are received in a web browser from a server, and the mapping and rendering are performed by the web browser. receiving a notification from the server indicating that the first user no longer exists;
re-rendering the three-dimensional virtual space without the three-dimensional model of the avatar for display in the web browser to the second user;
12. The method of claim 11, further comprising: receiving a notification from the server indicating that a third user has entered the three-dimensional virtual space;
receiving a second position and a second orientation of the third user in the three-dimensional virtual space;
receiving a second video stream captured from a camera of the third user's device, the camera positioned to capture a photographic image of the third user; ,
mapping the second video stream onto a second three-dimensional model of a second avatar;
including the second three-dimensional model positioned at the second location and oriented in the second direction for display to the second user from the viewpoint of the virtual camera of the second user; rendering the three-dimensional virtual space;
13. The method of claim 12, further comprising: Receiving data specifying the three-dimensional virtual space includes receiving a mesh specifying a conference space; receiving a background image; and rendering mapping the background onto a sphere. 6. The method of claim 5, comprising: A non-transitory tangible computer readable device storing instructions, the instructions, when executed by at least one computing device, instructing the at least one computing device to communicate between a first user and a second user. to perform an operation to enable video conferencing, said operation comprising:
receiving data specifying a three-dimensional virtual space;
receiving a position and orientation in the three-dimensional virtual space, the position and orientation being input by the first user;
receiving a captured video stream from a camera of the first user's device, the camera positioned to capture a photographic image of the first user;
mapping the video stream onto a three-dimensional model of an avatar;
Rendering, from the perspective of the second user's virtual camera, the three-dimensional virtual space including the three-dimensional model of the avatar positioned at the location and oriented for display to the second user. and
device, including The operation is
receiving an audio stream captured synchronously with the video stream from a microphone of the device of the first user, the microphone positioned to capture speech of the first user. , receiving and
outputting the audio stream for playback to the second user in synchronization with display of the video stream in the three-dimensional virtual space;
16. The device of claim 15, further comprising: The action comprises: receiving input from the second user indicating a desire to change the viewpoint of the virtual camera;
changing the viewpoint of the virtual camera of the second user;
re-rendering the three-dimensional virtual space including the three-dimensional model of the avatar positioned at the location and facing the direction for display to the second user from the altered viewpoint of the virtual camera; and
16. The device of claim 15, further comprising: 18. The device of claim 17, wherein the viewpoint of the virtual camera is defined by at least horizontal coordinates and pan and tilt values in the three-dimensional virtual space. When the action receives a new position and orientation of the first user in the three-dimensional virtual space,
16. The method of claim 15, further comprising re-rendering the three-dimensional virtual space including a three-dimensional model of the avatar positioned at the new location and oriented at the new orientation for display to the second user. Devices listed. 16. The device of claim 15, wherein said mapping comprises repeatedly mapping pixels to said three-dimensional model of said avatar for each frame of said video stream. 16. The device of claim 15, wherein the data, the position and orientation, and the video stream are received at a web browser from a server, and wherein the mapping and rendering is performed by the web browser. The operation is
receiving a notification from the server indicating that the first user no longer exists;
re-rendering the three-dimensional virtual space without the three-dimensional model of the avatar for display in the web browser to the second user;
22. The device of claim 21, further comprising: The operation is
receiving a notification from the server indicating that a third user has entered the three-dimensional virtual space;
receiving a second position and a second orientation of the third user in the three-dimensional virtual space;
receiving a second video stream captured from a camera of the third user's device, the camera positioned to capture a photographic image of the third user; ,
mapping the second video stream onto a second three-dimensional model of a second avatar;
including the second three-dimensional model positioned at the second location and oriented in the second direction for display to the second user from the viewpoint of the virtual camera of the second user; rendering the three-dimensional virtual space;
23. The device of claim 22, further comprising: said receiving data specifying said three-dimensional virtual space includes receiving a mesh specifying a conference space; receiving a background image; said rendering mapping said background onto a sphere; 16. The device of claim 15, comprising:

Images