KR102580110B1 - Web-based video conferencing virtual environment with navigable avatars and its applications - Google Patents

Abstract

Disclosed herein is a web-based video conferencing system that allows video avatars to navigate within a virtual environment. The system has a presentation mode that allows the presentation stream to be texture mapped to a presenter's screen located within the virtual environment. Relative left and right sounds are adjusted to provide a sense of the avatar's position in virtual space. The sound is further adjusted based on the area in which the avatar is located and the area in which the virtual camera is located. Video stream quality is adjusted based on relative position in virtual space. Three-dimensional modeling is available inside a virtual video conferencing environment.

Description

Web-based video conferencing virtual environment with navigable avatars and its applications

[Cross-reference to related applications]

This application is currently based on U.S. Patent No. 10,979,672, issued on April 13, 2021, and U.S. Utility Patent Application No. 17/075,338, filed on October 20, 2020, filed on March 11, 2021. U.S. Utility Patent Application No. 17/198,323, currently issued as U.S. Patent No. 11,095,857, issued on Aug. 17, 2021; U.S. Utility Patent Application No. 17/075,362, filed on October 20, 2020; U.S. patent application Ser. No. 17/075,390, filed Oct. 20, 2020, currently issued as U.S. Patent No. 10,952,006, issued March 16, 2021; U.S. Utility Patent Application No. 17/075,408, filed October 20, 2020, currently filed as U.S. Patent No. 11,070,768, currently filed as U.S. Patent No. 11,076,128, filed July 27, 2021 Priority is claimed on U.S. Utility Patent Application No. 17/075,428, filed Oct. 20, and U.S. Utility Patent Application No. 17/075,454, filed Oct. 20, 2020. The contents of each of these applications are incorporated herein by reference in their entirety.

[Technical field]

This field of technology generally relates to video conferencing.

[Related technology]

Videoconferencing involves the reception and transmission of audio-video signals by users at different locations for communication between people in real time. Video conferencing is widely available on many computing devices from a variety of different services, including the ZOOM service available from Zoom Communications Inc. of San Jose, California. Some video conferencing software, such as the FaceTime application available from Apple Inc. of Cupertino, California, comes standard on mobile devices.

Typically, these applications work by displaying video and outputting audio of other meeting participants. When there are multiple participants, the screen may be divided into multiple rectangular frames, each displaying a participant's video. Sometimes these services work by having larger frames that present a video of the speaker. When different individuals speak, the frame will switch between speakers. The application captures video from a camera integrated with the user's device and audio from a microphone integrated with the user's device. The application then transmits the audio and video to other applications running on other users' devices.

Many of these video conferencing applications have screen sharing capabilities. When a user decides to share their screen (or a portion of their screen), a stream is sent to other users' devices along with the contents of their screen. In some cases, other users can even control what's on your screen. This way, users can collaborate on projects or make presentations to other meeting participants.

Recently, the importance of video conferencing technology has increased. Many workplaces, trade fairs, conferences, conferences, schools, and places of worship have been closed or have encouraged people not to attend for fear of spreading the disease, especially COVID-19. Virtual meetings using videoconferencing technology are increasingly replacing physical meetings. Additionally, this technology offers advantages over physically meeting to avoid travel and commuting.

But often, the use of this video conferencing technology results in a loss of sense of place. There is an experiential aspect of being in the same location and physically meeting in person that is lost when meetings are conducted virtually. There is a social aspect to being able to pose and look at your peers. This feeling of experience is important in creating relationships and social connections. However, this feeling is lacking when it comes to conventional video conferences.

Moreover, when meetings start to gather multiple participants, additional problems arise with these video conferencing technologies. In physical meetings, people can have secondary conversations. You can project your voice so that only people close to you can hear what you are saying. In some cases, you may even have private conversations in the context of larger meetings. However, in a virtual meeting, when multiple people are speaking simultaneously, the software mixes the two audio streams substantially identically, causing participants to speak on top of each other. Therefore, when multiple people are involved in a virtual meeting, private conversations are not possible and conversations tend to take place more in the form of one-to-many speech. Here, too, virtual meetings rob participants of opportunities to make social connections, communicate more effectively, and network.

Moreover, due to limitations in network bandwidth and computing hardware, the performance of many video conferencing systems begins to slow down when many streams are deployed in a conference. Many computing devices are equipped to process video streams from a few participants, but are inadequately equipped to process video streams from a dozen or so participants. With many schools operating entirely virtually, a class of 25 students can seriously slow down school-issued computing devices.

Massively multiplayer online games (MMOG or MMO) can typically handle significantly more than 25 participants. These games often have hundreds or thousands of players on a single server. MMOs often allow players to navigate avatars around the virtual world. Sometimes these MMOs allow users to talk to each other or send messages to each other. Examples include the ROBLOX game available from Roblox Corporation of San Mateo, California and the MINECRAFT game available from Mojang Studios of Stockholm, Sweden.

Having bare avatars interact with each other also has limitations in terms of social interaction. These avatars are generally unable to convey the facial expressions that people often make inadvertently. These facial expressions can be observed in video conferences. Some publications may describe placing a video on an avatar in a virtual world. However, these systems typically require special software and have other limitations that limit their usefulness.

Improved methods for video conferencing are needed.

In one embodiment, the device enables video conferencing between a first user and a second user. The device includes a processor coupled to memory, a display screen, a network interface, and a web browser. The network interface includes: (i) data specifying a three-dimensional virtual space, (ii) a position and orientation in the three-dimensional virtual space, where the position and orientation are entered by the first user, and (iii) the first user's Configured to receive a video stream captured from a camera on the device. The first user's camera is positioned to capture photographic images of the first user. A web browser implemented in the processor is configured to download a web application from a server and execute the web application. The web application includes a texture mapper and renderer. The texture mapper is configured to texture map the video stream onto a three-dimensional model of the avatar. The renderer is configured to render, from the viewpoint of the second user's virtual camera, a three-dimensional virtual space including a texture-mapped three-dimensional model of the avatar positioned at a corresponding location and oriented in a corresponding direction for display to the second user. By managing texture mapping within the web application, embodiments eliminate the need to install special software.

In one embodiment, a computer-implemented method enables presentations in a virtual meeting including a plurality of participants. In this method, data specifying a three-dimensional virtual space is received. The position and orientation in three-dimensional virtual space are also received. The location and direction were entered by the first of the plurality of participants to the meeting. Finally, a video stream captured from the camera on the first participant's device is received. The camera was positioned to capture photographic images of the first participant. The video stream is texture mapped onto a three-dimensional model of the avatar. Additionally, a presentation stream is received from the first participant's device. The presentation stream is texture mapped onto a three-dimensional model of the presentation screen. Finally, from the viewpoint of the virtual camera of a second of the plurality of participants, a three-dimensional virtual space is rendered with a texture-mapped avatar and a texture-mapped presentation screen for display to the second participant. In this way, embodiments enable presentations in a social conference environment.

In one embodiment, a computer-implemented method provides audio for a virtual conference including a plurality of participants. In this method, from the viewpoint of the first user's virtual camera, a three-dimensional virtual space containing an avatar texture-mapped to the second user's video is rendered for display to the first user. The virtual camera is at a first position in the three-dimensional virtual space and the avatar is at a second position in the three-dimensional virtual space. An audio stream from the microphone of the second user's device is received. The microphone was positioned to capture the second user's speech. The volume of the received audio stream is adjusted to determine a left audio stream and a right audio stream to provide a sense of where the second location is relative to the first location in three-dimensional virtual space. The left audio stream and the right audio stream are output to be played to the first user in stereo.

In one embodiment, a computer-implemented method provides audio for a virtual conference. In this method, from the viewpoint of the first user's virtual camera, a three-dimensional virtual space containing an avatar texture-mapped to the second user's video is rendered for display to the first user. The virtual camera is at a first position in the three-dimensional virtual space and the avatar is at a second position in the three-dimensional virtual space. An audio stream from the microphone of the second user's device is received. It is determined whether the virtual camera and the avatar are located in the same area among the plurality of areas. When it is determined that the virtual camera and the avatar are not located in the same area, the audio stream is attenuated. The attenuated audio stream is output to be played to the first user. In this way, embodiments enable private, secondary conversations in a virtual video conferencing environment.

In one embodiment, a computer-implemented method efficiently streams video for a virtual conference. In this method, the distance between a first user and a second user in the virtual meeting space is determined. A video stream captured from a camera on a first user's device is received. The camera was positioned to capture photographic images of the first user. The resolution or bitrate of the video stream is reduced based on the determined distance so that closer distances result in greater resolution than longer distances. The video stream is transmitted to the second user's device at a reduced resolution or bitrate for display to the second user within the virtual meeting space. The video stream is texture mapped onto the first user's avatar for display to a second user within the virtual meeting space. In this way, embodiments allocate bandwidth and computing resources efficiently even when there are a large number of conference participants.

In one embodiment, a computer-implemented method enables modeling in virtual video conferencing. In this method, a three-dimensional model of a virtual environment, a mesh representing a three-dimensional model of an object, and a video stream from a participant in a virtual video conference are received. The video stream is texture mapped onto an avatar that can be navigated by the participant. A texture-mapped avatar and a mesh representing a three-dimensional model of objects within the virtual environment are rendered for display.

System, device, and computer program product embodiments are also disclosed.

Additional embodiments, features, and advantages of the invention, as well as the structure and operation of various embodiments, are described in detail below with reference to the accompanying drawings.

The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate the disclosure and, together with this description, further explain the principles of the disclosure and enable any person skilled in the art to make and use the disclosure. It plays a role in making things happen.
1 is a diagram illustrating an example interface for providing video conferencing in a virtual environment where video streams are mapped onto avatars.
2 is a diagram illustrating a three-dimensional model used to render a virtual environment with avatars for video conferencing.
3 is a diagram illustrating a system for providing video conferencing in a virtual environment.
Figures 4A-4C illustrate how data is transferred between the various components of the system in Figure 3 to provide video conferencing.
Figure 5 is a flow chart illustrating a method for adjusting relative left and right volumes to provide a sense of position in a virtual environment during a video conference.
Figure 6 is a chart illustrating how the volume rolls off as the distance between avatars increases.
Figure 7 is a flow chart illustrating a method for adjusting relative volume to provide different volume regions in a virtual environment during video conferencing.
8A and 8B are diagrams illustrating different volume areas in a virtual environment during a video conference.
9A-9C are diagrams illustrating traversing the hierarchy of volume areas in a virtual environment during a video conference.
10 illustrates an interface with a 3D model in a 3D virtual environment.
11 illustrates presentation screen sharing in a three-dimensional virtual environment used in video conferencing.
Figure 12 is a flow chart illustrating a method for allocating available bandwidth based on the relative positions of avatars within a three-dimensional virtual environment.
Figure 13 is a chart illustrating how priority values can drop as the distance between avatars increases.
Figure 14 is a chart illustrating how allocated bandwidth may vary based on relative priority.
Figure 15 is a diagram illustrating components of devices used to provide video conferencing within a virtual environment.
The drawing in which an element first appears is typically indicated by the leftmost number or digits in the corresponding reference number. In the drawings, like reference numbers may indicate identical or functionally similar elements.

Video conferencing using avatars in a virtual environment

1 is a diagram illustrating an example of an interface 100 providing video conferencing in a virtual environment where video streams are mapped onto avatars.

Interface 100 may be displayed to participants in a video conference. For example, interface 100 can be rendered for display to participants and continuously updated as the video conference progresses. A user can control the orientation of his or her virtual camera using, for example, keyboard inputs. In this way, the user can navigate around the virtual environment. In one embodiment, different inputs can change the X position and Y position and the pan angle and tilt angle of the virtual camera in the virtual environment. In further embodiments, the user may use inputs to change the height (Z coordinate) or yaw of the virtual camera. In still further embodiments, a user may enter inputs that cause the virtual camera to “hop” up and return to its original position, simulating gravity. The inputs available to navigate the virtual camera include, for example, the WASD keyboard keys to move the virtual camera forward, backward, left, and right in the X-Y plane, the space bar key to "hop" the virtual camera, and the pan and tilt angles. May include keyboard and mouse inputs, such as mouse movements specifying changes.

Interface 100 includes avatars 102A and 102B, each of which represents a different participant in the video conference. Avatars 102A and 102B are texture mapped to video streams 104A and 104B from the devices of the first participant and the second participant, respectively. A texture map is an image that is applied (mapped) to the surface of a shape or polygon. Here, the images are individual frames of video. Camera devices capturing video streams 104A and 104B are positioned to capture the faces of each participant. In this way, avatars are texture-mapped into moving images of the faces of conference participants as they speak and listen.

Similar to the way the virtual camera is controlled by the user viewing interface 100, the position and orientation of avatars 102A and 102B are controlled by the respective participants they represent. Avatars 102A and 102B are three-dimensional models represented by a mesh. Each avatar 102A and 102B may have the participant's name below the avatar.

Respective avatars 102A and 102B are controlled by various users. Each of these can be placed at a point within the virtual environment that corresponds to where its virtual cameras are located. Just as user viewing interface 100 can move the virtual camera around, various users can move their respective avatars 102A and 102B around.

The virtual environment rendered on the interface 100 includes a background image 120 and a three-dimensional model 118 of the arena. An arena may be a location or building where a video conference is to be held. The arena may include a floor area bounded by walls. The three-dimensional model 118 may include mesh and texture. Other ways to mathematically represent the surface of the three-dimensional model 118 may also be possible. For example, polygon modeling, curve modeling, and digital sculpting may be possible. For example, three-dimensional model 118 may be represented by voxels, splines, geometric primitives, polygons, or any other possible representation in three-dimensional space. The three-dimensional model 118 may also include specifications of light sources. Light sources may include, for example, point sources, directional sources, spotlight sources, and ambient sources. Objects can also have certain properties that describe how they reflect light. In examples, properties may include diffuse lighting, ambient lighting, and spectral lighting interactions.

In addition to the arena, the virtual environment may include various other three-dimensional models that illustrate different components of the environment. For example, the three-dimensional environment may include a decoration model 114, a speaker model 116, and a presentation screen model 122. Like model 118, these models can be expressed using any mathematical way of representing a geometric surface in three-dimensional space. These models may be separate from model 118 or may be combined into a single representation of the virtual environment.

Decorative models, such as model 114, serve to enhance realism and increase the aesthetic appeal of the arena. As will be described in greater detail below with respect to FIGS. 5 and 7 , speaker model 116 may virtually emit sounds such as presentations and background music. The presentation screen model 122 may serve to provide an outlet for presenting a presentation. The presenter's video or presentation screen share may be texture mapped onto the presentation screen model 122.

Button 108 may provide the user with a list of participants. In one example, after the user selects button 108, the user can chat with other participants by sending text messages individually or as a group.

Button 110 may enable a user to change attributes of the virtual camera used to render interface 100. For example, a virtual camera may have a field of view that specifies the angle at which data is rendered for display. Modeling data within the camera's field of view may be rendered, while modeling data outside the camera's field of view may not be rendered. Basically, the virtual camera's field of view can be set to somewhere between 60° and 110°, corresponding to a wide-angle lens and human vision. However, selecting button 110 can cause the virtual camera to increase its field of view beyond 170°, equivalent to a fisheye lens. This may enable users to have a broader peripheral awareness of their surroundings in a virtual environment.

Finally, button 112 allows the user to exit the virtual environment. Selecting button 112 may cause a notification to be sent to devices belonging to other participants signaling the devices of other participants to stop displaying the avatar corresponding to the user who was previously viewing interface 100. there is.

In this way, the interface virtual 3D space is used to conduct video conferencing. Every user controls an avatar, which can move, look around, jump, or do other things that change its position or direction. The virtual camera shows the user a virtual 3D environment and different avatars. Other users' avatars have as an integral part a virtual display showing the user's webcam image.

By providing users with a sense of space and allowing users to see each other's faces, embodiments provide a more social experience than traditional web conferencing or traditional MMO gaming. More social experiences have a variety of applications. For example, this can be used in online shopping. For example, interface 100 may be used for a virtual grocery store, house of worship, trade show, B2B sales, B2C sales, schooling, restaurant or cafeteria, product launch, or construction site (e.g., for architects, engineers, contractors). Visits, office spaces (e.g., people working virtually “at their desks”), remotely controlling machines (ships, vehicles, airplanes, submarines, drones, drilling equipment, etc.), plants/factories Control rooms, medical procedures, garden design, guided virtual bus tours, musical events (e.g. concerts), lectures (e.g. TED Talks), political party conferences, board meetings, underwater research, difficult-to-access locations. Research, training for emergencies (e.g. fire), cooking, shopping (including checkout and delivery), virtual arts and crafts (e.g. painting and pottery), weddings, funerals, baptisms, distance sports training, counseling. , fear treatment (e.g., face-to-face therapy), fashion shows, amusement parks, home decor, sports viewing, esports viewing, viewing performances captured using 3D cameras, board and role-playing game play, and medical imaging. Reviewing, viewing geological data, learning a language, meeting in a space for the visually impaired, meeting in a space for the hearing impaired, attending events for people who are normally unable to walk or stand, news or weather announcements, talk shows, book signings, voting. , MMOs, buying/selling virtual locations (such as those available in some MMOs, such as the SECOND LIFE games available from Linden Research, Inc., San Francisco, California), flea markets, garage sales, travel agencies, banks, archives, computer processes. Management, fencing/sword fighting/martial arts, re-enactments (e.g. crime scene and/or accident re-enactments), rehearsals of real events (e.g. weddings, presentations, shows, space walks), real events captured with 3D cameras. evaluating or viewing, livestock shows, zoos, experiencing the lives of tall/short/blind/deaf/white/black people (e.g. simulating the perspective from which the user would like to experience a response) modified video streams or still images for virtual worlds), interviews, game shows, interactive novels (e.g. murder mysteries), virtual fishing, virtual sailing, psychological research, behavioral analysis, virtual sports (e.g. hiking/bouldering), control of lights, etc. from home or other locations (domotics), memory palace, archeology, gift shop, virtual visit to make customers more comfortable during the actual visit, explaining procedures and making people more comfortable virtual medical procedures, and virtual exchanges/financial markets/stock markets (e.g. integrating real-time data and video feeds into the virtual world, real-time trading and analytics), as part of the job so that people actually meet each other organically. A virtual location you need to go to (for example, if you want to generate an invoice, this can only be done inside the virtual location) and augmented reality (which projects a person's face onto an AR headset (or helmet) so that you can see their facial expressions). (e.g., useful for military, law enforcement, firefighters, special operations, etc.), and has applications in providing reservations (e.g., for specific vacation homes/cars, etc.).

2 is a diagram 200 illustrating a three-dimensional model used to render a virtual environment with avatars for video conferencing. As illustrated in FIG. 1 , where the virtual environment includes a three-dimensional arena 118 and various three-dimensional models, including three-dimensional models 114 and 122 . As also illustrated in FIG. 1 , diagram 200 includes avatars 102A and 102B that navigate around the virtual environment.

As described above, interface 100 in Figure 1 is rendered from the viewpoint of a virtual camera. The virtual camera in question is illustrated in diagram 200 as virtual camera 204. As mentioned above, user viewing interface 100 in Figure 1 can control virtual camera 204 and navigate the virtual camera in three-dimensional space. Interface 100 is continuously updated with new positions of virtual camera 204 and any changes to models within the field of view of virtual camera 204. As described above, the field of view of virtual camera 204 may be a frustum defined, at least in part, by a horizontal field of view and a vertical field of view.

As described above with respect to Figure 1, a background image or texture may define at least a portion of the virtual environment. The background image may capture aspects of the virtual environment that are intended to appear from a distance. The background image may be a texture mapped onto the sphere 202. Virtual camera 204 may be at the origin of sphere 202. In this way, distant features of the virtual environment can be rendered efficiently.

In other embodiments, other shapes instead of sphere 202 may be used to texture map the background image. In various alternative embodiments, the shape may be a cylinder, cube, rectangular prism, or any other three-dimensional geometry.

3 is a diagram illustrating a system 300 that provides video conferencing in a virtual environment. System 300 includes a server 302 coupled to devices 306A and 306B through one or more networks 304.

Server 302 provides services for connecting a video conference session between device 306A and device 306B. As will be explained in more detail below, server 302 stores conference participants' devices (e.g., devices 306A and 306B) when new participants join the conference and when existing participants leave the conference. ) to communicate notifications. Server 302 communicates messages describing the position and orientation in the three-dimensional virtual space for each participant's virtual cameras within the three-dimensional virtual space. Server 302 also communicates video and audio streams between the participants' respective devices (e.g., devices 306A and 306B). Finally, the server 302 stores and transmits data describing data specifying the three-dimensional virtual space to the respective devices 306A and 306B.

In addition to the data needed for the virtual conference, server 302 may provide actionable information that informs devices 306A and 306B on how to render the data to provide an interactive conference.

Server 302 responds to the requests with a response. Server 302 may be a web server. A web server is software and hardware that uses Hypertext Transfer Protocol (HTTP) and other protocols to respond to client requests made over the World Wide Web. The main job of a web server is to display website content by storing, processing, and delivering web pages to users.

In an alternative embodiment, communication between devices 306A and 306B occurs on a peer-to-peer basis rather than through server 302. In a given embodiment, one or more of data describing each participant's location and orientation, notifications regarding new and existing participants, and each participant's video and audio streams are transmitted to the devices rather than via server 302. There is direct communication between (306A and 306B).

Network 304 enables communication between various devices 306A and 306B and server 302. The network 304 includes an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless wide area network (WWAN), and a MAN ( metropolitan area network, part of the Internet, part of the Public Switched Telephone Network (PSTN), a cellular telephone network, a wireless network, a WiFi network, a WiMax network, any other type of network, or any of two or more such networks. It can be a combination.

Devices 306A and 306B are the respective devices of respective participants in the virtual conference. Devices 306A and 306B each receive data necessary to conduct a virtual conference and render data necessary to provide a virtual conference. As will be described in more detail below, devices 306A and 306B include a display for presenting rendered meeting information, inputs that allow the user to control a virtual camera, and providing audio to the user for the meeting. It includes speakers (e.g., a headset) to capture the user's voice input, a microphone to capture the user's voice input, and a camera positioned to capture video of the user's face.

Devices 306A and 306B may be any type of computing device, including a laptop, desktop, smartphone, or tablet computer, or a wearable computer (eg, a smartwatch or an augmented or virtual reality headset).

Web browsers 308A and 308B may present network resources (e.g., web pages) addressed by a link identifier (such as a Uniform Resource Locator or URL) to retrieve and display the network resource. . Specifically, web browsers 308A and 308B are software applications for accessing information on the World Wide Web. Typically, web browsers 308A and 308B make this request using Hypertext Transfer Protocol (HTTP or HTTPS). When a user requests a web page from a particular website, the web browser retrieves the required content from the web server, interprets and executes the content, and then transmits the page to device 306A, shown as client/peer conference applications 308A and 308B. and 306B). In examples, the content may have HTML and client-side scripting, such as JavaScript. Once displayed, the user can enter information and make selections on the page, which may cause web browsers 308A and 308B to make additional requests.

Meeting applications 310A and 310B may be web applications downloaded from server 302 and configured to be executed by respective web browsers 308A and 308B. In one embodiment, meeting applications 310A and 310B may be JavaScript applications. In one example, meeting applications 310A and 310B may be written in a higher level language, such as the Typescript language, and translated or compiled into JavaScript. Meeting applications 310A and 310B are configured to interact with the WebGL JavaScript application programming interface. It can have control code specified in JavaScript and shader code written in GLSL ES (OpenGL ES Shading Language). Using the WebGL API, conferencing applications 310A and 310B can utilize graphics processing units (not shown) of devices 306A and 306B. Moreover, OpenGL rendering of interactive 2D and 3D graphics without using plug-ins.

Meeting applications 310A and 310B receive data describing the positions and orientations of other avatars and three-dimensional modeling information describing the virtual environment from server 302. Additionally, conference applications 310A and 310B receive video and audio streams of other conference participants from server 302.

Meeting applications 310A and 310B render three-dimensional modeling data, including data describing the three-dimensional environment and data representing the respective participant avatars. This rendering may involve rasterization, texture mapping, ray tracing, shading, or other rendering techniques. In one embodiment, rendering may involve ray tracing based on characteristics of the virtual camera. Ray tracing involves creating an image by tracing the path of light to pixels in the image plane and simulating the effects of encountering virtual objects. In some embodiments, to enhance realism, ray tracing can simulate optical effects such as reflection, refraction, scattering, and dispersion.

In this way, the user enters the virtual space using web browsers 308A and 308B. The scene is displayed on the user's screen. The user's webcam video stream and microphone audio stream are transmitted to server 302. When other users enter the virtual space, avatar models are created for them. This avatar's location is transmitted to the server and received by other users. Other users also receive notification from server 302 that the audio/video stream is available. A user's video stream is placed on an avatar created for that user. The audio stream is played as if coming from the avatar's location.

Figures 4A-4C illustrate how data is transferred between the various components of the system in Figure 3 to provide video conferencing. Like Figure 3, each of Figures 4A-4C depicts a connection between server 302 and devices 306A and 306B. In detail, Figures 4A-4C illustrate example data flows between the devices.

FIG. 4A illustrates a diagram 400 illustrating how server 302 transmits data describing the virtual environment to devices 306A and 306B. In particular, both devices 306A and 306B retrieve the 3D arena 404, background texture 402, spatial hierarchy 408, and any other 3D modeling information 406 from server 302. Receive.

As described above, background texture 402 is an image representing distant features of the virtual environment. The image may be regular (eg, a brick wall) or irregular. Background texture 402 may be encoded in any common image file format, such as bitmap, JPEG, GIF, or other file image format. This describes the background image that will be rendered, for example, for a sphere in the distance.

The 3D arena 404 is a 3D model of the space where the meeting will be held. As explained above, this may include, for example, a mesh and possibly its own texture information to be mapped onto the three-dimensional primitives it describes. This can define a space that the virtual camera and each avatar can navigate within the virtual environment. Accordingly, it may be bounded by edges (eg, walls or fences) that indicate to the user the perimeter of the navigable virtual environment.

Spatial hierarchy 408 is data that specifies partitions in the virtual environment. These partitions are used to determine how the sound is processed before being transmitted between participants. As will be explained below, this partition data may be hierarchical and may describe sound processing to enable areas in which participants of a virtual meeting can have private or secondary conversations.

The 3D model 406 is any other 3D modeling information needed to conduct a meeting. In one embodiment, this may include information describing the respective avatars. Alternatively or additionally, this information may include product demonstrations.

Once information needed to conduct a meeting is transmitted to participants, Figures 4B and 4C illustrate how server 302 passes information from one device to another. FIG. 4B illustrates a diagram 420 showing how server 302 receives information from respective devices 306A and 306B, and FIG. 4C illustrates how server 302 receives information from respective devices 306B and 306A. ) illustrates a diagram 420 showing how to transmit. Specifically, device 306A transmits location and orientation 422A, video stream 424A, and audio stream 426A to server 302, and server 302 transmits location and orientation 422A, video stream 424A, and audio stream 426A. Stream 424A and audio stream 426A are transmitted to device 306B. And the device 306B transmits the location and direction 422B, the video stream 424B, and the audio stream 426B to the server 302, and the server 302 transmits the location and direction 422B and the video stream 424B. ), and transmits the audio stream 426B to the device 306A.

Position and orientation 422A and 422B describe the position and orientation of the virtual camera relative to the user using device 306A. As described above, a position can be a coordinate in three-dimensional space (e.g., x, y, z coordinates), and a direction can be a direction in three-dimensional space (e.g., pan, tilt, roll). )) can be. In some embodiments, the user may not be able to control the roll of the virtual camera, so the direction may only specify the pan and tilt angles. Similarly, in some embodiments, the user may not be able to change the z-coordinate of the avatar (because the avatar is limited by virtual gravity), and the z-coordinate may be unnecessary. In this way, positions and directions 422A and 422B may each include coordinates and pan and tilt values on a horizontal plane in at least three-dimensional virtual space. Alternatively or additionally, the user may "jump" his or her avatar, so the Z position may be specified solely by an indication of whether the user is jumping his or her avatar.

In different examples, location and directions 422A and 422B may be sent and received using HTTP request responses or using socket messaging.

Video streams 424A and 424B are video data captured from the cameras of the respective devices 306A and 306B. Video can be compressed. For example, video may use any commonly known video codecs, including MPEG-4, VP8, or H.264. Video can be captured and transmitted in real time.

Similarly, audio streams 426A and 426B are audio data captured from the respective devices' microphones. Audio can be compressed. For example, audio can use any commonly known audio codecs, including MPEG-4 or vorbis. Audio can be captured and transmitted in real time. Video stream 424A and audio stream 426A are captured, transmitted, and presented in synchronization with each other. Similarly, video stream 424B and audio stream 426B are captured, transmitted, and presented in synchronization with each other.

Video streams 424A and 424B and audio streams 426A and 426B may be transmitted using the WebRTC application programming interface. WebRTC is an API available in JavaScript. As described above, devices 306A and 306B download and run web applications, such as meeting applications 310A and 310B, which may be implemented in JavaScript. Conferencing applications 310A and 310B can use WebRTC to receive and transmit video streams 424A and 424B and audio streams 426A and 426B by making API calls from their JavaScript.

As mentioned above, when a user leaves a virtual meeting, this exit is communicated to all other users. For example, if device 306A leaves the virtual meeting, server 302 will communicate that departure to device 306B. As a result, device 306B will stop rendering the avatar corresponding to device 306A, removing the avatar from the virtual space. Additionally, device 306B will stop receiving video stream 424A and audio stream 426A.

As described above, conferencing applications 310A and 310B periodically update the virtual space based on new information about their respective video streams 424A and 424B, position and orientation 422A and 422B, and the three-dimensional environment. You can re-render periodically or intermittently. For simplicity, each of these updates is now described from the perspective of device 306A. However, one of ordinary skill in the art will understand that device 306B will behave similarly given similar changes.

When device 306A receives video stream 424B, device 306A texture maps the frames from video stream 424A onto the avatar corresponding to device 306B. The texture-mapped avatar is re-rendered within a three-dimensional virtual space and presented to the user of device 306A.

When device 306A receives the new location and orientation 422B, device 306A creates an avatar corresponding to device 306B positioned at the new location and oriented in the new orientation. The created avatar is re-rendered within a three-dimensional virtual space and presented to the user of device 306A.

In some embodiments, server 302 may transmit updated model information describing the three-dimensional virtual environment. For example, server 302 may transmit updated information 402, 404, 406, or 408. When that happens, device 306A will re-render the virtual environment based on the updated information. This can be useful when your environment changes over time. For example, an outdoor event may change from daytime to sunset as the event progresses.

Again, when device 306B leaves the virtual conference, server 302 sends a notification to device 306A indicating that device 306B is no longer participating in the conference. In that case, device 306A will re-render the virtual environment without an avatar for device 306B.

Although FIGS. 3 and 4A-4C are illustrated with two devices for simplicity, those skilled in the art will understand that the techniques described herein can be extended to any number of devices. Additionally, although FIGS. 3 and 4A-4C illustrate a single server 302, those skilled in the art will understand that the functionality of server 302 may be distributed among a plurality of computing devices. In one embodiment, the data transmitted in FIG. 4A may come from one network address for server 302, while the data transmitted in FIGS. 4B and 4C may come from/to a different network address for server 302. can be transmitted.

In one embodiment, participants can set their webcam, microphone, speakers, and graphics settings before entering the virtual meeting. In an alternative embodiment, after starting the application, users can enter a virtual lobby, where they are greeted by an avatar controlled by a real person. This person can view and modify the user's webcam, microphone, speakers, and graphics settings. The guide may also instruct the user on how to use the virtual environment, for example by teaching the user about viewing, moving and interacting. When ready, the user automatically leaves the virtual waiting room and joins the actual virtual environment.

Adjustment of volume for video conferencing in a virtual environment

Embodiments also adjust volume to provide a sense of location and space within the virtual meeting. This is illustrated, for example, in Figures 5-7, 8A-8B and 9A-9C, each of which is described below.

5 is a flow chart illustrating a method 500 for adjusting relative left and right volumes to provide a sense of position in a virtual environment during a video conference.

At step 502, the volume is adjusted based on the distance between the avatars. As described above, an audio stream from the microphone of another user's device is received. The volume of both the first and second audio streams is adjusted based on the distance between the second location and the first location. This is illustrated in Figure 6.

Figure 6 shows a chart 600 illustrating how the volume rolls off as the distance between avatars increases. Chart 600 illustrates volume 602 on the x and y axes. As the distance between users increases, the volume remains constant until the reference distance 602 is reached. At that point, the volume begins to decline. In this way, all other things being equal, closer users will often sound louder than farther away users.

How fast the sound falls depends on the roll off factor. This may be a factor built into the settings of the video conferencing system or client device. As illustrated by lines 608 and 610, larger roll off rates will cause volume to drop more quickly than smaller roll off rates.

Returning to Figure 5, at step 504 the relative left and right audio is adjusted based on the direction in which the avatar is located. That is, the volume of audio output from the user's speakers (e.g., headset) will change to provide a sense of where the speaking user's avatar is located. Left and right based on the direction of the location of the user generating the audio stream (e.g., the location of the speaking user's avatar) relative to the location of the user receiving the audio (e.g., the location of the virtual camera). The relative volumes of the audio streams are adjusted. The locations may be on a horizontal plane within a three-dimensional virtual space. The relative volumes of the left and right audio streams are adjusted to provide a sense of where the second location is in three-dimensional virtual space relative to the first location.

For example, at step 504, the audio corresponding to the avatar on the left side of the virtual camera may be adjusted so that the audio is output at a higher volume in the receiving user's left ear than in the right ear. Similarly, the audio corresponding to the avatar on the right side of the virtual camera will be adjusted so that the audio is output at a higher volume in the receiving user's right ear than in the left ear.

At step 506, the relative left and right audio is adjusted based on the direction in which one avatar is oriented relative to the other. Based on the angle between the direction the virtual camera is facing and the direction the avatar is facing, such that more vertical this angle tends to have a larger volume difference between the left and right audio streams. The relative volume of the audio stream is adjusted.

For example, when the avatar is directly facing the virtual camera, the relative left and right volumes of the avatar's corresponding audio streams at step 506 may not be adjusted at all. When the avatar is facing the left side of the virtual camera, the relative left and right volumes of the avatar's corresponding audio streams can be adjusted so that the left side is louder than the right side. And, when the avatar is facing the right side of the virtual camera, the relative left and right volumes of the avatar's corresponding audio stream can be adjusted so that the right side is louder than the left side.

In one example, the calculation at step 506 may involve taking the cross product of the angle at which the virtual camera is facing and the angle at which the avatar is facing. The angles may be the direction they are facing in the horizontal plane.

In one embodiment, a check may be performed to determine which audio output device the user is using. If the audio output device is not a set of headphones or another type of speaker that provides a stereo effect, the adjustments in steps 504 and 506 may not occur.

Steps 502-506 are repeated for all audio streams received from all other participants. Based on the calculations in steps 502-506, left and right audio gains are calculated for all other participants.

In this way, the audio streams for each participant are adjusted to provide a sense of where the participant's avatar is located in the three-dimensional virtual environment.

Not only can the audio stream be adjusted to provide a sense of where the avatars are located, but in certain embodiments, the audio stream can be adjusted to provide private or semi-private volume regions. In this way, the virtual environment allows users to have private conversations. Additionally, it allows users to socialize with each other and have individual, secondary conversations not possible with conventional video conferencing software. This is illustrated, for example, in relation to Figure 7.

FIG. 7 is a flow chart illustrating a method 700 for adjusting relative volume to provide different volume regions in a virtual environment during video conferencing.

As described above, the server may provide specifications of sound or volume regions to client devices. The virtual environment can be partitioned into different volume areas. At step 702, the device determines in which sound areas the respective avatars and virtual camera are located.

For example, FIGS. 8A and 8B are diagrams illustrating different volume regions in a virtual environment during a video conference. FIG. 8A illustrates a diagram 800 with a volume area 802 that enables semi-private or secondary conversation between a user controlling an avatar 806 and a user controlling a virtual camera. In this way, users around conference table 810 can converse without disturbing others in the room. Sound from the user controlling the avatar 806 in the virtual camera may, but does not completely, drop when the virtual camera exits the volume area 802. This allows passers-by to join the conversation if they wish.

Interface 800 also includes buttons 804, 806, and 808, which will be described below.

FIG. 8B illustrates a diagram 800 with a volume area 804 that enables private conversation between a user controlling an avatar 808 and a user controlling a virtual camera. Once inside the volume area 804, audio from the user controlling the avatar 808 and the user controlling the virtual camera can only be output to those inside the volume area 804. Because audio from those users is not played at all to other users in the conference, their audio streams may not be transmitted to other user devices at all.

Volume spaces may be hierarchical, as illustrated in FIGS. 9A and 9B. Figure 9B is a diagram 930 showing a layout with different volume regions arranged in a hierarchy. Volume areas 934 and 935 are within volume area 933 and volume areas 933 and 932 are within volume area 931. These volume regions are represented as a hierarchical tree, as illustrated in diagram 900 and FIG. 9A.

In diagram 900, node 901 represents volume area 931 and is the root of the tree. Nodes 902 and 903 are children of node 901 and represent volume regions 932 and 933. Nodes 904 and 906 are children of node 903 and represent volume regions 934 and 935.

If a user located in area 934 wishes to listen to a user located in area 932, the audio stream must pass through a number of different virtual “walls,” each attenuating the audio stream. Specifically, the sound must pass through the walls of area 932, the walls of area 933, and the walls of area 934. Each wall is attenuated by a specific factor. This calculation is described in relation to steps 704 and 706 in Figure 7.

At step 704, the hierarchy is traversed to determine what various sound regions there are among the avatars. This is illustrated, for example, in Figure 9C. Starting from a node corresponding to the virtual region of the speaking voice (in this case node 904), a path to the receiving user's node (in this case node 902) is determined. To determine the path, links 952 from node to node are determined. In this way, a subset of the area between the area containing the avatar and the area containing the virtual camera is determined.

At step 706, the audio stream from the speaking user is attenuated based on the respective wall transmission rates of the area subset. Each respective wall transmittance specifies how much the audio stream is attenuated.

Additionally or alternatively, in that case different areas have different roll off rates and the distance based calculation shown in method 600 may be applied to the individual areas based on their respective roll off rates. In this way, different areas of the virtual environment project sound at different rates. The audio gains determined in a method as described above in relation to Figure 5 may be applied to the audio stream to determine left and right audio accordingly. In this way, wall penetration, roll-off rate, and side-to-side adjustments to provide a sense of direction of sound can all be applied together to provide a comprehensive audio experience.

Different audio areas may have different functions. For example, the volume area may be a podium area. If the user is located in the podium area, some or all of the attenuation described with respect to Figures 5 or 7 may not occur. For example, roll-off rates or wall penetration rates may cause no attenuation. In some embodiments, relative left and right audio can be adjusted to still provide a sense of direction.

For purposes of illustration, the methods described with respect to FIGS. 5 and 7 illustrate audio streams from a user with a corresponding avatar. However, the same methods can be applied to other sound sources other than avatars. For example, a virtual environment may have three-dimensional models of speakers. Sound may be emitted from speakers in the same way as the avatar models described above, either due to presentation or simply to provide background music.

As mentioned above, wall transmissivity can be used to completely isolate the audio. In one embodiment, this can be used to create a virtual office. In one example, each user may have a monitor in their physical (perhaps home) office that is constantly turned on and displays a conferencing application logged into the virtual office. There may be a feature that allows the user to indicate whether the user is in the office or should not be disturbed. If the Do Not Disturb indicator is turned off, co-workers or managers can enter the virtual space and knock or walk in just like they would in a real office. Visitors can leave notes if employees are not in the office. When the employee returns, he or she can read the note left by the visitor. The virtual office may have a whiteboard and/or interface that displays messages for users. Messages may be email and/or from a messaging application, such as the SLACK application available from Slack Technologies, Inc., San Francisco, California.

Users can customize or personalize their virtual offices. For example, users can set up models of posters or other wall decorations. Users can change the models or orientation of a desk or decorative item, such as a planter. Users can change the lighting or look out the window.

Returning again to Figure 8A, interface 800 includes various buttons 804, 806, and 808. When the user presses button 804, the attenuation described above with respect to the methods in FIGS. 5 and 7 may not occur, or may only occur in a smaller amount. In that situation, the user's voice is output uniformly to other users, allowing the user to speak to all participants in the meeting. As will be explained below, user video may also be output to a presentation screen within the virtual environment. When the user presses button 806, speaker mode is enabled. In that case, audio is output from sound sources within the virtual environment, such as playing background music. When the user presses button 808, a screen sharing mode may be enabled, allowing the user to share the contents of a screen or window on his or her device with other users. Content may be presented on a presentation model. This will also be explained below.

Presenting in a 3D environment

10 illustrates an interface 1000 with a 3D model 1004 in a 3D virtual environment. As described above with respect to Figure 1, interface 1000 may be displayed to a user who may navigate around the virtual environment. As illustrated in interface 1000, the virtual environment includes an avatar 1004 and a three-dimensional model 1002.

The 3D model 1002 is a 3D model of a product placed inside a virtual space. People can join this virtual space to observe the model and walk around it. Products can have localized sound to enhance the experience.

More specifically, when the presenter in the virtual space wants to show a 3D model, the presenter selects the desired model from the interface. This sends a message to the server to update the details (including the model's name and path). This will be automatically communicated to clients. In this way, the three-dimensional model can be rendered for display simultaneously with presenting the video stream. Users can navigate a virtual camera around a three-dimensional model of the product.

In different examples, the object may be a product demonstration, or may be an advertisement for a product.

11 illustrates an interface 1100 with presentation screen sharing in a three-dimensional virtual environment used in video conferencing. As described above with respect to Figure 1, interface 1100 may be displayed to a user who may navigate around the virtual environment. As illustrated in interface 1100, the virtual environment includes an avatar 1104 and a presentation screen 1106.

In this embodiment, a presentation stream is received from the devices of participants in a conference. The presentation stream is texture mapped onto a three-dimensional model of the presentation screen 1106. In one embodiment, the presentation stream may be a video stream from a camera on the user's device. In another embodiment, the presentation stream may be a screen share from the user's device, where a monitor or window is shared. The presentation video and audio streams may also be from external sources, such as a live stream of the event, through screen sharing or otherwise. When a user enables presenter mode, the user's presentation stream (and audio stream) is posted to the server, tagged with the name of the screen the user wishes to use. Other clients are notified that a new stream is available.

The presenter can also control the position and orientation of audience members. For example, a presenter may have the option of choosing to rearrange all other participants in the meeting so that they are positioned and oriented towards the presentation screen.

An audio stream is captured from the microphone of the first participant's device in synchronization with the presentation stream. The audio stream from the user's microphone may be heard by other users as if it were coming from the presentation screen 1106. In this way, presentation screen 1106 may be a sound source as described above. Because the user's audio stream is projected from the presentation screen 1106, it may be suppressed from coming from the user's avatar. In this way, the audio stream is output for playback in synchronization with displaying the presentation stream on screen 1106 within the three-dimensional virtual space.

Bandwidth allocation based on distance between users

FIG. 12 is a flow chart illustrating a method 1200 for allocating available bandwidth based on the relative positions of avatars within a three-dimensional virtual environment.

At step 1202, the distance between the first user and the second user in the virtual meeting space is determined. The distance may be the distance between them in a horizontal plane in three-dimensional space.

At step 1204, the received video streams are prioritized such that video streams from closer users are given priority over video streams from more distant users. The priority value may be determined as illustrated in FIG. 13.

13 shows a chart 1300 showing priority 1306 and distance 1302 on the y axis. As illustrated by line 1306, the priority state remains at a constant level until the reference distance 1304 is reached. After reaching the baseline distance, priority begins to drop.

At step 1206, the bandwidth available to the user device is distributed among the various video streams. This may be performed based on priority values determined at step 1204. For example, priorities can be adjusted proportionally so that they all add up to 1. For any videos for which insufficient bandwidth is available, the relative priority may be 0. Then, the priorities are adjusted again for the remaining video streams. Bandwidth is allocated based on these relative priority values. Additionally, bandwidth for audio streams may be reserved. This is illustrated in Figure 14.

14 illustrates a chart 1400 with the y-axis representing bandwidth 1406 and the x-axis representing relative priority. After the video has been allocated the minimum available bandwidth 1406, the bandwidth 1406 allocated to the video stream increases in proportion to its relative priority.

Once the allocated bandwidth is determined, the client can request video from the server at the bandwidth/bitrate/frame rate/resolution selected and allocated for that video. This can start the negotiation process between the client and server to start streaming video at the specified bandwidth. In this way, the available video and audio bandwidth is distributed fairly across all users, where users with double priority will receive double bandwidth.

In one possible implementation, using simulcast, all clients send multiple video streams, at different bitrates and resolutions, to the server. Other clients can then tell the server which of these streams they are interested in and want to receive.

At step 1208, it is determined whether the available bandwidth between the first user and the second user in the virtual meeting space makes display of the video ineffective at that distance. This decision can be made by the client or server. In the case of a client, the client sends a message to the server to stop sending video to the client. If ineffective, transmission of the video stream to the second user's device is stopped, and the second user's device is notified to replace the video stream with a still image. The still image may simply be the last video frames received (or one of the last video frames received).

In one embodiment, a similar process can be performed for audio, reducing quality given the size of the portion reserved for audio. In another embodiment, each audio stream is given a consistent bandwidth.

In this way, embodiments can increase performance for all users, while reducing video and audio stream quality for users who are more distant and/or less critical to the server. This is not done when sufficient bandwidth budget is available. Reduction is performed in both bitrate and resolution. This improves video quality because the bandwidth available to that user can be utilized more efficiently by the encoder.

Separately, video resolution is scaled down based on distance, so users twice as far away have half the resolution. In this way, given screen resolution limitations, unnecessary resolutions may not be downloaded. Therefore, bandwidth is conserved.

FIG. 15 is a diagram of system 1500 illustrating components of devices used to provide video conferencing within a virtual environment. In various embodiments, system 1500 may operate according to the methods described above.

Device 306A is a user computing device. Device 306A may be a desktop or laptop computer, smartphone, tablet, or wearable (eg, a watch or head mounted device). Device 306A includes microphone 1502, camera 1504, stereo speakers 1506, and input device 1512. Although not shown, device 306A also includes a processor and persistent, non-transitory and volatile memory. Processors may include one or more central processing units, graphics processing units, or any combination thereof.

Microphone 1502 converts sound into electrical signals. Microphone 1502 is positioned to capture the voice of the user of device 306A. In different examples, microphone 1502 may be a condenser microphone, an electret microphone, a moving coil microphone, a ribbon microphone, a carbon microphone, a piezoelectric microphone, a fiber optic microphone, a laser microphone, a water microphone, or a MEMS microphone.

Camera 1504 captures image data generally by capturing light through one or more lenses. Camera 1504 is positioned to capture photographic images of the user of device 306A. Camera 1504 includes an image sensor (not shown). The image sensor may be, for example, a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor. An image sensor may include one or more photodetectors that detect light and convert it into electrical signals. These electrical signals captured together in similar time frames make up a still photographic image. A series of still photographic images captured at regular intervals together make up a video. In this way, camera 1504 captures images and videos.

The stereo speaker 1506 is a device that converts electrical audio signals into corresponding left and right sounds. The stereo speaker 1506 outputs the left and right audio streams generated by the audio processor 1520 (below) to be played in stereo to the user of device 306A. Stereo speakers 1506 include both peripheral speakers and headphones designed to reproduce sound directly to the user's left and right ears. Exemplary speakers include moving-iron loudspeakers, piezoelectric loudspeakers, magnetostatic loudspeakers, electrostatic loudspeakers, ribbon and planar magnetic loudspeakers, bending wave loudspeakers, flat panel loudspeakers, and heil air motion transducers. air motion transducer), transparent ion conduction speaker, plasma arc speaker, thermoacoustic speaker, rotary woofer, moving coil, electrostatic, electret, planar magnetic, and balanced armature.

Network interface 1508 is a software or hardware interface between two devices or protocol layers in a computer network. Network interface 1508 receives video streams from server 302 for each participant of the conference. The video stream is captured from cameras on the devices of other participants in the video conference. Network interface 1508 also receives data from server 302 specifying a three-dimensional virtual space and arbitrary models therein. For each of the other participants, network interface 1508 receives a position and orientation in three-dimensional virtual space. Location and direction are entered by each of the different participants.

Network interface 1508 also transmits data to server 302. This transmits the location of the user's virtual camera on device 306A, which is used by renderer 1518, and transmits video and audio streams from camera 1504 and microphone 1502.

Display 1510 is an output device for presenting electronic information in visual or tactile form (the latter used, for example, in tactile electronic displays for the visually impaired). Display 1510 may be used in a television set, a computer monitor, a head mounted display, a heads-up display, the output of an augmented or virtual reality headset, a broadcast reference monitor, a medical monitor, a mobile display (for mobile devices), ( It may be a smartphone display (for a smartphone). To present information, display 1510 may be an electroluminescent (ELD) display, a liquid crystal display (LCD), a light emitting diode (LED) backlit LCD, a thin film transistor (TFT) LCD, a light emitting diode (LED) display, an OLED display, or an AMOLED display. It may include a display, a plasma (PDP) display, and a quantum dot (QLED) display.

Input device 1512 is equipment used to provide data and control signals to an information processing system, such as a computer or information device. Input device 1512 enables navigation in a three-dimensional environment by allowing a user to input a new desired location for a virtual camera used by renderer 1518. Examples of input devices include keyboards, mice, scanners, joysticks, and touchscreens.

Web browser 308A and web application 310A were described above with respect to FIG. 3 . Web application 310A includes a screen capturer 1514, texture mapper 1516, renderer 1518, and audio processor 1520.

Screen capturer 1514 captures the presentation stream, specifically screen sharing. Screen capturer 1514 can interact with APIs made available by web browser 308A. By calling functions available from the API, screen capturer 1514 can cause web browser 308A to ask the user which window or screen the user wishes to share. Based on the answer to that query, web browser 308A may return a video stream corresponding to the screen share to screen capturer 1514, which may then transmit the video stream to server 302 and ultimately other This is passed to the network interface 1508 for transmission to the participants' devices.

The texture mapper 1516 texture maps the video stream onto a 3D model corresponding to the avatar. Texture mapper 1516 may texture map individual frames from the video to the avatar. Additionally, the texture mapper 1516 can texture map the presentation stream to a three-dimensional model of the presentation screen.

Renderer 1518 generates texture-mapped images of the avatars for each participant, from the viewpoint of the user's virtual camera on device 306A, positioned and oriented in the corresponding positions received for output to display 1510. Render a 3D virtual space containing 3D models. Renderer 1518 also renders any other three-dimensional models, including, for example, presentation screens.

Audio processor 1520 adjusts the volume of the received audio stream to determine the left and right audio streams to provide a sense of where the second location is relative to the first location in three-dimensional virtual space. In one embodiment, audio processor 1520 adjusts the volume based on the distance between the second location and the first location. In another embodiment, audio processor 1520 adjusts the volume based on the direction from the second location to the first location. In another embodiment, the audio processor 1520 adjusts the volume based on the orientation of the second location relative to the first location on a horizontal plane within the three-dimensional virtual space. In another embodiment, audio processor 1520 may determine that, based on the direction the virtual camera is facing in three-dimensional virtual space, the left audio stream tends to have a higher volume when the avatar is positioned to the left of the virtual camera and the avatar is positioned to the left of the virtual camera. Adjust the volume so that the right audio stream tends to have a higher volume when positioned to the right of the virtual camera. Finally, in another embodiment, audio processor 1520 may, based on the angle between the direction the virtual camera is facing and the direction the avatar is facing, determine which angle is more perpendicular to where the avatar is facing than the left audio stream. Adjust the volume so that there tends to be a larger volume difference between the right and left audio streams.

The audio processor 1520 may also adjust the volume of the audio stream based on the area where the speaker is located relative to the area where the virtual camera is located. In this embodiment, the three-dimensional virtual space is divided into multiple regions. These areas can be hierarchical. When the speaker and virtual camera are located in different areas, wall transmittance can be applied to attenuate the volume of the speaking audio stream.

Server 302 includes attendance notifier 1522, stream coordinator 1524, and stream forwarder 1526.

Attendance notifier 1522 notifies meeting participants when they join and leave the meeting. When a new participant joins the conference, attendance notifier 1522 sends a message to the devices of other participants in the conference indicating that the new participant has joined. Attendance notifier 1522 signals stream forwarder 1526 to begin forwarding video, audio, and location/orientation information to other participants.

Stream coordinator 1524 receives a video stream captured from a camera on the first user's device. Stream coordinator 1524 determines available bandwidth to transmit data for the virtual conference to the second user. This determines the distance between the first user and the second user in the virtual meeting space. It then distributes the available bandwidth between the first and second video streams based on their relative distances. In this way, stream coordinator 1524 gives priority to video streams from closer users over video streams from more distant users. Additionally or alternatively, stream coordinator 1524 may be located on device 306A, perhaps as part of web application 310A.

Stream forwarder 1526 broadcasts received location/orientation information, video, audio, and screen sharing footage (along with any adjustments made by stream coordinator 1524). Stream forwarder 1526 may transmit information to device 306A in response to a request from conference application 310A. Meeting application 310A may send the request in response to a notification from attendance notifier 1522.

Network interface 1528 is a software or hardware interface between two devices or protocol layers in a computer network. Network interface 1528 transmits model information to the various participants' devices. Network interface 1528 receives video, audio, and screen sharing footage from various participants.

The screen capturer 1514, texture mapper 1516, renderer 1518, audio processor 1520, presence notifier 1522, stream handler 1524, and stream forwarder 1526 each include hardware, software, firmware, Or it may be implemented as any combination thereof.

Identifiers such as “(a),” “(b),” “(i),” “(ii),” etc. are sometimes used for different elements or steps. These identifiers are used for clarity and do not necessarily specify the order of elements or steps.

The invention has been described above with the help of functional building blocks that illustrate the implementation of specified functions and their relationships. The boundaries of these functional building blocks are arbitrarily defined herein for convenience of description. Alternative boundaries can be defined as long as the specified functions and their relationships are performed appropriately.

So that other embodiments can be readily modified and/or adapted for various applications, such as the specific embodiments, without undue experimentation and without departing from the general concept of the invention, by applying knowledge in the art, The foregoing description of specific embodiments will thus fully reveal the general nature of the invention. Accordingly, such adaptations and modifications are intended to be within the meaning and scope of equivalents of the disclosed embodiments based on the teachings and guidance presented herein. It should be understood that any terminology or terminology herein is for the purpose of description and not limitation, so that the terminology or terminology herein can be interpreted by a person skilled in the art based on the teachings and guidelines.

The breadth and scope of the invention should not be limited by any of the exemplary embodiments described above, but should be limited only by the appended claims and their equivalents.

Claims (146)

A system for enabling video conferencing between a first user and a second user, comprising:
A processor coupled to memory;
display screen;
(i) data specifying a three-dimensional virtual space, (ii) a position and orientation in the three-dimensional virtual space, wherein the position and orientation are input by the first user, and a camera on the first user's device. a network interface configured to receive a video stream captured from, wherein the camera is positioned to capture photographic images of the first user;
A web browser implemented on the processor and configured to download a web application from a server and execute the web application.
, wherein the web application:
a mapper configured to map the video stream onto a three-dimensional model of an avatar, and
The three-dimensional virtual space comprising, from the viewpoint of the second user's virtual camera, the three-dimensional model of the avatar with the mapped video stream positioned at the location and oriented in the direction for display to the second user. A renderer configured to render
A system that includes a. According to paragraph 1,
wherein the device further includes a graphics processing unit, and the mapper and the renderer include WebGL application calls that enable the web application to map or render using the graphics processing unit. 1. A computer-implemented method for enabling video conferencing between a first user and a second user, comprising:
transmitting a web application to a first client device of the first user and a second client device of the second user;
(i) a position and orientation in three-dimensional virtual space, wherein the position and orientation are input by the first user, and (ii) a video stream captured from a camera on the first client device, wherein the camera is receiving, from the first client device executing the web application - arranged to capture photographic images of a user; and
transmitting the location and direction and the video stream to the second client device of the second user.
Includes,
The web application, when executed on a web browser, maps the video stream onto a three-dimensional model of an avatar, positioned at the location and in the direction for display to the second user, from the viewpoint of the second user's virtual camera. and executable instructions for rendering the three-dimensional virtual space containing the three-dimensional model of the avatar mapped to the video stream oriented. According to paragraph 3,
wherein the web application includes WebGL application calls that enable the web application to map or render using a graphics processing unit of the second client device. 1. A computer-implemented method for enabling video conferencing between a first user and a second user, comprising:
Receiving data specifying a three-dimensional virtual space;
Receiving a position and orientation in the three-dimensional virtual space, wherein the position and orientation are input by the first user;
receiving a video stream captured from a camera on the first user's device, the camera positioned to capture photographic images of the first user;
mapping the video stream onto a three-dimensional model of an avatar by a web application implemented in a web browser; and
comprising the three-dimensional model of the avatar positioned at the location and oriented in the direction for display to the second user, from the viewpoint of the second user's virtual camera, by the web application implemented in the web browser. Rendering the three-dimensional virtual space
Method, including. According to clause 5,
receiving an audio stream captured in synchronization with the video stream from a microphone of the device of the first user, the microphone positioned to capture speech of the first user; and
Outputting the audio stream for playback to the second user in synchronization with the display of the video stream within the three-dimensional virtual space.
A method further comprising: According to clause 5,
When input is received from the second user indicating a desire to change the viewpoint of the virtual camera:
changing the viewpoint of the virtual camera of the second user; and
From the changed viewpoint of the virtual camera, re-rendering the three-dimensional virtual space including the three-dimensional model of the avatar positioned at the location and oriented in the direction for display to the second user.
A method further comprising: In clause 7,
The method wherein the viewpoint of the virtual camera is defined by at least coordinates on a horizontal plane in the three-dimensional virtual space and pan and tilt values. According to clause 5,
When the new location and new orientation of the first user in the three-dimensional virtual space are received:
The method further comprising re-rendering the three-dimensional virtual space including a three-dimensional model of the avatar located at the new location and oriented in the new orientation for display to the second user. According to clause 5,
The method of claim 1 , wherein the mapping step includes iteratively mapping pixels onto the three-dimensional model of the avatar for each frame of the video stream. According to clause 5,
wherein the data, the location and orientation, and the video stream are received from a server to a web browser, and the mapping and rendering are performed by the web browser. According to clause 11,
receiving, from the server, a notification indicating that the first user is no longer present; and
Re-rendering the three-dimensional virtual space without the three-dimensional model of the avatar for display to the second user on the web browser.
A method further comprising: According to clause 12,
receiving, from the server, a notification indicating that a third user has entered the three-dimensional virtual space;
Receiving a second location and a second direction of the third user in the three-dimensional virtual space;
receiving a second video stream captured from a camera on the third user's device, the camera positioned to capture photographic images of the third user;
mapping the second video stream onto a second three-dimensional model of a second avatar; and
From the viewpoint of the virtual camera of the second user, rendering the three-dimensional virtual space including the second three-dimensional model located at the second location and oriented in the second direction for display to the second user. steps to do
A method further comprising: According to clause 5,
The step of receiving data designating the three-dimensional virtual space includes receiving a mesh designating a meeting space and receiving a background image, and the rendering step includes converting the background image into a sphere image. A method comprising the step of mapping to. A non-transitory, tangible, computer-readable device having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations to enable video conferencing between a first user and a second user. , the operations are:
An act of receiving data specifying a three-dimensional virtual space;
Receiving a position and direction in the three-dimensional virtual space, wherein the position and direction are input by the first user;
receiving a video stream captured from a camera on the first user's device, the camera positioned to capture photographic images of the first user;
mapping the video stream onto a three-dimensional model of an avatar; and
Rendering, from the viewpoint of the second user's virtual camera, the three-dimensional virtual space including the three-dimensional model of the avatar located at the location and oriented in the direction for display to the second user.
A device containing a. According to clause 15,
The above operations are:
receiving an audio stream captured in synchronization with the video stream from a microphone of the device of the first user, the microphone positioned to capture utterances of the first user; and
Outputting the audio stream for playback to the second user in synchronization with display of the video stream within the three-dimensional virtual space
A device further comprising: According to clause 15,
The operations are: When input is received from the second user indicating a desire to change the viewpoint of the virtual camera:
Changing the viewpoint of the virtual camera of the second user; and
From the changed viewpoint of the virtual camera, re-rendering the three-dimensional virtual space including the three-dimensional model of the avatar positioned at the location and oriented in the direction for display to the second user.
A device further comprising: According to clause 17,
The device, wherein the viewpoint of the virtual camera is defined by at least coordinates and pan and tilt values on a horizontal plane in the three-dimensional virtual space. According to clause 15,
The operations are: When the new location and new orientation of the first user in the three-dimensional virtual space are received:
The device further comprising re-rendering the three-dimensional virtual space including the three-dimensional model of the avatar located at the new location and oriented in the new direction for display to the second user. According to clause 15,
The mapping operation includes repeatedly mapping pixels for each frame of the video stream onto the three-dimensional model of the avatar. According to clause 15,
The device, wherein the data, the location and orientation, and the video stream are received from a server to a web browser, and the mapping and rendering are performed by the web browser. According to clause 21,
The above operations are:
receiving, from the server, a notification indicating that the first user is no longer present; and
Re-rendering the three-dimensional virtual space without the three-dimensional model of the avatar for display to the second user on the web browser.
A device further comprising: According to clause 22,
The above operations are:
Receiving, from the server, a notification indicating that a third user has entered the three-dimensional virtual space;
receiving a second location and a second direction of the third user in the three-dimensional virtual space;
receiving a second video stream captured from a camera on the third user's device, the camera positioned to capture photographic images of the third user;
mapping the second video stream onto a second three-dimensional model of a second avatar; and
From the viewpoint of the virtual camera of the second user, rendering the three-dimensional virtual space including the second three-dimensional model located at the second location and oriented in the second direction for display to the second user. action
A device further comprising: According to clause 15,
The operation of receiving data specifying the three-dimensional virtual space includes receiving a mesh designating a meeting space and receiving a background image, and the rendering operation includes mapping the background image into a sphere. A device that does it. delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete