US20220232128A1 - Virtual Conferencing System with Layered Conversations - Google Patents

Virtual Conferencing System with Layered Conversations Download PDF

Info

Publication number
US20220232128A1
US20220232128A1 US17/562,649 US202117562649A US2022232128A1 US 20220232128 A1 US20220232128 A1 US 20220232128A1 US 202117562649 A US202117562649 A US 202117562649A US 2022232128 A1 US2022232128 A1 US 2022232128A1
Authority
US
United States
Prior art keywords
user
conversation
users
convo
volume
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
US17/562,649
Other languages
English (en)
Other versions
US11647123B2 (en
Inventor
Leo Glisic
Jesse Katz
Dustin Maltz
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Mycelium Inc
Original Assignee
Mycelium Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Mycelium Inc filed Critical Mycelium Inc
Priority to US17/562,649 priority Critical patent/US11647123B2/en
Assigned to Mycelium, Inc. reassignment Mycelium, Inc. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: GLISIC, LEO, MR., KATZ, JESSE, MR., MALTZ, DSTIN, MR.
Publication of US20220232128A1 publication Critical patent/US20220232128A1/en
Priority to US18/128,884 priority patent/US20230239408A1/en
Application granted granted Critical
Publication of US11647123B2 publication Critical patent/US11647123B2/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/40—Support for services or applications
    • H04L65/403—Arrangements for multi-party communication, e.g. for conferences
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/1066—Session management
    • H04L65/1083—In-session procedures
    • H04L65/1093—In-session procedures by adding participants; by removing participants
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/40—Support for services or applications
    • H04L65/403—Arrangements for multi-party communication, e.g. for conferences
    • H04L65/4046—Arrangements for multi-party communication, e.g. for conferences with distributed floor control
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04M—TELEPHONIC COMMUNICATION
    • H04M3/00—Automatic or semi-automatic exchanges
    • H04M3/42—Systems providing special services or facilities to subscribers
    • H04M3/56—Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities
    • H04M3/563—User guidance or feature selection
    • H04M3/564—User guidance or feature selection whereby the feature is a sub-conference
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04M—TELEPHONIC COMMUNICATION
    • H04M3/00—Automatic or semi-automatic exchanges
    • H04M3/42—Systems providing special services or facilities to subscribers
    • H04M3/56—Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities
    • H04M3/568—Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants

Definitions

  • the invention relates to virtual communications applications, and more particularly to a communications platform which enables multiple layers of conversations.
  • a big limitation in today's virtual communication applications is the inability to have dynamic, fluid conversations.
  • Users' video tiles are generally laid out in a grid pattern and they are all engaged in one single conversation, regardless of how many users are in the videoconferencing session. If two users speak at a time, the other users in the session will hear them both at the same time and it will sound as if they are talking over each other, making it difficult to understand what either user is saying. As a result, users have to be careful to take turns speaking, one at a time. This is equally frustrating for speakers and listeners, and disrupts the natural flow of conversations that are inhibited by these limitations.
  • a virtual communication system including logic for supporting layered conversations, the system comprising:
  • each computer represents a User
  • each computer including a display, audio input, audio output, video input (such as a camera), memory, data storage, and a processor
  • a knowledge base stored in memory, the knowledgebase containing an identifier identifying each conversation associated with a given user, the knowledgebase storing information identifying each participant in each of the conversations, the knowledgebase storing information identifying the type of each conversation associated with the given one of the at least two computers, where each User may participate in multiple simultaneous conversations;
  • layered conversation logic executed by the processor, the layered conversation logic controlling a volume of each participant in each conversation in which the User participates in accordance with the conversation type.
  • Example 1 The virtual communication system of Example 1, wherein a given User may communicate with one or more other Users in a conversation type selected from the group (Layer 1, Layer 2, and Layer 3), wherein LCX1, X2, . . . XN refers to a Layer C communication between Users X, X2, . . .
  • Example 2 The virtual communication system of Example 2, wherein if the User X1 is in L2X1,X3 then the layered conversation logic adjusts the volume of the User X3 heard by the User X1 to S2,2% volume and scales the volume of User X2 experienced by User X1 in conversation L1X1,X2 to S2,1% volume, where S2,1 and S2,2 are scaling factors and S2,2>S2,1.
  • Example 3 The virtual communication system of Example 3, wherein if the user X1 is in L3X1,X4 then the layered conversation logic adjusts the volume of the User X4 heard by the User X1 to S3,3% volume, scales the volume of User X3 experienced by User X1 in conversation L2X1,X3 to S3,2% volume, and scales the volume of User X2 experienced by User X1 in conversation L1X1,X2 to S3,1% volume, where S3,3, S3,2 and S3,1 are scaling factors and S3,3>S3,2>S3,1.
  • any User in conversation L1X1, X2, X3, . . . XN may invite one or more Users of conversation L1X1, X2, X3, . . . XN to participate in conversation L2X1, X2, X3, . . . XN ⁇ 1 contemporaneously with conversation L1X1, X2, X3, . . . XN.
  • any User in conversation L2X1, X2, X3, . . . XN may invite one or more Users of conversation L2X1, X2, X3, . . . XN to participate in conversation L3X1, X2, X3, . . . XN ⁇ 1 contemporaneously with conversation L2X1, X2, X3, . . . XN and conversation L1X1, X2, X3, . . . XN.
  • any User in conversation L1X1, X2, X3, . . . XN may join conversation L2X1, X2, X3, . . . XN ⁇ 1 contemporaneously with conversation L1X1, X2, X3, . . . XN.
  • any User in conversation L2X1, X2, X3, . . . XN may join conversation L3X1, X2, X3, . . . XN ⁇ 1 contemporaneously with conversation L1X1, X2, X3, . . . XN.
  • volume may be further scaled by a scaling factor EX,Y in proportion to the facing orientation of user X relative to user Y within the virtual communication system.
  • volume may be further scaled by a scaling factor FX,Y in proportion to the facing orientation of user Y relative to user X within the virtual communication system.
  • volume may be further scaled by a scaling factor GX,Y in proportion to the volume scaling preferences of user X.
  • Example 1 The virtual communication system of Example 1, wherein the layered conversation logic controls the volume and a visual layout of each participant in each conversation in which the User participates in accordance with the conversation type.
  • Example 2 The virtual communication system of Example 2, wherein users belonging to a given Layer 2 conversation are objectively positioned in a virtual space at a predefined distance, orientation and layout in relation to each other.
  • the virtual communication system of Example 14 further including visual indicators indicating users belonging to the given Layer 2 conversation.
  • Example 2 The virtual communication system of Example 2, wherein for every user in a Layer 2 conversation, the other users in the same Layer 2 conversations are subjectively positioned in a virtual space according to a predefined distance, orientation and layout in relation to each other and in relation to that user.
  • Example 2 The virtual communication system of Example 2, wherein users belonging to a given Layer 3 conversation are objectively positioned in a virtual space at a specified distance, orientation and layout in relation to each other.
  • Example 2 The virtual communication system of Example 2, wherein every user in a Layer 3 conversation is subjectively positioned in a virtual space according to a predefined distance, orientation and layout in relation to other users in the Layer 3 conversation.
  • FIG. 1 is a diagram depicting users A-D
  • FIG. 2 is a diagram depicting Users A-D with an arrow denoting a direction each user is facing;
  • FIG. 3 is a diagram depicting users A, B, C, D, and E in a Convo with each other, while users G and F are not in a Convo;
  • FIG. 4 is a diagram depicting users A, B, C, D, E in Convo 1, users H, I, J in Convo 2 and users F and K not in any Convo;
  • FIG. 5 is a diagram depicting users A, B, C, D, E in Convo 1, users F, G, H, I, J, K in Convo 2;
  • FIG. 6 is a diagram showing user A, B, C, D, E in Convo 1, users F, G, H, I, J, K in Convo 2, users L, M, N in Convo 3 and user P & Q not in any conversation;
  • FIG. 7 is a diagram depicting users A, B, C, D & E in a 5-person Convo, facing the center as shown by the arrows;
  • FIG. 8 is a diagram depicting users A, B, C, D & E in a 5-person Convo, facing the center as shown by the arrows;
  • FIG. 8 is a diagram depicting user C's field of view
  • FIG. 9 is a diagram illustrating User C's field of view from a first-person perspective, represented by the cone shape filled with the diagonal-line pattern and dashed lines on the outside of it;
  • FIG. 10 is a diagram illustrating what a wider field of view for User C might look like from a top-down view
  • FIG. 11 is a diagram illustrating what it might look like in a three-dimensional space from User C's first-person point of view;
  • FIG. 12 is a diagram of a subjective view in which, from User C's first-person point of view, they are able to see all the other users in their field of view, and their videos are tilted facing toward User C (either perfectly facing or only slightly tilted);
  • FIG. 13 depicts an example of User A's subjective view, allowing User A to see Users D, E, B, and C's video tiles all appearing as if they are facing User A.
  • FIG. 14 illustrates the example of FIG. 13 after User E leaves
  • FIG. 15 illustrates the example of FIG. 13 after User F joins
  • FIG. 16 is a block diagram of a client-server implementation
  • FIG. 17 is a block diagram of a peer-to-peer implementation
  • FIG. 18 is a block diagram of starting a convo in a peer-to-peer implementation
  • FIG. 19 is a block diagram of starting a convo in a client-server implementation
  • FIGS. 20A-20B are diagrams of a conversation before and after user A joins
  • FIG. 21 is a block diagram depicting the new positions and orientations within a Convo in a peer-to-peer implementation.
  • FIG. 22 is a block diagram depicting the new positions and orientations within a Convo in a client-server implementation.
  • a virtual conferencing system includes layers of conversations:
  • Layer 1 (entire session): the entire session including all of the users/participants at a virtual event or gathering.
  • Layer 2 (main conversations): a subset of the users/participants in layer 1 who have gathered into smaller group conversations.
  • Layer 3 within each Layer 2 conversation, a subset of users/participants involved in sidebar conversations.
  • each participant in the conversation described herein accesses the Virtual Conferencing System (“System”) using a computer, smartphone, gaming device, augmented or virtual reality device, or like device (hereinafter computer or User) which includes a processor or CPU or like hardware capable of executing software commands, a display, a data entry device such but not limited to a keyboard, memory, communications hardware facilitating telecommunications over a communications medium such as the internet, audio input (e.g., microphone) and audio output (e.g., speaker).
  • the audio input and audio output may be combined into a headset, or any other method of communicating and/or translating audio inputs and outputs, including but not limited to translating audio inputs and outputs to other mediums, as is often done with adaptive tools.
  • Each user may be visually represented to other users by a virtual representation of themselves such as avatars, video tiles (e.g., an image or placeholder graphic with the user's name, pseudonym, or initials), or the like.
  • the System may be implemented as a Peer-to-Peer communication system in which case each peer is a User.
  • the System may be implemented as a Client-Server communication system in which case Client is a User and the Server is a computer which facilitates and coordinates communication among the Clients (Users).
  • the Server need not include the audio input (e.g., microphone) and audio output, e.g., speaker.
  • the System of the invention includes logic for layering communications which may be implemented as software, hardware or a combination of both.
  • the logic may be implemented in code stored in a read-only-memory (ROM) device, an erasable programmable read-only-memory (EPROM or EEPROM) device or the like.
  • ROM read-only-memory
  • EPROM erasable programmable read-only-memory
  • EEPROM electrically erasable programmable read-only-memory
  • the logic may be implemented in software code or instructions stored in memory residing within the computer, on a removable/portable media such as a memory stick, or on a server accessible over the internet.
  • the description that follows lays out a technical solution to enable layered conversations within a virtual communication, augmented reality, or gaming application which should be understood to be any application within which communications between users is facilitated (hereinafter videoconferencing or virtual conferencing application).
  • the layered conversation is not actually a conversation but rather consists of music or other sounds which may be associated with a location and orientation of a User within the virtual environment.
  • the technical solution may be used to enable layered conversations within a videogame application including three-dimensional gaming applications.
  • the first step in enabling layered conversations is to add a spatial dimension to a videoconferencing application. Instead of all the users being represented as video tiles in a grid pattern, they will instead have virtual representation of themselves (such as avatars, video tiles, or some other representation) placed into a virtual world where they can move around.
  • virtual representation of themselves such as avatars, video tiles, or some other representation
  • This world can either be two or more dimensions.
  • most of the diagrams in this document will be based on a two-dimensional world with a top-down (birds' eye view) perspective, but they can just as easily be applied to a three-dimensional environment.
  • the users can move within this world by any combination of one or more mechanisms (for example, the arrow keys on their keyboard), joystick, gestures, or the like.
  • Users can move their avatars or video tile moves within the world, and they can see other users' avatars or video tiles move as well corresponding to those users' movements.
  • the volume of other users' audio may automatically adjust based on relative distance.
  • each user is represented by a gray circle, such that User A is the gray circle with the letter ‘A’ inside of it.
  • User A can hear User C at a relatively higher volume, User B at a relatively moderate volume, and User D at a relatively low volume.
  • V is the volume
  • d is the distance
  • p is the power exponent
  • orientation we mean where any user is directing their attention or focus toward or away from anyone or anything, which may be accomplished by any combination of one or more aspects of controls that might indicate orientation such as the direction a user is facing, moving, turning, looking, gesturing, etc.
  • FIG. 2 each user's orientation is represented by the arrow attached to that user.
  • users B, C and D are at equal distance to User A, but they are facing in different directions as denoted by the arrows.
  • User A would hear User B the loudest, User C at a lower volume than B, but higher than D, and User D at the lowest relative volume.
  • the volume calculations for both distance and orientation can be performed and applied independently, or in combination.
  • One possible implementation would be to cross-multiply the distance and orientation values to create a combined volume adjustment. For example, if User A would hear User B at 80% of full volume based on their relative distance, and at 70% based on User B's orientation, then the combined effect would be that User A would hear User B at 80% times 70%, or 56% of full volume.
  • Layer 1 allows users to move between conversations, which encourages more active participation, but will be a sub-optimal experience for several reasons:
  • Volume controls that rely exclusively on distance and orientation may result in an unpleasant combination of sounds that may distracting or disruptive if sounds are indistinguishable from the conversations they are engaged in, such as those from other nearby users with whom users are not having a conversation.
  • Convo a technical object called ‘Convo’ and refer to it as such going forward. Users will be able to create a Convo, join an existing Convo, or exit a Convo if they are inside of one.
  • users A, B, C, D, and E are in a Convo with each other, while users G and F are not in a Convo.
  • Users' volumes are adjusted such that for each user in a Convo, they can hear all other users inside that same Convo at full volume, and users outside of the Convo at a reduced volume.
  • User A could hear users B, C, D, and E at full volume, and users G and F at reduced volume.
  • the volume reduction applied to the volumes of User G and User F may be combined (for example, in a multiplicative manner) with volume reduction factors due to the distance and orientation of those users relative to User A.
  • An optional movement limitation may be imposed, such as not allowing users inside the Convo to move around freely within the world until they exit the Convo.
  • each user may see a subjective view of other users' video tiles which arranges them into a pattern, such as in the shape of a semi-circle, with respect to that user and faces their video tiles more in the direction of that user. See section titled ‘Subjective Convo View’ for further description.
  • Multiple Convos can exist in a single Layer 1 session.
  • Users A, B, C, D, and E are in one Convo (labeled Convo 1).
  • Users H, I and J are in another Convo (labeled Convo 2).
  • Users F, G and K are not in a Convo.
  • all users in Convo 1 may hear each other at full volume, and may hear all other users (H, I, J, K, F, and G) at reduced volume, based on a Convo volume reduction factor as well as additional volume reduction factors due to distance and orientation.
  • all users in Convo 2 may hear each other at full volume, and may hear all other users at reduced volume due to all three reduction factors (Convo, distance, orientation).
  • users K, F and G may hear all other users at a reduced volume due to distance and orientation, but not due to a Convo reduction factor since they are not in a Convo.
  • other factors can be introduced such that users who are not in a Convo hear users who are in a Convo at enhanced or reduced volume.
  • a convo is set to ‘public’. If a Convo is set to ‘private’, no users outside of the Convo may hear any of the users inside the Convo.
  • Layers 1 and 2 combined enable users to move around a virtual world, interact with each other, and freely organize into smaller conversations within one session, so that multiple conversations can take place at the same time.
  • matching patterns on the user circles denote that they are in a Sidebar with each other, while users with solid gray shading (no pattern) are in the Convo but not in a Sidebar.
  • users A, D and E are in a Sidebar.
  • a Sidebar can take place inside of a Convo between users who are part of the same Convo.
  • a Sidebar can consist of two or more users.
  • One example that can be included in the System is that if the number of users inside of a Sidebar reaches N-x, where N is the number of users in a Convo and x is either predetermined or pre-specified number (majority or supermajority), then the Sidebar has grown to overtake the Convo and may dissipate such that all users are returned to the Convo. In other words, the Sidebar may automatically cease to be a Sidebar if the number of participants exceeds a threshold value. In that case, the “Sidebar” is the main conversation and the Sidebar dissipates in favor of the Convo.
  • c) Users G and F will hear each other at full volume and will hear users H, I, J, and K at reduced volume.
  • An indicator of some kind may be employed to let users know who they are in a Sidebar with.
  • a different indicator will be used to let users know if other users are in a Sidebar with each other.
  • visual indicators such as matching borders/frames/rings can be placed around users' video tiles to indicate that they are in the same Sidebar. They can be matching based on any combination of colors, shapes, sizes or patterns to indicate which users are in the same Sidebar and which ones are in a different Sidebar or not in a Sidebar at all.
  • Convo 1 contains 5 users, with 3 users in a Sidebar within that Convo.
  • Convo 2 contains 6 users, with 2 Sidebars within that Convo, each containing 2 users.
  • Convo 3 contains 3 users with no Sidebars.
  • Three (3) additional users (P, Q, and R) are not in a Convo.
  • All volume adjustment rules outlined so far may be combined (for example, in a multiplicative manner), such that for each user, every other user's volume is adjusted based on: (1) Spatial distance to that user, Orientation relative to that user; (2) Whether that user is in a Convo and whether the other users are in a Convo; or (3) Whether that user is in a Sidebar and whether the other users are in a Sidebar.
  • user A may hear the other users shown in Table 1:
  • Convos A user can start a new Convo with another user as long as neither of them is currently in a Convo.
  • Some examples by which a Convo can be started are:
  • a user spatially approaches an existing Convo and stops moving, which commences a countdown. As long as the user does not move for a predetermined time interval (for example, 3 seconds), they join the Convo.
  • a predetermined time interval for example, 3 seconds
  • a user locates another user in a directory table accessible through a UI interface and from there selects an option to join a Convo with that user.
  • a user locates the Convo in a directory table accessible through a UI interface and from there selects an option to join that Convo.
  • An algorithm is used to add users to an existing Convo, either based on random placement or some type of selection rule (for example, matching based on shared interests). Users inside an existing Convo may place (bring) a user currently outside of that Convo inside their Convo.
  • a user can start a new Sidebar with another user as long as both of them are in the same Convo and the other user is not currently in a Sidebar.
  • Some examples by which a Sidebar could be started are:
  • a user can join an existing Sidebar as long as that Sidebar is within the same Convo as that user.
  • Some examples by which a Sidebar could be joined are:
  • a user can exit a Sidebar if they are inside of a Sidebar.
  • Some examples by which a Sidebar could be exited are:
  • FIG. 7 users A, B, C, D & E are in a 5-person Convo, facing the center as shown by the arrows. This is a top-down (bird's eye view) of the Convo.
  • FIG. 8 illustrates the problem that arises without a subjective view
  • FIG. 9 we represent User C's field of view from a first-person perspective, represented by the cone shape filled with the diagonal-line pattern and dashed lines on the outside of it.
  • User C may not be able to see user users A or B, only D and E.
  • FIG. 10 illustrates what a wider field of view for User C might look like from a top-down view.
  • FIG. 11 illustrates what it might look like in a three-dimensional space from User C's first-person point of view.
  • the pattern with squares represents the area where the video tiles would be displayed, and arrows denote the direction that the video tiles would be tilted.
  • Users D and E may be visible (from C's perspective), with their videos somewhat tilted, and users A and B may be mostly outside of the field-of-view and their videos may be significantly tilted.
  • FIG. 13 illustrates an example of User A's subjective view, allowing User A to see Users D, E, B, and C's video tiles all appearing as if they are facing User A.
  • This subjective view may automatically adapt to accommodate more or fewer users. For example, if User E leaves, the view may adjust to a 4-person Convo subjective view. See, FIG. 14 .
  • FIGS. 12-15 are purely illustrative, and different variations can be taken in the shape and size of the video tiles, and the layout in which they are arranged. For example, instead of oval-shaped video tiles being arranged in a semi-circle, we might have square-shaped tiles arranged in a triangle layout. Regardless of which shape and layout is chosen, each user may have a subjective view of the other users' video tiles to enable them to see all the users in the same convo, and to have them all tilted in the direction of that user.
  • the subjective view may or may not be applied to a Convo with only a few participants (for example, a 2-person or 3-person Convo), as a few users may simply be able to face each other without the problems outlined earlier.
  • the other Convo rules (such as volume adjustments) may still be applied while allowing the few users to move around and turn freely, as long as certain factors such as their distance and orientation remain within a certain range.
  • a certain level of coordination and communication between users may help to ensure that Convos and Sidebars function as previously described, such that each user knows:
  • This information may be contained in Convo and Sidebar objects, where each object contains a unique ID for that Convo or Sidebar and a list of users which are part of that Convo or Sidebar.
  • the collection of this and other information, in some storage medium will be referred to as the ‘knowledge base’.
  • Each user and server, if a server is part of the architecture) may keep and maintain its own knowledge base.
  • Communication between users and/or server can happen via any number of web-based communication protocols (such as TCP, IP, HTTP, or the like), or any communication channels built on those protocols (such as web sockets, BOSH, REST, or the like).
  • web-based communication protocols such as TCP, IP, HTTP, or the like
  • any communication channels built on those protocols such as web sockets, BOSH, REST, or the like.
  • This communication and coordination may happen through a peer-to-peer architecture (each user communicates directly with every other user), through a client-server architecture (where each user is a client and the clients communicate and coordinate through a server, which then communicates and coordinates with the other users/clients), or through combination of the two (for example, some communication or coordination might happen through peer-to-peer implementation while a different communication or coordination might happen through a client-server implementation).
  • FIG. 16 is an illustrative diagram of a client-server architecture in which Users A, B, C, and D communicate and/or coordinate with each other through a Server.
  • the blocks represent the nodes (Server and clients) and the lines represent the connections for communication. For example, if User A wanted to communicate something to User C, that message may be sent from User A to the Server and then from the Server to User C. If any calculations are required, they may be performed by individual users or they may be performed by the Server and then the results may be communicated to the users.
  • FIG. 17 is an illustrative diagram of a peer-to-peer architecture in which users A, B, C, and D communicate and/or coordinate directly with each other.
  • this architecture if User A wanted to communicate something to User C, that message may be sent directly from User A to User C. If any calculations are required, they may either be performed by User A and the results may be communicated directly to user C, or they may be performed by user C after the needed input data is received from user A.
  • a ‘Start Convo’ mechanism may be triggered by any number of potential events as outlined earlier in this document. Once this trigger occurs, the implementations for peer-to-peer and client-server might be as follows:
  • Peer-to-peer implementation User A would send a request to User B to start a new Convo with User A. Upon receiving the request, User B would verify that they are not in another Convo and then create a new Convo object (represented by a unique ID) and would send that ID back to User A confirming the start of a new Convo between A and B. User B would then place themselves in a Convo with User A. Upon receiving the confirmation and the ID from User B, User A would also place themselves in a Convo with User B. Both users are now in a new Convo with the other user and they both have the same Convo ID for that Convo object.
  • a new Convo object represented by a unique ID
  • Client-server implementation User A would send a request to the Server to start a new Convo with User B.
  • the Server would check its knowledge base to verify that neither User A nor User B are currently in a Convo.
  • the Server Upon successfully verifying, the Server would create a new Convo object (represented by a unique ID) containing Users A and B and would send that ID out to both Users A and B instructing them to place themselves in a Convo and who the other user is in that Convo.
  • Users A and B Upon receiving the instructions from the Server, Users A and B would both place themselves in a Convo with the other user communicated by the Server. Both users are now in a new Convo with the other user and they both have the same Convo ID for that Convo object.
  • the Server also updates its own knowledge base with the new Convo ID and which users it contains. Along with the completion of this process, the Server would also send out a communication to all other users in the same session letting them know that there is a new Convo object, providing the ID for that object, and that it contains Users A and B. All other users would then update their knowledge base with the new information. See, FIG. 19 .
  • a ‘Join Convo’ mechanism may be triggered by any number of potential actions or events as outlined earlier in this document. Once this trigger occurs, the implementations for peer-to-peer and client-server might be as follows:
  • Peer-to-peer implementation User A would already have in its knowledge base (updated prior to this) that users B and C are in a Convo and what the Convo ID is. User A would place themselves inside the Convo with users B and C, and send a communication to both users B and C that User A is joining that Convo with the same ID. Upon receiving the communication, users B and C would update their Convo to include User A. User A would also send out a communication to all other users in the same session letting them know that that Convo object with that Convo ID now also includes User A, and all other users would update their knowledge base with the new information.
  • a race condition arises when the correct operation of a program or algorithm is dependent on the right sequence or timing of certain processes. For example, if User D joined the Convo between users B and C at the same time, or close to the same time, as User A, then potentially we could end up with inconsistent versions of the Convo object where:
  • Some users receive the message from User A before the message from User D. As a result, they may accept the request from User A to join the Convo, and may then reject the request from User D to join an existing Convo of users B and C because by the time that message is received, the Convo from those users' perspective now contains users A, B, and C (so it is impossible to add User D to a Convo of users B and C since that Convo no longer exists).
  • an extra handshake or verification step may be added such that a user (such as User B or C, or some other user) serves as the arbiter and confirms to either User A or User D that they are allowed to join the Convo before they join, and using that as a forcing function to prevent two or more changes happening concurrently.
  • a client-server implementation may resolve race conditions by using the Server as the arbiter and single source of truth regarding Convo objects.
  • Client-server implementation User A would already have in its knowledge base (updated prior to this) that users B and C are in a Convo and what the Convo ID is. User A would send a request to the Server to join the Convo. To prevent race conditions, the Server would first verify that no other users are in the process of joining or exiting the same Convo, or in the process of starting a Convo with User A. Upon verifying, the Server would add User A to the same Convo object and send an update to all users in the session that User A is now a part of that Convo. Upon receiving the update,
  • an ‘Exit Convo’ mechanism may be triggered by any number of potential actions or events, such as those previously outlined in this document. Once this trigger occurs, the implementations for peer-to-peer and client-server might be as follows:
  • Peer-to-peer implementation User A would send a communication to all other users in the session that User A is exiting the Convo object, along with the Convo ID. User A would then remove themselves from the Convo. Upon receiving the communication from User A:
  • User A would send a communication to the Server that User A is exiting the Convo object, along with the Convo ID. User A would then remove themselves from the Convo.
  • the Server Upon receiving the communication from User A, the Server would update the Convo object in its knowledge base such that it no longer contains User A. If there were only two users in that Convo object prior to User A exiting, the Convo object may be terminated and neither user would belong to a Convo any longer.
  • the Server would then send the update to all other users in the session with the new information.
  • all users Upon receiving the communication from the Server, all users would update their knowledge base and Convo objects accordingly.
  • all users are arranged in a pattern, such as a circle.
  • a pattern such as a circle.
  • the positioning of all the users in that Convo might need to be updated so that they are again arranged in the desired pattern (e.g., a circle) and facing in a desired direction (e.g., facing the center of a circle).
  • FIG. 20A shows how User B, C, D, and E might be placed and arranged prior to User A joining a Convo
  • FIG. 20B shows how they might need to be moved and reoriented after User A joins the Convo.
  • a set of calculations might need be performed every time there is a change in the number of users in a Convo, taking into account the following inputs: (i) Number of users in that Convo and their positions prior to the calculation; and (ii) Changes (adding or removing a user, and which user).
  • the output of the calculation might then include the users' new positions and orientations.
  • the implementation for peer-to-peer and client-server might be as follows.
  • Peer-to-peer implementation In a peer-to-peer implementation, we would have each user in a Convo perform these calculations independently. Given the same set of inputs and the same arithmetic to be applied, each user would arrive at the same output on their own and then position and turn themselves accordingly. In the example above, upon User A joining, User C would calculate User C's new position and orientation, User D would calculate User D's new position and orientation, and so on. Upon each user completing their own calculation, they would communicate their new position and orientation out to all other users in the session so that they can update their knowledge base. One way this may be implemented is shown in FIG. 21 .
  • Client-server implementation In a client-server implementation, the Server would perform the calculation once and communicate out the new positions and orientations to all other users. In the example above, upon User A joining, the Server would calculate new positions and orientations for Users A, B, C, D and E, and would then send those out to all users. Upon receiving the information from the Server:
  • a ‘Start Sidebar’ mechanism may be triggered by any number of potential events as outlined earlier in this document. Once this trigger occurs, the implementations for peer-to-peer and client-server might be as follows.
  • Peer-to-peer implementation User A would send a request to User B to start a new Sidebar with User A. Upon receiving the request, User B would verify that they are not already in another Sidebar and then create a new Sidebar object (represented by a unique ID) and would send that ID back to User A confirming the start of a new Sidebar between A and B. User B would then place themselves in a Sidebar with User A. Upon receiving the confirmation and the ID from User B, User A would also place themselves in a Sidebar with User B. Both users are now in a new Sidebar with the other user and they both have the same Convo ID for that Convo object.
  • either User A or User B would send out a communication to all other users in the same session letting them know that there is a new Sidebar object, providing the ID for that object, and that it contains users A and B. All other users would update their knowledge base with the new information.
  • Client-server implementation User A would send a request to the Server to start a new Sidebar with User B.
  • the Server would check its knowledge base to verify that User B is not currently in a Sidebar.
  • the Server Upon successfully verifying, the Server would create a new Convo object (represented by a unique ID) containing users A and B and would send that ID out to both users A and B instructing them to place themselves in a Sidebar and who the other user is in that Sidebar.
  • users A and B Upon receiving the instructions from the Server, users A and B would both place themselves in a Sidebar with the other user. Both users are now in a new Sidebar with the other user and they both have the same Sidebar ID for that Sidebar object.
  • the Server also updates its own knowledge base with the new Sidebar object and which users it contains.
  • the Server would also send out a communication to all other users in the same session letting them know that there is a new Sidebar object, providing the ID for that object, and that it contains users A and B. All other users would update their knowledge base with the new information.
  • a ‘Join Sidebar’ mechanism may be triggered by any number of potential events as outlined earlier in this document. Once this trigger occurs, the implementations for peer-to-peer and client-server might be as follows.
  • Peer-to-peer implementation User A would already have in its knowledge base (updated prior to this) that users B and C are in a Sidebar and what the Sidebar ID is. User A would place themselves inside the Sidebar with users B and C, and send a communication to both users B and C that User A is joining that Sidebar with the same ID. Upon receiving the communication, users B and C would update their Sidebar to include User A. User A would also send out a communication to all other users in the same session letting them know that that Sidebar object with that Sidebar ID now also includes User A, and all other users would update their knowledge base with the new information.
  • Client-server implementation User A would already have in its knowledge base (updated prior to this) that users B and C are in a Sidebar and what the Sidebar ID is. User A would send a request to the Server to join the Sidebar. To prevent race conditions, the Server would first verify that no other users are in the process of joining or exiting the same Sidebar, or in the process of starting a Sidebar with User A. Upon verifying, the Server would add User A to the same Sidebar object and send an update to all users in the session that User A is now a part of that Sidebar. Upon receiving the update,
  • an ‘Exit Sidebar’ mechanism may be triggered by any number of potential events as outlined earlier in this document. Once this trigger occurs, the implementations for peer-to-peer and client-server might be as follows.
  • Peer-to-peer implementation User A would send a communication to all other users in the session that User A is exiting the Sidebar object, along with the Sidebar ID. User A would then remove themselves from the Sidebar. Upon receiving the communication from User A:
  • Client-server implementation User A would send a communication to the Server that User A is exiting the Sidebar object, along with the Sidebar ID. User A would then remove themselves from the Sidebar. Upon receiving the communication from User A, the Server would update the Sidebar object in its knowledge base such that it no longer contains User A. If there were only two users in that Sidebar object prior to User A exiting, the Sidebar object may be terminated and neither user would belong to a Sidebar any longer. The Server would then send the update to all other users in the session with the new information, all users would update their knowledge base and Convo objects accordingly.
US17/562,649 2021-01-15 2021-12-27 Virtual conferencing system with layered conversations Active US11647123B2 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
US17/562,649 US11647123B2 (en) 2021-01-15 2021-12-27 Virtual conferencing system with layered conversations
US18/128,884 US20230239408A1 (en) 2021-01-15 2023-03-30 Virtual Conferencing System with Layered Conversations

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202163138231P 2021-01-15 2021-01-15
US17/562,649 US11647123B2 (en) 2021-01-15 2021-12-27 Virtual conferencing system with layered conversations

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/128,884 Continuation US20230239408A1 (en) 2021-01-15 2023-03-30 Virtual Conferencing System with Layered Conversations

Publications (2)

Publication Number Publication Date
US20220232128A1 true US20220232128A1 (en) 2022-07-21
US11647123B2 US11647123B2 (en) 2023-05-09

Family

ID=82405674

Family Applications (2)

Application Number Title Priority Date Filing Date
US17/562,649 Active US11647123B2 (en) 2021-01-15 2021-12-27 Virtual conferencing system with layered conversations
US18/128,884 Pending US20230239408A1 (en) 2021-01-15 2023-03-30 Virtual Conferencing System with Layered Conversations

Family Applications After (1)

Application Number Title Priority Date Filing Date
US18/128,884 Pending US20230239408A1 (en) 2021-01-15 2023-03-30 Virtual Conferencing System with Layered Conversations

Country Status (4)

Country Link
US (2) US11647123B2 (fr)
EP (1) EP4278272A1 (fr)
CA (1) CA3204966A1 (fr)
WO (1) WO2022154958A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230032545A1 (en) * 2021-07-30 2023-02-02 Verizon Patent And Licensing Inc. Independent Control of Avatar Location and Voice Origination Location within a Virtual Collaboration Space

Citations (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030001890A1 (en) * 2001-06-13 2003-01-02 Brin Glen David Interactive communication between a plurality of users
DE602004004824T2 (de) * 2003-02-28 2007-06-28 Palo Alto Research Center Inc., Palo Alto Automatische Behandlung von Konversationsgruppen
US20070282989A1 (en) * 2006-05-30 2007-12-06 Microsoft Corporation Proximity filtering of multiparty VoIP communications
US20090249228A1 (en) * 2008-03-26 2009-10-01 Li-Te Cheng Computer method and apparatus for persisting pieces of a virtual world group conversation
WO2010071984A1 (fr) * 2008-12-28 2010-07-01 Nortel Networks Limited Indication visuelle de contexte audio dans un environnement virtuel généré par un ordinateur
US7840668B1 (en) * 2007-05-24 2010-11-23 Avaya Inc. Method and apparatus for managing communication between participants in a virtual environment
US20120147123A1 (en) * 2010-12-08 2012-06-14 Cisco Technology, Inc. System and method for exchanging information in a video conference environment
US8243903B1 (en) * 2008-12-27 2012-08-14 Avaya Inc. Method and apparatus for implementing a secure side conversation on a telephone conference call
US20160142677A1 (en) * 2012-11-14 2016-05-19 Rounds Entertainment Ltd. Multi-User Interactive Virtual Environment Including Broadcast Content and Enhanced Social Layer Content
US20160269504A1 (en) * 2015-03-10 2016-09-15 Cisco Technology, Inc. System, method, and logic for generating graphical identifiers
US20200162616A1 (en) * 2018-11-20 2020-05-21 International Business Machines Corporation Haptic feedback during phone calls
US20200404031A1 (en) * 2015-05-18 2020-12-24 Twilio Inc. System and method for providing a media communication conversation service
US10979671B1 (en) * 2020-05-20 2021-04-13 Leo McElroy Internet communication system that modifies users' perceptions based on their proximity within a virtual space
US11070768B1 (en) * 2020-10-20 2021-07-20 Katmai Tech Holdings LLC Volume areas in a three-dimensional virtual conference space, and applications thereof
US20220086203A1 (en) * 2020-09-14 2022-03-17 NWR Corporation Systems and methods for teleconferencing virtual environments

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7933226B2 (en) 2003-10-22 2011-04-26 Palo Alto Research Center Incorporated System and method for providing communication channels that each comprise at least one property dynamically changeable during social interactions
US20120182384A1 (en) * 2011-01-17 2012-07-19 Anderson Eric C System and method for interactive video conferencing
US9191618B2 (en) * 2012-10-26 2015-11-17 Speedcast, Inc. Method and system for producing and viewing video-based group conversations

Patent Citations (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030001890A1 (en) * 2001-06-13 2003-01-02 Brin Glen David Interactive communication between a plurality of users
DE602004004824T2 (de) * 2003-02-28 2007-06-28 Palo Alto Research Center Inc., Palo Alto Automatische Behandlung von Konversationsgruppen
US20070282989A1 (en) * 2006-05-30 2007-12-06 Microsoft Corporation Proximity filtering of multiparty VoIP communications
US7840668B1 (en) * 2007-05-24 2010-11-23 Avaya Inc. Method and apparatus for managing communication between participants in a virtual environment
US20090249228A1 (en) * 2008-03-26 2009-10-01 Li-Te Cheng Computer method and apparatus for persisting pieces of a virtual world group conversation
US8243903B1 (en) * 2008-12-27 2012-08-14 Avaya Inc. Method and apparatus for implementing a secure side conversation on a telephone conference call
WO2010071984A1 (fr) * 2008-12-28 2010-07-01 Nortel Networks Limited Indication visuelle de contexte audio dans un environnement virtuel généré par un ordinateur
US20120147123A1 (en) * 2010-12-08 2012-06-14 Cisco Technology, Inc. System and method for exchanging information in a video conference environment
US20160142677A1 (en) * 2012-11-14 2016-05-19 Rounds Entertainment Ltd. Multi-User Interactive Virtual Environment Including Broadcast Content and Enhanced Social Layer Content
US20160269504A1 (en) * 2015-03-10 2016-09-15 Cisco Technology, Inc. System, method, and logic for generating graphical identifiers
US20200404031A1 (en) * 2015-05-18 2020-12-24 Twilio Inc. System and method for providing a media communication conversation service
US20200162616A1 (en) * 2018-11-20 2020-05-21 International Business Machines Corporation Haptic feedback during phone calls
US10979671B1 (en) * 2020-05-20 2021-04-13 Leo McElroy Internet communication system that modifies users' perceptions based on their proximity within a virtual space
US20220086203A1 (en) * 2020-09-14 2022-03-17 NWR Corporation Systems and methods for teleconferencing virtual environments
US11070768B1 (en) * 2020-10-20 2021-07-20 Katmai Tech Holdings LLC Volume areas in a three-dimensional virtual conference space, and applications thereof

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230032545A1 (en) * 2021-07-30 2023-02-02 Verizon Patent And Licensing Inc. Independent Control of Avatar Location and Voice Origination Location within a Virtual Collaboration Space
US11902766B2 (en) * 2021-07-30 2024-02-13 Verizon Patent And Licensing Inc. Independent control of avatar location and voice origination location within a virtual collaboration space

Also Published As

Publication number Publication date
EP4278272A1 (fr) 2023-11-22
US20230239408A1 (en) 2023-07-27
WO2022154958A1 (fr) 2022-07-21
US11647123B2 (en) 2023-05-09
CA3204966A1 (fr) 2022-07-21

Similar Documents

Publication Publication Date Title
US8082297B2 (en) Method and apparatus for managing communication between participants in a virtual environment
TWI504271B (zh) 會議中最相關人員的自動標識和表示
US8144633B2 (en) Method and system for controlling audio in a collaboration environment
US20100169796A1 (en) Visual Indication of Audio Context in a Computer-Generated Virtual Environment
JP7377911B2 (ja) プログラム、システム及びプログラム制御方法
US20230239408A1 (en) Virtual Conferencing System with Layered Conversations
US11184362B1 (en) Securing private audio in a virtual conference, and applications thereof
US10979671B1 (en) Internet communication system that modifies users' perceptions based on their proximity within a virtual space
US20220150288A1 (en) Information processing device and computer readable medium
US11743430B2 (en) Providing awareness of who can hear audio in a virtual conference, and applications thereof
US10440129B2 (en) Unified location and presence communication across real and virtual worlds
TW202036263A (zh) 設定多使用者虛擬實境聊天環境的方法與裝置
US10891014B2 (en) Remote view manipulation in communication session
JP2022111966A (ja) 音声及びビデオ会議アプリケーションの音量調節
US20240029339A1 (en) Multi-screen presentation in a virtual videoconferencing environment
EP3716091A1 (fr) Procédé et appareil pour la visualisation d'activités d'assistance sociale
JP3456419B2 (ja) 仮想空間内における共有空間表示方法及びシステム及び仮想空間内における共有空間表示プログラムを格納した記憶媒体
KR20220049907A (ko) 위치 기반 아바타 메신저 서비스 제공 방법 및 장치
US11741652B1 (en) Volumetric avatar rendering
JP7389286B1 (ja) プログラム、情報処理装置、および情報処理システム
US11748939B1 (en) Selecting a point to navigate video avatars in a three-dimensional environment
US11741664B1 (en) Resituating virtual cameras and avatars in a virtual environment
JP7143874B2 (ja) 情報処理装置、情報処理方法およびプログラム
KR20180001610A (ko) 퍼즐 매칭 게임 방법 및 이를 이용하여 퍼즐 매칭 게임을 진행시키는 게임 시스템
WO2022235916A1 (fr) Sécurisation de l'audio privé dans une conférence virtuelle, et ses applications

Legal Events

Date Code Title Description
FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY

AS Assignment

Owner name: MYCELIUM, INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:GLISIC, LEO, MR.;KATZ, JESSE, MR.;MALTZ, DSTIN, MR.;REEL/FRAME:058541/0928

Effective date: 20211228

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO SMALL (ORIGINAL EVENT CODE: SMAL); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STCF Information on status: patent grant

Free format text: PATENTED CASE