EP3533222A1 - Companion devices for real-time collaboration in teleconference sessions - Google Patents
Companion devices for real-time collaboration in teleconference sessionsInfo
- Publication number
- EP3533222A1 EP3533222A1 EP17794863.5A EP17794863A EP3533222A1 EP 3533222 A1 EP3533222 A1 EP 3533222A1 EP 17794863 A EP17794863 A EP 17794863A EP 3533222 A1 EP3533222 A1 EP 3533222A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- teleconference
- computing device
- streams
- data
- session
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/1066—Session management
- H04L65/1083—In-session procedures
- H04L65/1093—In-session procedures by adding participants; by removing participants
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/40—Support for services or applications
- H04L65/401—Support for services or applications wherein the services involve a main real-time session and one or more additional parallel real-time or time sensitive sessions, e.g. white board sharing or spawning of a subconference
- H04L65/4015—Support for services or applications wherein the services involve a main real-time session and one or more additional parallel real-time or time sensitive sessions, e.g. white board sharing or spawning of a subconference where at least one of the additional parallel sessions is real time or time sensitive, e.g. white board sharing, collaboration or spawning of a subconference
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/40—Support for services or applications
- H04L65/403—Arrangements for multi-party communication, e.g. for conferences
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/2866—Architectures; Arrangements
- H04L67/30—Profiles
- H04L67/303—Terminal profiles
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/52—Network services specially adapted for the location of the user terminal
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/75—Indicating network or usage conditions on the user display
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M3/00—Automatic or semi-automatic exchanges
- H04M3/42—Systems providing special services or facilities to subscribers
- H04M3/42229—Personal communication services, i.e. services related to one subscriber independent of his terminal and/or location
- H04M3/42263—Personal communication services, i.e. services related to one subscriber independent of his terminal and/or location where the same subscriber uses different terminals, i.e. nomadism
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M3/00—Automatic or semi-automatic exchanges
- H04M3/42—Systems providing special services or facilities to subscribers
- H04M3/56—Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities
- H04M3/568—Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M7/00—Arrangements for interconnection between switching centres
- H04M7/0024—Services and arrangements where telephone services are combined with data services
- H04M7/0027—Collaboration services where a computer is used for data transfer and the telephone is used for telephonic communication
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/141—Systems for two-way working between two video terminals, e.g. videophone
- H04N7/147—Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/15—Conference systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/0482—Interaction with lists of selectable items, e.g. menus
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/2866—Architectures; Arrangements
- H04L67/30—Profiles
- H04L67/306—User profiles
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M2203/00—Aspects of automatic or semi-automatic exchanges
- H04M2203/20—Aspects of automatic or semi-automatic exchanges related to features of supplementary services
- H04M2203/2094—Proximity
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M3/00—Automatic or semi-automatic exchanges
- H04M3/42—Systems providing special services or facilities to subscribers
- H04M3/56—Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities
- H04M3/563—User guidance or feature selection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M3/00—Automatic or semi-automatic exchanges
- H04M3/42—Systems providing special services or facilities to subscribers
- H04M3/56—Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities
- H04M3/563—User guidance or feature selection
- H04M3/564—User guidance or feature selection whereby the feature is a sub-conference
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/141—Systems for two-way working between two video terminals, e.g. videophone
- H04N7/142—Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
- H04N2007/145—Handheld terminals
Definitions
- Communication and collaboration are key aspects in people's lives, both socially and in business.
- Communication and collaboration tools have been developed with the aim of connecting people to share experiences.
- the aim of these tools is to provide, over a network, an experience which mirrors real life interaction between individuals and groups of people. Interaction is typically provided by audio and/or visual elements.
- Such tools include instant messaging, voice calls, video calls, group chat, shared desktop, shared media and content, shared applications, etc.
- Such tools can perform capture, manipulation, transmission, and reproduction of audio and visual elements, and use various combinations of such elements in an attempt to provide a collaborative environment.
- a user can access such tools at a user device which may be provided by a laptop or desktop computer, mobile phone, tablet, games console, etc.
- Such devices can be linked in a variety of possible network architectures, such as peer-to-peer architectures or client-server architectures or a hybrid, such as a centrally managed peer-to-peer architecture. In general, some or all of these architectures can facilitate a teleconference session to attempt to enable rich communication and collaboration somewhat similar to an in-person collaboration meeting.
- the present disclosure provides a number of techniques for enhancing a user's experience when joining teleconference sessions with multiple devices.
- a system differentiates the devices as a primary device and at least one companion device.
- the primary device has a first control set for controlling a teleconference session and the at least one companion device has a companion control set for sharing content.
- the primary device also has one set of selected streams, e.g., a stage view, and the companion device has a subset of those streams or other streams based on an activity level.
- the present disclosure provides a number of techniques for enabling users to readily share content using the companion devices.
- a teleconference system can generate teleconference data comprising a plurality of streams associated with a teleconference session.
- the teleconference system can also receive a first request to join the teleconference session from a first computing device associated with a first user identity.
- the first user identity may be associated with a particular user or participant of the teleconference session.
- the request can be instantiated at the first computing device using a user interface.
- the first computing device can be a primary device configured to anchor an immersive teleconference experience.
- the teleconference system can communicate the teleconference data to the first computing device for displaying, or causing a display of, a primary user interface.
- the primary user interface includes a rendering of the plurality of streams.
- the rendering of the plurality of streams may include one or more renderings of individual streams arranged within a user interface.
- This display can be referred to herein as a "stage" view of a teleconference session.
- the teleconference system can also cause the first computing device to display a first control set for controlling aspects of the teleconference session.
- the first control set also referred to herein as "core controls,” can be used to and exit the session, mute the sound of the session, etc.
- the teleconference system can also receive a second request to join the teleconference session from a second computing device associated with the first user identity.
- the second computing device can be used by the same user associated with the first user identity.
- the teleconference system can cause the second computing device to display a secondary user interface having companion controls for sharing content.
- the teleconference system can cause the second computing device to display a selection of streams that can be arranged to present a reduced or otherwise streamlined portion of teleconference data for mitigating or avoiding distractions, interference, and other elements that can disrupt the teleconference session.
- the teleconference system can select at least one stream of the plurality of streams based on an activity level or other values. Having selected the at least one stream, the teleconference system can communicate the teleconference data to the second computing device causing the second computing device to display a companion user interface.
- the companion user interface can include a rendering of the at least one stream.
- the first device may receive teleconference data having the plurality of streams
- the second device may be configured to only receive a sub-portion, or a selected at least one stream.
- a teleconference system can also receive requests to join a teleconference session from a plurality of computing devices.
- the teleconference system can generate teleconference data associated with the teleconference session and distribute the teleconference data to primary and companion devices similarly as described above, or in any desired order.
- the teleconference system can differentiate across multiple devices joining a teleconference session to identify multiple devices associated with a single user identity. Thereafter, the teleconference system can communicate the teleconference data for causing a display of the plurality of streams on a primary device associated with a user and communicate teleconference data for causing a display of at least one selected stream in companion devices associated with the user.
- a first computing device can transmit a first request to join a teleconference session.
- the first computing device can be associated with a first user identity.
- the first computing device responsive to the request, can receive first teleconference data comprising a plurality of streams associated with the teleconference session.
- the first computing device can also display a rendering of the plurality of streams.
- a second computing device can transmit a second request to join the teleconference session.
- the second computing device can also be associated with the first user identity. Responsive to the second request, the second computing device can receive second teleconference data comprising a subset of streams of the plurality of streams.
- the second computing device can also display a rendering of the subset of streams of the plurality of streams.
- the subset of streams of the plurality of streams can be based at least in part on data indicating an activity level associated with the subset of streams.
- the user can also utilize control commands that are associated with a primary device and sharing commands that are associated with a companion device.
- the user can reconfigure the commands on each device in response to a user providing an input command, such as a reconfigure command.
- the teleconference system can facilitate a "device swap" by effectively switching primary and companion devices for the user. Reverse swaps are also possible in some implementations.
- the user can also utilize control commands and reconfigure commands to allow devices to seamlessly "drop out" of a teleconference session.
- the teleconference system can receive a reconfiguring command for causing a transition to the first computing device from the second computing device.
- the teleconference system can discontinue communicating the teleconference data to the second computing device.
- the second computing device may drop out, and the first computing device may take over as a sole primary device associated with the first user identity.
- commands for enriching a user experience are also disclosed.
- these commands can allow limiting or augmenting any particular device from one mode of operation to another mode of operation.
- two or more modes of operation may be established by the teleconference system for the first user identity.
- These modes of operation can include, for example, a primary mode of operation and a companion mode of operation. Therefore, once a device enters a teleconference session, different types of contextual user interfaces can be displayed for allowing a user to access and share relevant information based on these modes of operation.
- a companion control menu can be displayed for enabling a companion device to be used to readily select various types of media data to be shared.
- a primary control menu can be displayed for enabling control of the primary device.
- the combination of the control menus and the use of the modes of operation enable a user to join the teleconference session in a way that reduces distractions while also allowing the user to select and share the most relevant content across more than one device.
- the teleconference system can also manipulate components of the primary and companion devices to enhance the teleconference session and limit disruption.
- the teleconference system can remotely disable a camera component of one or more computing devices such that only a single "primary" or "face view” camera is associated with a single user identity.
- the teleconference system can remotely disable an audio component of one or more devices to limit audio feedback from the user's devices.
- Other manipulation of these components can also be possible, for example, through selective activation to allow a user to use a "roving camera" on a companion device to share a unique view from a companion device that is separate and unique from a camera view on the primary device.
- the technologies described herein related to the teleconference system can be combined in multiple manners to enable participation of users in a variety of scenarios including scenarios where users have multiple devices which are used to enrich a teleconference session.
- FIG. 1 is a diagram illustrating an example environment in which a system can operate to facilitate companion devices for real-time collaboration in teleconference sessions.
- FIG. 2 is a diagram illustrating components of an example computing device configured to facilitate companion devices for real-time collaboration in teleconference sessions.
- FIGS. 3A-3D illustrate several example scenarios where computing devices can be utilized to enable the techniques disclosed herein.
- FIG. 4 illustrates an example user interface arrangement of a primary computing device
- FIGS. 5A-5B illustrate several example user interface arrangements of a companion computing device
- FIGS. 6A-6C illustrate several example user interface arrangements of different forms of companion computing devices.
- FIGS. 7A-7D illustrate several example user interface arrangements with shared content using a roving camera view of a companion computing device.
- FIG. 8 is a flowchart of an example method of facilitating companion devices for real-time collaboration in teleconference sessions.
- a teleconference system Prior to joining a teleconference session, can generate teleconference data comprising a plurality of streams associated with a teleconference session.
- the teleconference system can also receive a first request tojoin the teleconference session from a first computing device associated with a first user identity.
- the first user identity may be associated with a particular user or participant of the teleconference session.
- the request can be instantiated at the first computing device using a user interface.
- the first computing device can be a primary device configured to anchor an immersive teleconference experience.
- the teleconference system can communicate the teleconference data to the first computing device for displaying, or causing to display, a primary user interface.
- the primary user interface includes a rendering of the plurality of streams.
- the rendering of the plurality of streams may include one or more renderings of individual streams arranged within a user interface.
- the first computing device can display renderings of one or more streams associated with the teleconference data.
- the teleconference system can also receive a second request to join the teleconference session from a second computing device associated with the first user identity.
- the second computing device can be used by the same user associated with the first user identity.
- the second computing device can be similar in arrangement to the first computing device, or can be a different form of computing device.
- the second computing device can be a computing device, such as a companion device, arranged to present a reduced or otherwise streamlined portion of teleconference data while avoiding distractions, interference, and other elements that can disrupt the teleconference session.
- the teleconference system can select at least one stream of the plurality of streams based on an activity level or other values. Having selected the at least one stream, the teleconference system can communicate the teleconference data to the second computing device causing the second computing device to display a companion user interface.
- the companion user interface can include a rendering of the at least one stream.
- the first device may receive teleconference data having a plurality of streams
- the second device may be configured to only receive a sub-portion, or a selected at least one stream.
- the teleconference system can overcome many technical drawbacks associated with conventional teleconference systems.
- a teleconference system can also receive requests to join a teleconference session from a plurality of computing devices.
- the teleconference system can generate teleconference data associated with the teleconference session and distribute the teleconference data to primary and companion devices similarly as described above, or in any desired order.
- the teleconference system can differentiate across multiple devices joining a teleconference session to identify multiple devices associated with a single user identity. Thereafter, the teleconference system can communicate the teleconference data for causing to display a rendering of the plurality of streams and communicate teleconference data for causing to display a rendering of at least one selected stream.
- a first computing device can transmit a first request to join a teleconference session. The first computing device can be associated with a first user identity. The first computing device, responsive to the request, can receive first teleconference data comprising a plurality of streams associated with the teleconference session. The first computing device can also display a rendering of the plurality of streams.
- a second computing device can transmit a second request to join the teleconference session.
- the second computing device can also be associated with the first user identity. Responsive to the second request, the second computing device can receive second teleconference data comprising a subset of streams of the plurality of streams.
- the second computing device can also display a rendering of the subset of streams of the plurality of streams.
- the subset of streams of the plurality of streams can be based at least in part on data indicating an activity level associated with the subset of streams.
- the user can also utilize control commands and reconfigure commands to transfer association as primary and companion modes of operation between the devices.
- the teleconference system can communicate different teleconference data to each device based on the control and reconfigure commands. Accordingly, the teleconference system can facilitate a "device swap" by effectively switching primary and companion modes of operation for devices for the user. Reverse swaps are also possible in some implementations.
- the user can also utilize control command and reconfigure commands to allow devices to seamlessly "drop out" of a teleconference session.
- the teleconference system can receive a reconfiguring command for causing a transition to the first computing device from the second computing device.
- the teleconference system can discontinue communicating the teleconference data to the second computing device.
- the second computing device may drop out, and the first computing device may take over as a sole primary device associated with the first user identity.
- commands for enriching a user experience are also disclosed.
- these commands can allow limiting or augmenting any particular device from a mode of operation to another mode of operation.
- two or more modes of operation may be established by the teleconference system for the first user identity.
- These modes of operation can include, for example, a primary mode of operation and a companion mode of operation. Therefore, once a device enters a teleconference session, different types of contextual user interfaces can be displayed for allowing a user to access and share relevant information based on these modes of operation.
- a companion control menu can be displayed for enabling a companion device to be used to readily select various types of media data to be shared.
- a primary control menu can be displayed for enabling control of the primary device.
- the combination of the control menus and the use of the modes of operation enable a user to join the teleconference session in a way that reduces distractions while also allowing the user to select and share the most relevant content across more than one device.
- the teleconference system can also manipulate components of the primary and companion devices to enhance the teleconference session and limit disruption.
- the teleconference system can remotely disable a camera component of one or more computing devices such that only a single "primary" or "face view” camera is associated with a single user identity.
- the teleconference system can remotely disable an audio component of one or more devices to limit audio feedback from the user's devices.
- Other manipulation of these components can also be possible, for example, through selective activation to allow a user to use a "roving camera" on a companion device to share a unique view from a companion device that is separate and unique from a camera view on the primary device.
- the technologies described herein related to the teleconference system can be combined in multiple manners to enable participation of users in a variety of scenarios including scenarios where users have multiple devices used to join a teleconference session, while avoiding duplicative experiences, increasing efficient use of resources, and enriching the teleconference session.
- scenarios including scenarios where users have multiple devices used to join a teleconference session, while avoiding duplicative experiences, increasing efficient use of resources, and enriching the teleconference session.
- FIGS. 1-8 Various examples, scenarios, and aspects are described below with reference to FIGS. 1-8.
- FIG. 1 a diagram illustrating an example of a teleconference system 100 is shown in which a system 102 can operate to provide a teleconference session 104 in accordance with an example implementation.
- the teleconference session 104 is between a number of client computing devices 106(1) through 106(N) (where N is a positive integer number having a value of two or greater).
- the client computing devices 106(1) through 106(N) enable users to participate in the teleconference session 104.
- the teleconference session 104 may be hosted, over one or more network(s) 108, by the system 102.
- the system 102 may provide a service that enables users of the client computing devices 106(1) through 106(N) to participate in the teleconference session 104.
- the teleconference session 104 may be hosted by one of the client computing devices 106(1) through 106(N) utilizing peer-to-peer technologies.
- the system 102 includes device(s) 110 and the device(s) 110 and/or other components of the system 102 may include distributed computing devices that communicate with one another, with the system 102, and/or with the client computing devices 106(1) through 106(N) via the one or more network(s) 108.
- the system 102 may be an independent system that is tasked with managing aspects of one or more teleconference sessions 104.
- the system 102 may be managed by entities such as SLACK®, WEBEX®, GOTOMEETING®, GOOGLE HANGOUTS®, etc.
- Network(s) 108 may include, for example, public networks such as the Internet, private networks such as an institutional and/or personal intranet, or some combination of private and public networks.
- Network(s) 108 may also include any type of wired and/or wireless network, including but not limited to local area networks ("LANs”), wide area networks ("WANs”), satellite networks, cable networks, Wi-Fi networks, WiMax networks, mobile communications networks (e.g., 3G, 4G, and so forth) or any combination thereof.
- Network(s) 108 may utilize communications protocols, including packet-based and/or datagram-based protocols such as Internet protocol (“IP”), transmission control protocol (“TCP”), user datagram protocol (“UDP”), or other types of protocols.
- IP Internet protocol
- TCP transmission control protocol
- UDP user datagram protocol
- network(s) 108 may also include a number of devices that facilitate network communications and/or form a hardware basis for the networks, such as switches, routers, gateways, access points, firewalls, base stations, repeaters, back
- network(s) 108 may further include devices that enable connection to a wireless network, such as a wireless access point ("WAP").
- WAP wireless access point
- Examples support connectivity through WAPs that send and receive data over various electromagnetic frequencies (e.g., radio frequencies), including WAPs that support Institute of Electrical and Electronics Engineers (“IEEE”) 802.11 standards (e.g., 802.1 lg, 802.1 In, and so forth), and other standards.
- IEEE Institute of Electrical and Electronics Engineers
- device(s) 110 may include one or more computing devices that operate in a cluster or other grouped configuration to share resources, balance load, increase performance, provide fail-over support or redundancy, or for other purposes.
- device(s) 110 may belong to a variety of classes of devices such as traditional server-type devices, desktop computer-type devices, and/or mobile-type devices.
- a server-type device device(s) 110 may include a diverse variety of device types and are not limited to a particular type of device.
- Device(s) 110 may represent, but are not limited to, server computers, desktop computers, web-server computers, personal computers, mobile computers, laptop computers, mobile phones, tablet computers, or any other sort of computing device.
- a client computing device may belong to a variety of classes of devices, which may be the same as, or different from, device(s) 110, such as traditional client-type devices, desktop computer-type devices, mobile-type devices, special purpose-type devices, embedded-type devices, and/or wearable-type devices.
- a client computing device can include, but is not limited to, a desktop computer, a game console and/or a gaming device, a tablet computer, a personal data assistant ("PDA”), a mobile phone/tablet hybrid, a laptop computer, a telecommunication device, a computer navigation type client computing device such as a satellite-based navigation system including a global positioning system (“GPS”) device, a wearable device, a virtual reality (“VR”) device, an augmented reality (AR) device, an implanted computing device, an automotive computer, a network-enabled television, a thin client, a terminal, an Internet of Things (“IoT”) device, a work station, a media player, a personal video recorder (“PVR”), a set-top box, a camera, an integrated component (e.g., a peripheral device) for inclusion in a computing device, an appliance, or any other sort of computing device.
- GPS global positioning system
- VR virtual reality
- AR augmented reality
- IoT Internet of Things
- a client computing device includes input/output (“I/O") interfaces that enable communications with input/output devices such as user input devices including peripheral input devices (e.g., a game controller, a keyboard, a mouse, a pen, a voice input device, a touch input device, a gestural input device, and the like) and/or output devices including peripheral output devices (e.g., a display, a printer, audio speakers, a haptic output device, and the like).
- peripheral input devices e.g., a game controller, a keyboard, a mouse, a pen, a voice input device, a touch input device, a gestural input device, and the like
- peripheral output devices e.g., a display, a printer, audio speakers, a haptic output device, and the like.
- Client computing device(s) 106(1) through 106(N) of the various classes and device types can represent any type of computing device having one or more processing unit(s) 112 operably connected to computer-readable media 114 such as via a bus 116, which in some instances can include one or more of a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and any variety of local, peripheral, and/or independent buses.
- a bus 116 which in some instances can include one or more of a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and any variety of local, peripheral, and/or independent buses.
- Executable instructions stored on computer-readable media 114 may include, for example, an operating system 128, a client module 130, a profile module 132, and other modules, programs, or applications that are loadable and executable by processing units(s) 112.
- Client computing device(s) 106(1) through 106(N) may also include one or more interface(s) 134 to enable communications with other input devices 148 such as network interfaces, cameras, keyboards, touch screens, and pointing devices (mouse).
- the interface(s) 134 can also enable communications between client computing device(s) 106(1) through 106(N) and other networked devices, such as device(s) 110 and/or devices of the system 102, over network(s) 108.
- Such network interface(s) 134 may include one or more network interface controllers (NICs) or other types of transceiver devices to send and receive communications and/or data over a network.
- NICs network interface controllers
- client computing devices 106(1) through 106(N) may use their respective client modules 130 to connect with one another and/or other external device(s) in order to participate in the teleconference session 104.
- client computing devices 106(1) through 106(N) may use their respective client modules 130 to connect with one another and/or other external device(s) in order to participate in the teleconference session 104.
- a first user may utilize a client computing device 106(1) to communicate with a second user of another client computing device 106(2).
- the users may share data, which may cause the client computing device 106(1) to connect to the system 102 and the other client computing devices 106(2) through 106(N) to connect over the network 108.
- the client module 130 of each client computing device 106(1) through 106(N) may include logic that detects user input and communicates control signals to the server module 136 to request changes in the teleconference session 104 on the display.
- the client module 130 in the first client computing device 106(1) in FIG. 1 may detect a user input at an input device 148.
- the user input may be sensed, for example, as a finger press on a user interface element displayed on a touchscreen (e.g., touchscreen 150(2)), or as a click of a mouse on a user interface element selected by a pointer on the display 150.
- the client module 130 translates the user input according to a function associated with the selected user interface element.
- the client module 130 may require the teleconference session 104 to perform a function.
- the client module 130 sends a control signal 156(1) to the server module 136 for service by the teleconference session 104.
- the user of the client computing device 106(1) may wish to cause a transition from a first mode of operation of the client computing device 106(1) to a second mode of operation, for the teleconference session 104.
- the user may click on the desired user interface element on the user's display 150.
- the client module 130 sends the control signal 156(1) to the server module 136.
- the server module 136 would, in response to the control signal 156(1) perform the desired transition based on the teleconference data 146(1), data indicating locations of one or more of the other computing devices 106, and other suitable information.
- the client computing device(s) 106(1) -through- 106(N) may use their respective profile modules 132 to generate participant profiles, and provide the participant profiles to other client computing devices and/or to the device(s) 110 of the system 102.
- a participant profile may include one or more of an identity of a participant (e.g., a name, user identity, a unique identifier ("ID"), etc.), participant data, such as personal data and location data may also be stored.
- Participant profiles may be utilized to register participants for teleconference sessions 104 and to indicate priority or preferences associated with a user identity and/or client computing device.
- the device(s) 110 of the system 102 includes a server module 136, a data store 138, and an output module 140.
- the server module 136 is configured to receive, from individual client computing devices 106(1) through 106(N), stream 142(1) through 142(M) (where M is a positive integer number equal to 2 or greater).
- M the number of instances submitted
- N the number of client computing devices.
- one or more of the client computing devices may be communicating an additional stream or transmission of media data that includes content, such as a document or other similar type of media intended to be shared during the teleconference session 104.
- the server module 136 is also configured to receive, generate, and communicate session data 144 and to store the session data 144 in the data store 138.
- the server module 136 may select aspects of the streams 142 that are to be shared with the client computing devices 106(1) through 106(N).
- the server module 136 may combine the streams 142 to generate teleconference data 146 defining aspects of the teleconference session 104.
- the teleconference data 146 can comprise select streams 142.
- the teleconference data 146 can define aspects of the teleconference session 104, such as a user interface arrangement of the user interfaces on the client computing devices 106, the type of data that is displayed, and other functions of the server and clients.
- the server module 136 may configure the teleconference data 146 for the individual client computing devices 106(1) through 106(N).
- Teleconference data 146 can include individual instances referenced as 146(1) through 146(N).
- teleconference data 146 can include first teleconference data 142(1) for communication with a primary computing device, and second teleconference data 142(2) through 142(N) for communication with a companion computing device.
- the output module 140 may communicate the teleconference data instances 146(1) through 146(N) to the client computing devices 106(1) through 106(N).
- the output module 140 communicates teleconference data 146(1) to client computing device 106(1), teleconference data 146(2) to client computing device 106(2), teleconference data 146(3) to client computing device 106(3), and teleconference data 146(N) to client computing device 106(N), respectively.
- the teleconference data instances 146(1) through 146(N) may communicate audio that may include video representative of the contribution of each participant in the teleconference session 104.
- Each teleconference data instance 146(1) through 146(N) may also be configured in a manner that is unique to the needs of each participant user of the client computing devices 106(1) through 106(N).
- Each client computing device 106(1) through 106(N) may be associated with a teleconference session view. Examples of the use of teleconference session views to control the views for each participant at the client computing devices are described with reference to FIG. 2.
- FIG. 2 a system block diagram is shown illustrating components of an example device 200 configured to provide the teleconference session 104 between the client computing devices, such as client computing devices 106(1) through 106(N) in accordance with an example implementation.
- the device 200 can be used to illustrate some components of one of the client computing devices 106.
- the device 200 may represent one of device(s) 110 where the device 200 includes one or more processing unit(s) 202, computer-readable media 204, and communication interface(s) 206.
- the components of the device 200 are operatively connected, for example, via a bus 207, which may include one or more of a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and any variety of local, peripheral, and/or independent buses.
- a bus 207 may include one or more of a system bus, a data bus, an address bus, a PCI bus, a Mini-PCI bus, and any variety of local, peripheral, and/or independent buses.
- processing unit(s) such as the processing unit(s) 202 and/or processing unit(s) 112 may represent, for example, a CPU-type processing unit, a GPU- type processing unit, a field-programmable gate array (“FPGA"), another class of digital signal processor (“DSP”), or other hardware logic components that may, in some instances, be driven by a CPU.
- FPGA field-programmable gate array
- DSP digital signal processor
- illustrative types of hardware logic components include Application-Specific Integrated Circuits ("ASICs”), Application-Specific Standard Products (“ASSPs”), System-on-a-Chip Systems (“SOCs”), Complex Programmable Logic Devices (“CPLDs”), etc.
- ASICs Application-Specific Integrated Circuits
- ASSPs Application-Specific Standard Products
- SOCs System-on-a-Chip Systems
- CPLDs Complex Programmable Logic Devices
- computer-readable media such as computer-readable media 204 and/or computer-readable media 114, may store instructions executable by the processing unit(s).
- the computer-readable media may also store instructions executable by external processing units such as by an external CPU, an external GPU, and/or executable by an external accelerator, such as an FPGA type accelerator, a DSP type accelerator, or any other internal or external accelerator.
- an external CPU such as by an external CPU, an external GPU, and/or executable by an external accelerator, such as an FPGA type accelerator, a DSP type accelerator, or any other internal or external accelerator.
- an external accelerator such as an FPGA type accelerator, a DSP type accelerator, or any other internal or external accelerator.
- at least one CPU, GPU, and/or accelerator is incorporated in a computing device, while in some examples one or more of a CPU, GPU, and/or accelerator is external to a computing device.
- Computer-readable media may include computer storage media and/or communication media.
- Computer storage media may include one or more of volatile memory, nonvolatile memory, and/or other persistent and/or auxiliary computer storage media, removable and non-removable computer storage media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data.
- computer storage media includes tangible and/or physical forms of media included in a device and/or hardware component that is part of a device or external to a device, including but not limited to random access memory (“RAM”), static random-access memory (“SRAM”), dynamic random-access memory (“DRAM”), phase change memory (“PCM”), read-only memory (“ROM”), erasable programmable read-only memory (“EPROM”), electrically erasable programmable read- only memory (“EEPROM”), flash memory, compact disc read-only memory (“CD-ROM”), digital versatile disks (“DVDs”), optical cards or other optical storage media, magnetic cassettes, magnetic tape, magnetic disk storage, magnetic cards or other magnetic storage devices or media, solid-state memory devices, storage arrays, network attached storage, storage area networks, hosted computer storage or any other storage memory, storage device, and/or storage medium that can be used to store and maintain information for access by a computing device.
- RAM random access memory
- SRAM static random-access memory
- DRAM dynamic random-access memory
- PCM phase change
- communication media may embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave, or other transmission mechanism.
- computer storage media does not include communication media. That is, computer storage media does not include communications media consisting solely of a modulated data signal, a carrier wave, or a propagated signal, per se.
- Communication interface(s) 206 may represent, for example, network interface controllers ("NICs") or other types of transceiver devices to send and receive communications over a network.
- NICs network interface controllers
- computer-readable media 204 includes the data store 138.
- the data store 138 includes data storage such as a database, data warehouse, or other type of structured or unstructured data storage.
- the data store 138 includes a corpus and/or a relational database with one or more tables, indices, stored procedures, and so forth to enable data access including one or more of hypertext markup language (“HTML”) tables, resource description framework (“RDF”) tables, web ontology language (“OWL”) tables, and/or extensible markup language (“XML”) tables, for example.
- HTML hypertext markup language
- RDF resource description framework
- OWL web ontology language
- XML extensible markup language
- the data store 138 may store data for the operations of processes, applications, components, and/or modules stored in computer-readable media 204 and/or executed by processing unit(s) 202 and/or accelerator(s). For instance, in some examples, the data store 138 may store session data 144, profile data 210, and/or other data.
- the session data 208 may include a total number of users in the teleconference session 104, and activity that occurs in the teleconference session 104 (e.g., behavior, activity of the participants), and/or other data related to when and how the teleconference session 104 is conducted or hosted.
- profile data 210 include, but are not limited to, a user identity ("ID"), priority value, and other data.
- the data store 138 stores data related to the view each user experiences on the display of the users' client computing devices 106.
- the data store 138 may include a teleconference session mode 250(1) through 250(N) corresponding to the operating mode of each client computing device 106(1) through 106(N) participating in the teleconference session 104.
- the teleconference session mode 250 may also be termed a "mode of operation" herein.
- the teleconference system 102 may support individual control over the view and mode each user experiences across multiple devices during the teleconference session 104. For example, as described in more detail below, the system 102 permits a user to participate with the user's client computing device 106 operating in one of several modes of operation.
- the modes of operation can include, for example, a primary mode of operation and a companion mode of operation.
- a computing device 106 enters the teleconference session 104, different types of contextual user interfaces can be displayed on the display 150 for enabling users to access and share relevant information based on these modes of operation.
- a companion control menu can be displayed for enabling a user to readily select various types of media data to be shared by a user's companion computing device.
- a primary control menu can be displayed for enabling control of a state of the teleconference session 104 at a user's primary computing device.
- control menus allow users to join the teleconference session 104 with multiple devices in a manner that reduces distractions while also allowing users to select and share the most relevant content.
- a primary device can be selected by the teleconference system 102 to operate in the primary mode of operation.
- one or more companion devices can be selected by the teleconference system 102 to operate in the companion mode of operation.
- a primary device can be selected based on an order of the devices attempting to join the teleconference session 104, when associated with a single user identity.
- a user's first device to attempt to join can be selected as the primary device.
- the teleconference system 102 can analyze priority values associated with one or more computing devices to determine that a user's first computing device has a priority value superseding priority values of the other computing devices associated with the same user identity.
- the teleconference system 102 can also manipulate components of companion devices to enhance the teleconference session 104 and limit disruption.
- the teleconference system 102 can remotely disable a camera component (e.g., input devices 148) of companion computing devices.
- the teleconference system 102 can remotely disable an audio component (e.g., input device 148) of companion computing devices.
- Other manipulation of these components can also be possible, for example, through selective activation to allow a user to "take the stage" or share content using a companion device rather than a primary device.
- the data store 138 may store the profile data 210, streams 142, teleconference session modes 250, teleconference data 146, and session data 144. Alternately, some or all of the above-referenced data can be stored on separate memories 224 on board one or more processing unit(s) 202 such as a memory on board a CPU-type processor, a GPU-type processor, an FPGA-type accelerator, a DSP-type accelerator, and/or another accelerator.
- processing unit(s) 202 such as a memory on board a CPU-type processor, a GPU-type processor, an FPGA-type accelerator, a DSP-type accelerator, and/or another accelerator.
- the computer-readable media 204 also includes an operating system 226 and application programming interface(s) 228 configured to expose the functionality and the data of the device(s) 110 (e.g., example device 200) to external devices associated with the client computing devices 106(1) through 106(N).
- the computer-readable media 204 includes one or more modules such as the server module 136 and an output module 140, although the number of illustrated modules is just an example, and the number may vary higher or lower. That is, functionality described herein in association with the illustrated modules may be performed by a fewer number of modules or a larger number of modules on one device or spread across multiple devices.
- the teleconference system 102 is configured to host the teleconference session 104 with the plurality of client computing devices 106(1) through 106(N).
- the teleconference system 102 includes one or more processing units 202 and a computer-readable medium 204 having encoded thereon computer-executable instructions to cause the one or more processing units 202 to receive streams 142(1) through 142(M) at the system 102 from a plurality of client computing devices 106(1) through 106(N), select streams 142 based, at least in part, on the teleconference session mode 250 for each user's computing device, and communicate a teleconference data 146 according to the teleconference session modes 250 corresponding to the client computing devices 106(1) through 106(N).
- the teleconference data 146(1) through 146(N) are communicated from the system 102 to the plurality of client computing devices 106(1) through 106(N).
- the teleconference session modes 250(1) through 250(N) cause the plurality of client computing devices 106(1) through 106(N) to display views of the teleconference session 104 under user control.
- the computer-executable instructions also cause the one or more processing units 202 to determine that the teleconference session 104 is to transition client computing devices 106(1) through 106(N) to a different teleconference session mode 250 based on a user communicated control signal 156, also referred to herein as a control command 156, and other suitable information.
- the control command 156 comprises at least one of a user identity, a request to join a teleconference session 104, an access request, a transfer request, or other data described herein.
- the techniques disclosed herein may utilize one or more predetermined modes of operation, also referred to as "modes,” or "teleconference session modes.”
- the system 102 performs a method that includes receiving the streams 142(1) through 142(M) at the system 102 from a plurality of client computing devices 106(1) through 106(N), combining and formatting the streams 142 based, at least in part, on user identities for each client computing device to form a teleconference data 146(1) through 146(N), and transmitting a teleconference data 146(1) through 146(N) to respective client computing devices.
- the predetermined modes of operation can include a first mode of operation (referred to herein as a "primary”) and a second mode of operation (referred to herein as a "companion").
- the primary and companion modes of operation can be configured automatically through intelligent selection of a primary and companion device using profile data accessible by the teleconference system 102.
- the predetermined modes of operation can also facilitate graphical elements providing control functionality ("control elements") for a teleconference session 104. For instance, a graphical element may be generated on the user interface enabling a user to provide content, end a session, mute one or more sounds, control other participants' streams, transition a particular device to a different mode of operation, etc.
- the techniques disclosed below can utilize a primary mode of operation.
- the primary mode of operation may be a "fully functional" form of operation facilitating the anchoring of an immersive experience for a particular user.
- only a single device associated with a user identity may be in the primary mode of operation.
- an audio device such as a microphone may be enabled to receive audio information (e.g., sounds, voices, etc.) from the user.
- a video device such as a forward-facing camera may be enabled to receive video information (e.g., video recording) from the user.
- the primary mode of operation and therefore a device operating in the primary mode of operation, may be fully functional and receiving multiple forms of audio/video input from a participant.
- the techniques disclosed herein can utilize a secondary mode of operation.
- the secondary mode of operation may be a "companion" form of operation facilitating the participation of a user with multiple devices while not detracting from the immersive experience provided by a primary device functioning in the primary mode of operation.
- one or more computing devices associated with a single user identity may function in the companion mode of operation.
- audio capture devices of the associated computing device may be disabled to mitigate audio feedback.
- a camera device such as a forward-facing camera, may be disabled depending upon whether video information is being actively shared from the associated device. In this manner, duplicative video streams and audio streams for the same user identity can be avoided while decreasing network usage and increasing efficiency of the teleconference system 102.
- a selection of devices 106 to operate in the primary and companion modes of operation may be based on user identities and/or profile data 210 associated with a plurality of computing devices. For example, a user may use two or more computing devices 106 to join the teleconference session 104 by issuing a request 156 from each computing device 106 to join the teleconference session 104.
- the teleconference system 102 can select a first computing device 106(1) as a primary computing device for operating in the primary mode of operation.
- the primary computing device can be used to anchor an immersive experience for the user and limit disruption caused by conventional teleconference systems.
- the teleconference system 102 can generate teleconference data 146 associated with a teleconference session 104.
- the teleconference data 146 can include participant streams 142 having video of one or more participants and content streams 142 having video or images of files, data structures, word processing documents, and other shareable content.
- the teleconference system 102 may communicate first teleconference data 146(1) to the primary computing device, and communicate second teleconference data 146(2) through 146(N) to the companion computing device(s). Accordingly, while the primary device is the only device receiving a first selection of teleconference data 146(1), the companion devices receive at least a sub- portion of the teleconference data, e.g., teleconference data 146(2) through 146N).
- the teleconference system 102 of this illustrative example can overcome many technical drawbacks associated with conventional teleconference systems including wasted bandwidth, excessive use of computing resources of all client computing devices to process a first selection of teleconference data, and other drawbacks.
- FIGS. 3A through 3D illustrate a number of scenarios where techniques of the present disclosure can be utilized.
- each user 301 is associated with one or more devices 106.
- the following description describes scenarios where individual users 301 join a teleconference session 104.
- the following examples show scenarios where devices can start in primary mode or companion mode.
- the first user 301 A is associated with a first computing device 106A at a first location 311, which is used to call into a teleconference session 104 that is already in communication with a fifth computing device 106E utilized by a fourth user 301D and a fifth user 301E in a second location 310. Since the first user 301 A is utilizing a single device, the first computing device 106 A can enter the teleconference session 104 in the primary mode of operation. Thus, the microphone and the speaker of the first computing device 106 A are turned on while the first computing device 106 A joins the teleconference session. Additionally, both of the first computing device 106 A and the fifth computing device 106E may operate in the primary mode of operation. Moreover, both of the first computing device 106 A and the fifth computing device 106E can receive first teleconference data 146(1), which includes a first selection of teleconference data or streams 142.
- the first user 301A is utilizing the first computing device 106A to join the teleconference session 104
- a second user 301B is utilizing a second computing device 106B to join the teleconference session 104
- a third user 301C is utilizing a third computing device 106C and a fourth computing device 106D to join the teleconference session 104
- the fourth user 301D is utilizing the fifth computing device 106E to join the teleconference session 104
- the fifth user 301E is utilizing the fifth computing device 106E to join the teleconference session 104.
- the first computing device 106 A, the second computing device 106B, and the third computing device 106C may all join the teleconference session 104 in the primary mode of operation.
- the third computing device 106C and the fourth computing device 106D are associated with a single user identity (e.g., as associated with the third user 301C) and the third user 301C is using the third computing device 106C and the fourth computing device 106D to join the same teleconference session 104.
- the third computing device 106C may join the teleconference session 104 in the primary mode of operation while the fourth computing device 104D may join the teleconference session 104 in the companion mode of operation.
- the third computing device 106C may join the teleconference session 104 in the primary mode of operation. Subsequently, if the third user 301C also attempts to join the teleconference session 104 with the fourth computing device 106D, the fourth computing device 106D may join the teleconference session 104 in the companion mode of operation.
- both of the third computing device 106C and the fourth computing device 106D may initially join the teleconference session 104 in the primary mode of operation. Thereafter, upon determination that both of the third computing device 106C and the fourth computing device 106D are detected as being associated with the same user identity (e.g., of the fourth user 301D), the teleconference system 102 can transition the third computing device 103C to the primary mode of operation for user 301D, and can transition the fourth computing device 104D to the companion mode of operation.
- the companion mode of operation may include different user interface arrangements, renderings of streams, and other aspects as compared to the primary mode of operation.
- the primary device can be selected based on one or more factors.
- the third computing device 106C can become the primary device if it is the first computing device to enter the teleconference session for a particular user identity.
- the third computing device 106C can also be selected as the primary device based on any suitable data, such as a priority level, user preference, or a user input.
- Third computing device 106C can also be selected as the primary device based on the device type.
- the third computing device 106C may be selected as the primary device based on the device type of the third computing device 106C having possibly increased capabilities as compared to the fourth computing device 104D. Other considerations in selecting primary devices may also be applicable.
- the teleconference system 102 can also select the fourth computing device 106D, as illustrated in FIG. 3B. As further shown, each of the primary and companion devices are configured to correspondingly operate in the primary or companion modes of operation.
- the techniques described herein also allow for select streams of data to be presented at a companion device and a primary device.
- the primary device 106C may include a rendering of a plurality of streams 142 and a set of core teleconference control elements.
- the companion device 106D may include a rendering of a selected plurality of streams 142(2) and a set of companion teleconference control elements.
- the primary device 106C may include a rendering of a plurality of streams 142 and a set of core teleconference control elements.
- the companion device 106D may include a rendering of a selected content stream 142(2) based on an activity level and a set of companion teleconference control elements.
- Other alterations and combinations of these above described example scenarios and interface arrangements are also applicable.
- a sixth user 301F is utilizing a sixth computing device 106F and a seventh computing device 106B to join the teleconference session 104.
- the sixth user 301F is at a remote location 312, separate from the second location 310.
- the sixth computing device 106F may join the teleconference session 104 in the primary mode such that the sixth user 301F participates in a fully immersive experience using the sixth computing device 106F.
- the seventh computing device 106G may join the teleconference session 104 in the companion mode of operation.
- the sixth user 301F may transfer functionality between each device such that the devices "swap" modes of operation from the primary mode to the secondary mode, and vice versa.
- FIG. 4 illustrates aspects of a first user interface 400 that can be displayed on a device 106 in communication with a telecommunication session 104.
- This example user interface 400 can be displayed on a device 106, such as a desktop computer, that is operating in the primary mode of operation.
- the first user interface 400 includes a primary portion 402 which provides a stage for the teleconference session 104.
- Selected media data such as streaming videos, files, and applications, shared in the teleconference session 104 can be displayed in the primary portion 402 of the user interface.
- the primary portion 402 can display a variety of participant streams and/or content streams 142 from the teleconference session 104. Accordingly, the primary portion 402 displays a rendering of the first teleconference data described above.
- the first teleconference data 146(1) can generally include a first selection of teleconference data as compared to a reduced sub- portion of data being communicated to companion devices.
- the first user interface 400 can also comprise a number of core teleconference control elements 403, which can be configured to control aspects of the teleconference session 104.
- the core teleconference control elements 403 are configured to control a state of the teleconference session 104.
- a first button of the core teleconference control elements 403 can disconnect a device 106 from the teleconference session 104.
- a second button of the core teleconference control elements 403 can control the microphone of the device 106, i.e., a mute button.
- a third button of the core teleconference control elements 403 can control the camera of the device 106, i.e., toggle the camera on or off.
- a fourth button or any positioned control element can also involve a transfer function.
- the primary device may transition to become a secondary device.
- the display of the core teleconference control elements 403 can fade over a predetermined period of time.
- the core teleconference control elements 403 can be redisplayed based on one or more actions, such as a hover or another suitable user interaction within a predetermined portion of the user interface 400.
- a menu having an arrangement of selectable elements providing access to various types of media can be displayed.
- the media can include, but is not limited to a desktop share, an application share, or a file share.
- the user interface 400 can be configured to allow the user to scroll through additional selectable elements by swiping or scrolling the selectable elements to the left or to the right.
- the display of the selectable elements can be expanded when a user provides a gesture, such as an upward swipe.
- FIG. 5A shows aspects of a companion user interface 500 for a companion device.
- the user interface 500 includes a reduced user interface configured to display the sub-portion of teleconference data 146 based at least in part on an activity level.
- the example of FIG. 5 A shows a control menu having sharing control elements 501 for displaying options for sharing media and/or options for promoting a companion device to a primary device.
- the interface 500 includes a session stage 502 somewhat similar to the session stage 402 described above.
- a first control 501A is configured to activate a camera device.
- the camera of the device In response to receiving a user actuation of the first control 501 A, the camera of the device is activated and a live video feed from the camera may be presented "on stage" (e.g., if allowed by a primary device), and a rendering of the live feed is displayed on other devices in communication with the teleconference session 104.
- the second control 50 IB is configured to activate file sharing and application sharing functionality.
- the secondary device enables a user to select an application or a file to be shared with the teleconference session 104. Once a file or an application is selected, a rendering of the selected media can be presented "on stage," e.g., a rendering of the selected media is communicated to devices in communication with the teleconference session 104.
- the third control 501C is configured to activate a transfer function.
- the companion device may become a primary device based on considerations of the teleconference system 102. For example, if a device 106 is associated with a particular user identity, the user actuation of the third control 501C is received, and there is no longer a primary device associated with the user identity, the teleconference system 102 may enable the device to switch to the primary mode of operation to retain the immersive experience for the user.
- the transfer to the primary mode may also be facilitated by demoting an existing primary device associated with the user identity to the companion mode of operation to allow the initial companion device to become a user's primary device.
- a swap from companion mode to primary mode can be facilitated by the techniques described herein.
- the fourth control 50 ID provides a connection control to the teleconference session 104, e.g., the fourth control 501D can be a "hang up" or disconnect button.
- the fourth control 501D can be a "hang up" or disconnect button.
- the interface 500 does not have core teleconference control elements for controlling a microphone or other forms of audio communication as a companion device is not configured to communicate audio signals in some implementations. Additionally, as shown in FIG. 5B, other variations on the interface 500 including a lack or hidden set of companion control elements is also applicable.
- the contextual menu controls, features, searchable content, recent or contextual content can be the open state of many differing conversation and collaboration core control needs such as: sharing, recording, remote control, pstn/voip calling, sensor detection, device detection/selection, etc.
- the layout configuration of the control menu can adapt to differing screen layouts as well: top, bottom, side, etc. Actionability can be influenced by differing gestures as well: swipe up, down, left, right, and tap, double tap, tap and hold, etc.
- a companion device can select different types of media data for display. For instance, a companion device can display received teleconference data that is based on an activity level.
- a companion device can display received teleconference data that is based on an activity level.
- the participant stream may be displayed on companion devices on the main portion 502 of the user interface 500. It follows that as different content streams or participant streams are displayed on a primary device, only a subset of teleconference data is communicated to the companion devices. Accordingly, reduced bandwidth usage is realized while also allowing a fully immersive teleconference experience.
- FIG. 6 A aspects of another user interface, interface 600, used for a device 106 operating in a companion mode of operation are described.
- the particular interface 600 is an illustrative example representing a user interface on a mobile device, such as a tablet computer, mobile phone, or other similar device. It can be appreciated that similar aspects of the interface 600 as compared to the interface 500 exist. Accordingly, exhaustive description of similar elements is omitted herein for the sake of brevity.
- the user interface 600 can comprise a control menu having sharing control elements 651 for providing access to various types of media.
- the sharing control elements 651 can be arranged to function similar to the sharing control elements 501 described in detail above.
- the first selectable element 651 A can provide access to a particular file.
- the second selectable element 65 IB can provide access to a desktop share, and the third selectable element 651C can provide access to an application share.
- These examples are provided for illustrative purposes and are not to be construed as limiting. It can be appreciated that the selectable elements 651 can provide access to any suitable type of shareable media and are arranged similarly to the selectable elements 501.
- the interface 600 can also include a display of the session stage 652, which is a video presentation of the second teleconference data or sub-portion of teleconference data communicated to the client devices.
- the display of the session stage 652 can show a salient feature of the session, such as the most dominant speaker or shared content.
- a graphical element such as the chevron displayed in the bottom center of the interface
- the display of the session stage 652 can expand to show additional sharing options available to companion devices. For example, as shown in FIG. 6B and FIG. 6C, examples of full swipe configurations of a user interface 600 of a mobile device operating in the companion mode are provided.
- the user interface 600 can comprise a control menu having sharing control elements 651A-651I for providing access to various types of media.
- the selectable elements 651 A-65 II can provide access to a particular file, a desktop share, and an application. These examples are provided for illustrative purposes and are not to be construed as limiting. It can be appreciated that the selectable elements 651 A-55 II can provide access to any suitable type of shareable media. It can also be appreciated that the interface 600 can have any suitable number of selectable elements 651. In addition, a user can scroll through additional selectable elements 651 A-65 II by actuating the arrows to view additional selectable elements 651.
- the user interface 600 can also include an "ADD" button for enabling users to add content, e.g., files, applications and other content, to the menu.
- Displayed controls and content can scale in rows and column behaviors to adapt to differing size devices, windows, user needs, etc.
- the user interface 600 can also provide access to a display of the session stage by the use of a gesture, such as a down swipe applied to the first UI element 662.
- a gesture such as a down swipe applied to the first UI element 662.
- the user interface 600 transitions to the example user interface 600 shown in FIG. 6C, which includes a video presentation of the second teleconference data 146(2) that is presented on the session stage 652.
- other share controls 664 e.g., a send button or a share button, can be positioned in the user interface 600.
- the user interface 600 can be configured with or without the share controls.
- the user interface 600 shown in FIG. 6C can transition back to the user interface 600 shown in FIG. 6B in response to receiving a predetermined input gesture, such as an up swipe at the second UI element 663.
- companion devices may implement a shared camera or "roving camera” functionality to allow a meeting participant to offer a real-world view of content and a surrounding environment while not disturbing an associated primary device.
- a companion device 106D comprises a camera 781 suitable for streaming video data to the teleconference session 104.
- the device 106D As the user directs the camera (referred to herein as a "roving camera") of the device 106D to content 780, the device 106D generates a video stream or photo that is communicated to the server module 136 for distribution of the content.
- the shared content can include a video stream or still images.
- FIG. 7A shows the fourth computing device 106D operating in the companion mode of operation and generating image data via the camera 781.
- the image data is shared with devices of the teleconference session 104, such as the first computing device 106 A and the second computing device 106B. Any content can be shared, including a whiteboard, a drawing, a photograph, a book page, or any other content from an analog environment.
- the teleconference system 102 can receive a request to share video content from a camera 781 of the computing device 106D.
- the teleconference system 102 can add the video content to the teleconference data and the sub-portion of the teleconference data for sharing amongst the devices participating in the teleconference session 104.
- any user in an online meeting can send live video or a still image to a shared video stage that treats roving camera video as a content stream rather than as a traditional egocentric participant stream (aka 'talking head' video).
- This feature might be triggered by a user using a companion device to share real-world content or other content with meeting participants.
- the content 780 can be displayed at any primary device, such as primary device 106E using an overlay view. Other views, such as persistent views, can also be applicable.
- Other devices associated with the teleconference session 104 may also receive and display the content 780, as shown in FIGS. 7C and 7D. Accordingly, while a companion device can be used to receive second teleconference data 146(2) or a sub-portion of teleconference data 146, a companion device can also be used to augment teleconference sessions.
- real-time mediated collaboration such as via an online meeting, videoconference, or a document collaboration system is typically accomplished using a single device per user ID.
- current collaboration systems if multiple devices are simultaneously joining and associated with a single user identity, it is possible the user experience is essentially duplicated across devices, and several inefficiencies including over utilization of bandwidth and use of multiple audio devices per user identity are apparent. This may have deleterious results for the user experience.
- audio feedback may be presented through "echoes" and "ghosting" of an audio stream. At worst this causes technical and practical problems, including but not limited to race conditions for synchronizations, notifications, control execution, audio echo and howl, battery drain, and extra bandwidth usage.
- This invention concerns the addition of second and third, up to an n number of devices associated with a single user identity which can be automatically differentiated upon join so as to augment the primary device experience rather than duplicate it.
- the invention distinguishes the states of any primary device in a real-time collaboration session from the states of any companion devices that join the same session while associated with a single user identity.
- FIG. 8 illustrates a flowchart of an example method 800 of providing companion devices for real-time collaboration in teleconference sessions.
- the method 800 may include generating teleconference data 146 comprising a plurality of streams 142 associated with a teleconference session 104, at block 802.
- the plurality of streams 142 may be received from a plurality of computing devices 106 associated with the teleconference session 104.
- the method 800 may further include receiving a first request to join the teleconference session 104 from a first computing device (e.g., the third computing device 106C of FIG. 3 A) associated with a first user identity (e.g., the third user 301C of FIG. 3 A), at block 804.
- a first computing device e.g., the third computing device 106C of FIG. 3 A
- a first user identity e.g., the third user 301C of FIG. 3 A
- the request may be substantially similar to the access request, configure command, and/or reconfigure commands described in detail above with reference to FIG. 1 (e.g., CTL 156(1)).
- the method 800 may further include communicating the teleconference data 146 to the first computing device 106C causing the first computing device (e.g., computing device 106C) to display a primary user interface 400, at block 806.
- the primary user interface 400 can include a rendering of the plurality of streams 142.
- the user interface 400 can be arranged to display multiple streams or a single stream, as well.
- the user interface 400 can comprise a plurality of core teleconference control elements 403.
- the core teleconference control elements 403 can include user interface elements configured to control a state of the first computing device (e.g., device 106C) and a state of the teleconference session 104.
- the method 800 includes receiving a second request to join the teleconference session 104 from a second computing device (e.g., device 106D) associated with the first user identity (e.g., user 301C), at block 808.
- a second computing device e.g., device 106D
- the user 301C may attempt to join the teleconference session 104 from the device 106C and the device 106D at substantially the same time.
- the user e.g., user 301C
- the user 301C may attempt to join the teleconference session 104 with a companion device 106D.
- the method 800 may include selecting at least one stream of the plurality of streams 142 based on an activity level, at block 810.
- the sub-portion of teleconference data or the second teleconference data 146(2) through 146(N) can be selected based on an activity level for data presented by participants of the teleconference session 104.
- the video or shared content in the media data for each participant may be analyzed to determine an activity level for any stream 142(1) through 142(M) received at the teleconference system 102.
- the activity level can be based on any type of activity including, but not limited to, any combination of the following:
- participant motion - the extent to which a participant moves in a video may
- participant's activity level Participants in the process of gesturing or otherwise moving in the video may be deemed to be participating at a relatively high level in the teleconference.
- participant lip motion - the video may be analyzed to determine the extent to
- Participants speaking at a relatively high level may be deemed to be participating at a corresponding relatively high level.
- participant facial expressions - the participant' s video may be analyzed to
- content modification - video of content being shared in the teleconference may be analyzed to determine if it is being modified. Thus, recently or actively modified content may have a high activity level.
- content page turning - video of content being shared may be analyzed to determine if there is page turning of a document, for example, and assigned a corresponding activity level.
- number of participants having content in a display area - video of content being shared may be assigned an activity level based on the number of participants that have a view of the content in a display area.
- participant entering teleconference session - media data from participants entering a teleconference may be assigned a high activity level. This value can be based on the order in which participants join a session.
- participant leaving teleconference session - media data from participants leaving a teleconference may be assigned a low activity level.
- participant streams and/or content streams having a relatively high activity level can be chosen for display in any of the companion user interface arrangements described above via communication of second teleconference data 146(2) through 146(N) or a sub-portion of the teleconference data 146 having the relatively high activity level. It is also understood that activity levels can be superseded based on other logic, for example, if a user having multiple devices desires a fixed view for companion computing devices, or other scenarios.
- the method 800 further includes communicating the teleconference data 146 to the second computing device 106D causing the second computing device 106D to display a companion user interface 500, at block 812.
- the companion user interface 500 can include a rendering of the at least one stream. It is understood that the companion user interface 500 can include renderings of multiple streams, content streams, participant streams, or any combination thereof. Additionally, the companion user interface 500 can comprise a plurality of companion teleconference control elements 501.
- the companion teleconference control elements 501 can include user interface elements configured to provide content sharing functionality. Generally, the content sharing functionality enables the second computing device 106D to share at least one of a selected file, a desktop view, or a stream generated by a camera 781 of the second computing device 106D.
- the plurality of streams 142 can also comprise at least one participant stream and at least one content stream, as described above. Therefore, in a scenario where participant streams and content streams are processed by the teleconference system 102, the teleconference system 102 can also communicate the at least one participant stream and the at least one content stream to the first computing device 106C and communicate the at least one content stream to the second computing device 106D. In this manner, reduced usage of bandwidth can be realized through intelligent selection of streams 142 to communicate to primary and companion devices.
- a user 301C can also request to join the teleconference session 104 using multiple devices.
- the user 301C can transmit a first request 156 to join a teleconference session 104 from a first computing device 106C associated with a first user identity 301C.
- the first computing device 106C can receive first teleconference data 146(1) comprising a plurality of streams 142 associated with the teleconference session 104 and display a rendering of the plurality of streams 142.
- the user 301C can transmit a second request to join the teleconference session 104 from a second computing device 106D associated with the first user identity 301C. Responsive to the second request, the second computing device 106D can receive second teleconference data 146(2) comprising a subset of streams of the plurality of streams 142 and display a rendering of the subset of streams of the plurality of streams 142.
- the subset of streams of the plurality of streams 142 is based at least in part on data indicating an activity level associated with the subset of streams 142, as described in detail above.
- the user 301C can transmit several commands using the user interface elements illustrated in FIG. 4 and FIG. 5 A.
- the user 301C can transmit one or more core teleconference control requests from the first computing device 106C to control the state of the teleconference session 104.
- the user 301C can transmit one or more companion teleconference control requests from the second computing device 106D to control content sharing functionality of the second computing device 106D.
- Other forms of requests and controls including the "swap" functionality described above are also applicable.
- the operations represent computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, enable the one or more processors to perform the recited operations.
- computer-executable instructions include routines, programs, objects, modules, components, data structures, and the like that perform particular functions or implement particular abstract data types.
- the order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be executed in any order, combined in any order, subdivided into multiple sub-operations, and/or executed in parallel to implement the described processes.
- the described processes can be performed by resources associated with one or more device(s) such as one or more internal or external CPUs or GPUs, and/or one or more pieces of hardware logic such as FPGAs, DSPs, or other types of accelerators.
- All of the methods and processes described above may be embodied in, and fully automated via, software code modules executed by one or more general purpose computers or processors.
- the code modules may be stored in any type of computer-readable storage medium or other computer storage device. Some or all of the methods may alternatively be embodied in specialized computer hardware.
- Conditional language such as, among others, "can,” “could,” “might” or “may,” unless specifically stated otherwise, are understood within the context to present that certain examples include, while other examples do not include, certain features, elements and/or steps. Thus, such conditional language is not generally intended to imply that certain features, elements and/or steps are in any way required for one or more examples or that one or more examples necessarily include logic for deciding, with or without user input or prompting, whether certain features, elements and/or steps are included or are to be performed in any particular example.
- Conjunctive language such as the phrase "at least one of X, Y or Z,” unless specifically stated otherwise, is to be understood to present that an item, term, etc. may be either X, Y, or Z, or a combination thereof.
- a method comprising generating teleconference data (146) comprising a plurality of streams (142) associated with a teleconference session (104); receiving a first request to join the teleconference session (104) from a first computing device (106C) associated with a first user identity (301C); communicating the teleconference data (146) to the first computing device (106C) causing the first computing device (106C) to display a primary user interface (400), the primary user interface (400) including a rendering of the plurality of streams (142); receiving a second request to join the teleconference session (104) from a second computing device (106D) associated with the first user identity (301C); selecting at least one stream of the plurality of streams (142) based on an activity level; and communicating the teleconference data (146) to the second computing device (106D) causing the second computing device (106D) to display a companion user interface (500), the companion user interface (500) including a rendering of the at least one stream.
- Clause 2 A method as in clause 1, wherein the primary user interface comprises a plurality of core teleconference control elements, the core teleconference control elements including user interface elements configured to control a state of the first computing device and a state of the teleconference session.
- Clause 3 A method as in any of the clauses above, wherein the companion user interface comprises a plurality of companion teleconference control elements, the companion teleconference control elements including user interface elements configured to provide content sharing functionality, wherein the content sharing functionality enables the second computing device to share at least one of a selected file, a desktop view, or a stream generated by a camera of the second computing device.
- Clause 4 A method as in any of the clauses above, wherein the activity level is representative of a level of activity of the at least one stream.
- Clause 5 A method as in any of the clauses above, wherein the plurality of streams comprise at least one participant stream and at least one content stream, wherein the method further comprises: communicating the at least one participant stream and the at least one content stream to the first computing device; and communicating the at least one content stream to the second computing device.
- Clause 6 A method as in any of the clauses above, wherein the at least one participant stream comprises a user media data of the teleconference session and the at least one content stream comprises content media data of the teleconference session.
- a method comprising: transmitting a first request to join a teleconference session (104) from a first computing device (106C) associated with a first user identity (301C); receiving first teleconference data (146(1)) comprising plurality of streams (142) associated with the teleconference session (104) at the first computing device (106C); displaying a rendering of the plurality of streams (142) on the first computing device (106C); transmitting a second request to join the teleconference session (104) from a second computing device (106D) associated with the first user identity (301C); receiving second teleconference data comprising a subset of streams (142) of the plurality of streams (142) at the second computing device (106D); and displaying a rendering of the subset of streams (142) of the plurality of streams (142) on the second computing device (106D), wherein the subset of streams (142) of the plurality of streams (142) is based at least in part on data indicating an activity level associated with the subset of streams (142).
- Clause 8 A method as in any of the clauses above, wherein the method further comprises: transmitting one or more core teleconference control requests from the first computing device to control the state of the teleconference session.
- Clause 9 A method as in any of the clauses above, wherein the method further comprises: transmitting one or more companion teleconference control requests from the second computing device to control content sharing functionality of the second computing device.
- Clause 10 A method as in any of the clauses above, wherein the plurality of streams comprise a plurality of participant streams and a plurality of content streams, wherein the plurality of participant streams comprise a user media data of the teleconference session and the plurality of content streams comprise content media data of the teleconference session.
- a method comprising: generating teleconference data (146) comprising plurality of streams (142) associated with a teleconference session (104); receiving a first request to join the teleconference session (104) from a first computing device (106C) associated with a first user identity (301C); receiving a second request to join the teleconference session (104) from a second computing device (106D) associated with the first user identity (301C); communicating the teleconference data (146) to the first computing device (106C); and communicating a sub-portion of the teleconference data (146(2)) to the second computing device (106D), wherein the sub-portion (146(2)) comprises at least one stream that is selected from the plurality of streams (142) based on an activity level.
- Clause 12 A method as in any of the clauses above, wherein the method further comprises: receiving a control command for causing a transition of the first computing device and the second computing device; and in response to receiving the control command, communicating the teleconference data to the second computing device for displaying a rendering of the plurality of streams; and communicating the sub-portion of the teleconference data to the first computing device for displaying a rendering of the at least one stream.
- Clause 13 A method as in any of the clauses above, wherein the method further comprises: receiving a reconfiguring command for causing a transition to the first computing device; and in response to receiving the reconfiguring command, discontinuing communicating the sub-portion of the teleconference data to the second computing device.
- Clause 14 A method as in any of the clauses above, wherein the method further comprises: disabling a camera component of the second computing device.
- Clause 15 A method as in any of the clauses above, wherein the method further comprises: disabling an audio component of the second computing device.
- Clause 16 A method as in any of the clauses above, wherein the method further comprises: receiving a request to share video content from a camera of the second computing device; and adding the video content to the teleconference data and the sub- portion of the teleconference data.
- a system comprising: one or more processing units (202); and a computer-readable medium (204) having encoded thereon computer-executable instructions to cause the one or more processing units (202) to generate teleconference data (146) comprising a plurality of streams (142) associated with a teleconference session (104); receive a first request to join the teleconference session (104) from a first computing device (106C) associated with a first user identity (301C); communicate the teleconference data (146) to the first computing device (106C) for displaying a primary user interface (400), the primary user interface (400) including a plurality of core teleconference control elements (403); receive a second request to join the teleconference session (104) from a second computing device (106D) associated with the first user identity (301C); select at least one stream (142) of the plurality of streams (142) based on an activity level; and communicate the teleconference data (146) to the second computing device (106D) causing the second computing device (106D)
- Clause 18 A system as in clause 17, wherein the plurality of streams comprise at least one participant stream and at least one content stream, wherein the computer- executable instructions further cause the one or more processing units to communicate the at least one participant stream and the at least one content stream to the first computing device; and communicate the at least one content stream to the second computing device.
- Clause 19 A system as in any of the clauses above, wherein the computer- executable instructions further cause the one or more processing units to receive a reconfiguring command for causing a transition to the first computing device; and in response to receiving the reconfiguring command, discontinue communicating the teleconference data to the second computing device.
- Clause 20 A system as in any of the clauses above, wherein the computer- executable instructions further cause the one or more processing units to receive a control command for causing a transition of the first computing device and the second computing device; and in response to receiving the control command, communicate the teleconference data to the second computing device causing a rendering of the plurality of streams; and communicate the teleconference data to the first computing device causing a rendering of the at least one stream.
Landscapes
- Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Computer Networks & Wireless Communication (AREA)
- General Engineering & Computer Science (AREA)
- Business, Economics & Management (AREA)
- General Business, Economics & Management (AREA)
- Telephonic Communication Services (AREA)
- Information Transfer Between Computers (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- Computer Vision & Pattern Recognition (AREA)
Abstract
Description
Claims
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201662415403P | 2016-10-31 | 2016-10-31 | |
US15/480,332 US11310294B2 (en) | 2016-10-31 | 2017-04-05 | Companion devices for real-time collaboration in communication sessions |
PCT/US2017/057948 WO2018081030A1 (en) | 2016-10-31 | 2017-10-24 | Companion devices for real-time collaboration in teleconference sessions |
Publications (1)
Publication Number | Publication Date |
---|---|
EP3533222A1 true EP3533222A1 (en) | 2019-09-04 |
Family
ID=62019994
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP17794195.2A Withdrawn EP3533221A1 (en) | 2016-10-31 | 2017-10-24 | Enhanced techniques for joining teleconferencing sessions |
EP17794863.5A Pending EP3533222A1 (en) | 2016-10-31 | 2017-10-24 | Companion devices for real-time collaboration in teleconference sessions |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP17794195.2A Withdrawn EP3533221A1 (en) | 2016-10-31 | 2017-10-24 | Enhanced techniques for joining teleconferencing sessions |
Country Status (4)
Country | Link |
---|---|
US (2) | US11212326B2 (en) |
EP (2) | EP3533221A1 (en) |
CN (2) | CN109891878A (en) |
WO (2) | WO2018081029A1 (en) |
Families Citing this family (48)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2003107146A2 (en) * | 2002-06-18 | 2003-12-24 | Wink Interactive, Llc | Method, apparatus and system for management of information content for enhanced accessibility over wireless communication networks |
WO2015183367A1 (en) | 2014-05-30 | 2015-12-03 | Apple Inc. | Continuity |
DK201670622A1 (en) | 2016-06-12 | 2018-02-12 | Apple Inc | User interfaces for transactions |
CN111343060B (en) | 2017-05-16 | 2022-02-11 | 苹果公司 | Method and interface for home media control |
US10728193B2 (en) * | 2017-11-17 | 2020-07-28 | International Business Machines Corporation | Receiving and sharing files in a group messaging environment |
DK201870364A1 (en) | 2018-05-07 | 2019-12-03 | Apple Inc. | Multi-participant live communication user interface |
CN109005443B (en) * | 2018-08-24 | 2021-05-28 | 重庆虚拟实境科技有限公司 | Real-person remote interaction method for VR-AR all-in-one machine terminal and system based on same |
CN109059901B (en) * | 2018-09-06 | 2020-02-11 | 深圳大学 | AR navigation method based on social application, storage medium and mobile terminal |
US11128792B2 (en) | 2018-09-28 | 2021-09-21 | Apple Inc. | Capturing and displaying images with multiple focal planes |
US10616151B1 (en) * | 2018-10-17 | 2020-04-07 | Asana, Inc. | Systems and methods for generating and presenting graphical user interfaces |
CN111258528B (en) * | 2018-12-03 | 2021-08-13 | 华为技术有限公司 | Voice user interface display method and conference terminal |
US11190568B2 (en) * | 2019-01-09 | 2021-11-30 | Bose Corporation | Multimedia communication encoding system |
PL3912312T3 (en) * | 2019-01-15 | 2022-11-21 | Telefonaktiebolaget Lm Ericsson (Publ) | Providing communication services using sets of i/o devices |
US10764442B1 (en) * | 2019-03-27 | 2020-09-01 | Lenovo (Singapore) Pte Ltd | Muting an audio device participating in a conference call |
US11271762B2 (en) * | 2019-05-10 | 2022-03-08 | Citrix Systems, Inc. | Systems and methods for virtual meetings |
US11010121B2 (en) | 2019-05-31 | 2021-05-18 | Apple Inc. | User interfaces for audio media control |
KR20240049648A (en) | 2019-05-31 | 2024-04-16 | 애플 인크. | User interfaces for audio media control |
EP3984032A1 (en) * | 2019-06-17 | 2022-04-20 | Google LLC | Methods, systems, and media for providing dynamic media sessions |
US11240470B2 (en) | 2019-07-08 | 2022-02-01 | Nextiva, Inc. | Multi-device teleconferences |
CN110430384B (en) * | 2019-08-23 | 2020-11-03 | 珠海格力电器股份有限公司 | Video call method and device, intelligent terminal and storage medium |
US11546391B2 (en) | 2019-11-01 | 2023-01-03 | Microsoft Technology Licensing, Llc | Teleconferencing interfaces and controls for paired user computing devices |
US11304246B2 (en) | 2019-11-01 | 2022-04-12 | Microsoft Technology Licensing, Llc | Proximity-based pairing and operation of user-specific companion devices |
US11256392B2 (en) * | 2019-11-01 | 2022-02-22 | Microsoft Technology Licensing, Llc | Unified interfaces for paired user computing devices |
US11057702B1 (en) * | 2019-12-20 | 2021-07-06 | Microsoft Technology Licensing, Llc | Method and system for reducing audio feedback |
CN112312058B (en) * | 2020-03-22 | 2023-06-27 | 北京字节跳动网络技术有限公司 | Interaction method and device and electronic equipment |
CN111596985B (en) * | 2020-04-24 | 2023-03-14 | 腾讯科技(深圳)有限公司 | Interface display method, device, terminal and medium in multimedia conference scene |
CN112311754B (en) * | 2020-06-02 | 2022-09-30 | 北京字节跳动网络技术有限公司 | Interaction method and device and electronic equipment |
US12113844B2 (en) * | 2020-09-15 | 2024-10-08 | Carrier Corporation | Digital data processing |
US11392291B2 (en) | 2020-09-25 | 2022-07-19 | Apple Inc. | Methods and interfaces for media control with dynamic feedback |
US11769115B1 (en) | 2020-11-23 | 2023-09-26 | Asana, Inc. | Systems and methods to provide measures of user workload when generating units of work based on chat sessions between users of a collaboration environment |
WO2022140557A1 (en) * | 2020-12-22 | 2022-06-30 | Dolby Laboratories Licensing Corporation | Acoustic feedback reduction in co-located audioconferencing devices |
US11843647B2 (en) * | 2021-04-30 | 2023-12-12 | Zoom Video Communications, Inc. | Control of dedicated meeting room devices for video communications |
US20220360949A1 (en) * | 2021-05-05 | 2022-11-10 | Lightspeed Technologies, Inc. | Registration and mode switching for devices in a distributed audio system |
US11893214B2 (en) | 2021-05-15 | 2024-02-06 | Apple Inc. | Real-time communication user interface |
US11907605B2 (en) * | 2021-05-15 | 2024-02-20 | Apple Inc. | Shared-content session user interfaces |
US11928303B2 (en) | 2021-05-15 | 2024-03-12 | Apple Inc. | Shared-content session user interfaces |
US11847378B2 (en) | 2021-06-06 | 2023-12-19 | Apple Inc. | User interfaces for audio routing |
US20230064462A1 (en) * | 2021-08-24 | 2023-03-02 | International Business Machines Corporation | Cognitive core network selection for conference calling |
GB2612015A (en) * | 2021-09-06 | 2023-04-26 | Laduma Ltd | System and method for interactive meeting with both in-room attendees and remote attendees |
US11677906B2 (en) * | 2021-09-10 | 2023-06-13 | Zoom Video Communications, Inc. | Secondary mode device software access for primary mode device users |
US11812135B2 (en) | 2021-09-24 | 2023-11-07 | Apple Inc. | Wide angle video conference |
CN113872778B (en) * | 2021-10-15 | 2023-07-14 | 北京百度网讯科技有限公司 | Device connection method, device and storage medium |
US11775606B2 (en) * | 2021-10-29 | 2023-10-03 | Weld North Education LLC | Inter-browser presentation control |
US20230199041A1 (en) * | 2021-12-21 | 2023-06-22 | Nevolane Business Gmbh | Remote collaboration platform |
WO2023196231A1 (en) * | 2022-04-04 | 2023-10-12 | Apple Inc. | User interfaces for camera sharing |
US11570218B1 (en) * | 2022-04-06 | 2023-01-31 | Discovery.Com, Llc | Systems and methods for synchronous group device transmission of streaming media and related user interfaces |
CN115242566B (en) * | 2022-06-28 | 2023-09-05 | 深圳乐播科技有限公司 | Cloud conference joining method and device |
US12081607B2 (en) | 2022-12-23 | 2024-09-03 | Dropbox, Inc. | Shared capture session |
Family Cites Families (89)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP1145096B1 (en) | 1998-11-24 | 2007-01-17 | Telefonaktiebolaget LM Ericsson (publ) | Mobile telephone auto pc logon |
FI110560B (en) | 2000-12-27 | 2003-02-14 | Nokia Corp | Grouping of wireless communication terminals |
US8934382B2 (en) | 2001-05-10 | 2015-01-13 | Polycom, Inc. | Conference endpoint controlling functions of a remote device |
US20030048174A1 (en) | 2001-09-11 | 2003-03-13 | Alcatel, Societe Anonyme | Electronic device capable of wirelessly transmitting a password that can be used to unlock/lock a password protected electronic device |
US7191233B2 (en) | 2001-09-17 | 2007-03-13 | Telecommunication Systems, Inc. | System for automated, mid-session, user-directed, device-to-device session transfer system |
US8081205B2 (en) | 2003-10-08 | 2011-12-20 | Cisco Technology, Inc. | Dynamically switched and static multiple video streams for a multimedia conference |
US7279060B2 (en) | 2004-05-04 | 2007-10-09 | Eastman Kodak Company | Guarded cover film for LCD polarizers |
US7450084B2 (en) | 2004-12-17 | 2008-11-11 | Microsoft Corporation | System and method for managing computer monitor configurations |
US7554571B1 (en) | 2005-03-18 | 2009-06-30 | Avaya Inc. | Dynamic layout of participants in a multi-party video conference |
US20060224882A1 (en) | 2005-03-31 | 2006-10-05 | Microsoft Corporation | Method and system for unlocking a computing device |
US8661540B2 (en) | 2005-10-07 | 2014-02-25 | Imation Corp. | Method and apparatus for secure credential entry without physical entry |
US7394366B2 (en) | 2005-11-15 | 2008-07-01 | Mitel Networks Corporation | Method of detecting audio/video devices within a room |
US20070300165A1 (en) | 2006-06-26 | 2007-12-27 | Microsoft Corporation, Corporation In The State Of Washington | User interface for sub-conferencing |
US8139752B2 (en) | 2006-07-28 | 2012-03-20 | Ubiquity Software Corporation Limited | Voice conference control from an instant messaging session using an automated agent |
US8503651B2 (en) | 2006-12-27 | 2013-08-06 | Nokia Corporation | Teleconferencing configuration based on proximity information |
US7973857B2 (en) | 2006-12-27 | 2011-07-05 | Nokia Corporation | Teleconference group formation using context information |
US8677270B2 (en) | 2007-05-04 | 2014-03-18 | Microsoft Corporation | Live companion user interface |
US20080281971A1 (en) | 2007-05-07 | 2008-11-13 | Nokia Corporation | Network multimedia communication using multiple devices |
US8446454B2 (en) | 2007-05-21 | 2013-05-21 | Polycom, Inc. | Dynamic adaption of a continuous presence videoconferencing layout based on video content |
KR20100007625A (en) | 2008-07-14 | 2010-01-22 | 엘지전자 주식회사 | Mobile terminal and method for displaying menu thereof |
US8350891B2 (en) | 2009-11-16 | 2013-01-08 | Lifesize Communications, Inc. | Determining a videoconference layout based on numbers of participants |
US20110119389A1 (en) * | 2009-11-19 | 2011-05-19 | Microsoft Corporation | Transferring multiple communication modalities during a conversation |
US20110271207A1 (en) * | 2010-04-30 | 2011-11-03 | American Teleconferencing Services Ltd. | Location-Aware Conferencing |
US20110292161A1 (en) | 2010-05-25 | 2011-12-01 | Vidyo, Inc. | Systems And Methods For Scalable Video Communication Using Multiple Cameras And Multiple Monitors |
NO332222B1 (en) | 2010-07-26 | 2012-07-30 | Cisco Tech Inc | Method and interaction server for transmitting an interaction session, as well as a multimedia endpoint |
US8379077B2 (en) | 2010-11-24 | 2013-02-19 | Cisco Technology, Inc. | Automatic layout and speaker selection in a continuous presence video conference |
US8647202B2 (en) | 2010-12-16 | 2014-02-11 | Microsoft Corporation | Companion object customization |
US20120182384A1 (en) | 2011-01-17 | 2012-07-19 | Anderson Eric C | System and method for interactive video conferencing |
CN102625304B (en) | 2011-01-27 | 2016-01-20 | 腾讯科技(深圳)有限公司 | Failure mobile terminal associated application remembers system, the device and method of password |
US20120280789A1 (en) | 2011-05-02 | 2012-11-08 | Apigy Inc. | Systems and methods for controlling a locking mechanism using a portable electronic device |
US8918722B2 (en) | 2011-05-06 | 2014-12-23 | David H. Sitrick | System and methodology for collaboration in groups with split screen displays |
US9369673B2 (en) * | 2011-05-11 | 2016-06-14 | Blue Jeans Network | Methods and systems for using a mobile device to join a video conference endpoint into a video conference |
US9565708B2 (en) | 2011-05-20 | 2017-02-07 | Microsoft Technology Licensing, Llc | Auto-connect in a peer-to-peer network |
US20160342784A1 (en) | 2011-07-15 | 2016-11-24 | Vmware, Inc. | Mobile device authentication |
US9800688B2 (en) | 2011-09-12 | 2017-10-24 | Microsoft Technology Licensing, Llc | Platform-enabled proximity service |
JP6135055B2 (en) | 2011-09-16 | 2017-05-31 | 株式会社リコー | Information providing apparatus, transmission system, and program |
US8896651B2 (en) | 2011-10-27 | 2014-11-25 | Polycom, Inc. | Portable devices as videoconferencing peripherals |
US9021557B2 (en) | 2011-10-27 | 2015-04-28 | Stmicroelectronics Pte Ltd | System and method for security using a sibling smart card |
US9131333B2 (en) | 2011-12-30 | 2015-09-08 | Linkedin Corporation | Systems and methods for mobile device pairing |
US20130219288A1 (en) | 2012-02-20 | 2013-08-22 | Jonathan Rosenberg | Transferring of Communication Event |
US8786517B2 (en) | 2012-02-21 | 2014-07-22 | Blackberry Limited | System and method for displaying a user interface across multiple electronic devices |
US20130219303A1 (en) | 2012-02-21 | 2013-08-22 | Research In Motion Tat Ab | Method, apparatus, and system for providing a shared user interface |
US20140006495A1 (en) | 2012-06-29 | 2014-01-02 | International Business Machines Corporation | Authorizing access to a web conference for a specified duration of time |
US20140026070A1 (en) | 2012-07-17 | 2014-01-23 | Microsoft Corporation | Dynamic focus for conversation visualization environments |
US20140028726A1 (en) | 2012-07-30 | 2014-01-30 | Nvidia Corporation | Wireless data transfer based spanning, extending and/or cloning of display data across a plurality of computing devices |
WO2014023851A1 (en) * | 2012-08-10 | 2014-02-13 | Nec Europe Ltd. | Method and system for providing content for user devices |
US9026053B2 (en) | 2013-02-17 | 2015-05-05 | Fitbit, Inc. | System and method for wireless device pairing |
US9369672B2 (en) | 2013-03-14 | 2016-06-14 | Polycom, Inc. | Intelligent layouts for call scaling and layout persistence |
US9400895B2 (en) | 2013-03-15 | 2016-07-26 | Intel Corporation | Mechanisms for locking computing devices |
US9137723B2 (en) | 2013-03-15 | 2015-09-15 | Facebook, Inc. | Portable platform for networked computing |
US20140315489A1 (en) | 2013-04-22 | 2014-10-23 | Htc Corporation | Method for performing wireless display sharing, and associated apparatus and associated computer program product |
US9407866B2 (en) | 2013-05-20 | 2016-08-02 | Citrix Systems, Inc. | Joining an electronic conference in response to sound |
US10243786B2 (en) | 2013-05-20 | 2019-03-26 | Citrix Systems, Inc. | Proximity and context aware mobile workspaces in enterprise systems |
US9172908B2 (en) | 2013-06-18 | 2015-10-27 | Microsoft Technology Licensing, Llc | Unpaired devices |
CN104469256B (en) | 2013-09-22 | 2019-04-23 | 思科技术公司 | Immersion and interactive video conference room environment |
US9088694B2 (en) | 2013-10-03 | 2015-07-21 | Avaya, Inc. | Adjusting video layout |
GB201319687D0 (en) | 2013-11-07 | 2013-12-25 | Microsoft Corp | Call handling |
US9391982B1 (en) | 2014-02-27 | 2016-07-12 | Cullen/Frost Bankers, Inc. | Network authentication of multiple profile accesses from a single remote device |
US20150271273A1 (en) | 2014-03-18 | 2015-09-24 | CafeX Communications Inc. | System for Using a Device as a Side Car |
US10055567B2 (en) | 2014-05-30 | 2018-08-21 | Apple Inc. | Proximity unlock and lock operations for electronic devices |
WO2015187941A1 (en) | 2014-06-05 | 2015-12-10 | Reel, Inc. | Apparatus and method for sharing content items among a plurality of mobile devices |
US9485414B2 (en) | 2014-06-20 | 2016-11-01 | John Visosky | Eye contact enabling device for video conferencing |
US9602771B2 (en) | 2014-12-10 | 2017-03-21 | Polycom, Inc. | Automated layouts optimized for multi-screen and multi-camera videoconferencing calls |
US9438643B2 (en) * | 2015-01-15 | 2016-09-06 | Cisco Technology, Inc. | Multi-device conference participation |
US10440757B2 (en) | 2015-02-17 | 2019-10-08 | Google Llc | Second-screen control automatic pairing using push notifications |
US10701120B2 (en) | 2015-03-06 | 2020-06-30 | Disney Enterprises, Inc. | Proximity based entitlement sharing |
US9819902B2 (en) | 2015-03-19 | 2017-11-14 | Microsoft Technology Licensing, Llc | Proximate resource pooling in video/audio telecommunications |
US9848075B1 (en) | 2015-05-14 | 2017-12-19 | Invoy Technologies, Llc | Communication system for pairing user devices with medical devices |
US9913079B2 (en) | 2015-06-05 | 2018-03-06 | Apple Inc. | Cloud-based proximity pairing and switching for peer-to-peer devices |
US9801219B2 (en) | 2015-06-15 | 2017-10-24 | Microsoft Technology Licensing, Llc | Pairing of nearby devices using a synchronized cue signal |
US11106417B2 (en) | 2015-06-23 | 2021-08-31 | Airwatch, Llc | Collaboration systems with managed screen sharing |
US9602956B1 (en) | 2015-08-25 | 2017-03-21 | Yahoo! Inc. | System and method for device positioning with bluetooth low energy distributions |
US10031577B2 (en) | 2015-10-05 | 2018-07-24 | International Business Machines Corporation | Gaze-aware control of multi-screen experience |
US10244057B2 (en) | 2015-10-09 | 2019-03-26 | Adobe Systems Incorporated | Techniques for associating and sharing data from multiple local devices |
US10334076B2 (en) | 2016-02-22 | 2019-06-25 | Google Llc | Device pairing in augmented/virtual reality environment |
CN105704426B (en) | 2016-03-17 | 2019-10-25 | 华为技术有限公司 | A kind of methods, devices and systems carrying out video conference |
US10999331B1 (en) | 2016-07-06 | 2021-05-04 | Google Llc | Reverse discovery and pairing of client devices to a media device |
US20180267774A1 (en) | 2017-03-16 | 2018-09-20 | Cisco Technology, Inc. | Conference assistant device with configurable user interfaces based on operational state |
US10838681B2 (en) | 2017-04-05 | 2020-11-17 | Panasonic Avionics Corporation | Screen mirroring from personal electronic devices to transportation vehicle display screens |
EP3759957A4 (en) | 2018-03-02 | 2022-02-09 | Nitto Denko Corporation | Device pairing system and method, and device communication control system and method |
JP7073526B2 (en) | 2018-04-02 | 2022-05-23 | グーグル エルエルシー | Methods, devices, and systems for interactive cloud gaming |
US10824384B2 (en) | 2018-04-30 | 2020-11-03 | Dell Products L.P. | Controller for providing sharing between visual devices |
DK201870364A1 (en) | 2018-05-07 | 2019-12-03 | Apple Inc. | Multi-participant live communication user interface |
US11095659B2 (en) | 2018-05-30 | 2021-08-17 | Cisco Technology, Inc. | Personalized services based on confirmed proximity of user |
US10841174B1 (en) | 2018-08-06 | 2020-11-17 | Apple Inc. | Electronic device with intuitive control interface |
US20210004454A1 (en) | 2019-07-07 | 2021-01-07 | Apple Inc. | Proof of affinity to a secure event for frictionless credential management |
US11304246B2 (en) | 2019-11-01 | 2022-04-12 | Microsoft Technology Licensing, Llc | Proximity-based pairing and operation of user-specific companion devices |
US11256392B2 (en) | 2019-11-01 | 2022-02-22 | Microsoft Technology Licensing, Llc | Unified interfaces for paired user computing devices |
US11546391B2 (en) | 2019-11-01 | 2023-01-03 | Microsoft Technology Licensing, Llc | Teleconferencing interfaces and controls for paired user computing devices |
-
2017
- 2017-04-05 US US15/480,325 patent/US11212326B2/en active Active
- 2017-04-05 US US15/480,332 patent/US11310294B2/en active Active
- 2017-10-24 CN CN201780067327.2A patent/CN109891878A/en not_active Withdrawn
- 2017-10-24 WO PCT/US2017/057947 patent/WO2018081029A1/en unknown
- 2017-10-24 EP EP17794195.2A patent/EP3533221A1/en not_active Withdrawn
- 2017-10-24 WO PCT/US2017/057948 patent/WO2018081030A1/en unknown
- 2017-10-24 EP EP17794863.5A patent/EP3533222A1/en active Pending
- 2017-10-24 CN CN201780067116.9A patent/CN109923859B/en active Active
Also Published As
Publication number | Publication date |
---|---|
WO2018081029A1 (en) | 2018-05-03 |
US20180124128A1 (en) | 2018-05-03 |
US20180124136A1 (en) | 2018-05-03 |
WO2018081030A1 (en) | 2018-05-03 |
CN109923859B (en) | 2021-07-30 |
CN109923859A (en) | 2019-06-21 |
CN109891878A (en) | 2019-06-14 |
US11212326B2 (en) | 2021-12-28 |
EP3533221A1 (en) | 2019-09-04 |
US11310294B2 (en) | 2022-04-19 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11310294B2 (en) | Companion devices for real-time collaboration in communication sessions | |
US10863136B2 (en) | Switch view functions for teleconference sessions | |
US10863227B2 (en) | Event conditioned views for communication sessions | |
EP3533180B1 (en) | Integrated multitasking interface for telecommunication sessions | |
US10509964B2 (en) | Toggle view functions for teleconferencing sessions | |
US9608950B2 (en) | Systems and methods for sharing videos and images in a texting environment | |
US10996839B2 (en) | Providing consistent interaction models in communication sessions | |
US20180232920A1 (en) | Contextually aware location selections for teleconference monitor views | |
US20180295158A1 (en) | Displaying group expressions for teleconference sessions | |
CN110169079B (en) | Media content control of a source device on a sink device | |
US11785194B2 (en) | Contextually-aware control of a user interface displaying a video and related user text | |
US20200201512A1 (en) | Interactive editing system | |
US9924136B1 (en) | Coordinated display transitions of people and content | |
US10942633B2 (en) | Interactive viewing and editing system | |
US20230388357A1 (en) | 2d and 3d transitions for renderings of users participating in communication sessions | |
US20230388355A1 (en) | Automation of visual indicators for distinguishing active speakers of users displayed as three-dimensional representations |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: UNKNOWN |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20190411 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: EXAMINATION IS IN PROGRESS |
|
17Q | First examination report despatched |
Effective date: 20210930 |
|
RAP3 | Party data changed (applicant data changed or rights of an application transferred) |
Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC |