Classifications

• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
  • H04L65/40—Support for services or applications
  • H04L65/403—Arrangements for multi-party communication, e.g. for conferences
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
  • H04L65/60—Network streaming of media packets
  • H04L65/75—Media network packet handling
  • H04L65/765—Media network packet handling intermediate
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
  • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
  • H04L65/80—Responding to QoS
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
  • H04N21/24—Monitoring of processes or resources, e.g. monitoring of server load, available bandwidth, upstream requests
  • H04N21/2402—Monitoring of the downstream path of the transmission network, e.g. bandwidth available
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
  • H04N21/258—Client or end-user data management, e.g. managing client capabilities, user preferences or demographics, processing of multiple end-users preferences to derive collaborative data
  • H04N21/25808—Management of client data
  • H04N21/25825—Management of client data involving client display capabilities, e.g. screen resolution of a mobile phone
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
  • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
  • H04N21/258—Client or end-user data management, e.g. managing client capabilities, user preferences or demographics, processing of multiple end-users preferences to derive collaborative data
  • H04N21/25808—Management of client data
  • H04N21/25833—Management of client data involving client hardware characteristics, e.g. manufacturer, processing or storage capabilities
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/41—Structure of client; Structure of client peripherals
  • H04N21/414—Specialised client platforms, e.g. receiver in car or embedded in a mobile appliance
  • H04N21/41407—Specialised client platforms, e.g. receiver in car or embedded in a mobile appliance embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
  • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
  • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
  • H04N21/440245—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
  • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
  • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
  • H04N21/440263—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by altering the spatial resolution, e.g. for displaying on a connected PDA
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
  • H04N21/47—End-user applications
  • H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
  • H04N21/4728—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
  • H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
  • H04N21/631—Multimode Transmission, e.g. transmitting basic layers and enhancement layers of the content over different transmission paths or transmitting with different error corrections, different keys or with different transmission protocols
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
  • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
  • H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
  • H04N21/632—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing using a connection between clients on a wide area network, e.g. setting up a peer-to-peer communication via Internet for retrieving video segments from the hard-disk of other client devices
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N7/00—Television systems
  • H04N7/14—Systems for two-way working
  • H04N7/15—Conference systems

Definitions

• Embodiments relate to exchanging portions of a video stream via different links during a communication session.
• Wireless communication systems have developed through various generations, including a first-generation analog wireless phone service (1G), a second-generation (2G) digital wireless phone service (including interim 2.5G and 2.75G networks) and a third- generation (3G) high speed data, Internet-capable wireless service.
• 1G first-generation analog wireless phone service
• 2G second-generation digital wireless phone service
• 3G third- generation
• technologies including Cellular and Personal Communications Service (PCS) systems.
• PCS Personal Communications Service
• Examples of known cellular systems include the cellular Analog Advanced Mobile Phone System (AMPS), and digital cellular systems based on Code Division Multiple Access (CDMA), Frequency Division Multiple Access (FDMA), Time Division Multiple Access (TDM A), the Global System for Mobile access (GSM) variation of TDMA, and newer hybrid digital communication systems using both TDMA and CDMA technologies.
• CDMA Code Division Multiple Access
• FDMA Frequency Division Multiple Access
• TDM A Time Division Multiple Access
• GSM Global System for Mobile access
• the method for providing CDMA mobile communications was standardized in the United States by the Telecommunications Industry Association/Electronic Industries Association in TIA/EIA/IS-95-A entitled "Mobile Station-Base Station Compatibility Standard for Dual-Mode Wideband Spread Spectrum Cellular System," referred to herein as IS-95.
• Combined AMPS & CDMA systems are described in TIA/EIA Standard IS-98.
• Other communications systems are described in the IMT-2000/UM, or International Mobile Telecommunications System 2000/Universal Mobile Telecommunications System, standards covering what are referred to as wideband CDMA (W-CDMA), CDMA2000 (such as CDMA2000 lxEV-DO standards, for example) or TD-SCDMA.
• Node Bs also referred to as cell sites or cells
• UEs user equipments
• Node Bs provide entry points to an access network (AN) or radio access network (RAN), which is generally a packet data network using standard Internet Engineering Task Force (IETF) based protocols that support methods for differentiating traffic based on Quality of Service (QoS) requirements.
• AN access network
• RAN radio access network
• IP Internet Protocol
• Push-to-talk (PTT) capabilities are becoming popular with service sectors and consumers.
• PTT can support a "dispatch" voice service that operates over standard commercial wireless infrastructures, such as W- CDMA, CDMA, FDMA, TDMA, GSM, etc.
• endpoints e.g., UEs
• a dispatch call or simply a PTT call.
• a PTT call is an instantiation of a group, which defines the characteristics of a call.
• a group in essence is defined by a member list and associated information, such as group name or group identification.
• Telepresence refers to a set of technologies which allow a person to feel as if they were present, to give the appearance of being present. Additionally, users may be given the ability to affect the remote location. In this case, the user's position, movements, actions, voice may be sensed, transmitted and duplicated in the remote location to bring about this effect. Therefore information may be traveling in both directions between the user and the remote location. Telepresence via video deploys greater technical sophistication and improved fidelity of both sight and sound than in traditional videoconferencing.
• a user can physically show ideas using touch points, movements and gestures, which can be communicated synchronously on other UEs.
• the present invention presents a means for scaling and/or representation of data stream in a real-time streaming mobile collaboration environment in accordance to UEs display capabilities and bandwidth allocation.
• Display capabilities of UEs can vary depending in screen size, color resolution, frame rate, display resolution, color resolution, and aspect ratio. Additionally, display capabilities of UEs can vary depending on processor speed, device memory, software application. Alternatively, bandwidth allocation and the performance level of the connection to each UE can vary. Therefore, allocation for exchanging data stream varies among different transmitting and receiving UEs depending on each UE's display capabilities.
• Embodiments of the invention allow for the determination of the display capabilities of each UE, in order to prevent the bandwidth allocation for each UE from being either underutilized or over-utilized.
• the present invention presents a means for determining the capability of each UE and translating the data stream to be transmitted accordingly.
• the present invention presents a means for a server to transition the display data stream based on a physical user input for transmission in a telepresence environment.
• the invention also provides a means for determining the data capability of the target UEs and connection performance to the target UEs, and for adjusting transmission of the display data accordingly.
• a UE is participating in a communication session that shares a video stream with target UE(s).
• the UE receives user input that identifies high priority portion(s) of the video stream.
• the UE generates a first video feed based on the high priority portion(s) and a second video feed based at least on other portion(s) of the video stream.
• the first and second video feeds are exchanged with the target UE(s) on first and second links, respectively.
• the first link that carries the first video feed can be allocated QoS.
• the target UE(s) combine the first and second video feeds to reconstruct a version of the video stream, and then present the reconstructed version of the video stream.
• FIG. 1 is a diagram of a wireless network architecture that supports access terminals and access networks in accordance with at least one embodiment of the invention.
• FIG. 2A illustrates the core network of FIG. 1 according to an embodiment of the present invention.
• FIG. 2B illustrates the core network of FIG. 1 according to another embodiment of the present invention.
• FIG. 2C illustrates an example of the wireless communications system of FIG. 1 in more detail.
• FIG. 3 is an illustration of a user equipment (UE) in accordance with at least one embodiment of the invention.
• FIG. 4 illustrates a communication device that includes logic configured to receive and/or transmit information.
• FIG. 5 illustrates a process of exchanging data representative of physical user input during a group communication in accordance with an embodiment of the present invention.
• FIG. 6 illustrates a communication flow that is based upon an execution of the process of FIG. 5 in accordance with an embodiment of the invention.
• FIG. 7A illustrates an example implementation of the process of FIG. 5 in accordance with an embodiment of the invention.
• FIG. 7B illustrates an example of the original representation of physical user input for a user that draws a circle on a display screen of a UE and corresponding representations of the physical user input at target UEs in accordance with an embodiment of the invention.
• FIG. 7C illustrates a more detailed implementation of FIG. 7A in accordance with an embodiment of the invention.
• FIG. 8A illustrates an implementation of a portion of FIG. 7A in accordance with an embodiment of the invention.
• FIG. 8B illustrates an example implementation of FIG. 8A in accordance with an embodiment of the invention.
• FIG. 9 illustrates a process of selectively adjusting display settings for one or more target UEs during a communication session based on received user-generated physical input.
• FIG. 10 illustrates an example implementation of the process described in FIG. 9 in accordance with an embodiment of the invention.
• FIG. 11 is directed to a process of selectively transmitting different video feeds associated with a video stream being displayed on a given UE to at least one target UE in accordance with an embodiment of the present invention.
• FIG. 12 is directed to a continuation of the process of FIG. 11 from the perspective of a target UE that receives the first and second video feeds via the first and second links, respectively, from the given UE in accordance with an embodiment of the invention.
• FIG. 13 illustrates an example implementation of how the set of high priority portions of the video stream can be identified during the process of FIG. 11 in accordance with an embodiment of the invention.
• FIG. 14 illustrates an explicit selection of a video stream portion which can occur during the process of FIG. 13 in accordance with an embodiment of the invention.
• FIG. 15A illustrates an example implementation of a portion of the process of FIG. 12 in accordance with an embodiment of the invention.
• FIG. 15B illustrates another example implementation of a portion of the process of FIG. 12 in accordance with an embodiment of the invention.
• FIG. 16 illustrates an example implementation of how the set of high priority portions of the video stream can be identified during the process of FIG. 11 in accordance with another embodiment of the invention.
• FIG. 17A illustrates an example of user input which can trigger a portion of the process of FIG. 16 in accordance with an embodiment of the invention.
• FIG. 17B illustrates another example of user input which can trigger a portion of the process of FIG. 16 in accordance with an embodiment of the invention.
• FIGS. 18A-18D illustrate an example implementation of the process of FIG. 17 A in conjunction with FIGS. 11 and 16 in accordance with an embodiment of the invention.
• FIG. 19A illustrates an example implementation of a portion of FIG. 12 in accordance with an embodiment of the invention.
• FIG. 19B illustrates another example implementation of a portion of FIG. 12 in accordance with an embodiment of the invention, a portion of FIG. 12 in accordance with an embodiment of the invention.
• a High Data Rate (HDR) subscriber station referred to herein as user equipment (UE), may be mobile or stationary, and may communicate with one or more access points (APs), which may be referred to as Node Bs.
• UE transmits and receives data packets through one or more of the Node Bs to a Radio Network Controller (RNC).
• RNC Radio Network Controller
• the Node Bs and RNC are parts of a network called a radio access network (RAN).
• RAN radio access network
• a radio access network can transport voice and data packets between multiple access terminals.
• the radio access network may be further connected to additional networks outside the radio access network, such core network including specific carrier related servers and devices and connectivity to other networks such as a corporate intranet, the Internet, public switched telephone network (PSTN), a Serving General Packet Radio Services (GPRS) Support Node (SGSN), a Gateway GPRS Support Node (GGSN), and may transport voice and data packets between each UE and such networks.
• PSTN public switched telephone network
• GPRS General Packet Radio Services
• SGSN Serving General Packet Radio Services
• GGSN Gateway GPRS Support Node
• a UE that has established an active traffic channel connection with one or more Node Bs may be referred to as an active UE, and can be referred to as being in a traffic state.
• a UE that is in the process of establishing an active traffic channel (TCH) connection with one or more Node Bs can be referred to as being in a connection setup state.
• TCH active traffic channel
• a UE may be any data device that communicates through a wireless channel or through a wired channel.
• a UE may further be any of a number of types of devices including but not limited to PC card, compact flash device, external or internal modem, or wireless or wireline phone.
• the communication link through which the UE sends signals to the Node B(s) is called an uplink channel (e.g., a reverse traffic channel, a control channel, an access channel, etc.).
• the communication link through which Node B(s) send signals to a UE is called a downlink channel (e.g., a paging channel, a control channel, a broadcast channel, a forward traffic channel, etc.).
• traffic channel can refer to either an uplink/reverse or downlink/forward traffic channel.
• FIG. 1 illustrates a block diagram of one exemplary embodiment of a wireless communications system 100 in accordance with at least one embodiment of the invention.
• System 100 can contain UEs, such as cellular telephone 102, in communication across an air interface 104 with an access network or radio access network (RAN) 120 that can connect the UE 102 to network equipment providing data connectivity between a packet switched data network (e.g., an intranet, the Internet, and/or core network 126) and the UEs 102, 108, 110, 112.
• a packet switched data network e.g., an intranet, the Internet, and/or core network 126)
• the UE can be a cellular telephone 102, a personal digital assistant 108, a pager 110, which is shown here as a two-way text pager, or even a separate computer platform 112 that has a wireless communication portal.
• Embodiments of the invention can thus be realized on any form of UE including a wireless communication portal or having wireless communication capabilities, including without limitation, wireless modems, PCMCIA cards, personal computers, telephones, or any combination or sub-combination thereof.
• UE in other communication protocols (i.e., other than W-CDMA) may be referred to interchangeably as an "access terminal”, “AT”, “wireless device”, “client device”, “mobile terminal”, “mobile station” and variations thereof.
• System 100 is merely exemplary and can include any system that allows remote UEs, such as wireless client computing devices 102, 108, 110, 112 to communicate over-the-air between and among each other and/or between and among components connected via the air interface 104 and RAN 120, including, without limitation, core network 126, the Internet, PSTN, SGSN, GGSN and/or other remote servers.
• remote UEs such as wireless client computing devices 102, 108, 110, 112 to communicate over-the-air between and among each other and/or between and among components connected via the air interface 104 and RAN 120, including, without limitation, core network 126, the Internet, PSTN, SGSN, GGSN and/or other remote servers.
• the RAN 120 controls messages (typically sent as data packets) sent to a RNC 122.
• the RNC 122 is responsible for signaling, establishing, and tearing down bearer channels (i.e., data channels) between a Serving General Packet Radio Services (GPRS) Support Node (SGSN) and the UEs 102/108/110/112. If link layer encryption is enabled, the RNC 122 also encrypts the content before forwarding it over the air interface 104.
• the function of the RNC 122 is well-known in the art and will not be discussed further for the sake of brevity.
• the core network 126 may communicate with the RNC 122 by a network, the Internet and/or a public switched telephone network (PSTN).
• PSTN public switched telephone network
• the RNC 122 may connect directly to the Internet or external network.
• the network or Internet connection between the core network 126 and the RNC 122 transfers data, and the PSTN transfers voice information.
• the RNC 122 can be connected to multiple Node Bs 124.
• the RNC 122 is typically connected to the Node Bs 124 by a network, the Internet and/or PSTN for data transfer and/or voice information.
• the Node Bs 124 can broadcast data messages wirelessly to the UEs, such as cellular telephone 102.
• the Node Bs 124, RNC 122 and other components may form the RAN 120, as is known in the art. However, alternate configurations may also be used and the invention is not limited to the configuration illustrated.
• the functionality of the RNC 122 and one or more of the Node Bs 124 may be collapsed into a single "hybrid" module having the functionality of both the RNC 122 and the Node B(s) 124.
• FIG. 2A illustrates the core network 126 according to an embodiment of the present invention.
• FIG. 2A illustrates components of a General Packet Radio Services (GPRS) core network implemented within a W-CDMA system.
• the core network 126 includes a Serving GPRS Support Node (SGSN) 160, a Gateway GPRS Support Node (GGSN) 165 and an Internet 175.
• SGSN Serving GPRS Support Node
• GGSN Gateway GPRS Support Node
• portions of the Internet 175 and/or other components may be located outside the core network in alternative embodiments.
• GPRS is a protocol used by Global System for Mobile communications (GSM) phones for transmitting Internet Protocol (IP) packets.
• GSM Global System for Mobile communications
• IP Internet Protocol
• the GPRS Core Network e.g., the GGSN 165 and one or more SGSNs 160
• the GPRS core network is an integrated part of the GSM core network, provides mobility management, session management and transport for IP packet services in GSM and W- CDMA networks.
• the GPRS Tunneling Protocol is the defining IP protocol of the GPRS core network.
• the GTP is the protocol which allows end users (e.g., UEs) of a GSM or W-CDMA network to move from place to place while continuing to connect to the internet as if from one location at the GGSN 165. This is achieved transferring the subscriber's data from the subscriber's current SGSN 160 to the GGSN 165, which is handling the subscriber's session.
• GTP-U is used for transfer of user data in separated tunnels for each packet data protocol (PDP) context.
• PDP packet data protocol
• GTP-C is used for control signaling (e.g., setup and deletion of PDP contexts, verification of GSN reach-ability, updates or modifications such as when a subscriber moves from one SGSN to another, etc.).
• GTP' is used for transfer of charging data from GSNs to a charging function.
• the GGSN 165 acts as an interface between the GPRS backbone network (not shown) and the Internet (i.e., an external packet data network) 175.
• the GGSN 165 extracts the packet data with associated packet data protocol (PDP) format (e.g., IP or PPP) from the GPRS packets coming from the SGSN 160, and sends the packets out on a corresponding packet data network.
• PDP packet data protocol
• the incoming data packets are directed by the GGSN 165 to the SGSN 160 which manages and controls the Radio Access Bearer (RAB) of the destination UE served by the RAN 120.
• RAB Radio Access Bearer
• the GGSN 165 stores the current SGSN address of the target UE and his/her profile in its location register (e.g., within a PDP context).
• the GGSN is responsible for IP address assignment and is the default router for the connected UE.
• the GGSN also performs authentication and charging functions.
• the SGSN 160 is representative of one of many SGSNs within the core network 126, in an example. Each SGSN is responsible for the delivery of data packets from and to the UEs within an associated geographical service area. The tasks of the SGSN 160 includes packet routing and transfer, mobility management (e.g., attach/detach and location management), logical link management, and authentication and charging functions.
• the location register of the SGSN stores location information (e.g., current cell, current VLR) and user profiles (e.g., IMSI, PDP address(es) used in the packet data network) of all GPRS users registered with the SGSN 160, for example, within one or more PDP contexts for each user or UE.
• location information e.g., current cell, current VLR
• user profiles e.g., IMSI, PDP address(es) used in the packet data network
• SGSNs are responsible for (i) de-tunneling downlink GTP packets from the GGSN 165, (ii) uplink tunnel IP packets toward the GGSN 165, (iii) carrying out mobility management as UEs move between SGSN service areas and (iv) billing mobile subscribers.
• SGSNs configured for GSM/EDGE networks have slightly different functionality as compared to SGSNs configured for W-CDMA networks.
• the RAN 120 communicates with the SGSN 160 via a Radio Access Network Application Part (RANAP) protocol.
• RANAP operates over a Iu interface (Iu-ps), with a transmission protocol such as Frame Relay or IP.
• Iu-ps Iu interface
• the SGSN 160 communicates with the GGSN 165 via a Gn interface, which is an IP-based interface between SGSN 160 and other SGSNs (not shown) and internal GGSNs, and uses the GTP protocol defined above (e.g., GTP-U, GTP-C, GTP', etc.).
• GTP protocol defined above
• the Gn between the SGSN 160 and the GGSN 165 carries both the GTP-C and the GTP-U. While not shown in FIG. 2A, the Gn interface is also used by the Domain Name System (DNS).
• DNS Domain Name System
• the GGSN 165 is connected to a Public Data Network (PDN) (not shown), and in turn to the Internet 175, via a Gi interface with IP protocols either directly or through a Wireless Application Protocol (WAP) gateway.
• PDN Public Data Network
• Gi Wireless Application Protocol
• FIG. 2B illustrates the core network 126 according to another embodiment of the present invention.
• FIG. 2B is similar to FIG. 2A except that FIG. 2B illustrates an implementation of direct tunnel functionality.
• Direct Tunnel is an optional function in Iu mode that allows the SGSN 160 to establish a direct user plane tunnel, GTP-U, between RAN and GGSN within the Packet Switched (PS) domain.
• a direct tunnel capable SGSN such as SGSN 160 in FIG. 2B, can be configured on a per GGSN and per RNC basis whether or not the SGSN can use a direct user plane connection.
• the SGSN 160 in FIG. 2B handles the control plane signaling and makes the decision of when to establish Direct Tunnel.
• RAB Radio Bearer
• the GTP-U tunnel is established between the GGSN 165 and SGSN 160 in order to be able to handle the downlink packets.
• the optional Direct Tunnel between the SGSN 160 and GGSN 165 is not typically allowed (i) in the roaming case (e.g., because the SGSN needs to know whether the GGSN is in the same or different PLMN), (ii) where the SGSN has received Customized Applications for Mobile Enhanced Logic (CAMEL) Subscription Information in the subscriber profile from a Home Location Register (HLR) and/or (iii) where the GGSN 165 does not support GTP protocol version 1.
• HLR Home Location Register
• iii) where the GGSN 165 does not support GTP protocol version 1.
• the CAMEL restriction if Direct Tunnel is established then volume reporting from SGSN 160 is not possible as the SGSN 160 no longer has visibility of the User Plane.
• a CAMEL server can invoke volume reporting at anytime during the life time of a PDP Context, the use of Direct Tunnel is prohibited for a subscriber whose profile contains CAMEL Subscription Information.
• the SGSN 160 can be operating in a Packet Mobility Management (PMM)- detached state, a PMM-idle state or a PMM-connected state.
• PMM Packet Mobility Management
• the GTP- connections shown in FIG. 2B for Direct Tunnel function can be established whereby the SGSN 160 is in the PMM-connected state and receives an Iu connection establishment request from the UE.
• the SGSN 160 ensures that the new Iu connection and the existing Iu connection are for the same UE, and if so, the SGSN 160 processes the new request and releases the existing Iu connection and all RABs associated with it.
• the SGSN 160 may perform security functions.
• the SGSN 160 sends an Update PDP Context Request(s) to the associated GGSN(s) 165 to establish the GTP tunnels between the SGSN 160 and GGSN(s) 165 in case the Iu connection establishment request is for signaling only.
• the SGSN 160 may immediately establish a new direct tunnel and send Update PDP Context Request(s) to the associated GGSN(s) 165 and include the RNC's Address for User Plane, a downlink Tunnel Endpoint Identifier (TEID) for data in case the Iu connection establishment request is for data transfer.
• TEID downlink Tunnel Endpoint Identifier
• the UE also performs a Routing Area Update (RAU) procedure immediately upon entering PMM-IDLE state when the UE has received an RRC Connection Release message with cause "Directed Signaling connection re-establishment" even if the Routing Area has not changed since the last update.
• RAU Routing Area Update
• the RNC will send the RRC Connection Release message with cause "Directed Signaling Connection re- establishment" when the RNC is unable to contact the Serving RNC to validate the UE due to lack of lur connection (e.g., see TS 25.331 [52]).
• the UE performs a subsequent service request procedure after successful completion of the RAU procedure to reestablish the radio access bearer when the UE has pending user data to send.
• the PDP context is a data structure present on both the SGSN 160 and the GGSN 165 which contains a particular UE's communication session information when the UE has an active GPRS session.
• the UE When a UE wishes to initiate a GPRS communication session, the UE must first attach to the SGSN 160 and then activate a PDP context with the GGSN 165. This allocates a PDP context data structure in the SGSN 160 that the subscriber is currently visiting and the GGSN 165 serving the UE's access point.
• FIG. 2C illustrates an example of the wireless communications system 100 of FIG. 1 in more detail.
• UEs 1...N are shown as connecting to the RAN 120 at locations serviced by different packet data network end-points.
• the illustration of FIG. 2C is specific to W-CDMA systems and terminology, although it will be appreciated how FIG. 2C could be modified to conform with a lx EV-DO system.
• UEs 1 and 3 connect to the RAN 120 at a portion served by a first packet data network end-point 162 (e.g., which may correspond to SGSN, GGSN, PDSN, a home agent (HA), a foreign agent (FA), etc.).
• a first packet data network end-point 162 e.g., which may correspond to SGSN, GGSN, PDSN, a home agent (HA), a foreign agent (FA), etc.
• the first packet data network end-point 162 in turn connects, via the routing unit 188, to the Internet 175 and/or to one or more of an authentication, authorization and accounting (AAA) server 182, a provisioning server 184, an Internet Protocol (IP) Multimedia Subsystem (IMS) / Session Initiation Protocol (SIP) Registration Server 186 and/or the application server 170.
• IP Internet Protocol
• IMS Internet Multimedia Subsystem
• SIP Session Initiation Protocol
• UEs 2 and 5...N connect to the RAN 120 at a portion served by a second packet data network end-point 164 (e.g., which may correspond to SGSN, GGSN, PDSN, FA, HA, etc.).
• the second packet data network end-point 164 in turn connects, via the routing unit 188, to the Internet 175 and/or to one or more of the AAA server 182, a provisioning server 184, an IMS / SIP Registration Server 186 and/or the application server 170.
• UE 4 connects directly to the Internet 175, and through the Internet 175 can then connect to any of the system components described above.
• UEs 1, 3 and 5...N are illustrated as wireless cell-phones, UE 2 is illustrated as a wireless tablet-PC and UE 4 is illustrated as a wired desktop station.
• the wireless communication system 100 can connect to any type of UE, and the examples illustrated in FIG. 2C are not intended to limit the types of UEs that may be implemented within the system.
• the AAA 182, the provisioning server 184, the IMS/SIP registration server 186 and the application server 170 are each illustrated as structurally separate servers, one or more of these servers may be consolidated in at least one embodiment of the invention.
• the application server 170 is illustrated as including a plurality of media control complexes (MCCs) 1...N 170B, and a plurality of regional dispatchers 1...N 170A.
• MCCs media control complexes
• the regional dispatchers 170A and MCCs 170B are included within the application server 170, which in at least one embodiment can correspond to a distributed network of servers that collectively functions to arbitrate communication sessions (e.g., half-duplex group communication sessions via IP unicasting and/or IP multicasting protocols) within the wireless communication system 100.
• communication sessions arbitrated by the application server 170 can theoretically take place between UEs located anywhere within the system 100, multiple regional dispatchers 170A and MCCs are distributed to reduce latency for the arbitrated communication sessions (e.g., so that an
• the regional dispatchers 170A are generally responsible for any functionality related to establishing a communication session (e.g., handling signaling messages between the UEs, scheduling and/or sending announce messages, etc.), whereas the MCCs 170B are responsible for hosting the communication session for the duration of the call instance, including conducting an in-call signaling and an actual exchange of media during an arbitrated communication session.
• a UE 200 (here a wireless device), such as a cellular telephone, has a platform 202 that can receive and execute software applications, data and/or commands transmitted from the RAN 120 that may ultimately come from the core network 126, the Internet and/or other remote servers and networks.
• the platform 202 can include a transceiver 206 operably coupled to an application specific integrated circuit ("ASIC" 208), or other processor, microprocessor, logic circuit, or other data processing device.
• ASIC 208 or other processor executes the application programming interface ("API') 210 layer that interfaces with any resident programs in the memory 212 of the wireless device.
• API' application programming interface
• the memory 212 can be comprised of read-only or random-access memory (RAM and ROM), EEPROM, flash cards, or any memory common to computer platforms.
• the platform 202 also can include a local database 214 that can hold applications not actively used in memory 212.
• the local database 214 is typically a flash memory cell, but can be any secondary storage device as known in the art, such as magnetic media, EEPROM, optical media, tape, soft or hard disk, or the like.
• the internal platform 202 components can also be operably coupled to external devices such as antenna 222, display 224, push-to-talk button 228 and keypad 226 among other components, as is known in the art.
• an embodiment of the invention can include a UE including the ability to perform the functions described herein.
• the various logic elements can be embodied in discrete elements, software modules executed on a processor or any combination of software and hardware to achieve the functionality disclosed herein.
• ASIC 208, memory 212, API 210 and local database 214 may all be used cooperatively to load, store and execute the various functions disclosed herein and thus the logic to perform these functions may be distributed over various elements.
• the functionality could be incorporated into one discrete component. Therefore, the features of the UE 200 in FIG. 3 are to be considered merely illustrative and the invention is not limited to the illustrated features or arrangement.
• the wireless communication between the UE 102 or 200 and the RAN 120 can be based on different technologies, such as code division multiple access (CDMA), W- CDMA, time division multiple access (TDMA), frequency division multiple access (FDMA), Orthogonal Frequency Division Multiplexing (OFDM), the Global System for Mobile Communications (GSM), or other protocols that may be used in a wireless communications network or a data communications network.
• CDMA code division multiple access
• W-CDMA time division multiple access
• FDMA frequency division multiple access
• OFDM Orthogonal Frequency Division Multiplexing
• GSM Global System for Mobile Communications
• the data communication is typically between the client device 102, Node B(s) 124, and the RNC 122.
• the RNC 122 can be connected to multiple data networks such as the core network 126, PSTN, the Internet, a virtual private network, a SGSN, a GGSN and the like, thus allowing the UE 102 or 200 access to a broader communication network.
• voice transmission and/or data can be transmitted to the UEs from the RAN using a variety of networks and configurations. Accordingly, the illustrations provided herein are not intended to limit the embodiments of the invention and are merely to aid in the description of aspects of embodiments of the invention.
• FIG. 4 illustrates a communication device 400 that includes logic configured to perform functionality.
• the communication device 400 can correspond to any of the above-noted communication devices, including but not limited to UEs 102, 108, 110, 112 or 200, Node Bs or base stations 120, the RNC or base station controller 122, a packet data network end-point (e.g., SGSN 160, GGSN 165, a Mobility Management Entity (MME) in Long Term Evolution (LTE), etc.), any of the servers 170 through 186, etc.
• MME Mobility Management Entity
• LTE Long Term Evolution
• communication device 400 can correspond to any electronic device that is configured to communicate with (or facilitate communication with) one or more other entities over a network.
• the communication device 400 includes logic configured to receive and/or transmit information 405.
• the logic configured to receive and/or transmit information 405 can include a wireless communications interface (e.g., Bluetooth, WiFi, 2G, 3G, etc.) such as a wireless transceiver and associated hardware (e.g., an RF antenna, a MODEM, a modulator and/or demodulator, etc.).
• a wireless communications interface e.g., Bluetooth, WiFi, 2G, 3G, etc.
• a wireless transceiver and associated hardware e.g., an RF antenna, a MODEM, a modulator and/or demodulator, etc.
• the logic configured to receive and/or transmit information 405 can correspond to a wired communications interface (e.g., a serial connection, a USB or Firewire connection, an Ethernet connection through which the Internet 175 can be accessed, etc.).
• a wired communications interface e.g., a serial connection, a USB or Firewire connection, an Ethernet connection through which the Internet 175 can be accessed, etc.
• the communication device 400 corresponds to some type of network-based server (e.g., SGSN 160, GGSN 165, application server 170, etc.)
• the logic configured to receive and/or transmit information 405 can correspond to an Ethernet card, in an example, that connects the network-based server to other communication entities via an Ethernet protocol.
• the logic configured to receive and/or transmit information 405 can include sensory or measurement hardware by which the communication device 400 can monitor its local environment (e.g., an accelerometer, a temperature sensor, a light sensor, an antenna for monitoring local RF signals, etc.).
• the logic configured to receive and/or transmit information 405 can also include software that, when executed, permits the associated hardware of the logic configured to receive and/or transmit information 405 to perform its reception and/or transmission function(s).
• the logic configured to receive and/or transmit information 405 does not correspond to software alone, and the logic configured to receive and/or transmit information 405 relies at least in part upon hardware to achieve its functionality.
• the communication device 400 further includes logic configured to process information 410.
• the logic configured to process information 410 can include at least a processor.
• Example implementations of the type of processing that can be performed by the logic configured to process information 410 includes but is not limited to performing determinations, establishing connections, making selections between different information options, performing evaluations related to data, interacting with sensors coupled to the communication device 400 to perform measurement operations, converting information from one format to another (e.g., between different protocols such as .wmv to .avi, etc.), and so on.
• the processor included in the logic configured to process information 410 can correspond to a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein.
• a general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine.
• a processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
• the logic configured to process information 410 can also include software that, when executed, permits the associated hardware of the logic configured to process information 410 to perform its processing function(s). However, the logic configured to process information 410 does not correspond to software alone, and the logic configured to process information 410 relies at least in part upon hardware to achieve its functionality.
• the communication device 400 further includes logic configured to store information 415.
• the logic configured to store information 415 can include at least a non-transitory memory and associated hardware (e.g., a memory controller, etc.).
• the non-transitory memory included in the logic configured to store information 415 can correspond to RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
• the logic configured to store information 415 can also include software that, when executed, permits the associated hardware of the logic configured to store information 415 to perform its storage function(s). However, the logic configured to store information 415 does not correspond to software alone, and the logic configured to store information 415 relies at least in part upon hardware to achieve its functionality.
• the communication device 400 further optionally includes logic configured to present information 420.
• the logic configured to display information 420 can include at least an output device and associated hardware.
• the output device can include a video output device (e.g., a display screen, a port that can carry video information such as USB, HDMI, etc.), an audio output device (e.g., speakers, a port that can carry audio information such as a microphone jack, USB, HDMI, etc.), a vibration device and/or any other device by which information can be formatted for output or actually outputted by a user or operator of the communication device 400.
• a video output device e.g., a display screen, a port that can carry video information such as USB, HDMI, etc.
• an audio output device e.g., speakers, a port that can carry audio information such as a microphone jack, USB, HDMI, etc.
• a vibration device e.g., a vibration device by which information can be formatted for output or actually outputted by a user or operator of the
• the logic configured to present information 420 can include the display 224.
• the logic configured to present information 420 can be omitted for certain communication devices, such as network communication devices that do not have a local user (e.g., network switches or routers, remote servers, etc.).
• the logic configured to present information 420 can also include software that, when executed, permits the associated hardware of the logic configured to present information 420 to perform its presentation function(s).
• the logic configured to present information 420 does not correspond to software alone, and the logic configured to present information 420 relies at least in part upon hardware to achieve its functionality.
• the communication device 400 further optionally includes logic configured to receive local user input 425.
• the logic configured to receive local user input 425 can include at least a user input device and associated hardware.
• the user input device can include buttons, a touch-screen display, a keyboard, a camera, an audio input device (e.g., a microphone or a port that can carry audio information such as a microphone jack, etc.), and/or any other device by which information can be received from a user or operator of the communication device 400.
• the logic configured to receive local user input 425 can include the display 224 (if implemented a touch-screen), keypad 226, etc.
• the logic configured to receive local user input 425 can be omitted for certain communication devices, such as network communication devices that do not have a local user (e.g., network switches or routers, remote servers, etc.).
• the logic configured to receive local user input 425 can also include software that, when executed, permits the associated hardware of the logic configured to receive local user input 425 to perform its input reception function(s).
• the logic configured to receive local user input 425 does not correspond to software alone, and the logic configured to receive local user input 425 relies at least in part upon hardware to achieve its functionality.
• any software used to facilitate the functionality of the configured logics of 405 through 425 can be stored in the non-transitory memory associated with the logic configured to store information 415, such that the configured logics of 405 through 425 each performs their functionality (i.e., in this case, software execution) based in part upon the operation of software stored by the logic configured to store information 405.
• hardware that is directly associated with one of the configured logics can be borrowed or used by other configured logics from time to time.
• the processor of the logic configured to process information 410 can format data into an appropriate format before being transmitted by the logic configured to receive and/or transmit information 405, such that the logic configured to receive and/or transmit information 405 performs its functionality (i.e., in this case, transmission of data) based in part upon the operation of hardware (i.e., the processor) associated with the logic configured to process information 410.
• the configured logics or "logic configured to" of 405 through 425 are not limited to specific logic gates or elements, but generally refer to the ability to perform the functionality describe herein (either via hardware or a combination of hardware and software).
• the configured logics or “logic configured to" of 405 through 425 are not necessarily implemented as logic gates or logic elements despite sharing the word "logic". Other interactions or cooperation between the configured logics 405 through 425 will become clear to one of ordinary skill in the art from a review of the embodiments described below in more detail.
• media e.g., audio, video, text, etc.
• media can be presented by UEs during a server- arbitrated group communication session.
• video media can be transmitted from a floor-holder to the application server 170, and the application server 170 can then re-transmit the video media to the group for presentation at target UE(s).
• one or more of the UEs participating in the session can receive physical user input from their respective users and then transmit control information indicative of the received physical user input to the rest of the group.
• the physical user input can correspond to a form of telestration (e.g., on-screen drawing) whereby a user of a given UE circles a portion of an associated display, a graphic representative of the user's circle is added to the display and then transmitted to the rest of the group where the circle is re-constituted.
• the given user's attempt to highlight a point of interest on his/her display via the circle is disseminated to the rest of the group and overlaid on top of the rendering of the video media at the respective target UE(s).
• UEs with different presentation capabilities and/or connection performance levels can participate in the same server- arbitrated communication session.
• a UE connected to a 3G network may be part of the same communication session as a UE connected to a 4G or WLAN.
• a UE with a high- resolution display can be part of the same communication session as a UE with a low- resolution display.
• an embodiment of the invention is directed to an implementation whereby visual representations of physical user input are shared between UEs participating a group communication session in accordance with the capabilities and/or performance levels associated with the target UE(s).
• FIG. 5 illustrates a process of exchanging data representative of physical user input during a group communication in accordance with an embodiment of the present invention.
• UEs 1...N e.g., where N>2
• the communication session can correspond to a video conferencing session whereby the same video and/or image media is displayed at each of UEs 1...N (e.g., a collaborative map session, etc.).
• different video and/or image media can be presented at two or more of UEs 1...N (e.g., UE 1 may view UE 2's video media, UE 2 may view UE l's video media, and so on).
• the video and/or image media need not be actively mediated through the application server 170, but could also be rendered independently at UEs 1...N.
• audio media can be mediated by the application server 170 (e.g., half duplex or full duplex), but the video and/or image media could be loaded separately at the UEs 1...N.
• UEs 1...N could each be independently rendering a map of New York while discussing their travel plans so that the map data does not need to be actively exchanged between the UEs during the communication session.
• the application server 170 receives data from a given UE (“UE 1") that is configured to visually represent a physical user input at the given UE, 500.
• the physical user input that is configured to be visually represented by the data received at 500 can include a user of UE 1 circling a relevant part of the display on the given UE with his/her finger, the user highlighting a portion of the display on the given UE, and so on.
• the data representative of the physical user input that is received at 500 can be received in many different formats or levels of precision.
• the received data at 500 can correspond to a set of screen coordinates that were associated with the physical user input (e.g., which when connected collectively form a circle, a squiggly line, etc.).
• the application server 170 determines data presentation capabilities of at least one target UE (e.g., one or more of UEs 2...N) and/or a connection performance level from the application server 170 to the at least one target UE, 505.
• the determined presentation capabilities of the at least one target UE can include display capability of the at least one target UE, such as, but not limited to, display size, color resolution, frame rate, display resolution, aspect ratio and so on.
• the determined data presentation capabilities of the at least one target UE can depend on a performance capability of the at least one target UE, such as, but not limited to, processor speed, memory capacity, type of memory, clock frequency, battery life and power conservation requirements.
• the application server 170 can also determine the connection performance level associated with the application server 170's connection to the at least one target UE.
• the performance level to the at least one target UE can be inferred based on packet loss, round trip delay or other in-call parameters.
• the performance level to the target UE can be based upon information related to a serving network of the at least one target UE.
• the application server 170 may generally determine higher performance capabilities for 4G-connected UEs as compared to 3G-connected UEs.
• the application server 170 may be aware of network-specific performance expectations (e.g., from prior interactions serving UEs over the same network or the same type of network, etc.)
• the application server 170 selectively transitions the received data from a first level of precision (e.g., a high-quality or full-quality format as received from the given UE at 500) to a second level of precision (e.g., a reduced quality format) based on the determined data presentation capabilities and/or connection performance level for the target UE, 510.
• a first level of precision e.g., a high-quality or full-quality format as received from the given UE at 500
• a second level of precision e.g., a reduced quality format
• the transitioning of 510 can reduce the number of screen coordinates to a number that is appropriate for delivery and/or presentation at the at least one target UE based on the determined presentation capabilities and/or connection performance level of the at least one target UE (e.g., 700 screen-coordinates, coordinates for a center-point and a size of a pre-defined shape, screen-coordinates that correspond to vertexes of a pre-defined polygon and an associated center-point, etc.).
• the application server 170 can prioritize the transition of the received data at 510 based on an expected level of human sensitivity to each aspect targeted for transition or reduction in order to improve the user experience (e.g., reduce resolution but not frame-rate, etc.).
• the received data representative of the physical user input from 500 includes complex forms, shading, and color coding.
• the transition of this data to the second level of precision at 510 can include simplifying the complex form, reducing or eliminating the shading and/or reducing the number of associated colors.
• the transitioned data at 510 can be scaled down to a thumbnail size and then "stretched" for presentation once received by the particular target UE in order to fill its display screen.
• the shape of the received data of 500 can be reconstructed to reduce image size.
• the received data from the given UE at 500 can contain image data that is representative of a user' s selections on a map, and the received data can be converted from the image data into a set of GPS location and/or Cartesian coordinates for the map, so that the image data can be reconstructed at the display of the at least one target UE.
• the manner in which the received data is transitioned in 510 can be the same for each target UE, or alternatively can vary between target UEs based on UE-specific determinations from 505.
• the application server 170 transmits the selectively transitioned data to the at least one target UE for presentation thereon, 515.
• FIG. 6 illustrates a communication flow that is based upon an execution of the process of FIG. 5 in accordance with an embodiment of the invention.
• UE 1 and UE 2 are engaged in a communication session that involves some type of collaborative graphical display of media.
• a user of UE 1 provides physical user input that results in a squiggly, complex shape shown in shape 600.
• the user of UE 1 may have waved his/her finger in proximity to a touchscreen display of UE 1, after which UE l's sensors record the finger- waving for display as the shape 600.
• the shape 600 may be referred to as an original or full-quality (or high precision) representation of the physical user input (e.g., at least relative to the initial recording or capturing of the physical user input).
• the original representation of the physical user input can include encoding of a plurality of coordinates and/or vertexes that collectively define the shape when rendered on the display of UE 1.
• Data representative of the shape 600 is transmitted by UE 1 to the application server 170, 605.
• FIG. 6 shows the transmission of the selectively transitioned data that is representative of the shape 600, albeit in a reduced or simplified level of precision.
• the target UE e.g., UE 2 receives the selectively transitioned data and presents a modified shape 615.
• the modified shape 615 is still representative of (or faithful to) the initial physical user input at UE 1, but the modified shape is somewhat simpler and/or reduced in comparison to the full-quality representation of the physical user input that was captured and then presented at UE 1.
• the reduction and/or simplification to the data representative of the physical user input can occur for a number of reasons as noted above with respect to FIG. 5, such as a low-bandwidth connection between the application server 170 and UE 2, display restrictions associated with UE 2, etc.
• FIG. 7A illustrates an example implementation of the process of FIG. 5 in accordance with an embodiment of the invention.
• the UE 1 receives a physical user input during a group communication session with UE 2...N, at 700A.
• UE 1 presents an original representation of the physical user input at a first level of precision, at 705A.
• 700B is an example of the original representation of the physical user input for a user that draws a circle on a display screen of UE 1 that is showing a map of New York.
• the circle includes a lot of detail and is fairly complex because user movement is imperfect or was deliberately nonlinear.
• Another example of the presentation of the original representation of the physical user input is the shape 600 discussed above with respect to FIG. 6.
• UE 1 sends data representative of the physical user input from 700A to the application server 170, 710A.
• the transmission of the data at 710A corresponds to FIG. 6 at 605 and/or FIG. 5 at 500.
• the application server 170 receives the data representative of the physical user input from UE 1, after which 715A, 720A and 725A correspond to 505, 510 and 515, respectively, of FIG. 5.
• the target UE(s) present the selectively transitioned representation of the physical user input, at 730A.
• the representation of the physical user input at target UEs 2...N can correspond to any of 705B, 710B or 715B.
• a relatively simple circle is shown instead of the complexity of the original representation of 700B.
• the application server 170 can transition the original data from UE 1 that is visually representative of the physical user input into a data format that defines a radius, thickness, color, and/or center point of the circle shown at 705B.
• 705B can be the representation presented at a relatively low performing target UE or a target UE with a poor connection.
• the presentation of 710B is the same as 700B (i.e., no transition).
• 710B can be the representation presented at a relatively high performing target UE or a target UE with a good connection because the original representation from UE 1 did not undergo a quality (or precision) reduction.
• 715B a relatively simple octagon is shown instead of the complexity of the original representation of 700B.
• the application server 170 can transition the original data from UE 1 that is representative of the physical user input into a data format (or level of precision) that defines the vertexes and/or center point of the octagon shown at 715B.
• 715B can be an example of another representation presented at a relatively low performing target UE or a target UE with a poor connection.
• FIG. 7C illustrates a more detailed implementation of FIG. 7A in accordance with an embodiment of the invention.
• 700C through 715C and 730C through 740C correspond to 700A through 730A of FIG. 7A, respectively, and will not be discussed further for the sake of brevity.
• FIG. 7C differs from FIG. 7A with the inclusion of 720C and 725C.
• the application server 170 determines a set of low-performing UE(s) among the target UE(s) based on the determined data presentation capabilities and/or the performance level of the connection, at 720C.
• a low performing UE can include, but is not limited to, a UE with low display performance specifications (e.g., cannot handle high-resolution video stream), or a UE with a low throughput bandwidth connection (e.g., a lx connection with the application server 170).
• the application server 170 may adjust the participation level of the set of low-performing nodes in the group session, at 725C.
• Examples of adjusting the participation level can include, but are not limited to: dropping the set of low performing UEs from a full duplex to a half-duplex interaction with respect to the communication session; lowering the frame rates transmitted to the set of low performing UEs; lowering an image resolution of image media transmitted from the application server 170 to the set of low performing nodes; and/or lowering an audio rate of audio media transmitted from the application server 170 to the set of low performing UEs.
• FIG. 8A illustrates an implementation of the process of 720A of FIG. 7A in accordance with an embodiment of the invention.
• UEs 2...N are described whereby N can equal 2 or alternatively N can be greater than 2.
• FIG. 8A assumes that N is greater than two such that the application server 170 is responsible for re-formatting data representing UE l's physical user input for a plurality of target UEs.
• FIG. 8 A illustrates an example where the application server 170 selectively transitions the received data into different formats (or levels of precision) for different sets of target UEs based on each set' s respective determined data presentation capabilities and/or connection level.
• the selective transitioning of 720A of FIG. 7A can include a first transition of the received data from the first level of precision (i.e., the received level of precision, such as shape 600 of FIG. 6 or 700B of FIG. 7B or FIG. 8B) to a second level of precision for the first group (e.g., as shown in 705B of FIG. 7B and/or FIG. 8B, for example), 800A, the selective transitioning of 720A of FIG. 7A can include a second transition of the received data from the first level of precision into a third level of precision for the second group (e.g., as shown in 715B of FIG. 7B and/or FIG.
• the first level of precision i.e., the received level of precision, such as shape 600 of FIG. 6 or 700B of FIG. 7B or FIG. 8B
• a second level of precision for the first group e.g., as shown in 705B of FIG. 7B and/or FIG. 8B, for example
• FIG. 9 is directed towards using physical user input at a given UE to control device display settings at one or more other UEs participating in the communication session.
• any of the processes described above with respect to FIGS. 5 through 8B can be executed in parallel with the processes of FIGS.
• FIGS. 9 and 10 can be executed in an independent manner, such that the physical user input described below as triggering a display adjustment at the target UE(s) need not be associated with the physical user input described above with respect to FIGS. 5 through 8B.
• FIG. 9 illustrates a process of selectively adjusting display settings for one or more target UEs during a communication session based on received user-generated physical input.
• FIG. 9 illustrates an example whereby physical user input at first UE (e.g., a user rotating UE 1) is reported to the application server 170 that is arbitrating a communication session for UEs 1...N, and the application server 170 selectively controls or adjusts the display settings of the target UE(s) 2...N (e.g., such as display settings of UE 2 is adjusted by rotating display orientation).
• first UE e.g., a user rotating UE 1
• the application server 170 selectively controls or adjusts the display settings of the target UE(s) 2...N (e.g., such as display settings of UE 2 is adjusted by rotating display orientation).
• UEs 1...N and the application server 170 exchange media between UEs 1...N during a group communication session, 900.
• UE 1 receives a physical user input (e.g., rotation of the phone) that is recognized as a prompt to adjust display settings at one or more of the target UE(s) 2...N, 905.
• UE 1 may be provisioned with a set of pre-defined user gestures (or physical user inputs) that are each associated with corresponding display setting adjustment(s) to be implemented at UEs in communication with UE 1.
• UE 1 may detect that the user temporarily reorients UE 1 (e.g., from portrait mode to landscape mode).
• UE 1 reports the detected physical user input to the application server 170 in 910 to request that the application server 170 change the display settings at the target UE(s) 2...N.
• Table 1 (below) lists a set of example pre-defined physical user inputs that, when detected at UE 1, are associated display setting adjustments for one or more of target UE(s) 2...N: Physical User Input at UE Display Setting Affected UE(s) 1 Adjustment at Target
• the application server 170 selectively adjusts the display settings for the target UE(s) (e.g., UE 2...N), 915.
• the selective adjustment to the display settings can be implemented within the media stream being mediated by the application server 170, or alternatively can be implemented indirectly at the target UE(s) 2...N based on control signaling from the application server 170.
• the orientation change can be server- implemented such that the application server 170 itself re-maps the graphical media to the target orientation.
• the orientation change can be UE-implemented, whereby the application server 170 sends an orientation adjustment command to UEs 2...N, after which UEs 2...N will re-orient the unchanged incoming media stream to the target orientation at their end.
• the display setting adjustment is server-implemented or UE-implemented, at 920, the display settings at the target UE(s) 2...N are adjusted based on the physical user input from 905.
• FIG. 10 illustrates an example implementation of the process described in FIG. 9 in accordance with an embodiment of the invention.
• FIG. 10 illustrates example screen shots at a transmitting UE 1 and a target UE 2 during an example implementation of the process of FIG. 9.
• 1000 and 1010 illustrate states of the group communication session at UE 1 and UE 2, respectively, during 900 and 905 of FIG. 9, respectively. Accordingly, as shown in 1000 and 1005, UE 1 and UE 2 are each held in an upright position (or vertical orientation) by their respective users and UEs 1 and 2 are displaying a vertically oriented smiley face graphic. Next, during the communication session, a user of UE 1 turns UE 1 90 degrees so that UE 1 obtains a horizontal orientation, 1010. The user' s turning of UE 1 corresponds to the physical user input detected by UE 1 at 905 of FIG. 9.
• the turning of UE 1 can be a flip of UE 1 by its user that is deliberately made to change the orientation at UE 2.
• the turning of UE 1 can simply arise as a result of UE l 's user preferring a different orientation on his/her own phone.
• UE 1 may include logic for adjusting the smiley face so that the smiley face still appears vertically oriented to the user of UE 1 even though UE 1 itself is horizontally oriented.
• UE 1 reports the orientation change of UE 1 to prompt the application server 170 to adjust the orientation of a graphic being displayed at UE 2. Accordingly, in 915, the application server 170 adjusts the orientation at UE 2. In an example, in 915, the application server 170 can modify the graphical media being streamed to UE 2 by 90 degrees to implement the orientation adjustment for UE 2. In an alternative example, in 915, the application server 170 can send unmodified graphical media to UE 2 (if necessary) and can simply send control commands to UE 2 to instruct UE 2 to offset its orientation for the graphical media by 90 degrees (clockwise).
• UE 2's state is shown in 1015.
• UE 2 is vertically oriented and the smiley face has been transitioned 90 degrees (clockwise) with a horizontal orientation. While not shown in FIG. 10 explicitly, the orientation transition can occur to encourage the user of UE 2 to engage in the communication session in landscape mode instead of portrait mode.
• state 1015 shows UE 2 with a vertical orientation
• the user of UE 2 may be likely to alter UE 2's orientation to conform to the smiley face orientation.
• target UEs may lack the ability to implement the display setting adjustment requested by the user of UE 1 via his/her physical user input. For example, some target UEs may not be able to adjust their display settings (e.g., rotate display orientation), therefore the application server 170 will not adjust the display settings for these target UEs.
• FIGS. 11-19B relate to identifying high-priority portion(s) of a video stream based on user input and separately transmitting the identified high-priority portion(s) from lower priority portion(s) of the video stream via different links to the target UE(s).
• FIG. 11 is directed to a process of selectively transmitting different video feeds associated with a video stream being displayed on a given UE ("UE 1") to at least one target UE ("UEs 2...N”) in accordance with an embodiment of the present invention.
• UEs 1...N e.g., where N>2
• the communication session can correspond to a video conferencing session whereby the same video (and/or image media) is displayed at each of UEs 1...N (e.g., a collaborative map session, etc.).
• different video and/or image media can be presented at two or more of UEs 1...N (e.g., UE 1 may view UE 2's video media, UE 2 may view UE l's video media, and so on).
• different versions of the same video and/or image media can be presented at two or more of UEs 1...N (e.g., UEs 1 and 2 may each be sharing their respective camera feeds so UE 1 ' s camera feed is delivered and presented on UE 2's display screen while UE 2's camera feed is delivered and presented on UE l 's display screen, with UE 1 and 2's respective display screens also presenting reduced-size versions of their own camera feeds, and so on).
• the video and/or image media need not be actively mediated through the application server 170, but could also be rendered independently at UEs 1...N.
• audio media can be mediated by the application server 170 (e.g., half duplex or full duplex), but the video and/or image media could be loaded separately at the UEs 1...N.
• UEs 1...N could each be independently rendering a map of New York while discussing their travel plans so that the map data does not need to be actively exchanged between the UEs during the communication session.
• the communication session between UEs 1...N could occur via peer-to-peer (PTP) protocols at least in part, with two or more of UEs 1...N connected via a PTP connection.
• PTP peer-to-peer
• UE 1 receives user input that identifies a set of high priority portions of a video stream that is being displayed on UE 1 and is being shared with UEs 2...N in association with the communication session, 1100.
• the user input received at 1100 can correspond to physical user input that expressly indicates certain portions of the video stream to have high priority (e.g., the user of UE 1 circles portions of the video stream with his/her finger via a touchscreen interface, see FIGS.
• the user input received at 1100 can correspond to user input that implicitly indicates certain portions of the video stream to have high priority (e.g., one or more objects are identified to be of interest to the user of UE 1, such as by the user of UE 1 having previously zoomed in on these objects, with these objects subsequently being detected as being in certain portions of the video stream with the portions obtaining high priority status by virtue of containing the objects, see FIGS. 16-19B which are discussed below in more detail).
• one or more objects are identified to be of interest to the user of UE 1, such as by the user of UE 1 having previously zoomed in on these objects, with these objects subsequently being detected as being in certain portions of the video stream with the portions obtaining high priority status by virtue of containing the objects, see FIGS. 16-19B which are discussed below in more detail).
• UE 1 optionally adds one or more additional portions of the video stream to the set of high priority portions identified at 1100 based on one or more secondary factors that are not based on user input (express or implied), 1105. For example, certain portions of the video stream may be identified as being high priority based on having a high degree of motion via an associated motion vector, etc.
• UE 1 generates a first video feed based on the set of high priority portions of the video stream identified at 1100, 1110, and UE 1 also generates a second video feed of the video stream, 1115.
• the first video feed may include only the set of high priority portions of the video stream while the second video feed includes the entire video stream, such that the first video feed has a lower resolution than the second video feed by virtue of including fewer portions of the video stream.
• the first video feed may also include the entire video stream and hence has the same resolution as the second video feed, except that the portions separate from the set of high priority portions in the first video feed are "blacked out" which reduces a size of the first video feed relative to the second video feed.
• UE 1 transmits the first video feed to UEs 2...N over a first link, 1120, and UE 1 also transmits the second video feed to UEs 2...N over a second link with less reliability relative to the first link, 1125.
• the first link may be allocated QoS (e.g., a threshold level of guaranteed bit rate (GBR)) while the second link is not allocated QoS.
• the first and second links may both be allocated QoS, with the first link having more QoS (e.g., a higher GBR) relative to the second link. In either case, the first link is generally expected to be more reliable than the second link, which is why the first video feed (which contains the set of high priority portions of the video stream) is allocated to the first link.
• FIG. 12 is directed to a continuation of the process of FIG. 11 from the perspective of a target UE (e.g., one of UEs 2...N) that receives the first and second video feeds via the first and second links, respectively, from UE 1 in accordance with an embodiment of the invention.
• a target UE e.g., one of UEs 2...N
• receives the first and second video feeds via the first and second links, respectively, from UE 1 in accordance with an embodiment of the invention.
• the target UE receives the first video feed from UE 1 over the first link as transmitted at 1120 of FIG. 11, 1200.
• the target UE also receives the second video feed from UE 1 over the second link as transmitted at 1125 of FIG. 11, 1205.
• the target UE combines the first and second video feeds in order to reconstruct a version of the video stream, 1210, and then presents the reconstructed version of the video stream via a display to a user of the target UE, 1215.
• the first link is generally expected to be more reliable than the second link, so the reconstructed version of the video stream is expected to be fairly accurate in depicting the set of high priority portions contained in the first video feed, while artifacts may occur in the other portions separate from the set of high priority portions which are only conveyed to the target UE via the second video feed over the second link. More detail regarding how the first and second video feeds are combined is provided below.
• FIG. 13 illustrates an example implementation of how the set of high priority portions of the video stream can be identified at 1100 of FIG. 11 in accordance with an embodiment of the invention.
• UE 1 receives physical user input that indicates an explicit selection of the set of high priority portions of the video stream being displayed on UE 1, 1300.
• 1300 of FIG. 13 constitutes an example implementation of 1100 of FIG. 11.
• the process advances to 1105 of FIG. 11 where more high priority portions are optionally added to the set of high priority portions.
• FIG. 14 display screen 1400 depicting a map of New York is shown, whereby the display screen 1400 includes a first circle 1405 and a second circle 1410 being drawn by a user of UE 1.
• the circles 1405-1410 constitute the explicit selection from the user that is received at 1300 of FIG. 13.
• the display screen of UE 1 is divided into nine (9) sections as shown at 1420, with the first circle 1405 corresponding to section #1 and the second circle 1410 corresponding to section #9.
• the display screen 1420 depicts sections #1 and #9 as high priority portions via shading, with section #3 also being shaded as a high priority section based on optional 1105 of FIG. 11 (e.g., one or more secondary factors indicating that section #3 should also be high-priority).
• the first video feed 1425 shows an example whereby the first video feed includes the sections #1, #3 and #9 in isolation, such that the aggregate resolution of the first video feed 1425 is less than resolution of display screen 1420 by virtue of omitting sections #2 and #4... #8.
• the alternative first video feed 1430 shows an example whereby the first video feed includes each of sections #1...#9, with sections #2 and #4... #8 being blacked out such that the aggregate resolution of the first video feed 1430 is equal to the resolution of display screen 1420.
• the second video feed 1435 includes each of sections #1...#9, although the first video feed 1430 can be sent more efficiently than the second video feed 1435 by virtue of blacking out sections #2 and #4... #8.
• FIG. 15A illustrates an example implementation of 1210 of FIG. 12, whereby the first video feed 1425 of FIG. 14 is transmitted to the target UE over the first link and the second video feed 1435 of FIG. 14 is transmitted to the target UE over the second link.
• the first and second video feeds are combined, 1210, to produce the reconstructed version of the video stream as shown at 1500A.
• sections #1, #3 and #9 from the second link are replaced with sections #1, #3 and #9 from the first link, to produce (or reconstruct) the video stream.
• the reconstructed version of the video stream is populated with the set of high priority portions of the video stream (i.e., sections #1, #3 and #9) plus one or more portions from the second video feed that do not correspond to the set of high priority portions within the first video feed (i.e., sections #2, and #4-#8).
• sections #1, #3 and #9 are sent over the more reliable first link, so sections #1, #3 and #9 are shaded in the reconstructed video stream 1500A to depict that these particular sections may have fewer errors relative to sections #2 and #4...#8 which are only sent over the less reliable second link.
• FIG. 15B illustrates another example implementation of 1210 of FIG. 12, whereby the first video feed 1430 of FIG. 14 is transmitted to the target UE over the first link and the second video feed 1435 of FIG. 14 is transmitted to the target UE over the second link.
• the first and second video feeds are combined, 1210, to produce the reconstructed version of the video stream as shown at 1500B.
• sections #1, #3 and #9 are sent over the more reliable first link, so sections #1, #3 and #9 are shaded in the reconstructed video stream 1500B to depict that these particular sections may have fewer errors relative to sections #2 and #4...#8 which are only sent over the less reliable second link.
• the blacked out sections of the first video feed 1430 are replaced (where possible) by the corresponding sections #2 and #4... #8 in the second video feed 1435 during the combination of 1210.
• the reconstructed version of the video stream is populated with the set of high priority portions (i.e., sections #1, #3 and #9) of the video stream and one or more portions (i.e., sections #2 and #4-#8) from the second video feed that correspond to the one or more blacked out portions from the first video feed.
• FIG. 16 illustrates an example implementation of how the set of high priority portions of the video stream can be identified at 1100 of FIG. 11 in accordance with another embodiment of the invention.
• UE 1 determines, based on user input, to lock onto a set of objects by tracking where the set of objects is located within the video stream being displayed on UE 1, 1600. Based on the tracking of the set of objects that occurs at 1600, UE 1 identifies a current set of portions of the video stream (e.g., in a field of view of the video stream as displayed on a display screen of UE 1) containing the set of objects (if any), 1605.
• the identified current set of portions at 1605 can correspond to the set of high priority portions from 1100 of FIG.
• recognition of certain high priority object(s) within certain portion(s) of the video stream can upgrade the status of those portion(s) to high priority status.
• the user of UE 1 may be able to control where the set of objects is displayed in the video stream to a certain degree (e.g., by moving a camera of UE 1 closer to the set of objects or by turning the camera of UE 1 to a different angle which changes the position of the set of objects in the video stream, or by executing a digital zoom operation on certain portions of the video stream, etc.), the identified current set of portions at 1605 is based at least in part upon user input.
• 1600- 1605 of FIG. 16 constitutes an example implementation of 1100 of FIG. 11. [00119] With respect to FIG. 16, 1605 may perform continuously or periodically while UE 1 is locking onto at least one object.
• the at least one object being locked upon “moves" in terms of its relative position in the video stream (e.g., due to object movement such as a baby crawling across the floor, or due to camera movement at UE 1 which changes the position of the object relative to the camera and hence its associated position in the video stream, or some combination thereof)
• different portions of the video stream may be recognized as containing the at least one object. So, by identifying the current set of portions at 1605 in a continuous or periodic manner, UE 1 accounts for relative object movement within the video stream, as will be described in more detail below.
• UE 1 determines whether to end the locking operation. For example, as shown in more detail below with respect to FIG.
• UE 1 may infer that the set of objects is no longer important to the communication session and thereby may decide to end the locking operation at 1615.
• UE 1 may determine to lock onto a different set of objects, which may function to terminate the locking operation for the previous set of objects.
• the user of UE 1 may expressly instruct the UE 1 to end the locking operation.
• the locking operation whereby high priority status is allocated to portion(s) of the video stream that contain the set of objects is terminated, 1620 (e.g., the first video feed may be canceled altogether, or the first video feed may be mapped to different sections to accommodate a different set of high priority portions, etc.). Otherwise, if UE 1 determines to continue the locking operation, the process returns to 1605. As will be appreciated, as the video stream of the communication session changes, the portions of the video stream which contain the "locked" set of objects may also change. So, the current set of portions identified at 1605 may change during the communication session even if the "locked" set of objects remains the same.
• UE 1 may stop generating and/or transmitting the first video feed over the first video feed so long as the set of objects is "off screen". During the off screen period, UE 1 retains the first link so that the first link is available once the set of objects re-enters the video stream. When the set of objects is determined to be off screen for more than a threshold period of time, this can trigger 1615 to end the locking operation, which can potentially trigger tear down of the first link if there are no other high priority portion(s) of the video stream after the locking operation is ended at 1620.
• FIG. 17A illustrates an example of the user input which can trigger the determination of 1600 of FIG. 16 in accordance with an embodiment of the invention.
• UE 1 detects a zoom event that zooms in upon a set of objects within a video stream being displayed on UE 1, 1700 A.
• the detection of the zoom event at 1700A triggers the set of objects to be allocated high priority status, such that the process advances from 1700A to 1600 of FIG. 16.
• the detected zoom event of 1700A may be required to occur for a threshold period of time before advancing to 1600 of FIG.
• UE 1 may start a timer in response to detection of a particular object (e.g., a baby) in the field of view on an associated display screen of UE 1 during the zoom event. If the particular object remains in the field of view on the display screen for more than the threshold period of time based on the execution of the timer, then UE 1 may determine to "lock" on this object, even after the zoom event ends and/or after the object is no longer within the field of view on the display screen.
• a particular object e.g., a baby
• the detected zoom event shown in FIG. 17 A can be considered as an implicit selection of the set of objects by the user of UE 1, because the user's interest in the set of objects is inferred by the user's manipulation of the video feed during the communication session.
• the user can expressly select certain objects to be populated within the set of objects for the purpose of the locking operation, as shown in FIG. 17B.
• FIG. 17B illustrates another example of the user input which can trigger the determination of 1600 of FIG. 16 in accordance with an embodiment of the invention.
• UE 1 detects an explicit user selection of a set of objects within a video stream being displayed on UE 1 (e.g., a user can physically touch objects in the video stream via a touch screen interface at UE 1 to facilitate a "lock" upon the objects, etc.), 1700B.
• the detection of the explicit user selection at 1700B triggers the set of objects to be allocated high priority status, such that the process advances from 1700B to 1600 of FIG. 16.
• the user of UE 1 may also de-select any previously selected objects to terminate the locking operation for the de-selected objects.
• FIGS. 18A-18D illustrate an example implementation of the process of FIG. 17 A in conjunction with FIGS. 11 and 16 in accordance with an embodiment of the invention.
• FIGS. 18A-18D illustrate an example series of screen-shots (or frames) that are marked as (l)-(9), with a discussion of how the changes to frames (l)-(9) cause changes to the communication session in accordance with execution of the process of FIG. 16.
• display screen 1800A of UE 1 depicting a nursery scene containing a baby, a baby bottle and a beach ball is shown. Assume that the display screen 1800A constitutes a frame in a communication session.
• a user of UE 1 zooms in on the baby as shown in display screen 1805 A, either by physically moving the camera of UE 1 closer to the baby or by performing a digital zoom operation with the camera of UE 1.
• the zoom that focuses on the baby as depicted in the display screen 1805A constitutes a zoom event as in 1700A of FIG. 17A.
• the user of UE 1 zooms out as shown in display screen 1810A.
• the baby has been identified as a high priority object for performing a locking operation as in 1600 of FIG. 16.
• UE 1 may start a timer having in response to detection of a particular object (e.g., in this case, a baby) in the field of view on the display screen 1805 A during the zoom event. If the particular object remains in the field of view on the display screen 1805 A for more than a threshold period of time based on the execution of the timer, then UE 1 may determine to "lock” on this object, even after the zoom event ends and/or after the object is no longer within the field of view on the display screen 1805 A. Once an object is "locked”, another timer may be started to trigger the object to be "unlocked” upon expiration.
• a particular object e.g., in this case, a baby
• the timer for unlocking a locked object can be extended so as to extend the locking period for the associated object in response to a lock extension event (e.g., the timer can be extended each time the locked object is detected as being centered in the field of view of the display screen, each time the locked object is zoomed in upon, etc.).
• a lock extension event e.g., the timer can be extended each time the locked object is detected as being centered in the field of view of the display screen, each time the locked object is zoomed in upon, etc.
• FIG. 18A assume that the display screen of UE 1 is divided into four (4) sections as shown at 1815A, with the baby being identified as currently located in section #3 in accordance with 1605 of FIG. 16.
• the display screen 1815A thereby depicts section #3 as high priority via shading because section #3 includes the "locked" object (i.e., baby), in accordance with 1605-1610 of FIG. 16.
• the first video feed 1800B shows an example whereby the first video feed includes the section #3 in isolation, such that the aggregate resolution of the first video feed 1800B is less than resolution of display screen 1815A by virtue of omitting sections #1, #2 and #4.
• the alternative first video feed 1805B shows an example whereby the first video feed includes each of sections #1...#4, with sections #1, #2 and #4 being blacked out such that the aggregate resolution of the first video feed 1805B is equal to the resolution of a display screen (or display screen portion) that is outputting the first video feed 1805B.
• the second video feed 1810B includes each of sections #1...#4, although the first video feed 1805B can be sent more efficiently than the second video feed 1810B by virtue of blacking out sections #1, #2 and #4.
• the baby's relative position on the display screen of UE 1 moves to section #2, as shown in 1800C.
• the baby' s "movement” is a relative screen position change that can occur based on actual object movement (e.g., baby physically crawls to a different position), or camera movement (e.g., a user of UE 1 moves UE l's camera to a different position relative to the baby) or by some combination thereof (e.g., the baby's absolute position moves while the camera's position also moves).
• object movement e.g., baby physically crawls to a different position
• camera movement e.g., a user of UE 1 moves UE l's camera to a different position relative to the baby
• some combination thereof e.g., the baby's absolute position moves while the camera's position also moves.
• section #2 in 1800C causes section #2 to replace section #3 as the high-priority section based on object locking, as depicted in 1805C where section #2 is shown as high-priority via shading because section #2 now includes the "locked" object (i.e., the baby), in accordance with 1605-1610 of FIG. 16.
• the user of UE 1 zooms in on the ball as shown in display screen 18 IOC, either by physically moving the camera of UE 1 closer to the ball or by performing a digital zoom operation with the camera of UE 1.
• the zoom that focuses on the ball as depicted in the display screen 18 IOC constitutes a zoom event as in 1700A of FIG. 17 A.
• the user of UE 1 zooms out as shown in display screen 1815C.
• the "ball" zoom event shown in 18 IOC triggers the baby to be unlocked and the ball to be locked for the communication session. Accordingly, the ball is now identified as a high priority object for performing a locking operation as in 1600 of FIG. 16.
• section #2 is the high-priority section based on object locking, as depicted in 1820C where section #2 is shown as high-priority via shading because section #2 now includes the new "locked" object (i.e., the ball), in accordance with 1605-1610 of FIG. 16.
• FIG. 19A illustrates an example implementation of 1210 of FIG. 12, whereby the first video feed 1800B of FIG. 18B is transmitted to the target UE over the first link and the second video feed 1810B of FIG. 18B is transmitted to the target UE over the second link.
• the first and second video feeds are combined, 1210, to produce the reconstructed version of the video stream as shown at 1900A.
• section #3 is sent over the more reliable first link, so section #1 is shaded in the reconstructed video stream 1900A to depict that this particular section may have fewer errors (or artifacts) relative to sections #1, #2 and #4 which are only sent over the less reliable second link.
• FIG. 19B illustrates another example implementation of 1210 of FIG. 12, whereby the first video feed 1805B of FIG. 18B is transmitted to the target UE over the first link and the second video feed 1810B of FIG. 18B is transmitted to the target UE over the second link.
• the first and second video feeds are combined, 1210, to produce the reconstructed version of the video stream as shown at 1900B.
• section #3 is sent over the more reliable first link, so section #3 is shaded in the reconstructed video stream 1900B to depict that this particular section may have fewer errors (or artifacts) relative to sections #1, #2 and #4 which are only sent over the less reliable second link.
• the blacked out sections of the first video feed 1805B are replaced (where possible) by the corresponding sections #1, #2 and #4 in the second video feed 1810B during the combination of 1210.
• FIGS. 10-19B are directed to scenarios where the video stream is output by UE 1 while also being streamed to one or more target UEs, it will be appreciated the video stream does not necessarily need to be displayed via UE l 's display screen in other embodiments.
• UE 1 could be capturing video media with his/her camera, and streaming the captured video media to the target UE(s). It is possible that the video media being captured in this case is not actually displayed on the display screen of UE 1.
• the user of UE 1 may not be able to circle portions of the video stream as shown in FIG. 14, the user of UE 1 could identify high-priority portion(s) of the video stream in other ways.
• the user of UE 1 could physically move the camera closer to an object to zoom on the object and thereby trigger an object lock irrespective of whether the video stream is actually being displayed on UE 1.
• DSP digital signal processor
• ASIC application specific integrated circuit
• FPGA field programmable gate array
• a general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine.
• a processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
• a software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
• An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium.
• the storage medium may be integral to the processor.
• the processor and the storage medium may reside in an ASIC.
• the ASIC may reside in a user terminal (e.g., UE).
• the processor and the storage medium may reside as discrete components in a user terminal.
• the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium.
• Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another.
• a storage media may be any available media that can be accessed by a computer.
• such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.
• any connection is properly termed a computer-readable medium.
• the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave
• the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium.
• Disk and disc includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Databases & Information Systems (AREA)
• Computer Graphics (AREA)
• Computer Networks & Wireless Communication (AREA)
• Human Computer Interaction (AREA)
• Computer Security & Cryptography (AREA)
• General Engineering & Computer Science (AREA)
• Mobile Radio Communication Systems (AREA)

Priority Applications (2)

Applications Claiming Priority (2)

Publications (1)

Family

ID=55587318

Family Applications (1)

Country Status (3)

Citations (5)

Family Cites Families (3)

• 2016
  • 2016-01-28 EP EP16711027.9A patent/EP3251371A1/en not_active Withdrawn
  • 2016-01-28 WO PCT/US2016/015386 patent/WO2016123353A1/en active Application Filing
  • 2016-01-28 CN CN201680007497.7A patent/CN107211177A/zh active Pending

Patent Citations (5)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events