WO2022206626A1 - 一种增强现实通信的方法、装置及系统 - Google Patents

一种增强现实通信的方法、装置及系统 Download PDF

Info

Publication number
WO2022206626A1
WO2022206626A1 PCT/CN2022/083191 CN2022083191W WO2022206626A1 WO 2022206626 A1 WO2022206626 A1 WO 2022206626A1 CN 2022083191 W CN2022083191 W CN 2022083191W WO 2022206626 A1 WO2022206626 A1 WO 2022206626A1
Authority
WO
WIPO (PCT)
Prior art keywords
network element
terminal device
media
processing network
media processing
Prior art date
Application number
PCT/CN2022/083191
Other languages
English (en)
French (fr)
Inventor
刘艳
高扬
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP22778808.0A priority Critical patent/EP4304166A4/en
Priority to JP2023559102A priority patent/JP2024511504A/ja
Publication of WO2022206626A1 publication Critical patent/WO2022206626A1/zh
Priority to US18/474,721 priority patent/US20240022619A1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/10—Architectures or entities
    • H04L65/1016—IP multimedia subsystem [IMS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/10—Architectures or entities
    • H04L65/102—Gateways
    • H04L65/1033—Signalling gateways
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/10—Architectures or entities
    • H04L65/1063—Application servers providing network services
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/1066—Session management
    • H04L65/1069—Session establishment or de-establishment
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/60—Network streaming of media packets
    • H04L65/61—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio
    • H04L65/612—Network streaming of media packets for supporting one-way streaming services, e.g. Internet radio for unicast
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/80—Responding to QoS
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/131—Protocols for games, networked simulations or virtual reality
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/14—Session management
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/14—Multichannel or multilink protocols
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/24—Negotiation of communication capabilities
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H04N7/141—Systems for two-way working between two video terminals, e.g. videophone
    • H04N7/147—Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals

Definitions

  • the embodiments of the present application relate to the field of communication technologies, and in particular, to a method, an apparatus, and a system for augmented reality communication.
  • VoLTE Voice over long term evolution
  • 4G fourth generation
  • VoLTE enables users to communicate with shorter connection waiting times and higher quality voice and video calls.
  • Augmented Reality is a technology that ingeniously integrates virtual information with the real world. After the virtual information such as text, images, three-dimensional models, music, and videos is simulated and simulated, it is applied to the real world, and the two kinds of information complement each other, thereby realizing the "enhancement" of the real world. Augmented reality technology can not only effectively reflect the content of the real world, but also promote the display of virtual information content. There is currently no effective way to integrate AR into voice and video calls.
  • Embodiments of the present application provide a method, device, and system for augmented reality communication, and provide an implementation manner in which AR is integrated into a voice and video call, thereby improving user experience.
  • an embodiment of the present application provides an augmented reality AR communication system, including an Internet Protocol Multimedia Subsystem IMS core network element and a first AR media processing network element; the IMS core network element is used to control the first AR media processing network element.
  • a first media stream channel and an auxiliary data channel are established between an AR media processing network element and the first terminal device; the first AR media processing network element is used to receive data from the first terminal through the established auxiliary data channel the first AR auxiliary data of the device, and receive the first media stream for AR communication between the first terminal device and the second terminal device through the established media stream channel;
  • the first AR media processing network element further is configured to perform enhancement processing on the media stream according to the first AR auxiliary data to obtain a first AR media stream, and send the first AR media stream to the first terminal device through the media stream channel.
  • an auxiliary data channel and a media stream channel are established between the terminal device and the AR media processing network element, thereby effectively integrating AR into the voice and video calls.
  • AR auxiliary data and media streams follow the same transmission path, which can reduce latency.
  • the first AR media processing network element may be an IMS access gateway with AR processing capability or a media resource function MRF.
  • the first AR media processing network element may be an IMS access gateway.
  • the terminal device previously held an AR conference the first AR media processing network element may be an MRF.
  • the first AR media processing network element is located in the IMS network to which the first terminal device belongs (or is located).
  • the AR media processing network element in the IMS network where the first terminal device is located has the AR processing capability, and the AR media processing network element in the IMS network where the first terminal device is located performs enhanced processing using the proximity principle, which can further reduce the delay.
  • the system further includes a second AR media processing network element located in the IMS network to which the second terminal device belongs (or is located); the first AR media processing network element is also used for Send the AR media stream to the second AR media processing network element; the second AR media processing network element is used to receive the AR media stream sent by the first AR media processing network element, and the AR media stream is sent by the first AR media processing network element.
  • the second terminal device sends the AR media stream.
  • the first AR media processing network element is located in the IMS network to which the second terminal device belongs (or where it is located), and the system further includes a network element located in (or where the first terminal device belongs)
  • the third AR media processing network element in the IMS network, the third AR media processing network element does not have the AR processing capability, and the media stream channel between the first AR media processing network element and the first terminal device includes the first AR media processing network element. a first media stream channel between the AR media processing network element and a third AR media processing network element and a second media stream channel between the third AR media processing network element and the first terminal device;
  • the third AR media processing network element is configured to receive the media stream from the first terminal device through the second media stream channel, and send the media stream to the first AR media processing network element through the first media stream channel forwarding the media stream; receiving the AR media stream from the first AR media processing network element through the first media stream channel, and sending the AR media stream to the first terminal device through the second media stream channel .
  • the AR media processing network element in the IMS network where the first terminal device is located does not have the AR processing capability
  • the AR media processing network element in the IMS network where the second terminal device is located is used to provide the first terminal device.
  • Enhanced processing capabilities It provides a feasible solution for integrating AR into audio and video calls.
  • the first AR media processing network element is further configured to receive target object information from the first terminal device through the auxiliary data channel; the AR media processing network element is further configured to receive target object information from the first terminal device through the auxiliary data channel; The third-party server obtains the target object corresponding to the target object information.
  • the third-party server can provide the material of the target object for AR communication of the terminal device, and in this application, the target object is obtained from the third-party server by means of an auxiliary data channel.
  • the target object is a virtual object or an identifier of a virtual object; the first AR media processing network element is further configured to send the information to the first terminal device through the auxiliary data channel A virtual object or an identification of the virtual object. Since virtual objects are required in various scenarios, through the above design, virtual objects are obtained from a third-party server by means of an auxiliary data channel.
  • the target object information includes a virtual object type requested by the first terminal device, and the first AR media processing network element sends multiple virtual object identifiers of the virtual object type through an auxiliary data channel. to the first terminal device.
  • the first AR media processing network element after receiving the multiple virtual object identifiers corresponding to the virtual object, saves the correspondence between the virtual object type and the multiple virtual object identifiers.
  • the first AR media processing network element when it receives the virtual object type through the auxiliary data channel, it may first determine whether it has saved the virtual object identifier corresponding to the virtual object type. Then, obtain the virtual object identifier corresponding to the virtual object type from the third-party server. If it is not saved, then obtain the virtual object identifier corresponding to the virtual object type from the third-party server, and in this way, unnecessary waste of resources caused by the obtaining can be reduced.
  • the method further includes:
  • the first AR media processing network element receives the first virtual object identifier from the first terminal device through the auxiliary data channel, where the first virtual object identifier is used to identify that the user of the first terminal device is in the plurality of virtual objects.
  • the first virtual object selected from the objects; the first AR media processing network element obtains the first virtual object corresponding to the identifier of the first virtual object from a third-party server, and sends the first virtual object to the first terminal device through the auxiliary data channel a virtual object.
  • the target object is a real background object
  • the first AR media processing network element is further configured to receive the first AR communication generated by the AR communication sent by the first terminal device through the media data channel.
  • the first AR media processing network element is further configured to render and synthesize the real background object and the second media stream to obtain a second AR media stream; send the first AR media stream to the first terminal device through the media stream channel; Two AR media streams.
  • the background image corresponding to the foreground object is relatively fixed.
  • the rendering operation between the media stream and the background image is performed by the AR media processing network element, which can reduce the pressure on the upstream bandwidth of the terminal device. .
  • the system further includes an application server;
  • the application server is configured to receive the first request message from the AR media processing network element, and forward the first request message to the third-party server; receive the first response message from the third-party server, and send it to the third-party server.
  • the first AR media processing network element forwards the second response message.
  • the application server is configured with the address of the third-party server; the application server is specifically configured to forward the first request to the third-party server according to the address of the third-party server message, and forwarding the second response message to the first AR media processing network element.
  • the application server is further configured to, in the process of establishing the auxiliary data channel between the first terminal device and the first AR media processing network element, send the request to the first AR
  • the media processing network element sends the address of the third-party server.
  • an embodiment of the present application provides an augmented reality AR communication method, including: an Internet Protocol Multimedia Subsystem IMS core network element receives a first request message from a first terminal device, where the first request message is used to request a establishing an auxiliary data channel, where the auxiliary data channel is used for transmitting auxiliary data of AR communication between the first terminal device and the AR media processing network element; the IMS core network element controls the AR media processing network element and the first The auxiliary data channel is established between terminal devices.
  • the above solution provides a simple and effective way to create an auxiliary data channel, so as to obtain AR auxiliary data of AR communication through the auxiliary data channel, thereby realizing the enhancement processing of the media stream of AR communication in the AR communication process.
  • the IMS core network element controlling the AR media processing network element and the first terminal device to establish the auxiliary data channel includes: the IMS core network element sends the AR to the AR The media processing network element sends a second request message, where the second request message is used to request the AR media processing network element and the first terminal device to establish the auxiliary data channel; the IMS core network element receives the AR A second response message sent by the media processing network element, where the second response message is used to indicate to the IMS core network element that the AR media processing network element confirms the establishment of the auxiliary data channel; the IMS core network element reports to the IMS core network element The first terminal device sends a first response message, where the first response message is used to indicate to the first terminal device that the AR media processing network element successfully establishes the auxiliary data channel.
  • the AR media processing network element is an IMS access gateway with AR processing capability or a media resource function MRF.
  • the first AR media processing network element may be an IMS access gateway.
  • the terminal device previously held an AR conference the first AR media processing network element may be an MRF.
  • the IMS core network element controls the establishment of a media stream channel between the AR media processing network element and the first terminal device.
  • auxiliary data channels and media streams are established between the IMS core network element and the AR media processing network element, thereby effectively integrating AR into voice and video calls.
  • AR auxiliary data and media streams follow the same transmission path, which can reduce latency.
  • the AR media processing network element is located in the IMS network to which the first terminal device belongs or in the IMS network to which the second terminal device belongs.
  • the AR media processing network element in the IMS network where the first terminal device is located has the AR processing capability, and the AR media processing network element in the IMS network where the first terminal device is located performs enhanced processing using the proximity principle, which can further reduce the delay.
  • the AR media processing network element in the IMS network where the first terminal device is located does not have the AR processing capability
  • the first terminal device is provided with enhanced processing functions by means of the AR media processing network element in the IMS network where the second terminal device is located. It provides a feasible solution for integrating AR into audio and video calls.
  • the first request message and the second request message carry a first description parameter of the first terminal device for establishing the auxiliary data channel
  • the first response message and the second response message carry the second description parameter of the AR media processing network element for establishing the auxiliary data channel.
  • the method further includes: the IMS core network element sends a query request to the AR media processing network element, where the query request is used to inquire whether the AR media processing network element has AR processing capabilities; the IMS core network element The network element receives a query response sent by the AR media processing network element, where the query response carries information that the AR media processing network element has AR processing capability.
  • the AR media processing network element is located in the IMS network to which the second terminal device belongs, and the method further includes:
  • the IMS core network element Before the IMS core network element sends the query message to the AR media processing network element located in the IMS network to which the second terminal device belongs, it is determined that the AR media processing network element providing the service for the first terminal device does not have AR processing capability.
  • the process of establishing AR communication between the first terminal device and the second terminal device is the process of establishing an AR session between the first terminal device and the second terminal device, or the process of establishing an AR session between the first terminal device and the second terminal device or the process of establishing an AR session between the first terminal device and the second terminal device.
  • the first request message is an initial session protocol SIP request INVITE message, or a SIP re-establishment request message, or a SIP update request message.
  • the method further includes: the IMS core network element controlling the AR media processing network element and the first terminal device to establish the media stream channel, the media stream channel using A media stream of AR communication content is transmitted between the first terminal device and the AR media processing network element.
  • receiving a third request message from the first terminal device, where the third request message is further used to request establishment of a media stream channel, the IMS core network element controls the AR media processing network element to communicate with Establishing the media stream channel between the first terminal devices includes: the IMS core network element sending a fourth request message to the AR media processing network element, where the fourth request message is used to request the AR media processing network The media stream channel is established between the AR media processing element and the first terminal device; the IMS core network element receives a fourth response message sent by the AR media processing network element, and the fourth response message is used to send the IMS core network element indicates that the AR media processing network element confirms the establishment of the media stream channel; the IMS core network element sends a third response message to the first terminal device, where the third response message is used to send the first terminal The device instructs the AR media processing network element to successfully establish the media stream channel.
  • auxiliary data channel may be established in the same process as the media stream channel, or the media stream channel may be established first, and then the auxiliary data channel may be established.
  • the first request message and the third request message may be carried in the same message
  • the second request message and the fourth request message may be carried in the same message
  • the first response message and the third response message may be carried in the same message
  • the The second response message and the fourth response message may be carried in the same message.
  • an embodiment of the present application provides an augmented reality AR-based communication method, including:
  • the AR media processing network element establishes an auxiliary data channel with the first terminal device.
  • a media stream channel for transmitting AR communication content is established between the AR media processing network element and the first terminal device.
  • the AR media processing network element receives a second request message sent by the Internet Protocol Multimedia Subsystem IMS core network element, where the second request message is used to request the AR media processing network element to communicate with the first terminal device.
  • An auxiliary data channel is established between the first terminal device and the AR media processing network element, the auxiliary data channel is used to transmit the auxiliary data of AR communication between the first terminal device and the AR media processing network element; the AR media processing network element sends the first terminal device to the IMS core network element Two response messages, where the second response message is used to indicate to the IMS core network element that the AR media processing network element confirms the establishment of the auxiliary data channel.
  • the method further includes: after the auxiliary data channel is established, the AR media processing network element receives AR auxiliary data from the first terminal device through the auxiliary data channel.
  • the method further includes: receiving, by the AR media processing network element, a media stream of AR communication from the first terminal device through the media stream channel;
  • the AR auxiliary data performs media enhancement processing on the AR communication media stream to obtain an AR media stream;
  • the AR media processing network element sends the enhanced AR media stream to the first terminal through the media stream channel equipment.
  • the method further includes: sending, by the AR media processing network element, the enhanced-processed AR media stream to the second terminal device.
  • the AR auxiliary data includes one or more items of operation data of the virtual model, identification data of facial expressions or body movements of the target object, or pose data.
  • the AR media processing network element is an IMS access gateway with AR processing capability or a media resource function MRF.
  • the AR media processing network element is located in the IMS network to which the first terminal device belongs (or is located), or is located in the IMS network to which the second terminal device belongs (or is located). middle.
  • the second request message carries a first description parameter of the first terminal device used to establish the auxiliary data channel
  • the method further includes: the AR media processing network element from the The second description parameter supported by the AR media processing network element is determined in the first description parameter; the second response message carries the second description parameter.
  • an embodiment of the present application provides a method for AR communication based on augmented reality, comprising: the first terminal device sending a first request message to an IMS core network element, where the first request message is used to request the establishment of auxiliary data channel; the first terminal device receives the first response message sent by the IMS core network element, where the first response message is used to instruct the AR media processing network element to complete the establishment of the auxiliary data channel, and the auxiliary data channel uses Auxiliary data for AR communication is transmitted between the first terminal device and the AR media processing network element.
  • the method further includes: establishing a media stream channel for transmitting the content of AR communication between the first terminal device and the AR media processing network element.
  • establishing a media stream channel for transmitting AR communication content between the first terminal device and the AR media processing network element includes:
  • the first terminal device receives a third response message sent by the IMS core network element, where the third response message is used to indicate to the first terminal device that the AR media processing network element successfully establishes the media stream channel.
  • the first request message is further used to request the establishment of the media stream channel; the first response message is further used to instruct the AR media processing network element to successfully establish the media stream channel.
  • the method further includes:
  • the first terminal device sends augmented reality AR auxiliary data to the AR media processing network element through the auxiliary data channel, and sends a media stream to the AR media processing network element through the media stream channel, where the media stream is the first terminal device in the AR media processing network element.
  • the AR auxiliary data is used by the AR media processing network element to perform media enhancement processing on the media stream of the AR communication;
  • the first terminal device receives the enhanced-processed AR media stream from the first IMS access gateway through the media stream channel.
  • the method further includes:
  • the first terminal device receives the AR media stream of the second terminal device sent by the AR media processing network element through the media stream channel.
  • the first request message carries a first description parameter of the first terminal device for establishing the auxiliary data channel
  • the first response message carries the second description parameter of the AR media processing network element for establishing the auxiliary data channel.
  • the AR media processing network element is located in the IMS network to which the first terminal device belongs (or is located) or is located in the IMS network to which the second terminal device belongs (or is located) .
  • an embodiment of the present application further provides an apparatus, and the apparatus is applied to an IMS core network element, and the beneficial effects can be referred to the description of the second aspect and will not be repeated here.
  • the apparatus has the function of implementing the behavior in the method example of the second aspect above.
  • the functions can be implemented by hardware, or can be implemented by hardware executing corresponding software.
  • the hardware or software includes one or more modules corresponding to the above functions.
  • the structure of the apparatus may include a sending unit and a receiving unit, and these units may perform the corresponding functions in the method example of the second aspect. For details, please refer to the detailed description in the method example, which will not be repeated here. .
  • an embodiment of the present application further provides an apparatus, where the apparatus is applied to an AR media processing network element, and the beneficial effects can be referred to the description of the third aspect and will not be repeated here.
  • the apparatus has the function of implementing the behavior in the method example of the third aspect above.
  • the functions can be implemented by hardware, or can be implemented by hardware executing corresponding software.
  • the hardware or software includes one or more modules corresponding to the above functions.
  • the structure of the apparatus may include a sending unit, a receiving unit, and a processing unit, and these units may perform the corresponding functions in the method example of the third aspect. For details, please refer to the detailed description in the method example, here I won't go into details.
  • an embodiment of the present application further provides an apparatus, and the apparatus is applied to the terminal device.
  • the apparatus has the functionality to implement the behavior in the method example of the fourth aspect above.
  • the functions can be implemented by hardware, or can be implemented by hardware executing corresponding software.
  • the hardware or software includes one or more modules corresponding to the above functions.
  • the structure of the apparatus includes a sending unit and a receiving unit, and these units can perform the corresponding functions in the method example of the fourth aspect. For details, please refer to the detailed description in the method example, which will not be repeated here.
  • an embodiment of the present application further provides an apparatus, and the apparatus is applied to the IMS core network element, and the beneficial effects can be referred to the description of the second aspect and will not be repeated here.
  • the structure of the communication apparatus includes a processor and a communication interface, and may further include a memory, the processor is configured to support the IMS core network element to perform the corresponding function in the method of the second aspect through the communication interface.
  • the memory is coupled to the processor and holds program instructions and data necessary for the communication device.
  • an embodiment of the present application further provides an apparatus, where the apparatus is applied to the AR media processing network element, and the beneficial effects can be referred to the description of the third aspect and will not be repeated here.
  • the structure of the apparatus includes a processor and a communication interface, and may further include a memory, the processor is configured to support the AR media processing network element to perform the corresponding function in the method of the third aspect through the communication interface.
  • the memory is coupled to the processor and holds program instructions and data necessary for the apparatus.
  • an embodiment of the present application further provides an apparatus, and the apparatus is applied to the terminal device, and the beneficial effects can be referred to the description of the fourth aspect and will not be repeated here.
  • the structure of the apparatus includes a processor and a communication interface, and may further include a memory, where the processor is configured to support the terminal device to perform the corresponding function in the method of the fourth aspect through the communication interface.
  • the memory is coupled to the processor and holds program instructions and data necessary for the apparatus.
  • an embodiment of the present application further provides an apparatus, where the apparatus is applied to an application server.
  • the structure of the communication apparatus includes a processor and a communication interface, and may further include a memory, the processor is configured to support the application server to perform the corresponding functions of the application server in any of the above aspects through the communication interface.
  • the memory is coupled to the processor and holds program instructions and data necessary for the communication device.
  • the present application further provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, when the computer-readable storage medium runs on a computer, the computer executes the methods described in the above aspects.
  • the present application also provides a computer program product comprising instructions, which, when run on a computer, cause the computer to perform the methods described in the above aspects.
  • the present application further provides a computer chip, the chip is connected to a memory, and the chip is configured to read and execute a software program stored in the memory, and execute the methods described in the above aspects.
  • FIG. 1 is a schematic diagram of a possible AR communication system architecture in an embodiment of the present application
  • FIG. 2A is a schematic diagram of another possible AR communication system architecture in an embodiment of the present application.
  • FIG. 2B is a schematic diagram of another possible AR communication system architecture in an embodiment of the present application.
  • FIG. 2C is a schematic diagram of another possible AR communication system architecture in an embodiment of the present application.
  • FIG. 3 is a schematic diagram of another possible AR communication system architecture in an embodiment of the present application.
  • FIG. 4 is a schematic diagram of another possible AR communication system architecture in an embodiment of the present application.
  • FIG. 5 is a schematic diagram of another possible AR communication system architecture in an embodiment of the present application.
  • FIG. 6 is a schematic flowchart of a method for AR communication in an embodiment of the present application.
  • FIG. 7A is a schematic flowchart of another method for AR communication in an embodiment of the present application.
  • FIG. 7B is a schematic flowchart of another method for AR communication in an embodiment of the present application.
  • FIG. 8A is a schematic flowchart of another method for AR communication in an embodiment of the present application.
  • 8B is a schematic diagram of a T1 port and a T2 port on an IMS AGW in an embodiment of the present application;
  • FIG. 9 is a schematic flowchart of another method for AR communication in an embodiment of the present application.
  • FIG. 10 is a schematic flowchart of another method for AR communication in an embodiment of the present application.
  • FIG. 11 is a schematic flowchart of another method for AR communication in an embodiment of the present application.
  • FIG. 12A is a schematic flowchart of another method for AR communication in an embodiment of the present application.
  • FIG. 12B is a schematic flowchart of another method for AR communication in an embodiment of the present application.
  • FIG. 12C is a schematic flowchart of another AR communication method in an embodiment of the present application.
  • FIG. 13 is a schematic flowchart of another method for AR communication in an embodiment of the present application.
  • FIG. 14 is a schematic flowchart of another method for AR communication in an embodiment of the present application.
  • FIG. 15 is a schematic flowchart of another method for AR communication in an embodiment of the present application.
  • 16 is a schematic flowchart of another method for AR communication in an embodiment of the present application.
  • FIG. 17 is a schematic flowchart of another method for AR communication in an embodiment of the present application.
  • FIG. 19 is a schematic flowchart of another method for AR communication in an embodiment of the present application.
  • FIG. 20 is a schematic structural diagram of a device in an embodiment of the present application.
  • 21 is a schematic structural diagram of another device in an embodiment of the present application.
  • 22 is a schematic structural diagram of another device in an embodiment of the application.
  • FIG. 23 is a schematic structural diagram of another device in an embodiment of the present application.
  • the present application provides an AR-based communication method and device, and provides an implementation manner of integrating AR into a voice and video call, thereby improving user experience.
  • Voice and video calls can, but are not limited to, adopt VoLTE, and can also be applied to voice and video calls provided by future technologies.
  • AR has three characteristics, namely virtual and real fusion, real-time interaction and three-dimensional registration.
  • Virtual-real fusion is to superimpose computer-generated virtual objects and information into real-world scenes to achieve a more intuitive and in-depth understanding of real-world scenes.
  • the enhanced information can be non-geometric information related to real objects, such as video and text, or geometric information, such as virtual three-dimensional objects and scenes.
  • Real-time interaction is through the interactive interface device in the augmented reality system, people interact with the augmented reality environment in real time in a natural way.
  • "Registration" in 3D registration can be interpreted as tracking and positioning.
  • 3D registration refers to the one-to-one correspondence between the virtual objects generated by the computer and the real environment, and the user will continue to maintain the correct alignment relationship when moving in the real environment.
  • At least one item(s) below or similar expressions thereof refer to any combination of these items, including any combination of single item(s) or plural items(s).
  • at least one item (a) of a, b, or c can represent: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, c may be single or multiple .
  • B corresponding to A means that B is associated with A, and B can be determined according to A.
  • determining B according to A does not mean that B is only determined according to A, and B may also be determined according to A and/or other information.
  • the terms “comprising” and “having” in the embodiments and claims of the present application and the drawings are not exclusive. For example, a process, method, system, product or device that includes a series of steps or modules is not limited to the listed steps or modules, and may also include unlisted steps or modules.
  • the AR communication system includes an AR media processing network element and an AR control network element.
  • AR media processing network elements provide media processing services for terminal devices with specific AR capabilities.
  • the AR media processing network element has the AR user plane (AR User plane, AR-U) function, and may also be called an AR-U network element.
  • the AR control network element has the function of the AR control plane, which can be called an AR control plane (AR Control plane, AR-C) network element.
  • the network element may be referred to as a device, for example, an AR media processing network element may be referred to as an AR media processing device.
  • a terminal device with AR capability can have one or more of the following logical functions: support the collection of audio and video data of real scenes through cameras and/or sensors, support logical operation functions (for example, a vision engine can be deployed), and support low-latency encoding And lightweight image rendering, support for virtual objects from the cloud, etc.
  • the terminal device in the embodiment of the present application may be a device equipped with a camera and having a video call function.
  • the terminal device may be a wearable device (eg, an electronic watch), and the terminal device may also be a mobile phone, a tablet computer, or other devices.
  • the specific form of the terminal device is not particularly limited in this embodiment of the present application.
  • AR media processing network elements specific AR media processing functions.
  • AR media processing functions include one or more of three-dimensional modeling, object recognition, tracking and positioning, interactive perception, virtual scene rendering, and combination of virtual and real.
  • 3D modeling is the 3D construction of the real environment based on the data collected by the terminal equipment.
  • Object recognition refers to identifying the object of interest of the terminal device user.
  • Tracking and positioning refers to tracking and positioning the objects of interest of terminal device users.
  • Interaction perception refers to the use of user interaction technology to achieve multi-sensory consistency.
  • Virtual scene rendering refers to performing rendering on a virtual scene.
  • the combination of virtual and real refers to the fusion of virtual objects and real objects.
  • the AR media processing network element supports the establishment of media stream channels and auxiliary data channels with terminal equipment.
  • the media stream channel is used to transmit AR core data (or referred to as media stream data, also referred to as media stream) between the terminal device and the AR media processing network element, such as audio and video streams.
  • the auxiliary data channel is used to transmit AR auxiliary data between the terminal device and the AR media processing network element.
  • AR auxiliary data AR specific data
  • the data types that can be included in the AR auxiliary data in the uplink direction and the video data types included in the AR core data are exemplarily described.
  • the data types that may be included in the AR assistance data in the downstream direction and the video data types included in the audio and video streams are exemplarily described.
  • the AR controls the network element and has the function of session association and service control.
  • the session association may include the association between the media flow between the terminal device and the AR media processing network element, and the association between the signaling flow between the AR media processing network element and the AS in the AR session type scenario. Or the association between the media flow between the terminal device and the AR media processing network element, the signaling flow between the AR media processing network element and the AS, and the message flow between the AS and a third-party server.
  • the third-party server is used to provide the required materials for AR processing, such as virtual objects and real background objects.
  • the service control may be to control AR media processing based on service characteristics and the like. For example, different service characteristics correspond to different AR media processing methods, and the AR control network element can indicate different AR media processing methods to the AR media processing network element according to different service characteristics of the terminal device.
  • the AR media processing network element may be an IP multimedia subsystem (IP multimedia subsystem, IMS) access gateway (AGW) or may be a media resource function (media resource function, MRF).
  • IP multimedia subsystem IP multimedia subsystem, IMS
  • AGW access gateway
  • MRF media resource function
  • the AR control network element may be an application server.
  • the AR communication system includes an IMS AGW and an IMS core (core).
  • the IMS core (core) network element controls the establishment of an auxiliary data channel and a media stream channel between the IMS AGW with AR processing capability and the terminal device with AR capability.
  • the auxiliary data channel is used to transmit AR auxiliary data for AR communication between at least two terminal devices.
  • the media stream channel is used to transmit media streams for AR communication between at least two terminal devices.
  • the IMS AGW can provide media processing services for terminal equipment. For example, when the IMS AGW receives the media stream through the media stream channel and the AR assistance data through the auxiliary data channel, the IMS AGW may use the AR assistance data to perform enhanced processing on the media stream.
  • the AR communication system may also include an application server (application service, AS).
  • the AR communication system may further include at least two terminal devices.
  • the IMS AGWs that respectively serve different terminal devices may be the same or different IMS AGWs.
  • the two terminal devices are taken as an example of the first terminal device and the second terminal device respectively, serving the first terminal device. Different from the IMS AGW of the second terminal device, for example, they are IMS AGW1 and IMS AGW2 respectively.
  • Both terminal devices in AR communication may have AR capabilities, or one of the terminal devices may have AR capabilities, and the other terminal device may not have AR capabilities.
  • FIG. 2A it is taken as an example that both the first terminal device and the second terminal device have AR capabilities.
  • the IMS AGWs serving different terminal devices are different.
  • Different IMS AGWs of two terminal devices serving AR communication may both have AR processing capabilities, or only one of the IMS AGWs may have AR processing capabilities, and the other IMS AGW may not have AR processing capabilities.
  • Both IMS AGW1 and IMS AGW2 in FIG. 2A have AR processing capability.
  • the IMS AGW1 serves the first terminal device, or the IMS AGW1 is an access gateway to which the first terminal device belongs, or the IMS AGW1 is located in the IMS network to which the first terminal device belongs (or is located).
  • the IMS AGW2 serves the second terminal device, or the IMS AGW2 is an access gateway to which the second terminal device belongs, or the IMS AGW2 is located in the IMS network to which the second terminal device belongs.
  • One access gateway may serve multiple terminal devices, for example, IMS AGW1 serves the first terminal device and also serves the third terminal device (not shown in FIG. 2A ).
  • the IMS core may include a call session control function (call session control function, CSCF) and/or a home subscriber server (home subscriber server, HSS), and the IMS core may also include other network elements, which will not be repeated in the embodiments of this application.
  • CSCF call session control function
  • HSS home subscriber server
  • the CSCF is the call control center of the IMS core, which implements functions such as user access, authentication, session routing and service triggering on the IP transmission platform.
  • the CSCF may include one of a serving-call session control function (S-CSCF), a proxy-CSCF (Proxy CSCF, P-CSCF), an interrogating-CSCF (Interrogating-CSCF, I-CSCF), or multiple.
  • S-CSCF serving-call session control function
  • P-CSCF proxy-CSCF
  • Interrogating-CSCF Interrogating-CSCF
  • I-CSCF interrogating-CSCF
  • the HSS is used to record user subscription data (such as user information and service data).
  • the IMS core network element serving the first terminal device and the IMS core network element serving the second terminal device may be different, for example, related to the location of the terminal device.
  • the IMS core network element mentioned in the embodiment of this application may include a CSCF.
  • a third-party server may also be included in the AR communication system.
  • the third-party server can be understood as the server of the third-party provider.
  • the third-party server is used to provide the target object required by the terminal device to perform AR communication. such as virtual objects. Another example is the real background object.
  • the third-party server and the application server are configured with each other's address information, such as an IP address and a port number.
  • the third-party server supports establishing a communication connection with the AR media processing network element, such as a hypertext transfer protocol (hypertext transfer protocol, http) connection.
  • the third-party server does not support establishing a communication connection with the AR media processing network element, as shown in FIG. 2C , the third-party server establishes a communication connection with the application server, such as a hypertext transfer protocol http connection.
  • one of the two terminal devices in AR communication does not have AR capability.
  • the first terminal device has the AR capability
  • the second terminal device does not have the AR capability.
  • the IMS AGW1 serving the first terminal device in FIG. 3 has AR capability.
  • one of the two terminal devices in AR communication does not have AR capability.
  • the first terminal device has the AR capability
  • the second terminal device does not have the AR capability.
  • the IMS AGW2 serving the second terminal device has the AR capability
  • the IMS AGW1 serving the first terminal device does not have the AR capability.
  • the AR media processing network element may be a media resource function (media resource function, MRF).
  • MRF media resource function
  • the AR communication system includes an MRF, an IMS core network (core), an application server (application service, AS) and a third-party server as an example .
  • MRF is usually divided into media resource function control (media resource function control, MRFC) and media resource function processing (media resource function processor, MRFP).
  • MRF media resource function processor
  • FIG. 5 takes the first terminal device, the second terminal device and the third terminal device as examples.
  • Multiple terminal devices of an AR conference may all have AR capabilities, or one or several terminal devices may have AR capabilities, and other terminal devices may not have AR capabilities.
  • the first terminal device and the second terminal device both have the AR capability, and the third terminal device does not have the AR capability as an example.
  • the MRFs serving multiple terminal devices may be the same or different MRFs.
  • the MRFs serving the first terminal device and the third terminal device are the same, both are MRF1, serving the second terminal device.
  • the MRF is MRF2.
  • Different MRFs serving multiple terminal devices for AR conferences may all have AR processing capabilities, or only one or several MRFs may have AR processing capabilities, and the other MRFs may not have AR processing capabilities.
  • Both MRF1 and MRF2 in Figure 5 have AR processing capabilities.
  • MRF1 serves the first terminal device and the third terminal device, and MRF2 serves the second terminal device.
  • the terminal device may establish different auxiliary data channels according to the type of the AR auxiliary data to be transmitted. For example, when pose data needs to be transmitted, auxiliary data channel 1 is established, and when operation data needs to be transmitted, auxiliary data channel 2 is established.
  • the terminal equipment can also transmit different types of AR assistance data through one assistance data channel.
  • the following takes the AR communication between the first terminal device and the second terminal device as an example, after the first terminal device and the second terminal device trigger AR communication, the process of establishing an auxiliary data channel between the first terminal device and the AR media processing network element Be explained. See Figure 6.
  • a first terminal device sends a first request message to an IMS core network element, where the first request message is used to request the establishment of an auxiliary data channel.
  • the first request message may carry the first description parameter of the first terminal device.
  • the first description parameter is used by the first terminal device to establish the auxiliary data channel with the AR media processing network element.
  • the first request message also carries the address of the first terminal device.
  • the first description parameter may use a session description protocol (session description protocol, SDP).
  • SDP session description protocol
  • other description protocols may also be used, which are not specifically limited in this application.
  • the SDP is used as an example for the first description parameter.
  • the SDP information carrying the first description parameter is referred to as the first SDP information (may be referred to as the first SDP for short). That is, the first request message carries the first SDP information of the first terminal device, and the first SDP information includes a description parameter for the first terminal device to establish the auxiliary data channel with the AR media processing network element.
  • the first SDP information may include one or more parameters such as a port number of the first terminal device for transmitting AR assistance data, a type of media stream, a supported codec format, and the like.
  • the media stream types may include video (video stream), audio (audio stream), and datachannel (AR auxiliary data).
  • AR auxiliary data For example, in this case, m lines for describing AR assistance data may be included in the first SDP information.
  • m lines are described as follows:
  • m application 10001 User Datagram Protocol (UDP)/Datagram Transport Layer Security (DTLS)/(Stream Control Transmission Protocol, SCTP) webrtc-datachannel.
  • UDP User Datagram Protocol
  • DTLS Datagram Transport Layer Security
  • SCTP Stream Control Transmission Protocol
  • the m line represents a description for a specific AR auxiliary data stream
  • 10001 can represent the port number of the quintuple
  • UDP/DTLS/SCTP represents the transmission protocol followed by the AR auxiliary data
  • webrtc-datachannel represents the establishment of the auxiliary data channel.
  • Other information may also be included in the first SDP information, such as line a for describing AR media data.
  • the description of line a is as follows:
  • dcmap indicates a parameter set that needs to be negotiated between the two ends for the auxiliary data channel, such as whether the data channel supports reliable transmission, whether to set the maximum number of retransmissions, the sub-protocol format transmitted in the data channel, and so on.
  • 0 indicates the auxiliary data channel with ID 1
  • subprotocol indicates the protocol type of AR auxiliary data transmitted by the auxiliary data channel with ID 0.
  • Label indicates the description of the AR auxiliary data transmitted by the auxiliary data channel.
  • the first SDP information may also include the following:
  • dcsa represents the SDP attribute of the auxiliary data channel
  • 0 represents the ID of the specific auxiliary data channel
  • accept-type represents the format of the AR auxiliary data transmitted by the auxiliary data channel.
  • accept-types:text/plain indicates that the format used by the auxiliary data channel is text format.
  • the first terminal device may send the first request message to the IMS core network element in the process of establishing AR communication with the second terminal device.
  • the first request message may use a Session Initiation Protocol (Session Initiation Protocol, SIP) request (invite) message.
  • SIP Session Initiation Protocol
  • the first request message may also carry address information of the first terminal device, for example, the address information of the first terminal device may include the IP address and/or port number of the first terminal device.
  • the first request message is further used to request the establishment of a media stream channel, where the media stream channel is used to transmit a media stream for AR communication between the first terminal device and the second terminal device.
  • the first request message also carries a description parameter used by the first terminal device to establish the media stream channel with the AR media processing network element.
  • the first SDP information further includes a description parameter for the first terminal device to establish the media stream channel with the AR media processing network element.
  • the first terminal device may send the first request message to the IMS core network element in the process of re-establishing the AR session with the second terminal device.
  • the first request message may be a SIP Re-invite message.
  • the first terminal device may send the first request message to the IMS core network element in the process of updating the AR session with the second terminal device.
  • the first request message may be a SIP update invite (update request) message.
  • the IMS core network element After receiving the first request message, the IMS core network element sends a second request message to the AR media processing network element.
  • the second request message is used to request to establish the auxiliary data channel between the AR media processing network element and the first terminal device.
  • the second request message also carries the first SDP information.
  • the communication between the first terminal device and the second terminal device in this embodiment may be point-to-point communication between the first terminal device and the second terminal device, or may be the establishment of conference communication between the first terminal device and the second terminal device.
  • the AR media processing network element is an IMS AGW or MRF capable of AR processing.
  • the AR media processing network element may be an IMS AGW.
  • the AR media processing network element may be an MRF.
  • the AR media processing network element may be located in the IMS network to which the first terminal device belongs (or where it is located), and the AR media processing network element is used to provide services for the first terminal device, or may be located in (or where the second terminal device belongs) In the IMS network where it is located, it is used to provide services for the second terminal device.
  • the second request message may be an H248 protocol message, such as an H248 add request message.
  • the second request message may be a SIP invite message.
  • the AR media processing network element sends a second response message to the IMS core network element, where the second response message is a response message corresponding to the second request message.
  • the second response message is used to indicate to the IMS core network element that the AR media processing network element confirms the establishment of the auxiliary data channel.
  • the second response message carries the second description parameter of the AR media processing network element for establishing the auxiliary data channel.
  • the AR media processing network element may determine the second description parameter that it supports from the first description parameter.
  • the second description parameter may adopt the SDP protocol, and may also adopt other protocols.
  • the SDP information of the AR media processing network element is referred to as the second SDP information (may be referred to as the second SDP for short).
  • the second SDP information includes a description parameter of the AR media processing network element for establishing the auxiliary data channel with the first terminal device.
  • the second SDP information includes parameters such as the port of the auxiliary data channel corresponding to the AR media processing network element, the type of media stream, and the supported codec format.
  • the media stream types may include video (video stream), audio (audio stream), and datachannel (AR auxiliary data).
  • the second SDP information may include m lines for describing AR assistance data negotiated by the AR media processing network element, and the media stream type of the m lines is AR assistance data.
  • the IMS core network element When receiving the second response message, the IMS core network element sends the first response message to the first terminal device, where the first response message carries the address of the AR media processing network element.
  • the first response message carries the second SDP information of the AR media processing network element.
  • the IMS core network element after receiving the first request message, forwards the first request message to the application server.
  • the application server determines that the first terminal device subscribes to the AR service, it may send its own address information to the AR media processing network element through the IMS core network element.
  • the application server may add its own address information to the first request message and send it to the IMS core network element. Therefore, the IMS core network element carries the address information of the application server in the second request message sent to the AR media processing network element, so that when the subsequent AR media processing network element needs to obtain the content provided by the third-party server, it can obtain the content from the third-party server through the AS.
  • the third-party server obtains the required content.
  • the application server may send a subscription query request to the HSS, for querying whether the subscription information of the first terminal device includes the information of subscribing to the AR service.
  • the HSS may feed back the subscription information of the first terminal device to the application server through the subscription query response.
  • the subscription query request may use a direct connection request (diameter request) message.
  • the subscription query response may use a direct connection response (diameter response) message.
  • the MS core network element after receiving the first request message, forwards the first request message to the application server.
  • the application server may send the stored address information of the third-party server to the AR media processing network element through the IMS core network element.
  • the application server may add the address information of the third-party server to the first request message and send it to the IMS core network element. Therefore, the IMS core network element carries the address information of the third-party server in the second request message sent to the AR media processing network element, so that when the subsequent AR media processing network element needs to obtain the content provided by the third-party server, it can obtain the content provided by the third-party server.
  • the server gets what it needs.
  • the terminal device may know in advance whether the AR media processing network element has the AR processing capability. For example, in the process of registering with the network side, the terminal device learns whether there is an AR media processing network element with AR processing capability in the IMS network where the terminal device is located.
  • the IMS core network element before sending the second request message to the AR media processing network element, the IMS core network element first inquires whether the AR media processing network element has the AR processing capability, for example, sends a query to the AR media processing network element The query message is used to query whether the AR media processing network element has the AR processing capability. After receiving the query message, the AR media processing network element will send indication information to the IMS core network element to indicate whether the AR media processing network element has the AR processing capability. AR processing power. When it is determined that the AR media processing network element has the AR processing capability, a second request message is sent to the AR media processing network element having the AR processing capability.
  • the AR media processing network element may be an IMS AGW or MRF capable of AR processing. Take the AR media processing network element as the IMS AGW as an example. 7A and 7B, for the convenience of distinction, the IMS AGW that will provide services for the first terminal device is called IMS AGW1, and the IMS AGW that will provide services to the second terminal device is called IMS AGW2.
  • 701a refer to 601, which will not be repeated here.
  • the IMS core network element when receiving the first request message from the first terminal device, the IMS core network element sends a query message 1 to the IMS AGW1, where the query message 1 is used to query whether the IMS AGW1 has the AR processing capability.
  • the query message 1 may include parameters for querying whether AR processing capability is available.
  • the parameter can be "ifSupportforAR_flag".
  • Query message 1 can be H248 add request.
  • H248 add request adds the parameter ifSupportforAR_flag.
  • the IMS AGW1 when the IMS AGW1 determines that it has the AR processing capability, it sends a first indication to the IMS core network element, where the first indication is used to indicate that the IMS AGW1 has the AR processing capability.
  • the IMS core network element sends a second request message to the IMS AGW1. Refer to 602, and details are not repeated here.
  • the IMS core network element when receiving the first request message from the first terminal device, the IMS core network element sends a query message 2 to the IMS AGW1, where the query message 2 is used to query whether the IMS AGW1 has the AR processing capability.
  • the query message 2 may include parameters for querying whether AR processing capability is available.
  • the parameter can be "ifSupportforAR_flag".
  • Query message 2 can be H248 add request. H248 add request adds the parameter ifSupportforAR_flag.
  • the IMS AGW1 when it determines that it does not have the AR processing capability, it sends a second indication to the IMS core network element, where the second indication is used to indicate that the IMS AGW1 does not have the AR processing capability.
  • the second indication may be carried in H248 add response.
  • the IMS core network element sends a second request message to the IMS AGW2.
  • the IMS core network element sends the second request message to the AR media processing network element to carry parameters for inquiring whether the AR media processing network element has the AR processing capability.
  • the second request message carries "ifSupportforAR_flag".
  • the AR media processing network element does not carry the "ifSupportforAR_flag” parameter in the second response message and carries the second description parameter, it is determined that the AR media processing network element has the AR processing capability.
  • an auxiliary data channel is established during the AR session creation process.
  • the AR media processing network element with AR processing function is the IMS AGW1 to which the first terminal device belongs.
  • FIG. 8A is a schematic flowchart of a communication method in the first possible application scenario of the present application.
  • the first terminal device sends a first request message to an IMS core network element (such as intermediate IMS1).
  • IMS core network element such as intermediate IMS
  • the IMS core network element used to provide services for the first terminal device is referred to as intermediate IMS1 as an example.
  • the first request message adopts the SIP INVITE message as an example, and for the convenience of distinction, the first request message is called SIP INVITE1 as an example.
  • SIP INVITE1 is used to request the establishment of an AR session.
  • SIP INVITE1 includes the first SDP of the first terminal device.
  • the first SDP includes description parameters for the first terminal device to establish the auxiliary data channel.
  • the first SDP further includes description parameters for the first terminal device to establish a media stream channel.
  • the SIP INVITE also includes address information of the first terminal device, including the IP address and port number of the first terminal device.
  • the IP address is IP1o
  • the port number is P1o.
  • the first SDP includes parameters such as address information (such as a port number), a type of media stream, and a supported codec format.
  • the media stream types may include video (video stream), audio (audio stream), and datachannel (AR auxiliary data).
  • the first SDP information includes m lines for describing AR auxiliary data, and may further include m lines for describing audio data and m lines for video data. Line m is a description for one media stream (video stream, audio stream or stream composed of AR auxiliary data).
  • the following three m lines are defined, which respectively represent the m lines of the video stream, the m lines of the audio stream, and the m lines of the stream composed of AR auxiliary data.
  • m video 49154RTP/AVPF 101 102.
  • the first parameter of the m line represents the media name, which is used to indicate that the supported media stream type is video (video).
  • the second parameter is the port number, which is used to indicate that the video stream is sent using port 49154.
  • the third parameter is the transmission protocol, which adopts real-time transport protocol (RTP)/audio video profile feedback (AVPF).
  • RTP real-time transport protocol
  • AVPF audio video profile feedback
  • the 4th-5th parameter indicates the supported payload type number.
  • the m audio 49152RTP/AVP 98 97 99.
  • the first parameter of the m line represents the media name, which is used to indicate that the supported media stream type is audio (audio).
  • the second parameter is the port number, which is used to indicate that the audio stream is sent using port 49152.
  • the third parameter is the transmission protocol, which adopts real-time transport protocol (RTP)/audio video profile (AVP).
  • RTP real-time transport protocol
  • AVP audio video profile
  • the 4th-6th parameter indicates the supported payload type number.
  • m application 10001UDP/DTLS/SCTP webrtc-datachannel.
  • the m line represents a description for a specific AR auxiliary data stream
  • 10001 can represent the port number of the quintuple
  • UDP/DTLS/SCTP represents the transmission protocol followed by the AR auxiliary data
  • webrtc-datachannel represents the establishment of the auxiliary data channel.
  • the first SDP information may also include other information.
  • the first SDP information may also include other information.
  • reference may be made to the description in the embodiment corresponding to FIG. 6 , and details are not repeated here.
  • the Intermediate IMS1 receives the SIP INVITE1 of the first terminal device, and forwards the SIP INVITE1 to the AS.
  • the Intermediate IMS1 determines that the first terminal device has subscribed to the AR media enhancement service according to the subscription data of the first terminal device, and then forwards the SIP INVITE1 to the AS.
  • the AS is used to provide control services for AR media enhancement services.
  • AS sends SIP INVITE2 to Intermediate IMS1.
  • the AS After receiving the SIP INVITE1, the AS determines that the AR service subscribed by the first terminal device triggers the corresponding AR service.
  • the AS triggers the establishment of an AR session through Intermediate IMS1, that is, triggers SIP INVITE2 to Intermediate IMS1.
  • the SIP INVITE2 triggered by the AS and the SIP INVITE1 sent by the first terminal device may use the same message, or may use different messages.
  • the AS identifies that the first terminal device triggers an AR service (or initiates an AR session)
  • the AS may record the session identifier of the AR session as an AR session, and establish a binding relationship between the session identifier and the identifier of the first terminal device. . After the binding relationship is established, the target object obtained from the third-party server through the AS can be used later, which will be described in detail later, and will not be repeated here.
  • the Intermediate IMS1 sends a query message for querying whether the IMS AGW1 has the AR capability to the IMS AGW1.
  • the query message for querying whether the IMS AGW1 has the AR capability is referred to as query message 1 herein.
  • Query message 1 can use H248 add request1(req1).
  • H248 add request1 may be used to request bearer establishment.
  • H248 add request1 carries a parameter for inquiring whether the IMS AGW1 has the AR processing capability.
  • the parameter for inquiring whether the IMS AGW1 has AR processing capability may be "IfSupportforAR_flag".
  • outgoing termination T2 (outgoing termination T2). Specifically, create the outgoing port of IMS AGW1.
  • the IMS AGW1 feeds back the address information corresponding to the T2 port and the indication information indicating whether it has the AR processing capability to the Intermediate IMS1.
  • the IMS AGW1 After receiving the H248 add request1, the IMS AGW1 sends a query response 1 to the Intermediate IMS1, and the query response 1 includes indication information, which is used to indicate whether the IMS AGW1 has the AR processing capability. For example, query response 1 can use H248 add response1.
  • Intermediate IMS1 sends H248 add response(resp)1, and H248 add response1 includes the indication information and address information corresponding to the T2 port.
  • the address information corresponding to the T2 port includes an IP address and a port number, and the IP address is IP2o and the port number is P2o as an example.
  • IMS AGW1 has AR processing capability
  • IMS AGW1 has AR processing capability
  • IMS AGW1 has AR processing capability ability, for example.
  • the Intermediate IMS1 After receiving the H248 add response1, the Intermediate IMS1 identifies that the IMS AGW1 has the AR processing capability, and the Intermediate IMS1 triggers the creation of an AR session to the Intermediate IMS1.
  • Send SIP INVITE3 to Intermediate IMS1, and SIP INVITE3 carries the address information corresponding to the T2 port of IMS AGW1.
  • the Intermediate IMS1 recognizes that the IMS AGW1 has AR processing capabilities, then only the auxiliary data channel needs to be established between the first terminal device and the IMS AGW1, and the AR session establishment request sent to the network side of the second terminal device does not need to be Carry SDP for requesting establishment of auxiliary data channel.
  • Intermediate IMS2 sends SIP INVITE3 to AS2, carrying the address information corresponding to the T2 port of IMS AGW1.
  • AS2 sends SIP INVITE3 to Intermediate IMS2, carrying the address information corresponding to the T2 port of IMS AGW1.
  • Intermediate IMS2 sends H248 add requset2 to IMS AGW2 to instruct IMS AGW2 to create outgoing port T2.
  • the IP address and port number are included, and the IP address is IP3o and the port number is P3o as an example.
  • IMS AGW2 creates outgoing port T2, and sends H248 add resp2 to Intermediate IMS2, carrying the address information corresponding to the T2 port of IMS AGW2.
  • the IMS AGW2 sends a SIP INVITE4 to the second terminal device, and the SIP INVITE4 includes the address information corresponding to the T2 port of the IMS AGW2.
  • the second terminal device sends a SIP 180 to the IMS AGW2, where the SIP 180 carries the address information (IP1a, P1a) of the second terminal device.
  • Intermediate IMS2 sends H248 mod requset1 to IMS AGW2 to instruct IMS AGW2 to configure outgoing port T1. For example, establish an association relationship between the address information of T1 of the IMS AGW2 and the address information of the second terminal device.
  • IMS AGW2 configures outgoing port T2, and sends H248 mod resp1 to Intermediate IMS2.
  • H248 mod resp1 can carry the address information corresponding to the IMS AGW2 T2 port and the address information of the second terminal device.
  • Intermediate IMS2 sends H248 add requset3 to IMS AGW2 to instruct IMS AGW2 to create incoming port T2.
  • H248 add requset3 includes address information corresponding to the T2 port of IMS AGW1.
  • the IMS AGW2 creates the incoming port T2, and sends H248 add resp2 to the Intermediate IMS2, carrying the address information (IP4o, P4o) corresponding to the T2 port of the IMS AGW2.
  • the Intermediate IMS1 receives the SIP 180 from the second terminal device, for example, the SIP 180 may include address information of the second terminal device.
  • the IP address of the second terminal device is IP1a
  • the port number of the second terminal device is P1a.
  • Intermediate IMS1 records the IP address and port number of the second terminal device.
  • the Intermediate IMS1 sends a configuration modification request to the IMS AGW1.
  • the configuration modification request may use H248Mod req3.
  • the H248Mod req includes the IP address (IP1a) and port number (P1a) of the second terminal device.
  • H248Mod req3 is used to request IMS AGW1 to configure outgoing port T2.
  • the IMS AGW1 configures the outgoing termination T2 (outgoing termination T2) according to the address information of the second terminal device, and sends a configuration modification response to the Intermediate IMS1, and the modification configuration response may adopt H248 mod response3.
  • the Intermediate IMS1 receives a configuration modification response from the IMS AGW1, and the Intermediate IMS1 sends a second request message to the IMS AGW1.
  • the second request message is used to request the establishment of the auxiliary data channel, and may also be used to request the establishment of the incoming port T1, for example, using H248 add req4.
  • H248 add req4 is also used to request the establishment of an auxiliary data channel.
  • H248 add req4 includes the first SDP of the first terminal device and the address information of the first terminal device.
  • the IMS AGW1 After receiving the H248 add req4, the IMS AGW1 creates an incoming port T1 (incoming termination T1) and confirms the establishment of the auxiliary data channel.
  • the IMS AGW1 sends a second response message to the Intermediate IMS1, including the response of the incoming configuration T1 information, and may also include the response of the auxiliary data channel establishment.
  • the second response message can use H248 add response4.
  • the H248 add response4 includes the second SDP information of the IMS AGW1 and the address information of the IMS AGW1.
  • the address information of the IMS AGW1 includes the IP address (IP2a) and the port number (P2a) of the IMS AGW1.
  • the H248 add response4 may further include address information of the first terminal device.
  • the second SDP information includes m lines for describing the auxiliary data channel to be established of the IMS AGW1.
  • the response of the IMS AGW1 to the m line of the first terminal device can be described as follows:
  • m application 10001UDP/DTLS/SCTP webrtc-datachannel.
  • the Intermediate IMS1 records the address information of the T1 port of the IMS AGW1, and determines the second SDP information established by the auxiliary data channel, and the first response message sent to the AS includes the second SDP information and the address information of the IMS AGW1.
  • SIP 180 can be used in the first response message.
  • the SIP 180 includes the second SDP information and the address information of the IMS AGW1.
  • the AS forwards SIP 180 to the Intermediate IMS1.
  • the Intermediate IMS1 forwards the SIP 180 to the first terminal device.
  • step 803 after the AS receives the SIP INVITE1, when the AS determines that the first terminal device subscribes to the AR service, it can send its own address information to the IMS AGW1 through the Intermediate IMS1. AS can add its own address information to SIP INVITE1 and send it to Intermediate IMS1. Therefore, the Intermediate IMS1 carries the address information of the AS in the second request message (H248 add request) sent to the IMS AGW1, so that when the subsequent IMS AGW1 needs to obtain the content provided by the third-party server, it can obtain all the information from the third-party server through the AS. content required.
  • the AS may send a subscription query request to the HSS, for querying whether the subscription information of the first terminal device includes the information of subscribing to the AR service.
  • the HSS may feed back the subscription information of the first terminal device to the AS through the subscription query response.
  • the subscription query request may use a direct connection request (diameter request) message.
  • the subscription query response may use a direct connection response (diameter response) message.
  • the Intermediate IMS1 forwards the SIP INVITE1 to the AS after receiving the SIP INVITE1.
  • the AS determines that the first terminal device subscribes to the AR service, it can send the stored address information of the third-party server to the IMS AGW1 through the Intermediate IMS1.
  • the AS may add the address information of the third-party server to the first request message and send it to the Intermediate IMS1.
  • Intermediate IMS1 carries the address information of the third-party server in the second request message (H248 add request) sent to IMS AGW1, so that when the subsequent IMS AGW1 needs to obtain the content provided by the third-party server, it can obtain all the information from the third-party server. content required.
  • the above-mentioned T1 port and T2 port will be described below with reference to FIG. 8B .
  • the T1 port is the port on the IMS AGW used to connect terminal devices through the access network.
  • the T2 port is the port on the IMS AGW used to connect to the IMS core network.
  • the IMS AGW can establish a connection with the terminal equipment of the communication peer through the IMS core network.
  • the establishment of the auxiliary data channel is triggered by triggering the AR session re-establishment process.
  • the IMS AGW1 to which the first terminal device belongs has the AR processing capability.
  • FIG. 9 it is a schematic flowchart of the communication method of the third possible application scenario of the present application.
  • the first terminal device After the first terminal device establishes the AR session with the second terminal device, the first terminal device sends a first request message to an IMS core network element (such as intermediate IMS1).
  • the first request message is an SIP re-establishment request (Re-INVITE1) as an example.
  • SIP Re-INVITE1 is used to request the creation of an auxiliary data channel.
  • SIP Re-INVITE1 includes the first SDP of the first terminal device. For the description of the first SDP of the first terminal device, refer to the description in the embodiment corresponding to FIG. 6 , and details are not repeated here.
  • the Intermediate IMS1 receives the SIP Re-INVITE1 of the first terminal device, and forwards the Re-INVITE1 to the AS.
  • the AS sends Re-INVITE1 to the Intermediate IMS1.
  • Intermediate IMS1 sends query message 1 to IMS AGW1.
  • Query message 1 can be carried in H248mod req.
  • H248 mod req is used to request bearer establishment.
  • H248 mod req may also carry a parameter for inquiring whether the IMS AGW1 has AR processing capability.
  • the parameter for inquiring whether the IMS AGW1 has AR processing capability may be "IfSupportforAR_flag".
  • the IMS AGW1 receives the H248 mod req and indicates to the Intermediate IMS1 whether it has AR processing capability.
  • the IMS AGW1 After receiving the H248 mod request, the IMS AGW1 sends a query response 1 to the Intermediate IMS1, and the query response 1 includes indication information, which is used to indicate whether the IMS AGW1 has the AR processing capability.
  • indication information which is used to indicate whether the IMS AGW1 has the AR processing capability.
  • Intermediate IMS1 sends H248 mod response, and H248 mod response includes the indication information.
  • IMS AGW1 has AR processing capability
  • IMS AGW1 has AR processing capability
  • IMS AGW1 has AR processing capability ability, for example.
  • the Intermediate IMS1 After receiving the H248 mod response, the Intermediate IMS1 identifies that the IMS AGW1 has the AR processing capability, that is, only the auxiliary data channel needs to be established between the first terminal device and the IMS AGW1, and the AR session establishment request to the second terminal device does not need to carry the AR session.
  • the AR session establishment request can use SIP Re-INVITE2.
  • the Intermediate IMS1 sends the SIP Re-INVITE2 to the second terminal device through the Intermediate IMS2 to which the second terminal device belongs.
  • the Intermediate IMS1 receives a SIP response from the second terminal device, such as SIP 200OK.
  • Intermediate IMS1 sends a second request message to IMS AGW1, where the second request message can be used to request the establishment of an auxiliary data channel, and can also be used to modify incoming configuration T1 information, such as sending a message to modify incoming configuration T1 information to IMS AGW1 through H248 add req. ask.
  • H248 add req is also used to request the establishment of an auxiliary data channel.
  • H248 add req includes the first SDP of the first terminal device.
  • the IMS AGW1 modifies the incoming termination T1 information (incoming termination T1 information) and confirms the establishment of the auxiliary data channel.
  • the IMS AGW1 sends a response of modifying the incoming configuration T1 information to the Intermediate IMS1.
  • the response to modifying the incoming configuration T1 information can be H248 add response.
  • H248 add response includes the second SDP information of IMS AGW1.
  • the second SDP reference may be made to the description in the embodiment corresponding to FIG. 6 , and details are not repeated here.
  • the Intermediate IMS1 extracts the second SDP information of the IMS AGW1, and the AR session establishment response sent to the AS includes the second SDP information.
  • SIP 200OK can be used in the AR session establishment response.
  • the SIP 200OK includes the second SDP information.
  • the AS forwards the SIP 200OK to the Intermediate IMS1.
  • the Intermediate IMS1 forwards the SIP 200 OK to the first terminal device.
  • step 903 after the AS receives the SIP Re-INVITE1, when the AS determines that the first terminal device subscribes to the AR service, it can send its own address information to the IMS AGW1 through the Intermediate IMS1. AS can add its own address information to SIP Re-INVITE1 and send it to Intermediate IMS1. Therefore, the Intermediate IMS1 carries the address information of the AS in the second request message (H248 add request) sent to the IMS AGW1, so that when the subsequent IMS AGW1 needs to obtain the content provided by the third-party server, it can obtain all the information from the third-party server through the AS. content required.
  • the Intermediate IMS1 forwards the SIP Re-INVITE1 to the AS after receiving the SIP INVITE1.
  • the AS determines that the first terminal device subscribes to the AR service, it can send the stored address information of the third-party server to the IMS AGW1 through the Intermediate IMS1.
  • the AS may add the address information of the third-party server to the first request message and send it to the Intermediate IMS1.
  • Intermediate IMS1 carries the address information of the third-party server in the second request message (H248 add request) sent to IMS AGW1, so that when the subsequent IMS AGW1 needs to obtain the content provided by the third-party server, it can obtain all the information from the third-party server. content required.
  • an auxiliary data channel is established during the AR session creation process.
  • the IMS AGW1 to which the first terminal device belongs does not have the AR processing capability and the IMS AGW2 to which the second terminal device belongs has the AR processing capability as an example.
  • FIG. 10 it is a schematic flowchart of the communication method of the third possible application scenario of the present application.
  • the IMS AGW1 sends indication information to the Intermediate IMS1 to indicate whether the IMS AGW1 has the AR processing capability.
  • the Intermediate IMS1 sends the H248 add response, and the H248 add response includes the indication information and the address information corresponding to the T2 port.
  • the address information corresponding to the T2 port includes an IP address and a port number, and the IP address is IP2o and the port number is P2o as an example.
  • IMS AGW1 has AR processing capability
  • IMS AGW1 has AR processing capability
  • IMS AGW1 does not have AR Take processing power, for example.
  • the Intermediate IMS1 After receiving the H248 add response, the Intermediate IMS1 identifies that the IMS AGW1 does not have the AR processing capability, and the Intermediate IMS1 sends a second request message (SIP INVITE2) to the IMS AGW2 through the Intermediate IMS2 and the AS to which the second terminal device belongs. There is no need to carry a request for establishing an auxiliary data channel in the SIP INVITE2 to the second terminal device.
  • the SIP INVITE2 sent to the second terminal device carries the address information corresponding to the T2 port of the IMS AGW1.
  • the Intermediate IMS2 may send a query message 2 to the IMS AGW2.
  • H248 add req2 may also carry a parameter for inquiring whether the IMS AGW2 has AR processing capability.
  • the parameter for inquiring whether the IMS AGW2 has AR processing capability may be "IfSupportforAR_flag".
  • H248 add req2 is also used to instruct the creation of the outgoing port T2. For details, see 809, which will not be repeated here.
  • the IMS AGW2 receives the H248 add req2, and indicates to the Intermediate IMS2 the indication information whether it has the AR processing capability.
  • the IMS AGW2 after receiving the H248 add request2, the IMS AGW2 sends a query response 2 to the Intermediate IMS2, and the query response 2 includes the indication information 2, which is used to indicate whether the IMS AGW2 has the AR processing capability.
  • Intermediate IMS2 sends H248 add response2 (that is, query response 2), and H248 add response2 includes the indication information 2.
  • IMS AGW1 has AR processing capability
  • IMS AGW1 has AR processing capability
  • IMS AGW2 has AR processing capability Take capability as an example.
  • H248 add response2 also includes the address information of T2.
  • the IMS AGW2 sends H248 add requset3 to the IMS AGW2, which is used to instruct the IMS AGW2 to create the incoming port T2.
  • H248 add requset3 is also used to request the establishment of an auxiliary data channel with the first terminal device.
  • H248 add requset3 includes the first SDP of the first terminal device.
  • H248 add requset3 also includes address information corresponding to the T2 port of IMS AGW1.
  • the IMS AGW2 creates the incoming port T2, and sends H248 add resp2 to the Intermediate IMS2, where the H248 add resp2 includes a confirmation for creating an auxiliary data channel, such as including the second SDP of the IMS AGW2. It also includes address information (IP4o, P4o) corresponding to the IMS AGW2 T2 port.
  • IP4o, P4o address information
  • Intermediate IMS2 sends SIP 180 to Intermediate IMS1.
  • Intermediate IMS1 sends a configuration modification request to IMS AGW1, for example, the configuration modification request may be H248 Mod req3.
  • H248 Mod req3 is used to request IMS AGW1 to configure outgoing port T2.
  • IMS AGW1 configures outgoing termination T2 (outgoing termination T2), and sends a configuration modification response message to Intermediate IMS1, where the configuration modification response message may be H248Mod response3.
  • the Intermediate IMS1 receives the configuration modification response message from the IMS AGW1, and then initiates the creation of the incoming port T1 to the IMS AGW1.
  • the request message for creating the incoming port T1 carries the establishment request of the auxiliary data channel and the address information of the first terminal device.
  • the request message to create the incoming port T1 can be H248 add req4.
  • IMS AGW1 sends H248 add response4 to Intermediate IMS1.
  • H248 add response4 includes address information of IMS AGW1.
  • the address information of the IMS AGW1 includes the IP address (IP2a) and the port number (P2a) of the IMS AGW1.
  • the H248 add response4 may further include address information of the first terminal device.
  • the Intermediate IMS1 records the address information of the T1 port of the IMS AGW1, and the SIP 180 sent to the AS includes the second SDP information of the IMS AGW2 and the address information of the IMS AGW1.
  • the AS forwards SIP 180 to the Intermediate IMS1.
  • the Intermediate IMS1 forwards the SIP 180 to the first terminal device.
  • AS2 after AS2 receives SIP INVITE3, it can send its own address information to IMS AGW2 through Intermediate IMS2.
  • AS2 can add its own address information to SIP INVITE3 and send it to Intermediate IMS3. Therefore, the Intermediate IMS2 carries the address information of the AS3 in the H 248 add req3 sent to the IMS AGW2, so that the subsequent IMS AGW2 can obtain the required content from the third-party server through the AS when it needs to obtain the content provided by the third-party server.
  • the Intermediate IMS2 forwards the SIP INVITE3 to the AS2 after receiving the SIP INVITE3.
  • AS2 can send the stored address information of the third-party server to IMS AGW2 through Intermediate IMS2.
  • Intermediate IMS2 can add the address information of the third-party server to H248 add req3 and send it to Intermediate IMS2, so that the subsequent IMS AGW1 can obtain the required content from the third-party server when it needs to obtain the content provided by the third-party server.
  • an auxiliary data channel is established during the AR conference creation process.
  • the MRF to which the first terminal device belongs has the AR processing capability as an example.
  • FIG. 11 it is a schematic flowchart of the communication method of the fourth possible application scenario of the present application.
  • the Intermediate IMS1 identifies that the first terminal device initiates the AR conference, and sends a second request message to the MRF, where the second request message includes the first SDP of the first terminal device.
  • the second request message may use SIP INVITE2.
  • the Intermediate IMS1 sends a second response message, where the second response message includes the second SDP of the MRF.
  • the MRF determines that it does not have the AR processing capability, it ignores the request of the first terminal device to establish an auxiliary data channel.
  • the second response message may employ SIP 180.
  • the Intermediate IMS1 sends a first request response to the AS, and the first request response may use SIP 180.
  • the first request response includes the second SDP of the MRF.
  • the AS sends a first request response to the Intermediate IMS1.
  • the Intermediate IMS1 sends a first request response to the first terminal device.
  • step 1103 after the AS receives the SIP INVITE1, when the AS determines that the first terminal device subscribes to the AR service, it can send its own address information to the MRF through the Intermediate IMS1. AS can add its own address information to SIP INVITE1 and send it to Intermediate IMS1. Therefore, the Intermediate IMS1 carries the address information of the AS in the second request message (SIP INVITE2) sent to the MRF, so that when the subsequent MRF needs to obtain the content provided by the third-party server, it can obtain the required content from the third-party server through the AS.
  • SIP INVITE2 second request message
  • the Intermediate IMS1 forwards the SIP INVITE1 to the AS after receiving the SIP INVITE1.
  • the AS may send the stored address information of the third-party server to the MRF through the Intermediate IMS1.
  • the AS may add the address information of the third-party server to the first request message and send it to the Intermediate IMS1. Therefore, the Intermediate IMS1 carries the address information of the third-party server in the second request message (SIP INVITE2) sent to the MRF, so that the subsequent IMS AGW1 can obtain the required content from the third-party server when it needs to obtain the content provided by the third-party server. .
  • An auxiliary data channel is established between the first terminal device and the AR media processing network element.
  • a media stream channel is also established between the first terminal device and the second terminal device. The media stream channel processes network elements through AR media.
  • the first terminal device sends AR auxiliary data to the AR media processing network element through the auxiliary data channel.
  • the AR media processing network element is located in the IMS network of the first terminal device or in the IMS network of the second terminal device.
  • the AR media processing network element can be an IMS AGW or an MRF.
  • the first terminal device sends a media stream to an AR media processing network element through a media stream channel.
  • the media stream is generated by the first terminal device during AR communication with the second terminal device.
  • the AR media processing network element performs media enhancement processing on the media stream according to the AR auxiliary data to obtain the AR media stream.
  • the AR media processing network element sends the AR media stream to the first terminal device.
  • the first terminal device receives the AR media stream.
  • the AR media processing network element sends the AR media stream to the second terminal device through the media stream channel.
  • the second terminal device AR media stream.
  • both IMS AGW1 and IMS AGW2 have AR media capabilities, and the present application adopts the principle of nearby processing of AR assistance data, and the AR assistance data (non-media data) received by IMS AGW1 is data related to the first terminal device , such as the pose data of the user of the first terminal device, then the AR auxiliary data is only used for pose rendering of the media stream shared by the first terminal device, so the IMS AGW1 receives 1 (original media stream), a (non-media data such as user pose data of the first terminal device) and b (non-media data such as user annotation data collected by the first terminal device) are combined and rendered into 1 ⁇ a+b ⁇ (AR media stream) and returned to the first Terminal equipment; IMS AGW1 renders 1 ⁇ b ⁇ (AR media stream)
  • IMS AGW2 According to the received media stream and UE2's c (non-media data), IMS AGW2 finally renders it into media stream 1 ⁇ b+c ⁇ (AR media stream) and sends it to UE2.
  • media stream 1 ⁇ b+c ⁇ AR media stream
  • IMS AGW2 if IMS AGW2 has no AR media processing capability, then c (non-media data) of UE2 will be transmitted to IMS AGW1 for media enhancement processing.
  • the following describes an enhanced processing solution for implementing media streams through an auxiliary data channel with reference to specific application scenarios.
  • the AR media processing network element is taken as an example of the IMS AGW.
  • both the IMS AGW1 to which the first terminal device belongs and the IMS AGW2 to which the second terminal device belongs have AR processing capabilities.
  • the first terminal device and the IMS AGW1 to which the first terminal device belongs have completed the establishment of the auxiliary data channel 1.
  • the second terminal device and the IMS AGW2 to which the second terminal device belongs have completed the establishment of the auxiliary data channel 2.
  • a media stream channel is established between the first terminal device and the second terminal device.
  • the media stream channel can be considered to be divided into three sections, including the media stream channel between the first terminal device and the IMS AGW1, the media stream channel between the IMS AGW1 and the IMS AGW2, and the media stream channel between the IMS AGW2 and the second terminal device. flow channel.
  • the first terminal device has AR capability, for example, the first terminal device supports the user to mark on the shared screen.
  • the user of the first terminal device is referred to as user 1
  • the user of the second terminal device is referred to as user 2.
  • the first terminal device sends a media stream to the IMS AGW1 through a media stream channel.
  • the first terminal device obtains the pose data 1 generated by the user 1 marking on the display interface corresponding to the media stream, and sends the pose data 1 as AR auxiliary data to the IMS AGW1 through the auxiliary data channel 1.
  • IMS AGW1 renders the media stream according to the pose data 1 to obtain AR media stream 1.
  • IMS AGW1 sends the AR media stream 1 to IMS AGW2.
  • the IMS AGW1 sends the AR media stream 1 to the first terminal device.
  • the IMS AGW2 sends the AR media stream 1 to the second terminal device through the media stream channel.
  • both the user 1 of the second terminal device and the user 2 of the second terminal device can view the display interface marked by the user 1 .
  • the second terminal device also has the AR capability, for example, the second terminal device supports the user to mark on the shared screen.
  • the second terminal device acquires the pose data 2 generated by the annotation on the display interface corresponding to the media stream by the user 1, and sends the pose data 2 as AR auxiliary data to the IMS AGW2 through the auxiliary data channel 2.
  • the IMS AGW2 renders the media stream according to the pose data 2 to obtain the AR media stream 2.
  • IMS AGW2 sends the AR media stream 2 to IMS AGW1.
  • the IMS AGW2 sends the AR media stream 2 to the second terminal device.
  • the IMS AGW1 sends the AR media stream 2 to the first terminal device through the media stream channel 1. Therefore, both the user 1 of the second terminal device and the user 2 of the second terminal device can view the display interface marked by the user 2 .
  • the IMS AGW1 to which the first terminal device belongs does not have the AR processing capability
  • the IMS AGW2 to which the second terminal device belongs has the AR processing capability.
  • the IMS AGW2 to which the first terminal device and the second terminal device belong has completed the establishment of the auxiliary data channel 1.
  • the second terminal device and the IMS AGW2 to which the second terminal device belongs have completed the establishment of the auxiliary data channel 2.
  • a media stream channel is established between the first terminal device and the second terminal device.
  • the media stream channel can be considered to be divided into three sections, including the media stream channel between the first terminal device and the IMS AGW1, the media stream channel between the IMS AGW1 and the IMS AGW2, and the media stream channel between the IMS AGW2 and the second terminal device. flow channel.
  • the first terminal device has AR capability, for example, the first terminal device supports the user to mark on the shared screen.
  • the user of the first terminal device is referred to as user 1
  • the user of the second terminal device is referred to as user 2.
  • the first terminal device sends a media stream to the IMS AGW1 through a media stream channel.
  • the first terminal device acquires the pose data 1 generated by the user 1 marking on the display interface corresponding to the media stream, and sends the pose data 1 as AR auxiliary data to the IMS AGW2 through the auxiliary data channel 1.
  • IMS AGW2 renders the media stream according to the pose data 1 to obtain AR media stream 1.
  • the IMS AGW2 sends the AR media stream 1 to the IMS AGW1 through the media stream channel.
  • the IMS AGW1 sends the AR media stream 1 to the first terminal device through the media stream channel.
  • the IMS AGW2 sends the AR media stream 1 to the second terminal device.
  • both the user 1 of the second terminal device and the user 2 of the second terminal device can view the display interface marked by the user 1 .
  • the second terminal device also has the AR capability, for example, the second terminal device supports the user to mark on the shared screen.
  • the second terminal device acquires the pose data 2 generated by the annotation on the display interface corresponding to the media stream by the user 1, and sends the pose data 2 as AR auxiliary data to the IMS AGW2 through the auxiliary data channel 2.
  • the IMS AGW2 renders the media stream according to the pose data 2 to obtain the AR media stream 2.
  • IMS AGW2 sends the AR media stream 2 to IMS AGW1.
  • the IMS AGW2 sends the AR media stream 2 to the second terminal device.
  • the IMS AGW1 sends the AR media stream 2 to the first terminal device through the media stream channel 1. Therefore, both the user 1 of the second terminal device and the user 2 of the second terminal device can view the display interface marked by the user 2 .
  • FIG. 15 a schematic diagram of a possible target object acquisition process provided by an embodiment of the present application is shown.
  • the first terminal device sends the first AR assistance data to the AR media processing network element through the assistance data channel.
  • the first AR assistance data includes information of a target object to be used in the AR communication.
  • the AR media processing network element mentioned here is an IMS access gateway or MRF capable of AR processing. It can be understood that an auxiliary data channel is established between the AR media processing network element and the first terminal device.
  • the AR media processing network element may be located in the IMS network to which the first terminal device belongs, that is, the IMS AGW or MRF corresponding to the first terminal device.
  • the AR media processing network element may be located in the IMS network to which the second terminal device belongs, that is, the IMS AGW or MRF corresponding to the second terminal device.
  • the AR media processing network element when the AR media processing network element receives the first AR assistance data from the first terminal device through the assistance data channel, acquires the target object from a third-party server according to the first AR assistance data. It can be understood that the target object is used to enhance the media stream of the AR communication between the first terminal device and the second terminal device.
  • virtual objects such as 2D/3D objects
  • terminal devices generally do not have the ability to store a large number of 2D/3D objects.
  • 2D/3D objects On demand, get 2D/3D objects from third-party servers.
  • the AR media processing network element can render the foreground and background images.
  • the terminal device can send the demand information of the real background picture to the AR media processing network element through the auxiliary data channel, so that the AR media processing network element can obtain the display background picture from the third-party server according to the demand information, and then realize the display
  • the background picture and the foreground media stream sent by the terminal device are rendered and then sent to the terminal device.
  • the AR media processing network element acquires the address information of the AS in advance, and when acquiring the target object, the AR media processing network element acquires the target object from a third-party server through the AS.
  • the AR media processing network element acquires the address information of the AS in advance, and when acquiring the target object, the AR media processing network element acquires the target object from a third-party server through the AS.
  • the AR media processing network element obtains the address information of the third-party server in advance, and when the AR media processing network element obtains the target object, it can directly obtain the address information from the third-party server according to the address information of the third-party server. target.
  • an AR media processing network element obtains a virtual object from a third-party server through an AS.
  • the AR media processing network element is IMS AGW or MRF (IMS AGW/MRF).
  • the IMS AGW may belong to the first terminal device or to the second terminal device of the opposite end of the call.
  • the MRF may belong to the first terminal device or to the second terminal device of the call opposite end.
  • the first terminal device sends the information of the virtual object to the IMS AGW/MRF through the auxiliary data channel. For example, the identifier of the virtual object or the type of the virtual object.
  • request message 1 when the IMS AGW/MRF receives the information of the virtual object, it sends a request message 1 to the application server, and the request message 1 includes the information of the virtual object.
  • request message 1 may be of the HTTP message type.
  • the request message 1 may be an HTTP request (request, req) message 1 .
  • the request message 1 includes the identification of the terminal device.
  • the application server receives the request message 1, and when it is determined that the first terminal device needs to request a virtual object, forwards the request message 1 to the third-party server.
  • the application server may extract the identifier of the terminal device from the request message 1, and determine the identifier of the session established by the terminal device according to the identifier of the terminal device.
  • the application server may establish a communication connection, such as an HTTP connection, with the third-party server according to the session identifier, so as to forward the request message 1 to the third-party server, so as to obtain the virtual object from the third-party server.
  • the session identifier of the first terminal device may be added to the request message 1 forwarded to the third-party server.
  • the third-party server receives the request message 1, and obtains a plurality of virtual object identifiers that satisfy the virtual object type according to the type of the virtual object. Taking a virtual object list composed of multiple virtual object identifiers as an example, of course, the multiple virtual object identifiers may also be sent in other forms, which are not specifically limited in this application.
  • the third-party server sends a response message 1 to the application server.
  • Response message 1 may include a list of virtual objects.
  • Response message 1 may be of HTTP message type.
  • the response message 1 may be an HTTP response (request, response) message 1 .
  • the response message 1 includes a session identification.
  • the application server forwards the response message 1 to the IMS AGW/MRF.
  • the server should determine the identifier of the terminal device according to the session identifier in the response message 1, and add the identifier of the first terminal device to the response message 1 forwarded by the IMS AGW/MRF.
  • the IMS AGW/MRF obtains the virtual object list from the response message 1 from the application server, and sends the virtual object list to the first terminal device through the auxiliary data channel.
  • the IMS AGW/MRF saves the virtual object list.
  • the virtual object list may no longer be obtained from the third-party server. It can be understood that, in step 1602, when the IMS AGW/MRF determines that it has stored a virtual object list corresponding to the information of the virtual object, the subsequent step of acquiring the virtual object list may not be performed.
  • the first terminal device sends the first virtual object identifier to the IMS AGW/MRF through the auxiliary data channel.
  • the first virtual object identification is one of a list of virtual objects.
  • request message 2 when the IMS AGW/MRF receives the first virtual object identifier, it sends a request message 2 to the application server, where the request message 2 includes the first virtual object identifier.
  • request message 2 may be of the HTTP message type.
  • the request message 2 may be an HTTP request (request, req) message 2 .
  • the application server receives the request message 2, and forwards the request message 2 to the third-party server.
  • the third-party server receives the request message 2, and obtains the first virtual object corresponding to the first virtual object identifier.
  • the third-party server sends a response message 2 to the application server.
  • the response message 2 may include the first virtual object.
  • Response message 2 may be of HTTP message type.
  • the response message 2 may be an HTTP response (response, resp) message 2 .
  • the application server forwards the response message 2 to the IMS AGW/MRF.
  • the IMS AGW/MRF obtains the first virtual object from the response message 2, and sends the first virtual object to the first terminal device through the auxiliary data channel.
  • the AR media processing network element directly obtains the virtual object from the third-party server.
  • the AR media processing network element is IMS AGW or MRF (IMS AGW/MRF).
  • the IMS AGW may belong to the first terminal device or to the second terminal device of the opposite end of the call.
  • the MRF may belong to the first terminal device or to the second terminal device of the call opposite end.
  • the first terminal device sends the information of the virtual object to the IMS AGW/MRF through the auxiliary data channel. For example, the identifier of the virtual object or the type of the virtual object.
  • the IMS AGW/MRF when the IMS AGW/MRF receives the information of the virtual object, it sends a request message 11 to the third-party application server.
  • the request message 11 may include information of the virtual object.
  • the third-party server receives the request message 11, and obtains multiple virtual object identifiers that satisfy the virtual object type according to the virtual object type. Taking a virtual object list composed of multiple virtual object identifiers as an example, of course, the multiple virtual object identifiers may also be sent in other forms, which are not specifically limited in this application.
  • the third-party server sends a response message 11 to the IMS AGW/MRF.
  • the response message 11 may include a list of virtual objects.
  • the response message 11 may be of the HTTP message type.
  • the response message 11 may be an HTTP response (response, resp) message 11 .
  • the IMS AGW/MRF obtains the virtual object list from the response message 11, and sends the virtual object list to the first terminal device through the auxiliary data channel.
  • the IMS AGW/MRF saves the virtual object list.
  • the virtual object list may no longer be obtained from the third-party server. It can be understood that, in step 1702, when the IMS AGW/MRF determines that it has stored a virtual object list corresponding to the information of the virtual object, the subsequent step of acquiring the virtual object list may not be performed.
  • the first terminal device sends the first virtual object identifier to the IMS AGW/MRF through the auxiliary data channel.
  • the first virtual object identification is one of a list of virtual objects.
  • the IMS AGW/MRF when the IMS AGW/MRF receives the first virtual object identifier, it sends the request message 12 to the third-party server.
  • the third-party server receives the request message 12, and obtains the first virtual object corresponding to the first virtual object identifier.
  • the third party server sends a response message 12 to the IMS AGW/MRF.
  • the response message 12 may include the first virtual object.
  • the response message 12 may be of the HTTP message type.
  • the response message 12 may be an HTTP response (response, resp) message 2 .
  • the IMS AGW/MRF obtains the first virtual object from the response message 2, and sends the first virtual object to the first terminal device through the auxiliary data channel.
  • an AR media processing network element obtains a real background object from a third-party server through an AS.
  • the AR media processing network element is IMS AGW or MRF (IMS AGW/MRF).
  • the IMS AGW may belong to the first terminal device or to the second terminal device of the opposite end of the call.
  • the MRF may belong to the first terminal device or to the second terminal device of the call opposite end.
  • the first terminal device sends the information of the real background object to the IMS AGW/MRF through the auxiliary data channel. For example, the identification of the real background object or the geographic location of the real background object, etc.
  • the first terminal device sends the media stream 1 generated by the first terminal device to the IMS AGW/MRF through the media stream channel.
  • the IMS AGW/MRF when the IMS AGW/MRF receives the information of the real background object, it sends a request message 21 to the application server, and the request message 21 includes the information of the real background object.
  • the request message 21 may be of the HTTP message type.
  • the request message 21 may be an HTTP request (request, req) message 21 .
  • the application server receives the request message 21, and when it is determined that the first terminal device needs to request the real background object, the application server forwards the request message 21 to the third-party server.
  • the third-party server receives the request message 21, and obtains the real background object according to the geographic location of the real background object.
  • the third-party server sends a response message 21 to the application server.
  • the response message 21 may include a list of real background objects.
  • the response message 21 may be of the HTTP message type.
  • the response message 21 may be an HTTP response (response, resp) message 21 .
  • the application server forwards the response message 21 to the IMS AGW/MRF.
  • the IMS AGW/MRF obtains the real background object from the response message 21, and renders the display background object into the media stream 1 to obtain the AR media stream 2.
  • the IMS AGW/MRF sends the AR media stream 2 to the first terminal device through the media stream channel.
  • an AR media processing network element directly obtains a real background object from a third-party server.
  • the AR media processing network element is IMS AGW or MRF (IMS AGW/MRF).
  • the IMS AGW may belong to the first terminal device or to the second terminal device of the opposite end of the call.
  • the MRF may belong to the first terminal device or to the second terminal device of the call opposite end.
  • the first terminal device sends the information of the real background object to the IMS AGW/MRF through the auxiliary data channel. For example, the identification of the real background object or the geographic location of the real background object, etc.
  • the first terminal device sends the media stream 1 generated by the first terminal device to the IMS AGW/MRF through the media stream channel.
  • the IMS AGW/MRF when it receives the information of the real background object, it sends a request message 22 to the third-party server, and the request message 22 includes the information of the real background object.
  • the request message 22 may be of the HTTP message type.
  • the request message 22 may be an HTTP request (request, req) message 22 .
  • the third-party server receives the request message 22, and obtains the real background object according to the geographic location of the real background object.
  • the third party server sends a response message 22 to the IMS AGW/MRF.
  • the response message 22 may include a list of real background objects.
  • the response message 22 may be of the HTTP message type.
  • the response message 22 may be an HTTP response (request, response) message 22 .
  • the IMS AGW/MRF obtains the real background object from the response message 22, and renders the display background object into the media stream 1 to obtain the AR media stream 2.
  • the IMS AGW/MRF sends the AR media stream 2 to the first terminal device through the media stream channel.
  • the embodiment of the present application provides an apparatus, which is specifically used to implement the method executed by the IMS core network element in the above method embodiment.
  • the structure of the apparatus is shown in FIG. 20 , including a receiving unit 2001 , a sending unit 2002 and a processing unit 2003 .
  • the receiving unit 2001 is configured to receive a first request message from a first terminal device, where the first request message is used to request the establishment of an auxiliary data channel, and the auxiliary data channel is used for the communication between the first terminal device and the AR media processing network element.
  • the processing unit 2003 is configured to control the establishment of the auxiliary data channel between the AR media processing network element and the first terminal device.
  • the processing unit 2003 controls the establishment of the auxiliary data channel between the AR media processing network element and the first terminal device through the receiving unit 2001 and the sending unit 2002 .
  • the sending unit 2002 is configured to send a second request message to the AR media processing network element, where the second request message is used to request to establish the auxiliary data between the AR media processing network element and the first terminal device aisle;
  • the receiving unit 2001 is further configured to receive a second response message sent by the AR media processing network element, where the second response message is used to indicate to the IMS core network element that the AR media processing network element is paired to establish the assistance Confirmation of data channel;
  • the sending unit 2002 is further configured to send a first response message to the first terminal device, where the first response message is used to indicate to the first terminal device that the AR media processing network element successfully establishes the auxiliary data channel .
  • the AR media processing network element is an IMS access gateway with AR processing capability or a media resource function MRF.
  • the AR media processing network element is located in the IMS network to which the first terminal device belongs or in the IMS network to which the second terminal device belongs.
  • the first request message and the second request message carry a first description parameter of the first terminal device for establishing the auxiliary data channel
  • the first response message and the second response message carry the second description parameter of the AR media processing network element for establishing the auxiliary data channel.
  • the sending unit 2002 is further configured to send a query request to the AR media processing network element, where the query request is used to inquire whether the AR media processing network element has the AR processing capability;
  • the receiving unit 2001 is further configured to receive a query response sent by the AR media processing network element, where the query response carries information that the AR media processing network element has an AR processing capability.
  • the first request message is further used to request the establishment of a media stream channel, and the media stream channel is used for transmitting AR communication between the first terminal device and the AR media processing network element media streaming of content;
  • the processing unit 2003 is further configured to control the establishment of the media stream channel between the AR media processing network element and the first terminal device.
  • the processing unit 2003 controls the establishment of the media stream channel between the AR media processing network element and the first terminal device through the sending unit 2002 and the receiving unit 2001 .
  • the sending unit 2002 is further configured to send a second request message to the AR media processing network element, where the second request message is used to request to establish the media between the AR media processing network element and the first terminal device flow channel;
  • the receiving unit 2001 is further configured to receive a second response message sent by the AR media processing network element, where the second response message is used to instruct the IMS core network element to establish the media stream channel for the AR media processing network element confirmation;
  • the sending unit 2002 is further configured to send a first response message to the first terminal device, where the first response message is used to indicate to the first terminal device that the AR media processing network element successfully establishes the media stream channel.
  • the embodiment of the present application provides an apparatus, which is specifically used to implement the method performed by the AR media processing network element in the above method embodiment.
  • the structure of the apparatus is shown in FIG. 21 , including a receiving unit. 2101 , a sending unit 2102 and a processing unit 2103 .
  • the receiving unit 2101 is configured to receive a second request message sent by the IMS core network element of the Internet Protocol Multimedia Subsystem, where the second request message is used to request the establishment of an auxiliary data channel between the AR media processing network element and the first terminal device.
  • the auxiliary data channel is used for transmitting auxiliary data of AR communication between the first terminal device and the AR media processing network element;
  • the sending unit 2102 is configured to send a second response message to the IMS core network element, where the second response message is used to indicate to the IMS core network element that the AR media processing network element has a right to establish the auxiliary data channel. confirm.
  • the receiving unit 2101 is specifically configured to receive AR assistance data from the first terminal device through the auxiliary data channel after the auxiliary data channel is established.
  • a media stream channel for transmitting AR communication content is established between the AR media processing network element and the first terminal device.
  • the receiving unit 2101 is configured to receive the media stream of AR communication from the first terminal device through the media stream channel;
  • a processing unit 2103 configured to perform media enhancement processing on the media stream of the AR communication according to the AR auxiliary data to obtain an AR media stream;
  • the sending unit 2102 is configured to send the AR media stream after enhanced processing to the first terminal device through the media stream channel.
  • the sending unit 2102 is further configured to send the enhanced-processed AR media stream to the second terminal device.
  • the AR auxiliary data includes one or more of operation data of the virtual model, identification data of facial expressions or body movements of the target object, or pose data.
  • the AR media processing network element is an IMS access gateway with AR processing capability or a media resource function MRF.
  • the AR media processing network element is located in the IMS network to which the first terminal device belongs, or is located in the IMS network to which the second terminal device belongs.
  • the second request message carries the first description parameter of the first terminal device for establishing the auxiliary data channel
  • the processing unit 2103 is specifically configured to obtain the first description parameter from the first terminal device.
  • the second description parameter supported by the AR media processing network element is determined in a description parameter; the second response message carries the second description parameter.
  • the embodiment of the present application provides an apparatus, which is specifically used to implement the method executed by the terminal device in the above method embodiment.
  • the structure of the apparatus is shown in FIG. unit 2202 and processing unit 2203.
  • the sending unit 2202 sends a first request message to the IMS core network element, where the first request message is used to request the establishment of an auxiliary data channel; the receiving unit 2201 receives the first response message sent by the IMS core network element, the first response The message is used to instruct the AR media processing network element to complete the establishment of the auxiliary data channel, and the auxiliary data channel is used for transmitting auxiliary data of AR communication between the first terminal device and the AR media processing network element.
  • a media stream channel for transmitting AR communication content is established between the first terminal device and the AR media processing network element.
  • the sending unit 2202 sends a third request message to the IMS core network element, where the third request message is used to request the establishment of the secondary media stream channel; the receiving unit 2201 receives the message sent by the IMS core network element.
  • a third response message where the third response message is used to indicate to the first terminal device that the AR media processing network element successfully establishes the media stream channel.
  • the first request message is further used to request the establishment of the media stream channel; the first response message is further used to instruct the AR media processing network element to successfully establish the media stream channel.
  • the sending unit 2202 sends augmented reality AR auxiliary data to the AR media processing network element through an auxiliary data channel, and sends a media stream to the AR media processing network element through the media stream channel, where the media stream is generated when the first terminal device performs AR communication with the second terminal device; the AR auxiliary data is used by the AR media processing network element to perform media enhancement processing on the media stream of the AR communication; the receiving unit 2201 uses the The media stream channel receives the enhanced-processed AR media stream from the first IMS access gateway.
  • the receiving unit 2201 receives the AR media stream of the second terminal device sent by the AR media processing network element through the media stream channel.
  • the first request message carries a first description parameter of the first terminal device for establishing the auxiliary data channel
  • the first response message carries the second description parameter of the AR media processing network element for establishing the auxiliary data channel.
  • the AR media processing network element is located in the IMS network to which the first terminal device belongs or in the IMS network to which the second terminal device belongs.
  • the division of units in the embodiments of the present application is schematic, and is only a logical function division. In actual implementation, there may be other division methods.
  • the functional units in the various embodiments of the present application may be integrated into one processing unit. In the device, it can also exist physically alone, or two or more units can be integrated into one module.
  • the above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of software function modules.
  • the integrated unit if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer-readable storage medium.
  • the technical solutions of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, and the computer software products are stored in a storage medium , including several instructions to make a terminal device (which may be a personal computer, a mobile phone, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the method in each embodiment of the present application.
  • the aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program codes .
  • the AR media processing network element, the application server, the IMS core network element, and the terminal device may be presented in the form of dividing each functional module in an integrated manner.
  • Module herein may refer to a specific ASIC, circuit, processor and memory executing one or more software or firmware programs, integrated logic circuit, and/or other device that may provide the functions described above.
  • the AR media processing network element, the application server, the IMS core network element, and the terminal device may be implemented using the structure shown in FIG. 23 .
  • the apparatus 2300 shown in FIG. 23 includes at least one processor 2301 and a communication interface 2302 .
  • a memory 2303 may also be included.
  • the processor 2301 in FIG. 23 can execute instructions by calling the computer stored in the memory 2303, so that the AR media processing
  • the network element may perform the method performed by the AR media processing network element in any of the above method embodiments.
  • the processor 2301 can communicate with other devices through the communication interface 2302. For example, the processor 2301 receives the request message sent by the IMS core network element through the communication interface 2302, and sends a response message to the IMS core network element.
  • the memory 2303 stores computer-executed instructions for realizing the functions of the sending unit, the receiving unit, and the processing unit in FIG. 21 , and the functions/implementation processes of the sending unit, the receiving unit, and the processing unit in FIG.
  • the processor 2301 in 23 invokes the computer-executed instructions stored in the memory 2303, and implements in conjunction with the communication interface 2302.
  • the processor 2301 in FIG. 23 can execute instructions by calling the computer stored in the memory 2303, so that the IMS core network
  • the element may perform the method performed by the IMS core network element (IMS AGW1 or IMS AGW2 or MRF1 or MRF2) in any of the above method embodiments.
  • the processor 2301 can communicate with other devices through the communication interface 2302. For example, the processor 2301 receives the message sent by the R media processing network element and the message sent to the AR media processing network element through the communication interface 2302.
  • the memory 2303 stores computer-executed instructions for realizing the functions of the sending unit, the receiving unit, and the processing unit in FIG. 20 , and the functions/implementation processes of the sending unit, the receiving unit, and the processing unit in FIG.
  • the processor 2301 in 23 invokes the computer-executed instructions stored in the memory 2303, and implements in conjunction with the communication interface 2302.
  • the processor 2301 in FIG. 23 can execute the instructions by calling the computer stored in the memory 2303, so that the terminal device can execute any of the above.
  • the processor 2301 can communicate with other devices through the communication interface 2302. For example, the processor 2301 receives the enhanced media stream sent by the AR media processing network element through the communication interface 2302, and sends AR auxiliary data and media stream to the AR media processing network element.
  • the memory 2303 stores computer-executed instructions for realizing the functions of the sending unit, the receiving unit, and the processing unit in FIG. 22 , and the functions/implementation processes of the sending unit, the receiving unit, and the processing unit in FIG.
  • the processor 2301 in 23 invokes the computer-executed instructions stored in the memory 2303, and implements in conjunction with the communication interface 2302.
  • the specific connection medium between the above-mentioned processor 2301 and the memory 2303 is not limited in this embodiment of the present application.
  • the memory 2303 and the processor 2301 are connected through a bus 2304 in the figure, and the bus 2304 is represented by a thick line in the figure. limited.
  • the bus 2304 can be divided into an address bus, a data bus, a control bus, and the like. For ease of presentation, only one thick line is shown in FIG. 23, but it does not mean that there is only one bus or one type of bus.
  • the embodiments of the present application further provide a computer storage medium, where a software program is stored in the storage medium, and when the software program is read and executed by one or more processors, it can implement any one or more of the above Embodiments provide methods performed by a business platform or edge computing device.
  • the computer storage medium may include: a U disk, a removable hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk and other mediums that can store program codes.
  • an embodiment of the present application further provides a chip, where the chip includes a processor for implementing the functions of the AR media server or the terminal device or the functions of the application server involved in any one or more of the above embodiments .
  • the chip further includes a memory, and the memory is used for necessary program instructions and data to be executed by the processor.
  • the chip may consist of chips, or may include chips and other discrete devices.
  • references throughout the specification to "one embodiment,” “one implementation,” “one implementation,” or “an example” mean that a particular feature, structure, or characteristic associated with the embodiment is included within the scope of this application. in at least one embodiment. Thus, appearances of "in one embodiment,” “one implementation,” “one implementation,” or “in an example” in various places throughout this specification are not necessarily necessarily referring to the same embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that, in various embodiments of the present application, the size of the sequence numbers of the above-mentioned processes does not mean the sequence of execution, and the execution sequence of each process should be determined by its functions and internal logic, and should not be dealt with in the embodiments of the present application. implementation constitutes any limitation.
  • system and “network” are often used interchangeably herein.
  • the term “and/or” in this article is only an association relationship to describe the associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, it can mean that A exists alone, A and B exist at the same time, and A and B exist independently B these three cases.
  • the character "/" in this document generally indicates that the related objects are an "or” relationship.
  • the term “at least one” referred to in this application refers to one, or more than one, including one, two, three and more; “multiple” refers to two, or more than two, including two, three or more.
  • At least one item(s) below or similar expressions thereof refer to any combination of these items, including any combination of single item(s) or plural items(s).
  • at least one item (a) of a, b, or c can represent: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, c may be single or multiple .
  • B corresponding to A means that B is associated with A, and B can be determined according to A.
  • determining B according to A does not mean that B is only determined according to A, and B may also be determined according to A and/or other information.
  • the terms “comprising” and “having” in the embodiments and claims of the present application and the drawings are not exclusive. For example, a process, method, system, product or device that includes a series of steps or modules is not limited to the listed steps or modules, and may also include unlisted steps or modules.
  • processor in the embodiments of the present application may be a central processing unit (central processing unit, CPU), and may also be other general-purpose processors, digital signal processors (digital signal processors, DSP), application-specific integrated circuits (application specific integrated circuit, ASIC), field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.
  • CPU central processing unit
  • DSP digital signal processors
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • a general-purpose processor may be a microprocessor or any conventional processor.
  • the method steps in the embodiments of the present application may be implemented in a hardware manner, or may be implemented in a manner in which a processor executes software instructions.
  • Software instructions can be composed of corresponding software modules, and software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (programmable ROM) , PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically erasable programmable read-only memory (electrically EPROM, EEPROM), registers, hard disks, removable hard disks, CD-ROMs or known in the art in any other form of storage medium.
  • An exemplary storage medium is coupled to the processor, such that the processor can read information from, and write information to, the storage medium.
  • the storage medium can also be an integral part of the processor.
  • the processor and storage medium may reside in an ASIC.
  • the ASIC may be located in a network device or in an end device.
  • the processor and the storage medium may also exist in the network device or the terminal device as discrete components.
  • the computer program product includes one or more computer programs or instructions.
  • the computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable device.
  • the computer program or instructions may be stored in or transmitted over a computer-readable storage medium.
  • the computer-readable storage medium can be any available media that can be accessed by a computer or a data storage device such as a server that integrates one or more available media.
  • the usable media can be magnetic media, such as floppy disks, hard disks, magnetic tapes; optical media, such as DVD; and semiconductor media, such as solid state disks (SSD).

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Computer Security & Cryptography (AREA)
  • Business, Economics & Management (AREA)
  • General Business, Economics & Management (AREA)
  • Computer And Data Communications (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Abstract

一种增强现实通信的方法、装置及系统，用以提供一种AR融入语音视频通话的实现方式，以提升用户体验。在终端设备和AR媒体处理网元之间建立辅助数据通道和媒体流通道，从而有效的将AR融入到语音视频通话中。此外，AR辅助数据和媒体流所走的传输路径相同，可以减少时延。AR媒体处理网元可以为具备AR处理能力的IMS接入网关或者媒体资源功能MRF。

Description

一种增强现实通信的方法、装置及系统
相关申请的交叉引用
本申请要求在2021年03月27日提交中国专利局、申请号为202110329252.9、申请名称为“一种增强现实通信的方法、装置及系统”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及通信技术领域，尤其涉及一种增强现实通信的方法、装置及系统。
背景技术
长期演进语音承载(voice over long term evolution，VoLTE)是架构在第四代(the 4th generation，4G)网络上全IP条件下的端到端语音方案。VoLTE使得用户之间通信时，接通等待时间更短，并且语音视频通话质量更高。增强现实(Augmented Reality，AR)是一种将虚拟信息与真实世界巧妙融合的技术，广泛运用了多媒体、三维建模、实时跟踪及注册、智能交互、传感等多种技术手段，将计算机生成的文字、图像、三维模型、音乐、视频等虚拟信息模拟仿真后，应用到真实世界中，两种信息互为补充，从而实现对真实世界的“增强”。增强现实技术不仅能够有效体现出真实世界的内容，也能够促使虚拟的信息内容显示出来。如何将AR融入语音视频通话中，目前并没有一种有效的实现方式。
发明内容
本申请实施例提供一种增强现实通信的方法、装置及系统，提供一种AR融入语音视频通话的实现方式，进而提升用户体验。
第一方面，本申请实施例提供一种增强现实AR通信的系统，包括互联网协议多媒体子系统IMS核心网元和第一AR媒体处理网元；所述IMS核心网元，用于控制所述第一AR媒体处理网元与第一终端设备之间建立第一媒体流通道和辅助数据通道；所述第一AR媒体处理网元，用于通过所建立的所述辅助数据通道接收来自第一终端设备的第一AR辅助数据，以及通过所建立的所述媒体流通道接收所述第一终端设备和第二终端设备进行AR通信的第一媒体流；所述第一AR媒体处理网元，还用于根据所述第一AR辅助数据对所述媒体流进行增强处理得到第一AR媒体流，并通过所述媒体流通道向所述第一终端设备发送所述第一AR媒体流。通过上述方案，在终端设备和AR媒体处理网元之间建立辅助数据通道和媒体流通道，从而有效的将AR融入到语音视频通话中。此外，AR辅助数据和媒体流所走的传输路径相同，可以减少时延。
在一种可能的设计中，所述第一AR媒体处理网元可以为具备AR处理能力的IMS接入网关或者媒体资源功能MRF。比如，在终端设备之间进行点对点通信时，第一AR媒体处理网元可以是IMS接入网关。再比如，终端设备之前进行AR会议时，第一AR媒体处理网元可以是MRF。
在一种可能的设计中，所述第一AR媒体处理网元位于第一终端设备所属的(或所在 的)IMS网络中。第一终端设备所在的IMS网络中的AR媒体处理网元具备AR处理能力，采用就近原则通过第一终端设备所在的IMS网络中的AR媒体处理网元来执行增强处理，可以进一步降低时延。
在一种可能的设计中，所述系统还包括位于第二终端设备所属的(或所在的)IMS网络中的第二AR媒体处理网元；所述第一AR媒体处理网元，还用于向所述第二AR媒体处理网元发送所述AR媒体流；所述第二AR媒体处理网元，用于接收所述第一AR媒体处理网元发送的所述AR媒体流，并所述第二终端设备发送所述AR媒体流。
在一种可能的设计中，所述第一AR媒体处理网元位于第二终端设备所属的(或所在的)IMS网络中，所述系统还包括位于第一终端设备所属的(或所在的)IMS网络中的第三AR媒体处理网元，第三AR媒体处理网元不具备AR处理能力，所述第一AR媒体处理网元与第一终端设备之间的媒体流通道包括所述第一AR媒体处理网元与第三AR媒体处理网元之间的第一媒体流通道和所述第三AR媒体处理网元与所述第一终端设备之间的第二媒体流通道；
所述第三AR媒体处理网元，用于通过所述第二媒体流通道接收来自第一终端设备的所述媒体流，通过所述第一媒体流通道向所述第一AR媒体处理网元转发所述媒体流；通过所述第一媒体流通道接收来自第一AR媒体处理网元的AR媒体流，以及通过所述第二媒体流通道向所述第一终端设备发送所述AR媒体流。
上述设计中，第一终端设备所在的IMS网络中的AR媒体处理网元不具备AR处理能力时，借助于第二终端设备所在的IMS网络中的AR媒体处理网元来为第一终端设备提供增强处理功能。为AR融入音视频通话提供一种可行的方案。
在一种可能的设计中，所述第一AR媒体处理网元，还用于通过所述辅助数据通道接收来自第一终端设备的目标对象信息；所述AR媒体处理网元，还用于从第三方服务器获取所述目标对象信息对应的目标对象。
上述设计中，第三方服务器可以为终端设备的AR通信提供目标对象的素材，本申请中借助辅助数据通道来从第三方服务器获取目标对象。
在一种可能的设计中，所述目标对象为虚拟对象或者虚拟对象的标识；所述第一AR媒体处理网元，还用于通过所述辅助数据通道向所述第一终端设备发送所述虚拟对象或者所述虚拟对象的标识。由于多种场景需要用到虚拟对象，通过上述设计，借助辅助数据通道从第三方服务器来获取虚拟对象。
一种可能的设计中，目标对象信息包括所述第一终端设备所请求的虚拟对象类型，所述第一AR媒体处理网元通过辅助数据通道将所述虚拟对象类型的多个虚拟对象标识发送给第一终端设备。
在一种可能的设计中，所述第一AR媒体处理网元接收虚拟对象对应的多个虚拟对象标识后，保存所述虚拟对象类型与所述多个虚拟对象标识的对应关系。
在一种可能的设计中，所述第一AR媒体处理网元在通过辅助数据通道接收到虚拟对象类型时，可以先确定自身是否保存有该虚拟对象类型对应的虚拟对象标识，如果保存可以不再向第三方服务器获取该虚拟对象类型对应的虚拟对象标识。如果未保存，再向第三方服务器获取该虚拟对象类型的对应的虚拟对象标识，通过该方式可以减少获取带来的不必要的资源浪费。
在一种可能的设计中，所述方法还包括：
所述第一AR媒体处理网元通过辅助数据通道接收来自第一终端设备的第一虚拟对象标识，所述第一虚拟对象标识用于标识所述第一终端设备的用户在所述多个虚拟对象中选择的第一虚拟对象；第一AR媒体处理网元从第三方服务器获取所述第一虚拟对象标识对应的第一虚拟对象，并通过所述辅助数据通道向第一终端设备发送该第一虚拟对象。
在一种可能的设计中，所述目标对象为现实背景对象，所述第一AR媒体处理网元，还用于通过所述媒体数据通道接收第一终端设备发送的所述AR通信产生的第二媒体流；
所述第一AR媒体处理网元，还用于将所述现实背景对象与第二媒体流渲染合成得到第二AR媒体流；通过所述媒体流通道向所述第一终端设备发送所述第二AR媒体流。在一些应用场景中，前景对象对应的背景画面是相对固定的，在该场景下，媒体流与背景画面之间的渲染操作由AR媒体处理网元来执行，可以降低终端设备的上行带宽的压力。
在一种可能的设计中，所述系统还包括应用服务器；
所述应用服务器，用于接收来自AR媒体处理网元的第一请求消息，并向所述第三方服务器转发所述第一请求消息；接收来自所述第三方服务器的第一响应消息，并向所述第一AR媒体处理网元转发所述第二响应消息。
在一种可能的设计中，所述应用服务器配置有所述第三方服务器的地址；所述应用服务器，具体用于根据所述第三方服务器的地址向所述第三方服务器转发所述第一请求消息，以及向所述第一AR媒体处理网元转发所述第二响应消息。
在一种可能的设计中，所述应用服务器，还用于在建立所述第一终端设备与所述第一AR媒体处理网元之间的辅助数据通道的流程中，向所述第一AR媒体处理网元发送所述第三方服务器的地址。
第二方面，本申请实施例提供一种增强现实AR通信的方法，包括：互联网协议多媒体子系统IMS核心网元接收来自第一终端设备的第一请求消息，所述第一请求消息用于请求建立辅助数据通道，所述辅助数据通道用于第一终端设备与AR媒体处理网元之间传输AR通信的辅助数据；所述IMS核心网元控制所述AR媒体处理网元与所述第一终端设备之间建立所述辅助数据通道。上述方案，提供一种简单有效的创建辅助数据通道的方式，从而通过辅助数据通道来获取AR通信的AR辅助数据，进而实现AR通信过程中的AR通信的媒体流进行增强处理。
在一种可能的设计中，所述IMS核心网元控制所述AR媒体处理网元与所述第一终端设备之间建立所述辅助数据通道，包括：所述IMS核心网元向所述AR媒体处理网元发送第二请求消息，所述第二请求消息用于请求AR媒体处理网元与所述第一终端设备之间建立所述辅助数据通道；所述IMS核心网元接收所述AR媒体处理网元发送的第二响应消息，所述第二响应消息用于向IMS核心网元指示所述AR媒体处理网元对建立所述辅助数据通道的确认；所述IMS核心网元向所述第一终端设备发送第一响应消息，所述第一响应消息用于向第一终端设备指示所述AR媒体处理网元成功建立所述辅助数据通道。
在一种可能的设计中，所述AR媒体处理网元为具备AR处理能力的IMS接入网关或者媒体资源功能MRF。比如，在终端设备之间进行点对点通信时，第一AR媒体处理网元可以是IMS接入网关。再比如，终端设备之前进行AR会议时，第一AR媒体处理网元可以是MRF。
在一种可能的设计中，IMS核心网元控制AR媒体处理网元与第一终端设备之间建立媒体流通道。通过上述方案，在IMS核心网元和AR媒体处理网元之间建立辅助数据通道 和媒体流，从而有效的将AR融入到语音视频通话中。此外，AR辅助数据和媒体流所走的传输路径相同，可以减少时延。
在一种可能的设计中，所述AR媒体处理网元位于所述第一终端设备所属的IMS网络中或者位于所述第二终端设备所属的IMS网络中。第一终端设备所在的IMS网络中的AR媒体处理网元具备AR处理能力，采用就近原则通过第一终端设备所在的IMS网络中的AR媒体处理网元来执行增强处理，可以进一步降低时延。第一终端设备所在的IMS网络中的AR媒体处理网元不具备AR处理能力时，借助于第二终端设备所在的IMS网络中的AR媒体处理网元来为第一终端设备提供增强处理功能。为AR融入音视频通话提供一种可行的方案。
在一种可能的设计中，所述第一请求消息和所述第二请求消息携带所述第一终端设备的用于建立所述辅助数据通道的第一描述参数；
所述第一响应消息和所述第二响应消息携带所述AR媒体处理网元的用于建立所述辅助数据通道的第二描述参数。
在一种可能的设计中，所述方法还包括：IMS核心网元向AR媒体处理网元发送查询请求，所述查询请求用于询问所述AR媒体处理网元是否具备AR处理能力；IMS核心网元接收所述AR媒体处理网元发送的查询响应，所述查询响应携带所述AR媒体处理网元具备AR处理能力的信息。上述设计中，提供一种可行且有效的确定AR媒体处理网元是否AR处理能力的方式。
在一种可能的设计中，所述AR媒体处理网元位于所述第二终端设备所属的IMS网络中，所述方法还包括：
IMS核心网元向位于所述第二终端设备所属的IMS网络中的AR媒体处理网元发送查询消息之前，确定为所述第一终端设备提供服务的AR媒体处理网元不具备AR处理能力。
在一种可能的设计中，第一终端设备与第二终端设备建立AR通信的流程为所述第一终端设备与所述第二终端设备建立AR会话的流程或者所述第一终端设备与所述第二终端设备更新AR会话的流程或者所述第一终端设备与所述第二终端设备重建立AR会话的流程。
在一种可能的设计中，所述第一请求消息为初始会话协议SIP请求INVITE消息，或者为SIP重建立请求消息，或者为SIP更新请求消息。
在一种可能的设计中，所述方法还包括：所述IMS核心网元控制所述AR媒体处理网元与所述第一终端设备之间建立所述媒体流通道，所述媒体流通道用于所述第一终端设备与所述AR媒体处理网元之间传输AR通信内容的媒体流。
在一种可能的设计中，接收来自第一终端设备的第三请求消息，所述第三请求消息还用于请求建立媒体流通道，所述IMS核心网元控制所述AR媒体处理网元与所述第一终端设备之间建立所述媒体流通道，包括：所述IMS核心网元向所述AR媒体处理网元发送第四请求消息，所述第四请求消息用于请求AR媒体处理网元与所述第一终端设备之间建立所述媒体流通道；所述IMS核心网元接收所述AR媒体处理网元发送的第四响应消息，所述第四响应消息用于向IMS核心网元指示所述AR媒体处理网元对建立所述媒体流通道的确认；所述IMS核心网元向所述第一终端设备发送第三响应消息，所述第三响应消息用于向第一终端设备指示所述AR媒体处理网元成功建立所述媒体流通道。
需要说明的是，辅助数据通道可以媒体流通道可以在同一个流程中建立，或者可以先 建立媒体流通道，再建立辅助数据通道。
示例性地，第一请求消息和第三请求消息可以承载于同一消息，第二请求消息和第四请求消息可以承载于同一消息，第一响应消息和第三响应消息可以承载与同一消息，第二响应消息和第四响应消息可以承载于同一消息。
第三方面，本申请实施例提供一种基于增强现实AR的通信方法，包括：
AR媒体处理网元在第一终端设备的触发下，与第一终端设备之间建立辅助数据通道。
在一种可能的设计中，AR媒体处理网元与第一终端设备之间建立传输AR通信内容的媒体流通道。
在一种可能的设计中，AR媒体处理网元接收互联网协议多媒体子系统IMS核心网元发送的第二请求消息，所述第二请求消息用于请求AR媒体处理网元与第一终端设备之间建立辅助数据通道，所述辅助数据通道用于第一终端设备与所述AR媒体处理网元之间传输AR通信的辅助数据；所述AR媒体处理网元向所述IMS核心网元发送第二响应消息，所述第二响应消息用于向所述IMS核心网元指示所述AR媒体处理网元对建立所述辅助数据通道的确认。
在一种可能的设计中，所述方法还包括：在所述辅助数据通道建立完成后，所述AR媒体处理网元通过所述辅助数据通道接收来自所述第一终端设备的AR辅助数据。
在一种可能的设计中，所述方法还包括：所述AR媒体处理网元通过所述媒体流通道接收来自所述第一终端设备的AR通信的媒体流；所述AR媒体处理网元根据所述AR辅助数据对所述AR通信的媒体流进行媒体增强处理得到AR媒体流；所述AR媒体处理网元通过所述媒体流通道将增强处理后的AR媒体流发送给所述第一终端设备。通过上述方案，在终端设备和AR媒体处理网元之间建立辅助数据通道和媒体流，从而有效的将AR融入到语音视频通话中。此外，AR辅助数据和媒体流所走的传输路径相同，可以减少时延。
在一种可能的设计中，所述方法还包括：所述AR媒体处理网元向所述第二终端设备发送所述增强处理后的AR媒体流。
[根据细则91更正 30.03.2022]　
在一种可能的设计中，所述AR辅助数据包括虚拟模型的操作数据、目标对象的标注面部表情或者身体动作的识别数据或者位姿数据中的一项或多项。
在一种可能的设计中所述AR媒体处理网元为具备AR处理能力的IMS接入网关或者媒体资源功能MRF。
在一种可能的设计中，所述AR媒体处理网元位于所述第一终端设备所属的(或所在的)IMS网络中，或者位于所述第二终端设备所属的(或所在的)IMS网络中。
在一种可能的设计中，所述第二请求消息携带所述第一终端设备的用于建立所述辅助数据通道的第一描述参数，所述方法还包括：所述AR媒体处理网元从所述第一描述参数中确定所述AR媒体处理网元所支持的第二描述参数；所述第二响应消息携带所述第二描述参数。
第四方面，本申请实施例提供一种基于增强现实AR通信的方法，包括：所述第一终端设备向IMS核心网元发送第一请求消息，所述第一请求消息用于请求建立辅助数据通道；所述第一终端设备接收所述IMS核心网元发送的第一响应消息，所述第一响应消息用于指示AR媒体处理网元完成建立所述辅助数据通道，所述辅助数据通道用于第一终端设备与所述AR媒体处理网元之间传输AR通信的辅助数据。
在一种可能的设计中，所述方法还包括：所述第一终端设备与所述AR媒体处理网元之间建立传输AR通信的内容的媒体流通道。
在一种可能的设计中，所述第一终端设备与所述AR媒体处理网元之间建立传输AR通信的内容的媒体流通道，包括：
所述第一终端设备向IMS核心网元发送第三请求消息，所述第三请求消息用于请求建立辅所述媒体流通道；
所述第一终端设备接收IMS核心网元发送的第三响应消息，所述第三响应消息用于向第一终端设备指示所述AR媒体处理网元成功建立所述媒体流通道。
在一种可能的设计中，所述第一请求消息还用于请求建立所述媒体流通道；所述第一响应消息还用于指示AR媒体处理网元成功建立所述媒体流通道。
在一种可能的设计中，所述方法还包括：
所述第一终端设备通过辅助数据通道向AR媒体处理网元发送增强现实AR辅助数据，以及通过所述媒体流通道向AR媒体处理网元发送媒体流，所述媒体流是第一终端设备在与第二终端设备进行AR通信时产生的；所述AR辅助数据用于所述AR媒体处理网元对所述AR通信的媒体流进行媒体增强处理；
所述第一终端设备通过所述媒体流通道接收来自所述第一IMS接入网关的增强处理后的AR媒体流。
在一种可能的设计中，所述方法还包括：
所述第一终端设备通过所述媒体流通道接收所述AR媒体处理网元发送的所述第二终端设备的AR媒体流。
在一种可能的设计中，所述第一请求消息携带所述第一终端设备的用于建立所述辅助数据通道的第一描述参数；
所述第一响应消息携带所述AR媒体处理网元的用于建立所述辅助数据通道的第二描述参数。
在一种可能的设计中，所述AR媒体处理网元位于所述第一终端设备所属的(或所在的)IMS网络中或者位于所述第二终端设备所属的(或所在的)IMS网络中。
第五方面，本申请实施例还提供了一种装置，所述装置应用于IMS核心网元，有益效果可以参见第二方面的描述此处不再赘述。该装置具有实现上述第二方面的方法实例中行为的功能。所述功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。所述硬件或软件包括一个或多个与上述功能相对应的模块。在一个可能的设计中，所述装置的结构中可以包括发送单元、接收单元，这些单元可以执行上述第二方面方法示例中的相应功能，具体参见方法示例中的详细描述，此处不做赘述。
第六方面，本申请实施例还提供了一种装置，所述装置应用于AR媒体处理网元，有益效果可以参见第三方面的描述此处不再赘述。该装置具有实现上述第三方面的方法实例中行为的功能。所述功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。所述硬件或软件包括一个或多个与上述功能相对应的模块。在一个可能的设计中，所述装置的结构中可以包括发送单元、接收单元和处理单元，这些单元可以执行上述第三方面方法示例中的相应功能，具体参见方法示例中的详细描述，此处不做赘述。
第七方面，本申请实施例还提供了一种装置，所述装置应用于所述终端设备，有益效果可以参见第四方面的描述此处不再赘述。该装置具有实现上述第四方面的方法实例中行 为的功能。所述功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。所述硬件或软件包括一个或多个与上述功能相对应的模块。在一个可能的设计中，所述装置的结构中包括发送单元、接收单元，这些单元可以执行上述第四方面方法示例中的相应功能，具体参见方法示例中的详细描述，此处不做赘述。
第八方面，本申请实施例还提供了一种装置，所述装置应用于所述IMS核心网元，有益效果可以参见第二方面的描述此处不再赘述。所述通信装置的结构中包括处理器和通信接口，还可以包括存储器，所述处理器被配置为支持IMS核心网元通过通信接口执行上述第二方面方法中相应的功能。所述存储器与所述处理器耦合，其保存所述通信装置必要的程序指令和数据。
第九方面，本申请实施例还提供了一种装置，所述装置应用于所述AR媒体处理网元，有益效果可以参见第三方面的描述此处不再赘述。所述装置的结构中包括处理器和通信接口，还可以包括存储器，所述处理器被配置为支持所述AR媒体处理网元通过通信接口执行上述第三方面方法中相应的功能。所述存储器与所述处理器耦合，其保存所述装置必要的程序指令和数据。
第十方面，本申请实施例还提供了一种装置，所述装置应用于所述终端设备，有益效果可以参见第四方面的描述此处不再赘述。所述装置的结构中包括处理器和通信接口，还可以包括存储器，所述处理器被配置为支持所述终端设备通过通信接口执行上述第四方面方法中相应的功能。所述存储器与所述处理器耦合，其保存所述装置必要的程序指令和数据。
第十一方面，本申请实施例还提供了一种装置，所述装置应用于应用服务器。所述通信装置的结构中包括处理器和通信接口，还可以包括存储器，所述处理器被配置为支持所述应用服务器通过通信接口执行上述任一方面中应用服务器相应的功能。所述存储器与所述处理器耦合，其保存所述通信装置必要的程序指令和数据。
第十二方面，本申请还提供一种计算机可读存储介质，所述计算机可读存储介质中存储有指令，当其在计算机上运行时，使得计算机执行上述各方面所述的方法。
第十三方面，本申请还提供一种包含指令的计算机程序产品，当其在计算机上运行时，使得计算机执行上述各方面所述的方法。
第十四方面，本申请还提供一种计算机芯片，所述芯片与存储器相连，所述芯片用于读取并执行所述存储器中存储的软件程序，执行上述各方面所述的方法。
附图说明
图1为本申请实施例中一种可能的AR通信系统架构示意图；
图2A为本申请实施例中另一种可能的AR通信系统架构示意图；
图2B为本申请实施例中又一种可能的AR通信系统架构示意图；
图2C为本申请实施例中又一种可能的AR通信系统架构示意图；
图3为本申请实施例中又一种可能的AR通信系统架构示意图；
图4为本申请实施例中又一种可能的AR通信系统架构示意图；
图5为本申请实施例中又一种可能的AR通信系统架构示意图；
图6为本申请实施例中一种AR通信的方法流程示意图；
图7A为本申请实施例中另一种AR通信的方法流程示意图；
图7B为本申请实施例中又一种AR通信的方法流程示意图；
图8A为本申请实施例中又一种AR通信的方法流程示意图；
图8B为本申请实施例中IMS AGW上的T1端口和T2端口示意图；
图9为本申请实施例中又一种AR通信的方法流程示意图；
图10为本申请实施例中又一种AR通信的方法流程示意图；
图11为本申请实施例中又一种AR通信的方法流程示意图；
图12A为本申请实施例中又一种AR通信的方法流程示意图；
图12B为本申请实施例中又一种AR通信的方法流程示意图；
图12C为本申请实施例中又一种AR通信的方法流程示意图；
图13为本申请实施例中又一种AR通信的方法流程示意图；
图14为本申请实施例中又一种AR通信的方法流程示意图；
图15为本申请实施例中又一种AR通信的方法流程示意图；
图16为本申请实施例中又一种AR通信的方法流程示意图；
图17为本申请实施例中又一种AR通信的方法流程示意图；
图18为本申请实施例中又一种AR通信的方法流程示意图；
图19为本申请实施例中又一种AR通信的方法流程示意图；
图20为本申请实施例中一种装置的结构示意图；
图21为本申请实施例中另一种装置的结构示意图；
图22为本申请实施例中又一种装置的结构示意图；
图23为本申请实施例中又一种装置的结构示意图。
具体实施方式
本申请提供一种基于AR的通信方法及装置，提供一种AR融入语音视频通话的实现方式，进而提升用户体验。语音视频通话可以但不仅限于采用VoLTE，还可以适用于未来技术提供的语音视频通话。AR具备三个特征，分别是虚实融合、实时交互和三维注册。虚实融合是将计算机生成的虚拟物体和信息叠加到真实世界的场景，以实现对现实场景更直观深入了解。增强的信息可是与真实物体相关的非几何信息，如视频、文字，也可以是几何信息，如虚拟的三维物体和场景。实时交互是通过增强现实系统中的交互接口设备，人们以自然方式与增强现实环境进行实时交互操作。三维注册中“注册”可解释为跟踪和定位，三维注册是指将计算机产生的虚拟物体与真实环境进行一一对应，且用户在真实环境中运动时，也将继续维持正确的对准关系。
本申请中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。本申请涉及的术语“至少一个”，是指一个，或一个以上，即包括一个、两个、三个及以上；“多个”，是指两个，或两个以上，即包括两个、三个及以上。另外，需要理解的是，在本申请的描述中，“第一”、“第二”等词汇，仅用于区分描述的目的，而不能理解为指示或暗示相对重要性，也不能理解为指示或暗示顺序。“以下至少一项(个)”或其类似表达，是指的这些项中的任意组合，包括单项(个)或复数项(个)的任意组合。例如，a,b,或c中的至少一项(个)，可以表示：a,b,c,a-b,a-c,b-c,或a-b-c，其中a,b,c可以是单个，也可以是多个。应理解，在本申 请实施例中，“与A相应的B”表示B与A相关联，根据A可以确定B。但还应理解，根据A确定B并不意味着仅仅根据A确定B，还可以根据A和/或其它信息确定B。此外，本申请实施例和权利要求书及附图中的术语“包括”和“具有”不是排他的。例如，包括了一系列步骤或模块的过程、方法、系统、产品或设备没有限定于已列出的步骤或模块，还可以包括没有列出的步骤或模块。
下面结合附图对本申请实施例提供的方案进行详细说明。
参见图1所示为本申请实施例的一种AR通信系统架构示意图。AR通信系统包括AR媒体处理网元和AR控制网元。AR媒体处理网元为具体AR能力的终端设备提供媒体处理服务。AR媒体处理网元具备AR用户面(AR User plane，AR-U)功能，也可以称为AR-U网元。AR控制网元具备AR控制面功能，可以称为AR控制面(AR Control plane，AR-C)网元。在本实施例中，网元可以称为设备，例如AR媒体处理网元可以称为AR媒体处理设备。
具备AR能力的终端设备可以具备如下一种或者多种逻辑功能：支持通过摄像头和/或传感器采集真实场景的音视频数据，支持逻辑运算功能(比如可以部署视觉引擎)，支持低时延的编码和轻量级的图像渲染，支持从云端获取虚拟对象等。本申请实施例中的终端设备可以是配置有摄像头，具有视频通话功能的设备，比如，终端设备可以是可穿戴设备(例如电子手表)，终端设备还可以是手机、平板电脑等设备。本申请实施例对终端设备的具体形式不作特殊限制。
AR媒体处理网元，具体AR媒体处理功能。比如AR媒体处理功能包括三维建模、对象识别、跟踪定位、交互感知、虚景渲染、虚实结合等中的一项或多项。三维建模是基于终端设备采集的数据等对真实环境进行三维构建。对象识别是指对终端设备用户的关注对象进行识别。跟踪定位，是指对终端设备用户的关注对象进行跟踪和定位。交互感知是指借助用户交互技术来实现多感官的一致性。虚景渲染是指对虚拟场景执行渲染。虚实结合是指虚拟对象与真实对象的融合。
AR媒体处理网元支持与终端设备之间建立媒体流通道和辅助数据通道。媒体流通道用于终端设备与AR媒体处理网元之间传输AR核心数据(或者称为媒体流数据，也可以简称媒体流)，比如音视频流。辅助数据通道用于终端设备与AR媒体处理网元之间传输AR辅助数据。AR辅助数据(AR specific data)终端设备与AR媒体网元之间传输的非媒体数据的统称。
作为一种举例，参见表1所示，示例性地描述上行方向的AR辅助数据可以包括的数据类型以及AR核心数据包括的视频数据类型。
表1
[根据细则91更正 30.03.2022]　
Figure WO-DOC-FIGURE-1
[根据细则91更正 30.03.2022]　
Figure WO-DOC-FIGURE-2
作为一种举例，参见表2所示，示例性地描述下行方向的AR辅助数据可以包括的数据类型以及音视频流包括的视频数据类型。
表2
[根据细则91更正 30.03.2022]　
Figure WO-DOC-FIGURE-3
应理解的是，以上表1和表2仅作为一种示例，并不构成对AR辅助数据和AR核心数据的具体限定。
AR控制网元，具备会话关联功能和业务控制功能。会话关联可以包括在AR会话类场景下终端设备与AR媒体处理网元之间的媒体流、AR媒体处理网元与AS之间的信令流之间的关联。或者终端设备与AR媒体处理网元之间的媒体流、AR媒体处理网元与AS的信令流以及AS与第三方服务器之间的消息流，以上三者之间的关联。第三方服务器用于提供AR处理的所需的素材，比如虚拟对象、现实背景对象等。业务控制可以是基于业务特征等对AR媒体处理进行控制。比如不同的业务特征对应不同的AR媒体处理方式，AR控制网元可以根据终端设备的不同的业务特征向AR媒体处理网元指示不同的AR媒体处理方式。
一种可能的实施方式中，AR媒体处理网元可以是IP多媒体系统(IP multimedia subsystem，IMS)接入网关(access gateway，AGW)或者可以是媒体资源功能(media resource function，MRF)。AR控制网元可以是应用服务器。
作为一种示例，参见图2A所示，以AR媒体处理网元为IMS AGW为例，AR通信系统中包括IMS AGW、IMS核心(core)。IMS核心(core)网元控制具备AR处理能力的IMS AGW与具备AR能力的终端设备之间建立辅助数据通道和媒体流通道。辅助数据通道用于传输至少两个终端设备之间进行AR通信的AR辅助数据。媒体流通道用于传输至少两个终端设备之间进行AR通信的媒体流。IMS AGW可以为终端设备提供媒体处理服 务。比如，IMS AGW在通过媒体流通道接收到媒体流和通过辅助数据通道接收到AR辅助数据时，可以使用AR辅助数据对媒体流进行增强处理。
AR通信系统还可以包括应用服务器(application service，AS)。AR通信系统中还可以包括至少两个终端设备。分别服务于不同终端设备的IMS AGW可以是同一个，也可以是不同的IMS AGW，图2A中以两个终端设备分别为第一终端设备和第二终端设备为例，服务于第一终端设备和第二终端设备的IMS AGW不同为例，分别为IMS AGW1和IMS AGW2。AR通信的两个终端设备之间可以均具备AR能力，也可以有其中一个终端设备具备AR能力，另一种终端设备不具备AR能力。图2A中以第一终端设备和第二终端设备均具有AR能力为例。例如，不同的终端设备接入不同的IMS网络中，服务于不同的终端设备的IMS AGW不同。服务于AR通信的两个终端设备的不同IMS AGW可以均具备AR处理能力，也可以仅其中一个IMS AGW具备AR处理能力，另一个IMS AGW不具备AR处理能力。图2A中的IMS AGW1和IMS AGW2均具有AR处理能力。IMS AGW1服务于第一终端设备，或者说IMS AGW1为第一终端设备所属的接入网关，或者说IMS AGW1位于第一终端设备所属的(或所在的)IMS网络中。IMS AGW2服务于第二终端设备，或者说IMS AGW2为第二终端设备所属的接入网关，或者说IMS AGW2位于第二终端设备所属的IMS网络中。一个接入网关可以服务于多个终端设备，比如IMS AGW1服务于第一终端设备，还服务于第三终端设备(图2A中未示出)。IMS core中可以包括呼叫会话控制功能(call session control function，CSCF)和/或归属签约用户服务器(home subscriber server，HSS)，IMS core还可以包括其它网元，本申请实施例中不再赘述。CSCF是IMS core的呼叫控制中心，是在IP传输平台上实现用户接入，鉴权、会话路由和业务触发等功能。CSCF可以包括服务-呼叫会话控制功能(serving-call session control function，S-CSCF)、代理-CSCF(Proxy CSCF，P-CSCF)、查询-CSCF(Interrogating-CSCF，I-CSCF)中的一个或多个。HSS用于记录用户的签约数据(比如用户信息、业务数据)。需要说明的是，为第一终端设备提供服务的IMS核心网元和为第二终端设备提供服务的IMS核心网元可以不同，比如与终端设备所在的位置相关。作为一种示例，本申请实施例所提及的IMS核心网元可以包括CSCF。
在一些实施例中，AR通信系统中还可以包括第三方服务器。第三方服务器可以理解为第三方提供商的服务器。第三方服务器用于提供终端设备执行AR通信所需的目标对象。比如虚拟对象。再比如现实背景对象。第三方服务器与应用服务器相互配置有对方的地址信息，比如可以包括IP地址和端口号。一种可能的示例中，参见图2B所示，第三方服务器支持与AR媒体处理网元建立通信连接，比如可以是超文本传输协议(hypertext transfer protocol，http)连接。另一种可能的示例中，第三方服务器不支持与AR媒体处理网元建立通信连接，参见图2C所示，第三方服务器与应用服务器建立通信连接，比如可以是超文本传输协议http连接。
作为另一种示例，AR通信的两个终端设备中其中一个终端设备不具备AR能力。例如，参见图3所示，第一终端设备具备AR能力，第二终端设备不具备AR能力。图3中服务于第一终端设备的IMS AGW1具备AR能力。
作为又一种示例，AR通信的两个终端设备中其中一个终端设备不具备AR能力。例如，参见图4所示，第一终端设备具备AR能力，第二终端设备不具备AR能力。图4中服务于第二终端设备的IMS AGW2具备AR能力，服务于第一终端设备的IMS AGW1不 具备AR能力。
另一种可能的实施方式中，AR媒体处理网元可以是媒体资源功能(media resource function，MRF)。作为一种示例，参见图5所示，以AR媒体处理网元为MRF为例，AR通信系统中包括MRF、IMS核心网(core)、应用服务器(application service，AS)和第三方服务器为例。MRF通常分为媒体资源功能控制(media resource function control，MRFC)和媒体资源功能处理(media resource function processor，MRFP)。在会议场景中，一个终端设备借助MRF与其它终端设备进行通信。MRF是会话消息与媒体流的中心节点。图5中以第一终端设备、第二终端设备和第三终端设备为例。AR会议的多个终端设备可以均具备AR能力，也可以有其中一个或者几个终端设备具备AR能力，其它终端设备不具备AR能力。图5中以第一终端设备、第二终端设备均具有AR能力、第三终端设备不具备AR能力为例。分别服务于多个终端设备的MRF可以是同一个，也可以是不同的MRF，图5中以服务于第一终端设备和第三终端设备的MRF相同，均为MRF1，服务于第二终端设备的MRF为MRF2。服务于进行AR会议的多个终端设备的不同MRF可以均具备AR处理能力，也可以仅其中一个或者几个MRF具备AR处理能力，其余MRF不具备AR处理能力。图5中的MRF1和MRF2均具有AR处理能力。MRF1服务于第一终端设备和第三终端设备，MRF2服务于第二终端设备。
可选地，本申请实施例中终端设备可以根据传输的AR辅助数据的类型建立不同的辅助数据通道。比如在需要传输位姿类数据时，建立辅助数据通道1，在需要传输操作类数据时，建立辅助数据通道2。终端设备也可以通过一个辅助数据通道传输不同类型的AR辅助数据。
如下以第一终端设备和第二终端设备进行AR通信为例，第一终端设备与第二终端设备触发AR通信后，对第一终端设备与AR媒体处理网元之间建立辅助数据通道的流程进行说明。参见图6所示。
601，第一终端设备向IMS核心网元发送第一请求消息，第一请求消息用于请求建立辅助数据通道。
第一请求消息可以携带第一终端设备的第一描述参数。第一描述参数是第一终端设备用于与所述AR媒体处理网元建立所述辅助数据通道的。可选地，第一请求消息还携带第一终端设备的地址。
作为一种举例，第一描述参数可以采用会话描述协议(session description protocol，SDP)。当然还可以采用其它的描述协议，本申请对此不作具体限定。后续描述时，以第一描述参数采用SDP为例。为了便于区分，将承载第一描述参数的SDP信息称为第一SDP信息(可以简称为第一SDP)。即第一请求消息携带所述第一终端设备的第一SDP信息，所述第一SDP信息包括用于第一终端设备与所述AR媒体处理网元建立所述辅助数据通道的描述参数。比如，第一SDP信息中可以包括第一终端设备的用于传输AR辅助数据的端口号、媒体流的类型、支持的编解码格式等参数中一项或多项。媒体流类型可以包括video(视频流)、audio(音频流)、以及datachannel(AR辅助数据)。比如，在该情况下，可以在第一SDP信息包括用于描述AR辅助数据的m行。
作为一种举例，m行的描述如下：
m＝application 10001用户数据报协议(user datagram protocol，UDP)/数据包传输层安全性协议(datagram transport layer security，DTLS)/(流控制传输协议Stream Control  Transmission Protocol，SCTP)webrtc-datachannel。
m行表示针对一个具体的AR辅助数据流的一个描述，10001可以表示五元组的端口号，UDP/DTLS/SCTP表示AR辅助数据遵循的传输协议，webrtc-datachannel表示建立辅助数据通道。第一SDP信息中还可以其它信息，比如用于描述AR媒体数据的a行。作为一种举例，a行的描述如下：
a＝dcmap:0subprotocol＝""；ordered＝true；max-retr＝2；label＝"AR-specific data"。
其中，dcmap表示用于辅助数据通道需要两端协商的参数集，比如该数据通道是否支持可靠传输，是否设置最大重传次数，该数据通道内传递的子协议格式等等。如上面的举例，0表示ID为0的辅助数据通道，subprotocol表示ID为0的辅助数据通道传输的AR辅助数据的协议类型，例如，subprotocol＝””，表示两端未制定协议类型；例如subprotocol＝”http”，表示辅助数据通道内部传递的AR辅助数据是采用http格式。Orderer表示是否可靠有序传输，max-retr表示最大重传次数，例如max-retr＝2，表示最大重传次数为2，label表示该辅助数据通道传输的AR辅助数据的描述。
作为另一种举例，第一SDP信息中还可以包括如下：
a＝dcsa:0accept-types:text/plain。
其中，dcsa表示辅助数据通道的SDP属性，0表示具体的辅助数据通道的ID，accept-type表示辅助数据通道传输的AR辅助数据的格式。accept-types:text/plain表示辅助数据通道采用的格式是文本格式。
一种示例中，第一终端设备可以在与第二终端设备建立AR通信的流程中，向IMS核心网元发送第一请求消息。例如，第一请求消息可以采用会话初始协议(Session Initiation Protocol，SIP)请求(invite)消息。第一请求消息中还可以携带第一终端设备的地址信息，例如，第一终端设备的地址信息可以包括第一终端设备的IP地址和/或端口号。
可选地，第一请求消息还用于请求建立媒体流通道，所述媒体流通道用于传输所述第一终端设备与第二终端设备之间进行AR通信的媒体流。第一请求消息还携带用于第一终端设备与所述AR媒体处理网元建立所述媒体流通道的描述参数。例如，第一SDP信息还包括用于第一终端设备与所述AR媒体处理网元建立所述媒体流通道的描述参数。
另一种示例中，第一终端设备可以在与第二终端设备重建立AR会话的流程中，向IMS核心网元发送第一请求消息。例如，第一请求消息可以是SIP Re-invite(重请求)消息。
又一种示例中，第一终端设备可以在与第二终端设备更新AR会话的流程中，向IMS核心网元发送第一请求消息。例如，第一请求消息可以是SIP update invite(更新请求)消息。
602，IMS核心网元接收到第一请求消息后，向AR媒体处理网元发送第二请求消息。所述第二请求消息用于请求AR媒体处理网元与第一终端设备之间建立所述辅助数据通道。示例性地，第二请求消息中还携带第一SDP信息。
本实施例中的第一终端设备与第二终端设备之间的通信可以是第一终端设备与第二终端设备点对点的通信，也可以是第一终端设备与第二终端设备建立会议通信。AR媒体处理网元为具备AR处理能力的IMS AGW或者MRF。示例性地，当第一终端设备与第二终端设备之间点对点通信时，AR媒体处理网元可以为IMS AGW。当第一终端设备与第二终端设备之间AR会议通信时，AR媒体处理网元可以为MRF。AR媒体处理网元可以是位于第一终端设备所属的(或所在的)IMS网络中，AR媒体处理网元用于为第一终端设 备提供服务，也可以是位于第二终端设备所属的(或所在的)IMS网络中，用于为第二终端设备提供服务。
比如，AR媒体处理网元为IMS AGW时，第二请求消息可以是H248协议消息，比如H248 add request消息。再比如，AR媒体处理网元为MRF时，第二请求消息可以是SIP invite消息。
603，AR媒体处理网元接收到第二请求消息时，向IMS核心网元发送第二响应消息，第二响应消息是所述第二请求消息对应的响应消息。例如，第二响应消息用于向IMS核心网元指示所述AR媒体处理网元对建立所述辅助数据通道的确认。比如，第二响应消息携带AR媒体处理网元的用于建立辅助数据通道的第二描述参数。AR媒体处理网元可以从第一描述参数中确定自身支持的第二描述参数。第二描述参数可以采用SDP协议，也可以采用其它协议。为了便于区分，将AR媒体处理网元的SDP信息称为第二SDP信息(可以简称为第二SDP)。所述第二SDP信息包括所述AR媒体处理网元的用于与第一终端设备建立所述辅助数据通道的描述参数。比如，第二SDP信息中包括AR媒体处理网元对应的辅助数据通道的端口、媒体流的类型、支持的编解码格式等参数。媒体流类型可以包括video(视频流)、audio(音频流)、以及datachannel(AR辅助数据)。比如，在该情况下，可以在第二SDP信息包括用于描述AR媒体处理网元协商得到的AR辅助数据的m行，该m行的媒体流类型是AR辅助数据。
604，IMS核心网元接收到第二响应消息时，向第一终端设备发送第一响应消息，第一响应消息中携带AR媒体处理网元的地址。第一响应消息携带所述AR媒体处理网元的第二SDP信息。
在一些实施例中，IMS核心网元在接收到第一请求消息后，将第一请求消息转发给应用服务器。应用服务器确定第一终端设备签约有AR业务时，可以将自身的地址信息通过IMS核心网元发送给AR媒体处理网元。应用服务器可以将自身的地址信息添加到第一请求消息中发送给IMS核心网元。从而IMS核心网元在向AR媒体处理网元发送的第二请求消息中承载该应用服务器的地址信息，从而后续AR媒体处理网元在需要获取第三方服务器提供的内容时，可以通过AS从第三方服务器获取所需内容。
示例性地，应用服务器可以向HSS发送签约查询请求，用于查询第一终端设备的签约信息是否包括签约AR业务的信息。HSS可以通过签约查询响应将第一终端设备的签约信息反馈给应用服务器。签约查询请求可以采用直连请求(diameter request)消息。签约查询响应可以采用直连响应(diameter response)消息。
在另一些实施例中，MS核心网元在接收到第一请求消息后，将第一请求消息转发给应用服务器。应用服务器确定第一终端设备签约有AR业务时，可以将保存的第三方服务器的地址信息通过IMS核心网元发送给AR媒体处理网元。应用服务器可以将第三方服务器的地址信息添加到第一请求消息中发送给IMS核心网元。从而IMS核心网元在向AR媒体处理网元发送的第二请求消息中承载该第三方服务器的地址信息，从而后续AR媒体处理网元在需要获取第三方服务器提供的内容时，可以从第三方服务器获取所需内容。
在一种可能的实施方式中，终端设备可以预先获知AR媒体处理网元是否具备AR处理能力。比如，终端设备在向网络侧的注册流程中，获知终端设备所在的IMS网络中是否存在具备AR处理能力的AR媒体处理网元。
在另一种可能的实施方式中，IMS核心网元在向AR媒体处理网元发送第二请求消息 之前，先询问AR媒体处理网元是否具备AR处理能力，比如向AR媒体处理网元发送查询消息，查询消息用于查询AR媒体处理网元是否具备AR处理能力，AR媒体处理网元在接收到查询消息后，会向IMS核心网元发送指示信息，用于指示AR媒体处理网元是否具备AR处理能力。在确定AR媒体处理网元具备AR处理能力时，再向该具备AR处理能力的AR媒体处理网元发送第二请求消息。
AR媒体处理网元可以为具备AR处理能力的IMS AGW或者MRF。以AR媒体处理网元为IMS AGW为例。参见图7A和图7B所示，为了便于区分，将为第一终端设备提供服务的IMS AGW称为IMS AGW1，将为第二终端设备提供服务的IMS AGW称为IMS AGW2。
701a，参见601，此处不再赘述。
702a，IMS核心网元在接收到第一终端设备的第一请求消息时，向IMS AGW1发送查询消息1，查询消息1用于查询IMS AGW1是否具备AR处理能力。
示例性地，查询消息1中可以包括用于询问是否具备AR处理能力的参数。例如，参数可以为“ifSupportforAR_flag”。
查询消息1可以是H248 add request。H248 add request增加参数ifSupportforAR_flag。
703a，IMS AGW1确定具备AR处理能力时，向IMS核心网元发送第一指示，第一指示用于指示IMS AGW1具备AR处理能力。示例性地，第一指示可以是“ifSupportforAR_flag＝true”。可选地，第一指示可以承载在H248 add response中。H248 add response包括“ifSupportforAR_flag＝true”。
704a，IMS核心网元向IMS AGW1发送第二请求消息。参见602，此处不再赘述。
705a-706a，参见603-604，此处不再赘述。
参见图7B所示，701b，参见601，此处不再赘述。
702b，IMS核心网元在接收到第一终端设备的第一请求消息时，向IMS AGW1发送查询消息2，查询消息2用于查询IMS AGW1是否具备AR处理能力。
示例性地，查询消息2中可以包括用于询问是否具备AR处理能力的参数。例如，参数可以为“ifSupportforAR_flag”。查询消息2可以是H248 add request。H248 add request增加参数ifSupportforAR_flag。
703b，IMS AGW1确定不具备AR处理能力时，向IMS核心网元发送第二指示，第二指示用于指示IMS AGW1不具备AR处理能力。示例性地，第二指示可以是“ifSupportforAR_flag＝false”。可选地，第二指示可以承载在H248 add response中。H248 add response包括“ifSupportforAR_flag＝false”。或者H248 add response不包括该参数ifSupportforAR_flag＝false，以指示IMS AGW1不具备AR处理能力。
704b，IMS核心网元向IMS AGW2发送第二请求消息。
705b-706b，参见603-604，此处不再赘述。
在另一种可能的实施方式中，IMS核心网元在向AR媒体处理网元发送第二请求消息中携带用于询问AR媒体处理网元是否具备AR处理能力的参数。比如，第二请求消息中携带“ifSupportforAR_flag”。AR媒体处理网元在第二响应消息中携带“ifSupportforAR_flag＝true”时，确定AR媒体处理网元具备AR处理能力。或者AR媒体处理网元在第二响应消息中未携带“ifSupportforAR_flag”参数，且携带有第二描述参数，则确定AR媒体处理网元具备AR处理能力。或者，AR媒体处理网元在第二响应消息中携 带“ifSupportforAR_flag＝false”参数，则确定AR媒体处理网元不具备AR处理能力。
下面结合具体应用场景对本申请实施例提供的辅助数据通道的建立流程进行描述。
第一种可能的应用场景中：在AR会话创建过程中建立辅助数据通道。该场景中以具备AR处理功能的AR媒体处理网元为第一终端设备所属的IMS AGW1为例。
参见图8A示所示，为本申请第一种可能的应用场景的通信方法流程示意图。
801，第一终端设备向IMS core网元(比如intermediate IMS1)发送第一请求消息。
此实施例中，将用于为第一终端设备提供服务的IMS core网元称为intermediate IMS1为例。在图8A中以第一请求消息采用SIP INVITE消息为例，为了便于区分，将第一请求消息称为SIP INVITE1为例。SIP INVITE1用于请求建立AR会话。SIP INVITE1包括第一终端设备的第一SDP。第一SDP包括用于第一终端设备建立所述辅助数据通道的描述参数。第一SDP还包括用于第一终端设备建立媒体流通道的描述参数。SIP INVITE还包括第一终端设备的地址信息，包括第一终端设备的IP地址和端口号。图8A中以IP地址为IP1o，端口号为P1o。比如，第一SDP中包括地址信息(比如端口号)、媒体流的类型、支持的编解码格式等参数。媒体流类型可以包括video(视频流)、audio(音频流)、以及datachannel(AR辅助数据)。比如，在该情况下，第一SDP信息包括用于描述AR辅助数据的m行，还可以包括用于描述音频数据的m行和视频数据的m行。m行是针对一个媒体流(视频流、音频流或者AR辅助数据构成的流)的描述。
作为一种举例，定义如下的3个m行，分别表示视频流的m行、音频流的m行和AR辅助数据构成的流的m行。
m＝video 49154RTP/AVPF 101 102。该m行的第一个参数表示媒体名称，用于表明支持的媒体流类型为视频(video)。第二个参数为端口号，用于表示采用49154端口发送该视频流。第三个参数为传输协议，采用实时传输协议(real-time transport protocol，RTP)/音视频配置文件反馈(audio video profile feedback，AVPF)。第4个-第5个参数表示所支持的净荷类型编号。
m＝audio 49152RTP/AVP 98 97 99。该m行的第一个参数表示媒体名称，用于表明支持的媒体流类型为音频(audio)。第二个参数为端口号，用于表示采用49152端口发送该音频流。第三个参数为传输协议，采用实时传输协议(real-time transport protocol，RTP)/音视频配置文件(audio video profile，AVP)。第4个-第6个参数表示所支持的净荷类型编号。
m＝application 10001UDP/DTLS/SCTP webrtc-datachannel。
m行表示针对一个具体的AR辅助数据流的一个描述，10001可以表示五元组的端口号，UDP/DTLS/SCTP表示AR辅助数据遵循的传输协议，webrtc-datachannel表示建立辅助数据通道。
第一SDP信息还可以包括其它信息。具体针对第一SDP描述可以参见图6对应的实施例中的描述，此处不再赘述。
802，Intermediate IMS1收到第一终端设备的SIP INVITE1，将SIP INVITE1转发给AS。
示例性地，Intermediate IMS1在接收到SIP INVITE1后，根据第一终端设备的签约数据确定第一终端设备签约了AR媒体增强业务，进而将SIP INVITE1转发给AS。该AS用于为AR媒体增强业务提供控制服务。
803，AS向Intermediate IMS1发送SIP INVITE2。
AS在接收到SIP INVITE1后，确定第一终端设备签约的AR业务触发相应的AR业务。AS通过Intermediate IMS1触发建立AR会话，即向Intermediate IMS1触发SIP INVITE2。可选地，AS触发的SIP INVITE2与第一终端设备发送的SIP INVITE1可以采用相同的消息，也可以采用不同的消息。
一些实施例中，AS识别第一终端设备触发AR业务(或者说发起AR会话)，AS可以记录该AR会话的会话标识为AR会话，并且建立会话标识和第一终端设备的标识的绑定关系。建立该绑定关系后，在后续通过AS从第三方服务器获取目标对象可以使用，后续会详细描述，此处不再赘述。
804，Intermediate IMS1向IMS AGW1发送用于查询IMS AGW1是否具备AR能力的查询消息。为了便于区分，此处将查询IMS AGW1是否具备AR能力的查询消息称为查询消息1。查询消息1可以采用H248 add request1(req1)。H248 add request1可以用于请求建立承载。示例性地，在H248 add request1携带用于询问IMS AGW1是否具备AR处理能力的参数。例如，用于询问IMS AGW1是否具备AR处理能力的参数可以为“IfSupportforAR_flag”。
可选地，收到H248 add requset1，创建出局端口T2(outgoing termination T2)。具体是，创建IMS AGW1的出局端口。IMS AGW1向Intermediate IMS1反馈T2端口对应地址信息，以及指示是否具备AR处理能力的指示信息。
805，IMS AGW1接收到H248 add request1后，向Intermediate IMS1发送查询响应1，查询响应1包括指示信息，用于指示IMS AGW1是否具备AR处理能力。例如，查询响应1可以采用H248 add response1。Intermediate IMS1发送H248 add response(resp)1，H248 add response1包括所述指示信息和T2端口对应的地址信息。例如，T2端口对应的地址信息包括IP地址和端口号，以IP地址为IP2o，端口号为P2o为例。例如，IMS AGW1具备AR处理能力，H248 add response包括IfSupportforAR_flag＝True”，IMS AGW1具备AR处理能力，H248 add response携带“IfSupportforAR_flag＝False”或者不携带该参数。该实施例中以IMS AGW1具备AR处理能力为例。
806，Intermediate IMS1接收到H248 add response1后，识别IMS AGW1具有AR处理能力，Intermediate IMS1向Intermediate IMS1触发创建AR会话。向Intermediate IMS1发送SIP INVITE3，SIP INVITE3中携带IMS AGW1的T2端口对应的地址信息。
需要说明的是，Intermediate IMS1识别IMS AGW1具有AR处理能力，则仅需在第一终端设备与IMS AGW1建立辅助数据通道，则在向第二终端设备的网络侧发送的AR会话建立请求中不需携带用于请求建立辅助数据通道的SDP。
807，Intermediate IMS2向AS2发送SIP INVITE3，携带IMS AGW1的T2端口对应的地址信息。
808，AS2向Intermediate IMS2发送SIP INVITE3，携带IMS AGW1的T2端口对应的地址信息。
809，Intermediate IMS2向IMS AGW2发送H248 add requset2，用于指示IMS AGW2创建出局端口T2。例如，包括IP地址和端口号，以IP地址为IP3o，端口号为P3o为例。
810，IMS AGW2创建出局端口T2，并向Intermediate IMS2发送H248 add resp2，携带IMS AGW2 T2端口对应的地址信息。
811，IMS AGW2向第二终端设备发送SIP INVITE4，SIP INVITE4包括IMS AGW2 T2端口对应的地址信息。
812，第二终端设备向IMS AGW2发送SIP 180，SIP 180携带第二终端设备的地址信息(IP1a，P1a)。
813，Intermediate IMS2向IMS AGW2发送H248 mod requset1，用于指示IMS AGW2配置出局端口T1。例如，建立IMS AGW2的T1的地址信息与第二终端设备的地址信息的关联关系。
814，IMS AGW2配置出局端口T2，并向Intermediate IMS2发送H248 mod resp1。H248 mod resp1可以携带IMS AGW2 T2端口对应的地址信息和第二终端设备的地址信息。
815，Intermediate IMS2向IMS AGW2发送H248 add requset3，用于指示IMS AGW2创建入局端口T2。例如，H248 add requset3包括IMS AGW1的T2端口对应的地址信息。
816，IMS AGW2创建入局端口T2，并向Intermediate IMS2发送H248 add resp2，携带IMS AGW2 T2端口对应的地址信息(IP4o，P4o)。
817，Intermediate IMS1接收来自第二终端设备的SIP 180，比如SIP 180中可以包括第二终端设备的地址信息。例如，第二终端设备的IP地址为IP1a，第二终端设备的端口号为P1a。Intermediate IMS1记录第二终端设备的IP地址和端口号。
818，Intermediate IMS1向IMS AGW1发送修改配置请求，比如修改配置请求可以采用H248Mod req3。H248Mod req包括第二终端设备的IP地址(IP1a)和端口号(P1a)。H248Mod req3用于请求IMS AGW1配置出局端口T2。
819，IMS AGW1根据第二终端设备的地址信息配置出局端口T2(outgoing termination T2)，并向Intermediate IMS1发送修改配置响应，修改配置响应可以采用H248 mod response3。
820，Intermediate IMS1收到IMS AGW1的修改配置响应，Intermediate IMS1向IMS AGW1发送第二请求消息。第二请求消息用于请求建立辅助数据通道，还可以用于请求创建入局端口T1，比如采用H248 add req4。H248 add req4还用于请求建立辅助数据通道。H248 add req4包括第一终端设备的第一SDP和第一终端设备的地址信息。
IMS AGW1接收到H248 add req4后，创建入局端口T1(incoming termination T1)，并对辅助数据通道的建立进行确认。
821，IMS AGW1向Intermediate IMS1发送第二响应消息，包括入局配置T1信息的响应，还可以包括辅助数据通道建立的响应。第二响应消息可以采用H248 add response4。H248 add response4包括IMS AGW1的第二SDP信息和IMS AGW1的地址信息。例如，IMS AGW1的地址信息包括IMS AGW1的IP地址(IP2a)和端口号(P2a)。可选地，H248 add response4还可以包括第一终端设备的地址信息。
示例性地，第二SDP信息包括用于描述IMS AGW1的所需建立的辅助数据通道的m行。作为一种举例，IMS AGW1对第一终端设备的m行的响应可以描述如下：
m＝application 10001UDP/DTLS/SCTP webrtc-datachannel。
822，Intermediate IMS1记录IMS AGW1的T1端口的地址信息，并确定辅助数据通道建立的第二SDP信息，向AS发送的第一响应消息中包括第二SDP信息和IMS AGW1的地址信息。第一响应消息中比如可以采用SIP 180。SIP 180包括第二SDP信息和IMS AGW1的地址信息。
823，AS向Intermediate IMS1转发SIP 180。
824，Intermediate IMS1向第一终端设备转发SIP 180。
在一些实施例中，步骤803中，AS接收到SIP INVITE1后，AS确定第一终端设备签约有AR业务时，可以将自身的地址信息通过Intermediate IMS1发送给IMS AGW1。AS可以将自身的地址信息添加到SIP INVITE1中发送给Intermediate IMS1。从而Intermediate IMS1在向IMS AGW1发送的第二请求消息(H248 add request)中承载该AS的地址信息，从而后续IMS AGW1在需要获取第三方服务器提供的内容时，可以通过AS从第三方服务器获取所需内容。
示例性地，AS可以向HSS发送签约查询请求，用于查询第一终端设备的签约信息是否包括签约AR业务的信息。HSS可以通过签约查询响应将第一终端设备的签约信息反馈给AS。签约查询请求可以采用直连请求(diameter request)消息。签约查询响应可以采用直连响应(diameter response)消息。
在另一些实施例中，Intermediate IMS1在接收到SIP INVITE1后，将SIP INVITE1转发给AS。AS确定第一终端设备签约有AR业务时，可以将保存的第三方服务器的地址信息通过Intermediate IMS1发送给IMS AGW1。AS可以将第三方服务器的地址信息添加到第一请求消息中发送给Intermediate IMS1。从而Intermediate IMS1在向IMS AGW1发送的第二请求消息(H248 add request)中承载该第三方服务器的地址信息，从而后续IMS AGW1在需要获取第三方服务器提供的内容时，可以从第三方服务器获取所需内容。
下面结合图8B对上述涉及的T1端口和T2端口进行说明。T1端口是IMS AGW上用于通过接入网连接终端设备的端口。T2端口是IMS AGW上用于连接IMS核心网的端口，IMS AGW通过IMS核心网能够与通信对端的终端设备建立连接。
第二种可能的应用场景中，通过触发AR会话重建立过程触发建立辅助数据通道。该场景中以第一终端设备所属的IMS AGW1具备AR处理能力为例。
参见图9所示，为本申请第三种可能的应用场景的通信方法流程示意图。
901，在第一终端设备与第二终端设备建立AR会话后，第一终端设备向IMS core网元(比如intermediate IMS1)发送第一请求消息。在图9中以第一请求消息为SIP重建立请求(Re-INVITE1)为例。SIP Re-INVITE1用于请求创建辅助数据通道。SIP Re-INVITE1包括第一终端设备的第一SDP。针对第一终端设备第一SDP的描述参见图6对应的实施例中的描述，此处不再赘述。
902，Intermediate IMS1收到第一终端设备的SIP Re-INVITE1，将Re-INVITE1转发给AS。
903，AS向Intermediate IMS1发送Re-INVITE1。
904，Intermediate IMS1向IMS AGW1发送查询消息1。查询消息1可以承载在H248mod req。H248 mod req用于请求建立承载。示例性地，H248 mod req还可以携带用于询问IMS AGW1是否具备AR处理能力的参数。例如，用于询问IMS AGW1是否具备AR处理能力的参数可以为“IfSupportforAR_flag”。
可选地，IMS AGW1收到H248 mod req，向Intermediate IMS1指示是否具备AR处理能力的指示信息。
905，IMS AGW1接收到H248 mod request后，向Intermediate IMS1发送查询响应1，查询响应1包括指示信息，用于指示IMS AGW1是否具备AR处理能力。例如，Intermediate  IMS1发送H248 mod response，H248 mod response包括所述指示信息。例如，IMS AGW1具备AR处理能力，H248 mod response包括IfSupportforAR_flag＝True”，IMS AGW1具备AR处理能力，H248 mod response携带“IfSupportforAR_flag＝False”或者不携带该参数。该实施例中以IMS AGW1具备AR处理能力为例。
906，Intermediate IMS1接收到H248 mod response后，识别IMS AGW1具有AR处理能力，即仅需在第一终端设备与IMS AGW1建立辅助数据通道，则在向第二终端设备AR会话建立请求不需携带用于请求建立辅助数据通道的请求。AR会话建立请求可以采用SIP Re-INVITE2。
示例性地，Intermediate IMS1通过第二终端设备所属的Intermediate IMS2向第二终端设备发送SIP Re-INVITE2。
907，Intermediate IMS1接收来自第二终端设备的SIP响应，比如SIP 200OK。
908，Intermediate IMS1向IMS AGW1发送第二请求消息，第二请求消息可以用于请求建立辅助数据通道，还用于修改入局配置T1信息，比如通过H248 add req向IMS AGW1发送修改入局配置T1信息的请求。H248 add req还用于请求建立辅助数据通道。H248 add req包括第一终端设备的第一SDP。
IMS AGW1接收到H248 add req后，修改入局配置T1信息(incoming termination T1信息)，并对辅助数据通道的建立进行确认。
909，IMS AGW1向Intermediate IMS1发送修改入局配置T1信息的响应。修改入局配置T1信息的响应可以采用H248 add response。H248 add response包括IMS AGW1的第二SDP信息。针对第二SDP的说明可以参见图6对应的实施例中的描述，此处不再赘述。
910，Intermediate IMS1提取IMS AGW1的第二SDP信息，向AS发送的AR会话建立响应中包括第二SDP信息。AR会话建立响应中比如可以采用SIP 200OK。SIP 200OK包括第二SDP信息。
911，AS向Intermediate IMS1转发SIP 200OK。
912，Intermediate IMS1向第一终端设备转发SIP 200OK。
在一些实施例中，步骤903中，AS接收到SIP Re-INVITE1后，AS确定第一终端设备签约有AR业务时，可以将自身的地址信息通过Intermediate IMS1发送给IMS AGW1。AS可以将自身的地址信息添加到SIP Re-INVITE1中发送给Intermediate IMS1。从而Intermediate IMS1在向IMS AGW1发送的第二请求消息(H248 add request)中承载该AS的地址信息，从而后续IMS AGW1在需要获取第三方服务器提供的内容时，可以通过AS从第三方服务器获取所需内容。
在另一些实施例中，Intermediate IMS1在接收到SIP INVITE1后，将SIP Re-INVITE1转发给AS。AS确定第一终端设备签约有AR业务时，可以将保存的第三方服务器的地址信息通过Intermediate IMS1发送给IMS AGW1。AS可以将第三方服务器的地址信息添加到第一请求消息中发送给Intermediate IMS1。从而Intermediate IMS1在向IMS AGW1发送的第二请求消息(H248 add request)中承载该第三方服务器的地址信息，从而后续IMS AGW1在需要获取第三方服务器提供的内容时，可以从第三方服务器获取所需内容。
第三种可能的应用场景中，在AR会话创建过程中建立辅助数据通道。该场景中以第一终端设备所属的IMS AGW1不具备AR处理能力且第二终端设备所属的IMS AGW2具备AR处理能力为例。
参见图10所示，为本申请第三种可能的应用场景的通信方法流程示意图。
1001-1004，参见801-804，此处不再赘述。
1005，IMS AGW1接收到H248 add request后，向Intermediate IMS1发送指示信息，用于指示IMS AGW1是否具备AR处理能力。例如，Intermediate IMS1发送H248 add response，H248 add response包括所述指示信息和T2端口对应的地址信息。例如，T2端口对应的地址信息包括IP地址和端口号，以IP地址为IP2o，端口号为P2o为例。例如，IMS AGW1具备AR处理能力，H248 add response包括IfSupportforAR_flag＝True”，IMS AGW1具备AR处理能力，H248 add response携带“IfSupportforAR_flag＝False”或者不携带该参数。该实施例中以IMS AGW1不具备AR处理能力为例。
Intermediate IMS1接收到H248 add response后，识别IMS AGW1不具有AR处理能力，Intermediate IMS1通过第二终端设备所属的Intermediate IMS2和AS向IMS AGW2发送第二请求消息(SIP INVITE2)。在向第二终端设备的SIP INVITE2中不需携带用于请求建立辅助数据通道的请求。向第二终端设备发送的SIP INVITE2中携带IMS AGW1的T2端口对应的地址信息。
1006-1008，参见806-808此处不再赘述。
1009，Intermediate IMS2在接收到来自AS的SIP INVITE2后，可以向IMS AGW2发送查询消息2。
查询消息2可以承载在H248 add req2。示例性地，H248 add req2还可以携带用于询问IMS AGW2是否具备AR处理能力的参数。例如，用于询问IMS AGW2是否具备AR处理能力的参数可以为“IfSupportforAR_flag”。H248 add req2还用于指示创建出局端口T2，具体参见809，此处不再赘述。
可选地，IMS AGW2收到H248 add req2，向Intermediate IMS2指示是否具备AR处理能力的指示信息。
1010，IMS AGW2接收到H248 add request2后，向Intermediate IMS2发送查询响应2，查询响应2包括指示信息2，用于指示IMS AGW2是否具备AR处理能力。例如，Intermediate IMS2发送H248 add response2(即查询响应2)，H248 add response2包括所述指示信息2。例如，IMS AGW1具备AR处理能力，H248 add response2包括IfSupportforAR_flag＝True”，IMS AGW1具备AR处理能力，H248 add response携带“IfSupportforAR_flag＝False”或者不携带该参数。该实施例中以IMS AGW2具备AR处理能力为例。H248 add response2还包括T2的地址信息。
1011-1014，参见811-814，此处不再赘述。
1015，IMS AGW2向IMS AGW2发送H248 add requset3，用于指示IMS AGW2创建入局端口T2。H248 add requset3还用于请求与第一终端设备之间建立辅助数据通道。H248 add requset3包括第一终端设备的第一SDP。例如，H248 add requset3还包括IMS AGW1的T2端口对应的地址信息。
1016，IMS AGW2创建入局端口T2，并向Intermediate IMS2发送H248 add resp2，H248 add resp2包括对创建辅助数据通道的确认，比如包括IMS AGW2的第二SDP。还包括IMS AGW2 T2端口对应的地址信息(IP4o，P4o)。
1017，Intermediate IMS2向Intermediate IMS1发送SIP 180。
1018，Intermediate IMS1向IMS AGW1发送修改配置请求，比如修改配置请求可以是 H248 Mod req3。H248 Mod req3用于请求IMS AGW1配置出局端口T2。
1019，IMS AGW1配置出局端口T2(outgoing termination T2)，并向Intermediate IMS1发送修改配置响应消息，修改配置响应消息可以是H248Mod response3。
1020，Intermediate IMS1收到IMS AGW1的修改配置响应消息，然后向IMS AGW1发起创建入局端口T1。示例性地，在用于创建入局端口T1的请求消息中携带辅助数据通道的建立请求和第一终端设备的地址信息。创建入局端口T1的请求消息可以是H248 add req4。
1021，IMS AGW1向Intermediate IMS1发送H248 add response4。H248 add response4包括IMS AGW1的地址信息。例如，IMS AGW1的地址信息包括IMS AGW1的IP地址(IP2a)和端口号(P2a)。可选地，H248 add response4还可以包括第一终端设备的地址信息。
1022，Intermediate IMS1记录IMS AGW1的T1端口的地址信息，向AS发送的SIP 180中包括IMS AGW2的第二SDP信息和IMS AGW1的地址信息。
1023，AS向Intermediate IMS1转发SIP 180。
1024，Intermediate IMS1向第一终端设备转发SIP 180。
在一些实施例中，AS2接收到SIP INVITE3后，可以将自身的地址信息通过Intermediate IMS2发送给IMS AGW2。AS2可以将自身的地址信息添加到SIP INVITE3中发送给Intermediate IMS3。从而Intermediate IMS2在向IMS AGW2发送的H 248 add req3中承载该AS3的地址信息，从而后续IMS AGW2在需要获取第三方服务器提供的内容时，可以通过AS从第三方服务器获取所需内容。
在另一些实施例中，Intermediate IMS2在接收到SIP INVITE3后，将SIP INVITE3转发给AS2。AS2可以将保存的第三方服务器的地址信息通过Intermediate IMS2发送给IMS AGW2。Intermediate IMS2可以将第三方服务器的地址信息添加到H248 add req3中发送给Intermediate IMS2，从而后续IMS AGW1在需要获取第三方服务器提供的内容时，可以从第三方服务器获取所需内容。
第四种可能的应用场景中，在AR会议创建过程中建立辅助数据通道。该场景中以第一终端设备所属的MRF具备AR处理能力为例。
参见图11所示，为本申请第四种可能的应用场景的通信方法流程示意图。
1101-1103，参见801-803，此处不再赘述。
1104，Intermediate IMS1识别第一终端设备发起AR会议，向MRF发送第二请求消息，所述第二请求消息包括第一终端设备的第一SDP。第二请求消息可以采用SIP INVITE2。
1105，MRF在确定具备AR处理能力，则Intermediate IMS1发送第二响应消息，第二响应消息包括MRF的第二SDP。MRF在确定不具备AR处理能力时，忽略第一终端设备建立辅助数据通道的请求。例如，第二响应消息可以采用SIP 180。
1106，Intermediate IMS1向AS发送第一请求响应，第一请求响应可以采用SIP 180。第一请求响应包括MRF的第二SDP。
1107，AS向Intermediate IMS1发送第一请求响应。
1108，Intermediate IMS1向第一终端设备发送第一请求响应。
在一些实施例中，步骤1103中，AS接收到SIP INVITE1后，AS确定第一终端设备签约有AR业务时，可以将自身的地址信息通过Intermediate IMS1发送给MRF。AS可以 将自身的地址信息添加到SIP INVITE1中发送给Intermediate IMS1。从而Intermediate IMS1在向MRF发送的第二请求消息(SIP INVITE2)中承载该AS的地址信息，从而后续MRF在需要获取第三方服务器提供的内容时，可以通过AS从第三方服务器获取所需内容。
在另一些实施例中，Intermediate IMS1在接收到SIP INVITE1后，将SIP INVITE1转发给AS。AS确定第一终端设备签约有AR业务时，可以将保存的第三方服务器的地址信息通过Intermediate IMS1发送给MRF。AS可以将第三方服务器的地址信息添加到第一请求消息中发送给Intermediate IMS1。从而Intermediate IMS1在向MRF发送的第二请求消息(SIP INVITE2)中承载该第三方服务器的地址信息，从而后续IMS AGW1在需要获取第三方服务器提供的内容时，可以从第三方服务器获取所需内容。
下面结合实施例对本申请实施例涉及的通过建立完成的辅助数据通道来实现媒体流的增强的流程进行描述。
参见图12A所示，针对第一终端设备与第二终端设备进行AR通信过程中的媒体流的增强流程进行说明。第一终端设备与AR媒体处理网元中建立有辅助数据通道。第一终端设备与第二终端设备之间还建立有媒体流通道。该媒体流通道通过AR媒体处理网元。
1201，第一终端设备在与第二终端设备进行AR通信的过程中，通过辅助数据通道向AR媒体处理网元发送AR辅助数据。AR媒体处理网元位于第一终端设备的IMS网络中或者位于第二终端设备的IMS网络中。AR媒体处理网元可以是IMS AGW或者MRF。
1202，第一终端设备通过媒体流通道向AR媒体处理网元发送媒体流。该媒体流是第一终端设备在与第二终端设备进行AR通信的过程中产生的。
1203，AR媒体处理网元根据AR辅助数据对媒体流进行媒体增强处理得到AR媒体流。
1204，AR媒体处理网元向第一终端设备发送AR媒体流。第一终端设备接收AR媒体流。
可选地，1205，AR媒体处理网元通过媒体流通道向第二终端设备发送AR媒体流。第二终端设备AR媒体流。
以AR媒体处理网元为IMS AGW为例。为第一终端设备服务的IMS AGW称为IMS AGW1，为第二终端设备提供服务的IMS AGW称为IMS AGW2。参见图12B所示，以IMS AGW1与IMS AGW2均具有AR媒体能力，本申请采用AR辅助数据就近处理原则，IMS AGW1收到的AR辅助数据(非媒体数据)是与第一终端设备相关的数据，比如第一终端设备的用户的位姿数据，那么该AR辅助数据仅用于对第一终端设备共享的媒体流做位姿渲染，所以IMS AGW1根据接收到的1(原始媒体流)、a(非媒体数据比如第一终端设备的用户位姿数据)和b(非媒体数据比如第一终端设备采集的用户标注数据)结合渲染成1{a+b}(AR媒体流)返回给第一终端设备；IMS AGW1再根据1(原始媒体流)和b(非媒体数据比如UE1采集的用户标注数据)渲染成1{b}(AR媒体流)发给IMS AGW2。IMS AGW2根据接收的媒体流和UE2的c(非媒体数据)，最后渲染成媒体流1{b+c}(AR媒体流)发给UE2。参见图12C所示，如果IMS AGW2没有AR媒体处理能力，那么UE2的c(非媒体数据)将传送到IMS AGW1进行媒体增强处理。
下面结合具体应用场景，对通过辅助数据通道实现媒体流的增强处理方案进行描述。
以第一终端设备的用户向第二终端设备的用户共享视频为例。第一终端设备的用户与 第二终端设备的用户能够同时观看第一终端设备共享的视频。后续描述时，以AR媒体处理网元为IMS AGW为例。
参见图13，以第一终端设备所属的IMS AGW1和第二终端设备所属的IMS AGW2均具备AR处理能力为例。第一终端设备与第一终端设备所属的IMS AGW1已经完成辅助数据通道1的建立。第二终端设备与第二终端设备所属的IMS AGW2已经完成辅助数据通道2的建立。此外，第一终端设备与第二终端设备之间建立有媒体流通道。该媒体流通道可以认为分为三段，分别包括第一终端设备与IMS AGW1之间的媒体流通道、IMS AGW1和IMS AGW2之间的媒体流通道、IMS AGW2与第二终端设备之间的媒体流通道。第一终端设备具备AR能力，比如第一终端设备支持用户在共享的画面中进行标注。为了便于区分，将第一终端设备的用户称为用户1，将第二终端设备的用户称为用户2。
1301，第一终端设备通过媒体流通道向IMS AGW1发送媒体流。
1302，第一终端设备获取用户1在媒体流对应的显示界面上标注产生的位姿数据1，将位姿数据1作为AR辅助数据通过辅助数据通道1发送给IMS AGW1。
1303，IMS AGW1根据所述位姿数据1对所述媒体流进行渲染得到AR媒体流1。
1304a，IMS AGW1将所述AR媒体流1发送给IMS AGW2。
1304b，IMS AGW1将AR媒体流1发送给第一终端设备。
1305，IMS AGW2将AR媒体流1通过媒体流通道发送给第二终端设备。
从而第二终端设备的用户1和第二终端设备用户2均能观看到经过用户1标注的显示界面。
一些实施例中，第二终端设备也具备AR能力，比如第二终端设备支持用户在共享的画面中进行标注。
1306，第二终端设备获取用户1在媒体流对应的显示界面上标注产生的位姿数据2，将位姿数据2作为AR辅助数据通过辅助数据通道2发送给IMS AGW2。
1307，IMS AGW2根据所述位姿数据2对所述媒体流进行渲染得到AR媒体流2。
1308a，IMS AGW2将所述AR媒体流2发送给IMS AGW1。
1308b，IMS AGW2将AR媒体流2发送给第二终端设备。
1309，IMS AGW1将AR媒体流2通过媒体流通道1发送给第一终端设备。从而第二终端设备的用户1和第二终端设备用户2均能观看到经过用户2标注的显示界面。
参见图14，以第一终端设备所属的IMS AGW1不具备AR处理能力，第二终端设备所属的IMS AGW2具备AR处理能力为例。第一终端设备与第二终端设备所属的IMS AGW2已经完成辅助数据通道1的建立。第二终端设备与第二终端设备所属的IMS AGW2已经完成辅助数据通道2的建立。此外，第一终端设备与第二终端设备之间建立有媒体流通道。该媒体流通道可以认为分为三段，分别包括第一终端设备与IMS AGW1之间的媒体流通道、IMS AGW1和IMS AGW2之间的媒体流通道、IMS AGW2与第二终端设备之间的媒体流通道。第一终端设备具备AR能力，比如第一终端设备支持用户在共享的画面中进行标注。为了便于区分，将第一终端设备的用户称为用户1，将第二终端设备的用户称为用户2。
1401，第一终端设备通过媒体流通道向IMS AGW1发送媒体流。
1402，第一终端设备获取用户1在媒体流对应的显示界面上标注产生的位姿数据1， 将位姿数据1作为AR辅助数据通过辅助数据通道1发送给IMS AGW2。
1403，IMS AGW2根据所述位姿数据1对所述媒体流进行渲染得到AR媒体流1。
1404a，IMS AGW2将所述AR媒体流1通过媒体流通道发送给IMS AGW1。
1405，IMS AGW1将所述AR媒体流1通过媒体流通道发送给第一终端设备。
1404b，IMS AGW2将AR媒体流1发送给第二终端设备。
从而第二终端设备的用户1和第二终端设备用户2均能观看到经过用户1标注的显示界面。
一些实施例中，第二终端设备也具备AR能力，比如第二终端设备支持用户在共享的画面中进行标注。
1406，第二终端设备获取用户1在媒体流对应的显示界面上标注产生的位姿数据2，将位姿数据2作为AR辅助数据通过辅助数据通道2发送给IMS AGW2。
1407，IMS AGW2根据所述位姿数据2对所述媒体流进行渲染得到AR媒体流2。
1408a，IMS AGW2将所述AR媒体流2发送给IMS AGW1。
1408b，IMS AGW2将AR媒体流2发送给第二终端设备。
1409，IMS AGW1将AR媒体流2通过媒体流通道1发送给第一终端设备。从而第二终端设备的用户1和第二终端设备用户2均能观看到经过用户2标注的显示界面。
下面结合实施例对本申请实施例中目标对象的获取流程进行描述。
参见图15所示，为本申请实施例提供的一种可能的目标对象获取流程示意图。
1501，在第一终端设备与第二终端设备的AR通信过程中，第一终端设备通过辅助数据通道向AR媒体处理网元发送第一AR辅助数据。其中，第一AR辅助数据包括所述AR通信需使用的目标对象的信息。这里所提及的AR媒体处理网元为具备AR处理能力的IMS接入网关或者MRF。可以理解的是，AR媒体处理网元与第一终端设备之间建立有辅助数据通道。该AR媒体处理网元可以位于第一终端设备所属的IMS网络中，即第一终端设备对应的IMS AGW或者MRF。AR媒体处理网元可以位于第二终端设备所属的IMS网络中，即第二终端设备对应的IMS AGW或者MRF。
1502，AR媒体处理网元通过辅助数据通道接收来自第一终端设备的第一AR辅助数据时，根据第一AR辅助数据从第三方服务器获取所述目标对象。可以理解的是，所述目标对象用于对第一终端设备与第二终端设备的AR通信的媒体流进行增强。
示例性地，一些应用场景中，需要用到虚拟对象，比如2D/3D对象，终端设备一般不具备存储大量的2D/3D对象的能力，因此可以通过上述方式利用辅助数据通道接收终端设备用户的需求，从第三方服务器获取2D/3D对象。
示例性地，一些应用场景中，前景对象需要的现实背景画面是相对固定时，因此为了降低终端设备侧的上行带宽的压力，可以由AR媒体处理网元来将前景与背景画面进行渲染。在该情况下，可以由终端设备通过辅助数据通道将现实背景画面的需求信息发送给AR媒体处理网元，从而由AR媒体处理网元根据需求信息从第三方服务器获取显示背景画面，进而实现显示背景画面与终端设备发来的前景媒体流进行渲染再发送给终端设备。
一种可能的实施方式中，AR媒体处理网元提前获取有AS的地址信息，AR媒体处理网元在获取目标对象时，通过AS从第三方服务器获取目标对象。具体获取方式，参见前述，此处不再赘述。
另一种可能的实施方式中，AR媒体处理网元提前获取有第三方服务器的地址信息，进而AR媒体处理网元在获取目标对象时，可以直接根据第三方服务器的地址信息从第三方服务器获取目标对象。
下面结合具体实施例对获取虚拟对象的流程进行描述。参见图16和图17所示。
参见图16所示，以AR媒体处理网元通过AS从第三方服务器获取虚拟对象为例。以AR媒体处理网元为IMS AGW或者MRF(IMS AGW/MRF)。该IMS AGW可以是第一终端设备所属的或者通话对端的第二终端设备所属的。该MRF可以是第一终端设备所属的或者通话对端的第二终端设备所属的。
1601，第一终端设备通过辅助数据通道向IMS AGW/MRF发送虚拟对象的信息。比如虚拟对象的标识或者虚拟对象的类型。
1602，IMS AGW/MRF接收到虚拟对象的信息时，向应用服务器发送请求消息1，请求消息1中包括虚拟对象的信息。以虚拟对象的类型为例。例如，请求消息1可以采用HTTP消息类型。当然还可以采用其它消息类型，本申请对此不作限定。比如请求消息1可以是HTTP请求(request，req)消息1。示例性地，请求消息1这种包括终端设备的标识。
1603，应用服务器接收到请求消息1，确定第一终端设备需要请求虚拟对象时，将所述请求消息1转发给第三方服务器。示例性地，应用服务器可以从请求消息1中提取终端设备的标识，根据终端设备的标识确定终端设备建立的会话标识。应用服务器可以根据该会话标识与第三方服务器之间建立通信连接，比如，HTTP连接，从而将请求消息1转发该第三方服务器，以从第三方服务器获取虚拟对象。转发给第三方服务器的请求消息1中可以增加第一终端设备的会话标识。
1604，第三方服务器接收到请求消息1，根据虚拟对象的类型获取满足虚拟对象类型的多个虚拟对象标识。以将多个虚拟对象标识组成虚拟对象列表为例，当然多个虚拟对象标识还可以采用其它形式发送，本申请对此不作具体限定。第三方服务器向应用服务器发送响应消息1。响应消息1可以包括虚拟对象列表。响应消息1可以采用HTTP消息类型。比如，响应消息1可以是HTTP响应(request，response)消息1。示例性地，响应消息1包括会话标识。
1605，应用服务器向IMS AGW/MRF转发响应消息1。应该服务器根据响应消息1中的会话标识，确定终端设备的标识，向IMS AGW/MRF转发的响应消息1中增加第一终端设备的标识。
1606，IMS AGW/MRF从来自应用服务器的响应消息1中获取虚拟对象列表，并通过辅助数据通道向第一终端设备发送虚拟对象列表。可选地，IMS AGW/MRF获取虚拟对象列表后，将虚拟对象列表保存。后续再通过辅助数据通道接收到虚拟对象列表对应的虚拟对象信息时，可以不再从第三方服务器获取该虚拟对象列表。可以理解的是，在步骤1602，IMS AGW/MRF确定自身保存有虚拟对象的信息对应的虚拟对象列表时，可以不再执行后续获取虚拟对象列表的步骤。
1607，第一终端设备通过辅助数据通道向IMS AGW/MRF发送第一虚拟对象标识。该第一虚拟对象标识是虚拟对象列表中的一个。
1608，IMS AGW/MRF接收到第一虚拟对象标识时，向应用服务器发送请求消息2，请求消息2中包括第一虚拟对象标识。例如，请求消息2可以采用HTTP消息类型。比如请求消息2可以是HTTP请求(request，req)消息2。
1609，应用服务器接收到请求消息2，将所述请求消息2转发给第三方服务器。
1610，第三方服务器接收到请求消息2，获取第一虚拟对象标识对应的第一虚拟对象。第三方服务器向应用服务器发送响应消息2。响应消息2可以包括第一虚拟对象。响应消息2可以采用HTTP消息类型。比如，响应消息2可以是HTTP响应(response，resp)消息2。
1611，应用服务器向IMS AGW/MRF转发响应消息2。
1612，IMS AGW/MRF从响应消息2中获取第一虚拟对象，并通过辅助数据通道向第一终端设备发送第一虚拟对象。
参见图17所示，以AR媒体处理网元直接从第三方服务器获取虚拟对象为例。以AR媒体处理网元为IMS AGW或者MRF(IMS AGW/MRF)。该IMS AGW可以是第一终端设备所属的或者通话对端的第二终端设备所属的。该MRF可以是第一终端设备所属的或者通话对端的第二终端设备所属的。
1701，第一终端设备通过辅助数据通道向IMS AGW/MRF发送虚拟对象的信息。比如虚拟对象的标识或者虚拟对象的类型。
1702，IMS AGW/MRF接收到虚拟对象的信息时，向第三方应用服务器发送请求消息11。请求消息11可以包括虚拟对象的信息。
1703，第三方服务器接收到请求消息11，根据虚拟对象的类型获取满足虚拟对象类型的多个虚拟对象标识。以将多个虚拟对象标识组成虚拟对象列表为例，当然多个虚拟对象标识还可以采用其它形式发送，本申请对此不作具体限定。第三方服务器向IMS AGW/MRF发送响应消息11。响应消息11可以包括虚拟对象列表。响应消息11可以采用HTTP消息类型。比如，响应消息11可以是HTTP响应(response，resp)消息11。
1704，IMS AGW/MRF从响应消息11中获取虚拟对象列表，并通过辅助数据通道向第一终端设备发送虚拟对象列表。可选地，IMS AGW/MRF获取虚拟对象列表后，将虚拟对象列表保存。后续再通过辅助数据通道接收到虚拟对象列表对应的虚拟对象信息时，可以不再从第三方服务器获取该虚拟对象列表。可以理解的是，在步骤1702，IMS AGW/MRF确定自身保存有虚拟对象的信息对应的虚拟对象列表时，可以不再执行后续获取虚拟对象列表的步骤。
1705，第一终端设备通过辅助数据通道向IMS AGW/MRF发送第一虚拟对象标识。该第一虚拟对象标识是虚拟对象列表中的一个。
1706，IMS AGW/MRF接收到第一虚拟对象标识时，向第三方服务器发送请求消息12。
1707，第三方服务器接收到请求消息12，获取第一虚拟对象标识对应的第一虚拟对象。第三方服务器向IMS AGW/MRF发送响应消息12。响应消息12可以包括第一虚拟对象。响应消息12可以采用HTTP消息类型。比如，响应消息12可以是HTTP响应(response，resp)消息2。
1708，IMS AGW/MRF从响应消息2中获取第一虚拟对象，并通过辅助数据通道向第一终端设备发送第一虚拟对象。
下面结合具体实施例对获取使用现实背景对象的媒体增强流程进行描述。参见图18和图19所示。
参见图18所示，以AR媒体处理网元通过AS从第三方服务器获取现实背景对象为例。以AR媒体处理网元为IMS AGW或者MRF(IMS AGW/MRF)。该IMS AGW可以是第一终端设备所属的或者通话对端的第二终端设备所属的。该MRF可以是第一终端设备所属的或者通话对端的第二终端设备所属的。
1801，第一终端设备通过辅助数据通道向IMS AGW/MRF发送现实背景对象的信息。比如现实背景对象的标识或者现实背景对象的地理位置等。
1802，第一终端设备通过媒体流通道向IMS AGW/MRF发送第一终端设备产生的媒体流1。
1803，IMS AGW/MRF接收到现实背景对象的信息时，向应用服务器发送请求消息21，请求消息21中包括现实背景对象的信息。以现实背景对象的地理位置为例。例如，请求消息21可以采用HTTP消息类型。当然还可以采用其它消息类型，本申请对此不作限定。比如请求消息21可以是HTTP请求(request，req)消息21。
1804，应用服务器接收到请求消息21，确定第一终端设备需要请求现实背景对象时，将所述请求消息21转发给第三方服务器。
1805，第三方服务器接收到请求消息21，根据现实背景对象的地理位置获取现实背景对象。第三方服务器向应用服务器发送响应消息21。响应消息21可以包括现实背景对象列表。响应消息21可以采用HTTP消息类型。比如，响应消息21可以是HTTP响应(response，resp)消息21。
1806，应用服务器向IMS AGW/MRF转发响应消息21。
1807，IMS AGW/MRF从响应消息21中获取现实背景对象，将显示背景对象渲染到媒体流1中得到AR媒体流2。
1808，IMS AGW/MRF通过媒体流通道向第一终端设备发送AR媒体流2。
参见图19所示，以AR媒体处理网元直接从第三方服务器获取现实背景对象为例。以AR媒体处理网元为IMS AGW或者MRF(IMS AGW/MRF)。该IMS AGW可以是第一终端设备所属的或者通话对端的第二终端设备所属的。该MRF可以是第一终端设备所属的或者通话对端的第二终端设备所属的。
1901，第一终端设备通过辅助数据通道向IMS AGW/MRF发送现实背景对象的信息。比如现实背景对象的标识或者现实背景对象的地理位置等。
1902，第一终端设备通过媒体流通道向IMS AGW/MRF发送第一终端设备产生的媒体流1。
1903，IMS AGW/MRF接收到现实背景对象的信息时，向第三方服务器发送请求消息22，请求消息22中包括现实背景对象的信息。以现实背景对象的地理位置为例。例如，请求消息22可以采用HTTP消息类型。当然还可以采用其它消息类型，本申请对此不作限定。比如请求消息22可以是HTTP请求(request，req)消息22。
1904，第三方服务器接收到请求消息22，根据现实背景对象的地理位置获取现实背景对象。第三方服务器向IMS AGW/MRF发送响应消息22。响应消息22可以包括现实背景对象列表。响应消息22可以采用HTTP消息类型。比如，响应消息22可以是HTTP响应(request，response)消息22。
1905，IMS AGW/MRF从响应消息22中获取现实背景对象，将显示背景对象渲染到 媒体流1中得到AR媒体流2。
1906，IMS AGW/MRF通过媒体流通道向第一终端设备发送AR媒体流2。
基于与方法实施例同一发明构思，本申请实施例提供一种装置，具体用于实现上述方法实施例中IMS核心网元所执行的方法，该装置的结构如图20所示，包括接收单元2001、发送单元2002以及处理单元2003。
接收单元2001，用于接收来自第一终端设备的第一请求消息，所述第一请求消息用于请求建立辅助数据通道，所述辅助数据通道用于第一终端设备与AR媒体处理网元之间传输AR通信的辅助数据；
处理单元2003，用于控制所述AR媒体处理网元与所述第一终端设备之间建立所述辅助数据通道。
在一种可能的实现方式中，处理单元2003，通过接收单元2001、发送单元2002实现控制所述AR媒体处理网元与所述第一终端设备之间建立所述辅助数据通道。
所述发送单元2002，用于向所述AR媒体处理网元发送第二请求消息，所述第二请求消息用于请求AR媒体处理网元与所述第一终端设备之间建立所述辅助数据通道；
所述接收单元2001，还用于接收所述AR媒体处理网元发送的第二响应消息，所述第二响应消息用于向IMS核心网元指示所述AR媒体处理网元对建立所述辅助数据通道的确认；
所述发送单元2002，还用于向所述第一终端设备发送第一响应消息，所述第一响应消息用于向第一终端设备指示所述AR媒体处理网元成功建立所述辅助数据通道。
在一种可能的实现方式中，所述AR媒体处理网元为具备AR处理能力的IMS接入网关或者媒体资源功能MRF。
在一种可能的实现方式中，所述AR媒体处理网元位于所述第一终端设备所属的IMS网络中或者位于所述第二终端设备所属的IMS网络中。
在一种可能的实现方式中，所述第一请求消息和所述第二请求消息携带所述第一终端设备的用于建立所述辅助数据通道的第一描述参数；
所述第一响应消息和所述第二响应消息携带所述AR媒体处理网元的用于建立所述辅助数据通道的第二描述参数。
在一种可能的实现方式中，所述发送单元2002，还用于向AR媒体处理网元发送查询请求，所述查询请求用于询问所述AR媒体处理网元是否具备AR处理能力；
所述接收单元2001，还用于接收所述AR媒体处理网元发送的查询响应，所述查询响应携带所述AR媒体处理网元具备AR处理能力的信息。
在一种可能的实现方式中，所述第一请求消息还用于请求建立媒体流通道，所述媒体流通道用于所述第一终端设备与所述AR媒体处理网元之间传输AR通信内容的媒体流；
所述处理单元2003，还用于控制所述AR媒体处理网元与所述第一终端设备之间建立所述媒体流通道。
在一种可能的实现方式中，所述处理单元2003通过发送单元2002和接收单元2001实现控制所述AR媒体处理网元与所述第一终端设备之间建立所述媒体流通道。
所述发送单元2002，还用于向所述AR媒体处理网元发送第二请求消息，所述第二请求消息用于请求AR媒体处理网元与所述第一终端设备之间建立所述媒体流通道；
接收单元2001，还用于接收所述AR媒体处理网元发送的第二响应消息，所述第二响应消息用于向IMS核心网元指示所述AR媒体处理网元对建立所述媒体流通道的确认；
发送单元2002，还用于向所述第一终端设备发送第一响应消息，所述第一响应消息用于向第一终端设备指示所述AR媒体处理网元成功建立所述媒体流通道。
基于与方法实施例同一发明构思，本申请实施例提供一种装置，具体用于实现上述方法实施例中AR媒体处理网元所执行的方法，该装置的结构如图21所示，包括接收单元2101、发送单元2102以及处理单元2103。
接收单元2101，用于接收互联网协议多媒体子系统IMS核心网元发送的第二请求消息，所述第二请求消息用于请求AR媒体处理网元与第一终端设备之间建立辅助数据通道，所述辅助数据通道用于第一终端设备与所述AR媒体处理网元之间传输AR通信的辅助数据；
发送单元2102，用于向所述IMS核心网元发送第二响应消息，所述第二响应消息用于向所述IMS核心网元指示所述AR媒体处理网元对建立所述辅助数据通道的确认。
在一种可能的实现方式中，接收单元2101，具体用于在所述辅助数据通道建立完成后，所通过所述辅助数据通道接收来自所述第一终端设备的AR辅助数据。
在一种可能的实现方式中，所述AR媒体处理网元与所述第一终端设备之间建立用于传输AR通信内容的媒体流通道。
在一种可能的实现方式中，接收单元2101，用于通过所述媒体流通道接收来自所述第一终端设备的AR通信的媒体流；
处理单元2103，用于根据所述AR辅助数据对所述AR通信的媒体流进行媒体增强处理得到AR媒体流；
发送单元2102，用于通过所述媒体流通道将增强处理后的AR媒体流发送给所述第一终端设备。
在一种可能的实现方式中，发送单元2102，还用于向所述第二终端设备发送所述增强处理后的AR媒体流。
[根据细则91更正 30.03.2022]　
在一种可能的实现方式中，所述AR辅助数据包括虚拟模型的操作数据、目标对象的标注面部表情或者身体动作的识别数据或者位姿数据中的一项或多项。
在一种可能的实现方式中，所述AR媒体处理网元为具备AR处理能力的IMS接入网关或者媒体资源功能MRF。
在一种可能的实现方式中，所述AR媒体处理网元位于所述第一终端设备所属的IMS网络中，或者位于所述第二终端设备所属的IMS网络中。
在一种可能的实现方式中，所述第二请求消息携带所述第一终端设备的用于建立所述辅助数据通道的第一描述参数，所述处理单元2103，具体用于从所述第一描述参数中确定所述AR媒体处理网元所支持的第二描述参数；所述第二响应消息携带所述第二描述参数。
基于与方法实施例同一发明构思，本申请实施例提供一种装置，具体用于实现上述方法实施例中终端设备所执行的方法，该装置的结构如图22所示，包括接收单元2201、发送单元2202以及处理单元2203。
发送单元2202向IMS核心网元发送第一请求消息，所述第一请求消息用于请求建立辅助数据通道；接收单元2201接收所述IMS核心网元发送的第一响应消息，所述第一响应消息用于指示AR媒体处理网元完成建立所述辅助数据通道，所述辅助数据通道用于第 一终端设备与所述AR媒体处理网元之间传输AR通信的辅助数据。
所述第一终端设备与所述AR媒体处理网元之间建立传输AR通信的内容的媒体流通道。
在一种可能的实现方式中，发送单元2202向IMS核心网元发送第三请求消息，所述第三请求消息用于请求建立辅所述媒体流通道；接收单元2201接收IMS核心网元发送的第三响应消息，所述第三响应消息用于向第一终端设备指示所述AR媒体处理网元成功建立所述媒体流通道。
在一种可能的实现方式中，所述第一请求消息还用于请求建立所述媒体流通道；所述第一响应消息还用于指示AR媒体处理网元成功建立所述媒体流通道。
在一种可能的实现方式中，发送单元2202通过辅助数据通道向AR媒体处理网元发送增强现实AR辅助数据，以及通过所述媒体流通道向AR媒体处理网元发送媒体流，所述媒体流是第一终端设备在与第二终端设备进行AR通信时产生的；所述AR辅助数据用于所述AR媒体处理网元对所述AR通信的媒体流进行媒体增强处理；接收单元2201通过所述媒体流通道接收来自所述第一IMS接入网关的增强处理后的AR媒体流。
在一种可能的实现方式中，接收单元2201通过所述媒体流通道接收所述AR媒体处理网元发送的所述第二终端设备的AR媒体流。
在一种可能的实现方式中，所述第一请求消息携带所述第一终端设备的用于建立所述辅助数据通道的第一描述参数；
所述第一响应消息携带所述AR媒体处理网元的用于建立所述辅助数据通道的第二描述参数。
在一种可能的实现方式中，所述AR媒体处理网元位于所述第一终端设备所属的IMS网络中或者位于所述第二终端设备所属的IMS网络中。
本申请实施例中对单元的划分是示意性的，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，另外，在本申请各个实施例中的各功能单元可以集成在一个处理器中，也可以是单独物理存在，也可以两个或两个以上单元集成在一个模块中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。
该集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台终端设备(可以是个人计算机，手机，或者网络设备等)或处理器(processor)执行本申请各个实施例该方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read-only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
在本申请实施例中，所述AR媒体处理网元和所述应用服务器、IMS核心网元以及终端设备均可以采用集成的方式划分各个功能模块的形式来呈现。这里的“模块”可以指特定ASIC，电路，执行一个或多个软件或固件程序的处理器和存储器，集成逻辑电路，和/或其他可以提供上述功能的器件。
在一个简单的实施例中，AR媒体处理网元和所述应用服务器、IMS核心网元以及终端设备可采用图23所示的结构来实现。
如图23所示的装置2300，包括至少一个处理器2301，通信接口2302。可选地，还可以包括存储器2303。
一种可能的实现方式中，当AR媒体处理网元采用图23所示的结构来实现时，图23中的处理器2301可以通过调用存储器2303中存储的计算机执行指令，使得所述AR媒体处理网元可以执行上述任一方法实施例中的所述AR媒体处理网元执行的方法。
处理器2301可以通过通信接口2302与其他设备进行通信。比如，处理器2301通过通信接口2302接收IMS核心网元发送的请求消息，以及向IMS核心网元发送响应消息。
具体的，存储器2303中存储有用于实现图21中的发送单元、接收单元和处理单元的功能的计算机执行指令，图21中的发送单元、接收单元和处理单元的功能/实现过程均可以通过图23中的处理器2301调用存储器2303中存储的计算机执行指令，并结合通信接口2302来实现。
又一种可能的实现方式中，当IMS核心网元采用图23所示的结构来实现时，图23中的处理器2301可以通过调用存储器2303中存储的计算机执行指令，使得所述IMS核心网元可以执行上述任一方法实施例中的所述IMS核心网元(IMS AGW1或者IMS AGW2或者MRF1或者MRF2)执行的方法。
处理器2301可以通过通信接口2302与其他设备进行通信。比如，处理器2301通过通信接口2302接收R媒体处理网元发送的消息，以及向AR媒体处理网元发送的消息。
具体的，存储器2303中存储有用于实现图20中的发送单元、接收单元和处理单元的功能的计算机执行指令，图20中的发送单元、接收单元和处理单元的功能/实现过程均可以通过图23中的处理器2301调用存储器2303中存储的计算机执行指令，并结合通信接口2302来实现。
又一种可能的实现方式中，当终端设备采用图23所示的结构来实现时，图23中的处理器2301可以通过调用存储器2303中存储的计算机执行指令，使得终端设备可以执行上述任一方法实施例中的所述第一终端设备或者第二终端设备执行的方法。
处理器2301可以通过通信接口2302与其他设备进行通信。比如，处理器2301通过通信接口2302接收AR媒体处理网元发送的增强后的媒体流等，以及向AR媒体处理网元发送AR辅助数据和媒体流等。
具体的，存储器2303中存储有用于实现图22中的发送单元、接收单元和处理单元的功能的计算机执行指令，图22中的发送单元、接收单元和处理单元的功能/实现过程均可以通过图23中的处理器2301调用存储器2303中存储的计算机执行指令，并结合通信接口2302来实现。
本申请实施例中不限定上述处理器2301以及存储器2303之间的具体连接介质。本申请实施例在图中以存储器2303和处理器2301之间通过总线2304连接，总线2304在图中以粗线表示，其它部件之间的连接方式，仅是进行示意性说明，并不引以为限。该总线2304可以分为地址总线、数据总线、控制总线等。为便于表示，图23中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。
基于以上实施例，本申请实施例还提供了一种计算机存储介质，该存储介质中存储软件程序，该软件程序在被一个或多个处理器读取并执行时可实现上述任意一个或多个实施例提供业务平台或边缘计算设备执行的方法。所述计算机存储介质可以包括：U盘、移动硬盘、只读存储器、随机存取存储器、磁碟或者光盘等各种可以存储程序代码的介质。
基于以上实施例，本申请实施例还提供了一种芯片，该芯片包括处理器，用于实现上述任意一个或多个实施例所涉及的AR媒体服务器的功能或者终端设备或者应用该服务器的功能。可选地，所述芯片还包括存储器，所述存储器，用于处理器所执行必要的程序指令和数据。该芯片，可以由芯片构成，也可以包含芯片和其他分立器件。
应理解，说明书通篇中提到的“一个实施例”、“一个实现方式”、“一个实施方式”或“一示例”意味着与实施例有关的特定特征、结构或特性包括在本申请的至少一个实施例中。因此，在整个说明书各处出现的“在一个实施例中”、“一个实现方式”、“一个实施方式”或“在一示例中”未必一定指相同的实施例。此外，这些特定的特征、结构或特性可以任意适合的方式结合在一个或多个实施例中。应理解，在本申请的各种实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。
另外，本文中术语“系统”和“网络”在本文中常被可互换使用。本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。本申请涉及的术语“至少一个”，是指一个，或一个以上，即包括一个、两个、三个及以上；“多个”，是指两个，或两个以上，即包括两个、三个及以上。另外，需要理解的是，在本申请的描述中，“第一”、“第二”等词汇，仅用于区分描述的目的，而不能理解为指示或暗示相对重要性，也不能理解为指示或暗示顺序。“以下至少一项(个)”或其类似表达，是指的这些项中的任意组合，包括单项(个)或复数项(个)的任意组合。例如，a,b,或c中的至少一项(个)，可以表示：a,b,c,a-b,a-c,b-c,或a-b-c，其中a,b,c可以是单个，也可以是多个。应理解，在本申请实施例中，“与A相应的B”表示B与A相关联，根据A可以确定B。但还应理解，根据A确定B并不意味着仅仅根据A确定B，还可以根据A和/或其它信息确定B。此外，本申请实施例和权利要求书及附图中的术语“包括”和“具有”不是排他的。例如，包括了一系列步骤或模块的过程、方法、系统、产品或设备没有限定于已列出的步骤或模块，还可以包括没有列出的步骤或模块。
可以理解的是，本申请的实施例中的处理器可以是中央处理单元(central processing unit，CPU)，还可以是其它通用处理器、数字信号处理器(digital signal processor，DSP)、专用集成电路(application specific integrated circuit，ASIC)、现场可编程门阵列(field programmable gate array，FPGA)或者其它可编程逻辑器件、晶体管逻辑器件，硬件部件或者其任意组合。通用处理器可以是微处理器，也可以是任何常规的处理器。
本申请的实施例中的方法步骤可以通过硬件的方式来实现，也可以由处理器执行软件指令的方式来实现。软件指令可以由相应的软件模块组成，软件模块可以被存放于随机存取存储器(random access memory，RAM)、闪存、只读存储器(Read-Only Memory，ROM)、可编程只读存储器(programmable ROM，PROM)、可擦除可编程只读存储器(erasable PROM，EPROM)、电可擦除可编程只读存储器(electrically EPROM，EEPROM)、寄存器、硬盘、移动硬盘、CD-ROM或者本领域熟知的任何其它形式的存储介质中。一种示例性的存储介质耦合至处理器，从而使处理器能够从该存储介质读取信息，且可向该存储介质写入信息。当然，存储介质也可以是处理器的组成部分。处理器和存储介质可以位于ASIC中。另外，该ASIC可以位于网络设备或终端设备中。当然，处理器和存储介质也可以作为分立组件存在于网络设备或终端设备中。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机程序或指令。在计算机上加载和执行所述计算机程序或指令时，全部或部分地执行本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其它可编程装置。所述计算机程序或指令可以存储在计算机可读存储介质中，或者通过所述计算机可读存储介质进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是集成一个或多个可用介质的服务器等数据存储设备。所述可用介质可以是磁性介质，例如，软盘、硬盘、磁带；也可以是光介质，例如，DVD；还可以是半导体介质，例如，固态硬盘(solid state disk，SSD)。
在本申请的各个实施例中，如果没有特殊说明以及逻辑冲突，不同的实施例之间的术语和/或描述具有一致性、且可以相互引用，不同的实施例中的技术特征根据其内在的逻辑关系可以组合形成新的实施例。

Claims (58)

  1. 一种增强现实AR通信的系统，其特征在于，包括互联网协议多媒体子系统IMS核心网元和第一AR媒体处理网元；
    所述IMS核心网元，用于控制所述第一AR媒体处理网元与第一终端设备之间建立第一媒体流通道和辅助数据通道；
    所述第一AR媒体处理网元，用于通过所建立的所述辅助数据通道接收来自第一终端设备的第一AR辅助数据，以及通过所建立的所述媒体流通道接收所述第一终端设备和第二终端设备进行AR通信的第一媒体流；
    所述第一AR媒体处理网元，还用于根据所述第一AR辅助数据对所述媒体流进行增强处理得到第一AR媒体流，并通过所述媒体流通道向所述第一终端设备发送所述第一AR媒体流。
  2. 如权利要求1所述的系统，其特征在于，所述第一AR媒体处理网元为具备AR处理能力的IMS接入网关或者媒体资源功能MRF。
  3. 如权利要求1或2所述的系统，其特征在于，所述第一AR媒体处理网元位于第一终端设备所属的IMS网络中。
  4. 如权利要求3所述的系统，其特征在于，所述系统还包括位于第二终端设备所属的IMS网络中的第二AR媒体处理网元；
    所述第一AR媒体处理网元，还用于向所述第二AR媒体处理网元发送所述AR媒体流；
    所述第二AR媒体处理网元，用于接收所述第一AR媒体处理网元发送的所述AR媒体流，并所述第二终端设备发送所述AR媒体流。
  5. 如权利要求1或2所述的系统，其特征在于，所述第一AR媒体处理网元位于第二终端设备所属的IMS网络中，所述系统还包括位于第一终端设备所属的IMS网络中的第三AR媒体处理网元，第三AR媒体处理网元不具备AR处理能力，所述第一AR媒体处理网元与第一终端设备之间的媒体流通道包括所述第一AR媒体处理网元与第三AR媒体处理网元之间的第一媒体流通道和所述第三AR媒体处理网元与所述第一终端设备之间的第二媒体流通道；
    所述第三AR媒体处理网元，用于通过所述第二媒体流通道接收来自第一终端设备的所述媒体流，通过所述第一媒体流通道向所述第一AR媒体处理网元转发所述媒体流；通过所述第一媒体流通道接收来自第一AR媒体处理网元的AR媒体流，以及通过所述第二媒体流通道向所述第一终端设备发送所述AR媒体流。
  6. 如权利要求1-5任一项所述的系统，其特征在于，所述第一AR媒体处理网元，还用于通过所述辅助数据通道接收来自第一终端设备的目标对象信息；
    所述AR媒体处理网元，还用于从第三方服务器获取所述目标对象信息对应的目标对象。
  7. 如权利要求6所述的系统，其特征在于，所述目标对象为虚拟对象或者虚拟对象的标识；
    所述第一AR媒体处理网元，还用于通过所述辅助数据通道向所述第一终端设备发送所述虚拟对象或者所述虚拟对象的标识。
  8. 如权利要求6所述的系统，其特征在于，所述目标对象为现实背景对象，所述第一AR媒体处理网元，还用于通过所述媒体数据通道接收第一终端设备发送的所述AR通信产生的第二媒体流；
    所述AR媒体处理网元，还用于将所述现实背景对象与第二媒体流渲染合成得到第二AR媒体流；通过所述媒体流通道向所述第一终端设备发送所述第二AR媒体流。
  9. 如权利要求5-8任一项所述的系统，其特征在于，所述系统还包括应用服务器；
    所述应用服务器，用于接收来自AR媒体处理网元的第一请求消息，并向所述第三方服务器转发所述第一请求消息；接收来自所述第三方服务器的第一响应消息，并向所述第一AR媒体处理网元转发所述第二响应消息。
  10. 如权利要求9所述的系统，其特征在于，所述应用服务器配置有所述第三方服务器的地址；
    所述应用服务器，具体用于根据所述第三方服务器的地址向所述第三方服务器转发所述第一请求消息，以及向所述AR媒体处理网元转发所述第二响应消息。
  11. 如权利要求5-8任一项所述的系统，其特征在于，所述应用服务器，还用于在建立所述第一终端设备与所述第一AR媒体处理网元之间的辅助数据通道的流程中，向所述第一AR媒体处理网元发送所述第三方服务器的地址。
  12. 一种增强现实AR通信的方法，其特征在于，包括：
    互联网协议多媒体子系统IMS核心网元接收来自第一终端设备的第一请求消息，所述第一请求消息用于请求建立辅助数据通道，所述辅助数据通道用于第一终端设备与AR媒体处理网元之间传输AR通信的辅助数据；
    所述IMS核心网元控制所述AR媒体处理网元与所述第一终端设备之间建立所述辅助数据通道。
  13. 如权利要求12所述的方法，其特征在于，所述IMS核心网元控制所述AR媒体处理网元与所述第一终端设备之间建立所述辅助数据通道，包括：
    所述IMS核心网元向所述AR媒体处理网元发送第二请求消息，所述第二请求消息用于请求AR媒体处理网元与所述第一终端设备之间建立所述辅助数据通道；
    所述IMS核心网元接收所述AR媒体处理网元发送的第二响应消息，所述第二响应消息用于向IMS核心网元指示所述AR媒体处理网元对建立所述辅助数据通道的确认；
    所述IMS核心网元向所述第一终端设备发送第一响应消息，所述第一响应消息用于向第一终端设备指示所述AR媒体处理网元成功建立所述辅助数据通道。
  14. 如权利要求12或13所述的方法，其特征在于，所述AR媒体处理网元为具备AR处理能力的IMS接入网关或者媒体资源功能MRF。
  15. 如权利要求12-13任一项所述的方法，其特征在于，所述AR媒体处理网元位于所述第一终端设备所属的IMS网络中或者位于所述第二终端设备所属的IMS网络中。
  16. 如权利要求12-15任一项所述的方法，其特征在于，所述第一请求消息和所述第二请求消息携带所述第一终端设备的用于建立所述辅助数据通道的第一描述参数；
    所述第一响应消息和所述第二响应消息携带所述AR媒体处理网元的用于建立所述辅助数据通道的第二描述参数。
  17. 如权利要求13-16任一项所述的方法，其特征在于，所述方法还包括：
    IMS核心网元向AR媒体处理网元发送查询请求，所述查询请求用于询问所述AR媒 体处理网元是否具备AR处理能力；
    IMS核心网元接收所述AR媒体处理网元发送的查询响应，所述查询响应携带所述AR媒体处理网元具备AR处理能力的信息。
  18. 如权利要求12所述的方法，其特征在于，所述方法还包括：
    所述IMS核心网元控制所述AR媒体处理网元与所述第一终端设备之间建立所述媒体流通道，所述媒体流通道用于所述第一终端设备与所述AR媒体处理网元之间传输AR通信内容的媒体流。
  19. 如权利要求18所述的方法，其特征在于，还包括：
    所述IMS核心网元接收来自第一终端设备的第三请求消息，所述第三请求消息还用于请求建立媒体流通道，
    所述IMS核心网元控制所述AR媒体处理网元与所述第一终端设备之间建立所述媒体流通道，包括：
    所述IMS核心网元向所述AR媒体处理网元发送第四请求消息，所述第四请求消息用于请求AR媒体处理网元与所述第一终端设备之间建立所述媒体流通道；
    所述IMS核心网元接收所述AR媒体处理网元发送的第四响应消息，所述第四响应消息用于向IMS核心网元指示所述AR媒体处理网元对建立所述媒体流通道的确认；
    所述IMS核心网元向所述第一终端设备发送第三响应消息，所述第三响应消息用于向第一终端设备指示所述AR媒体处理网元成功建立所述媒体流通道。
  20. 一种基于增强现实AR的通信方法，其特征在于，包括：
    AR媒体处理网元接收互联网协议多媒体子系统IMS核心网元发送的第二请求消息，所述第二请求消息用于请求AR媒体处理网元与第一终端设备之间建立辅助数据通道，所述辅助数据通道用于第一终端设备与所述AR媒体处理网元之间传输AR通信的辅助数据；
    所述AR媒体处理网元向所述IMS核心网元发送第二响应消息，所述第二响应消息用于向所述IMS核心网元指示所述AR媒体处理网元对建立所述辅助数据通道的确认。
  21. 如权利要求20所述的方法，其特征在于，所述方法还包括：
    在所述辅助数据通道建立完成后，所述AR媒体处理网元通过所述辅助数据通道接收来自所述第一终端设备的AR辅助数据。
  22. 如权利要求20或21所述的方法，其特征在于，所述方法还包括：
    所述AR媒体处理网元与所述第一终端设备之间建立用于传输AR通信内容的媒体流通道。
  23. 如权利要求22所述的方法，其特征在于，所述方法还包括：
    所述AR媒体处理网元通过所述媒体流通道接收来自所述第一终端设备的AR通信的媒体流；
    所述AR媒体处理网元根据所述AR辅助数据对所述AR通信的媒体流进行媒体增强处理得到AR媒体流；
    所述AR媒体处理网元通过所述媒体流通道将增强处理后的AR媒体流发送给所述第一终端设备。
  24. 如权利要求23所述的方法，其特征在于，所述方法还包括：
    所述AR媒体处理网元向所述第二终端设备发送所述增强处理后的AR媒体流。
  25. [根据细则91更正 30.03.2022]　
    如权利要求21-24任一项所述的方法，其特征在于，所述AR辅助数据包括虚拟模 型的操作数据、目标对象的标注、面部表情或者身体动作的识别数据或者位姿数据中的一项或多项。
  26. 如权利要求20-25任一项所述的方法，其特征在于，所述AR媒体处理网元为具备AR处理能力的IMS接入网关或者媒体资源功能MRF。
  27. 如权利要求20-26任一项所述的方法，其特征在于，所述AR媒体处理网元位于所述第一终端设备所属的IMS网络中，或者位于所述第二终端设备所属的IMS网络中。
  28. 如权利要求20-27任一项所述的方法，其特征在于，所述第二请求消息携带所述第一终端设备的用于建立所述辅助数据通道的第一描述参数，所述方法还包括：
    所述AR媒体处理网元从所述第一描述参数中确定所述AR媒体处理网元所支持的第二描述参数；
    所述第二响应消息携带所述第二描述参数。
  29. 一种基于增强现实AR通信的方法，其特征在于，包括：
    第一终端设备向IMS核心网元发送第一请求消息，所述第一请求消息用于请求建立辅助数据通道；
    所述第一终端设备接收所述IMS核心网元发送的第一响应消息，所述第一响应消息用于指示AR媒体处理网元完成建立所述辅助数据通道，所述辅助数据通道用于第一终端设备与所述AR媒体处理网元之间传输AR通信的辅助数据。
  30. 如权利要求29所述的方法，其特征在于，所述方法还包括：
    所述第一终端设备与所述AR媒体处理网元之间建立传输AR通信的内容的媒体流通道。
  31. 如权利要求29或30所述的方法，其特征在于，所述第一终端设备与所述AR媒体处理网元之间建立传输AR通信的内容的媒体流通道，包括：
    所述第一终端设备向IMS核心网元发送第三请求消息，所述第三请求消息用于请求建立辅所述媒体流通道；
    所述第一终端设备接收IMS核心网元发送的第三响应消息，所述第三响应消息用于向第一终端设备指示所述AR媒体处理网元成功建立所述媒体流通道。
  32. 如权利要求29所述的方法，其特征在于，所述第一请求消息还用于请求建立所述媒体流通道；所述第一响应消息还用于指示AR媒体处理网元成功建立所述媒体流通道。
  33. 如权利要求30-32任一项所述的方法，其特征在于，所述方法还包括：
    所述第一终端设备通过辅助数据通道向AR媒体处理网元发送增强现实AR辅助数据，以及通过所述媒体流通道向AR媒体处理网元发送媒体流，所述媒体流是第一终端设备在与第二终端设备进行AR通信时产生的；所述AR辅助数据用于所述AR媒体处理网元对所述AR通信的媒体流进行媒体增强处理；
    所述第一终端设备通过所述媒体流通道接收来自所述第一IMS接入网关的增强处理后的AR媒体流。
  34. 如权利要求33所述的方法，其特征在于，所述方法还包括：
    所述第一终端设备通过所述媒体流通道接收所述AR媒体处理网元发送的所述第二终端设备的AR媒体流。
  35. 如权利要求29-34任一项所述的方法，其特征在于，所述第一请求消息携带所述第一终端设备的用于建立所述辅助数据通道的第一描述参数；
    所述第一响应消息携带所述AR媒体处理网元的用于建立所述辅助数据通道的第二描述参数。
  36. 如权利要求29-35任一项所述的方法，其特征在于，所述AR媒体处理网元位于所述第一终端设备所属的IMS网络中或者位于所述第二终端设备所属的IMS网络中。
  37. 一种通信装置，其特征在于，应用于互联网协议多媒体子系统IMS核心网元，包括：
    接收单元，用于接收来自第一终端设备的第一请求消息，所述第一请求消息用于请求建立辅助数据通道，所述辅助数据通道用于第一终端设备与AR媒体处理网元之间传输AR通信的辅助数据；
    处理单元，用于控制所述AR媒体处理网元与所述第一终端设备之间建立所述辅助数据通道。
  38. 如权利要求37所述的装置，其特征在于，还包括：
    发送单元，用于向所述AR媒体处理网元发送第二请求消息，所述第二请求消息用于请求AR媒体处理网元与所述第一终端设备之间建立所述辅助数据通道；
    所述接收单元，还用于接收所述AR媒体处理网元发送的第二响应消息，所述第二响应消息用于向IMS核心网元指示所述AR媒体处理网元对建立所述辅助数据通道的确认；
    所述发送单元，还用于向所述第一终端设备发送第一响应消息，所述第一响应消息用于向第一终端设备指示所述AR媒体处理网元成功建立所述辅助数据通道。
  39. 如权利要求37-38任一项所述的装置，其特征在于，所述第一请求消息和所述第二请求消息携带所述第一终端设备的用于建立所述辅助数据通道的第一描述参数；
    所述第一响应消息和所述第二响应消息携带所述AR媒体处理网元的用于建立所述辅助数据通道的第二描述参数。
  40. 如权利要求38-39任一项所述的装置，其特征在于，所述发送单元，还用于向AR媒体处理网元发送查询请求，所述查询请求用于询问所述AR媒体处理网元是否具备AR处理能力；
    所述接收单元，还用于接收所述AR媒体处理网元发送的查询响应，所述查询响应携带所述AR媒体处理网元具备AR处理能力的信息。
  41. 如权利要求37所述的装置，其特征在于，所述处理单元，还用于控制所述AR媒体处理网元与所述第一终端设备之间建立所述媒体流通道，所述媒体流通道用于所述第一终端设备与所述AR媒体处理网元之间传输AR通信内容的媒体流。
  42. 如权利要求41所述的装置，其特征在于，所述接收单元，还用于接收来自第一终端设备的第三请求消息，所述第三请求消息还用于请求建立媒体流通道，
    所述发送单元，还用于向所述AR媒体处理网元发送第四请求消息，所述第四请求消息用于请求AR媒体处理网元与所述第一终端设备之间建立所述媒体流通道；
    所述接收单元，还用于接收所述AR媒体处理网元发送的第四响应消息，所述第四响应消息用于向IMS核心网元指示所述AR媒体处理网元对建立所述媒体流通道的确认；
    所述发送单元，还用于向所述第一终端设备发送第三响应消息，所述第三响应消息用于向第一终端设备指示所述AR媒体处理网元成功建立所述媒体流通道。
  43. 一种通信装置，其特征在于，应用于AR媒体处理网元，包括：
    接收单元，用于接收互联网协议多媒体子系统IMS核心网元发送的第二请求消息，所 述第二请求消息用于请求AR媒体处理网元与第一终端设备之间建立辅助数据通道，所述辅助数据通道用于第一终端设备与所述AR媒体处理网元之间传输AR通信的辅助数据；
    发送单元，用于向所述IMS核心网元发送第二响应消息，所述第二响应消息用于向所述IMS核心网元指示所述AR媒体处理网元对建立所述辅助数据通道的确认。
  44. 如权利要求43所述的装置，其特征在于，所述接收单元，还用于在所述辅助数据通道建立完成后，通过所述辅助数据通道接收来自所述第一终端设备的AR辅助数据。
  45. 如权利要求43或44所述的装置，其特征在于，还包括：
    处理单元，用于在所述AR媒体处理网元与所述第一终端设备之间建立用于传输AR通信内容的媒体流通道。
  46. 如权利要求45所述的装置，其特征在于，所述接收单元，还用于通过所述媒体流通道接收来自所述第一终端设备的AR通信的媒体流；
    所述处理单元，还用于根据所述AR辅助数据对所述AR通信的媒体流进行媒体增强处理得到AR媒体流；
    所述发送单元，还用于通过所述媒体流通道将增强处理后的AR媒体流发送给所述第一终端设备。
  47. 如权利要求46所述的装置，其特征在于，所述发送单元，还用于向所述第二终端设备发送所述增强处理后的AR媒体流。
  48. 如权利要求43-47任一项所述的装置，其特征在于，所述第二请求消息携带所述第一终端设备的用于建立所述辅助数据通道的第一描述参数；
    处理单元，用于从所述第一描述参数中确定所述AR媒体处理网元所支持的第二描述参数；
    所述第二响应消息携带所述第二描述参数。
  49. 一种通信装置，其特征在于，应用于第一终端设备，包括：
    发送单元，用于向IMS核心网元发送第一请求消息，所述第一请求消息用于请求建立辅助数据通道；
    接收单元，用于接收所述IMS核心网元发送的第一响应消息，所述第一响应消息用于指示AR媒体处理网元完成建立所述辅助数据通道，所述辅助数据通道用于第一终端设备与所述AR媒体处理网元之间传输AR通信的辅助数据。
  50. 如权利要求49所述的装置，其特征在于，还包括：
    处理单元，用于在所述第一终端设备与所述AR媒体处理网元之间建立传输AR通信的内容的媒体流通道。
  51. 如权利要求49或50所述的装置，其特征在于，所述发送单元，还用于向IMS核心网元发送第三请求消息，所述第三请求消息用于请求建立辅所述媒体流通道；
    所述接收单元，还用于接收IMS核心网元发送的第三响应消息，所述第三响应消息用于向第一终端设备指示所述AR媒体处理网元成功建立所述媒体流通道。
  52. 如权利要求49所述的装置，其特征在于，所述第一请求消息还用于请求建立所述媒体流通道；所述第一响应消息还用于指示AR媒体处理网元成功建立所述媒体流通道。
  53. 如权利要求50-52任一项所述的装置，其特征在于，所述发送单元，还用于通过辅助数据通道向AR媒体处理网元发送增强现实AR辅助数据，以及通过所述媒体流通道向AR媒体处理网元发送媒体流，所述媒体流是第一终端设备在与第二终端设备进行AR通 信时产生的；所述AR辅助数据用于所述AR媒体处理网元对所述AR通信的媒体流进行媒体增强处理；
    所述接收单元，还用于通过所述媒体流通道接收来自所述第一IMS接入网关的增强处理后的AR媒体流。
  54. 如权利要求53所述的装置，其特征在于，所述接收单元，还用于通过所述媒体流通道接收所述AR媒体处理网元发送的所述第二终端设备的AR媒体流。
  55. 如权利要求49-54任一项所述的装置，其特征在于，所述第一请求消息携带所述第一终端设备的用于建立所述辅助数据通道的第一描述参数；
    所述第一响应消息携带所述AR媒体处理网元的用于建立所述辅助数据通道的第二描述参数。
  56. 一种通信装置，其特征在于，所述装置包括通信接口和处理器；
    所述通信接口，用于收发信号；
    所述处理器，用于通过所述通信接口收发信号并实现如权利要求12至19中任一项所述的方法。
  57. 一种通信装置，其特征在于，所述装置包括通信接口和处理器；
    所述通信接口，用于收发信号；
    所述处理器，用于通过所述通信接口收发信号并实现如权利要求20至28中任一项所述的方法。
  58. 一种通信装置，其特征在于，所述装置包括通信接口和处理器；
    所述通信接口，用于收发信号；
    所述处理器，用于通过所述通信接口收发信号并实现如权利要求29至36中任一项所述的方法。
PCT/CN2022/083191 2021-03-27 2022-03-25 一种增强现实通信的方法、装置及系统 WO2022206626A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP22778808.0A EP4304166A4 (en) 2021-03-27 2022-03-25 AUGMENTED REALITY COMMUNICATION METHOD, APPARATUS AND SYSTEM
JP2023559102A JP2024511504A (ja) 2021-03-27 2022-03-25 拡張現実通信方法、装置、及びシステム
US18/474,721 US20240022619A1 (en) 2021-03-27 2023-09-26 Augmented reality communication method, apparatus, and system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110329252.9 2021-03-27
CN202110329252.9A CN115134335A (zh) 2021-03-27 2021-03-27 一种增强现实通信的方法、装置及系统

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/474,721 Continuation US20240022619A1 (en) 2021-03-27 2023-09-26 Augmented reality communication method, apparatus, and system

Publications (1)

Publication Number Publication Date
WO2022206626A1 true WO2022206626A1 (zh) 2022-10-06

Family

ID=83374764

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/083191 WO2022206626A1 (zh) 2021-03-27 2022-03-25 一种增强现实通信的方法、装置及系统

Country Status (5)

Country Link
US (1) US20240022619A1 (zh)
EP (1) EP4304166A4 (zh)
JP (1) JP2024511504A (zh)
CN (1) CN115134335A (zh)
WO (1) WO2022206626A1 (zh)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230044527A1 (en) * 2021-07-20 2023-02-09 Samsung Electronics Co, Ltd. System and methods for handling immersive service in ip multimedia subsystem and mission critical services
US20240054688A1 (en) * 2022-08-11 2024-02-15 Qualcomm Incorporated Enhanced Dual Video Call with Augmented Reality Stream
WO2024184397A1 (en) * 2023-03-06 2024-09-12 Telefonaktiebolaget Lm Ericsson (Publ) Network nodes, user equipment and methods performed therein

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110221962A1 (en) * 2010-03-10 2011-09-15 Microsoft Corporation Augmented reality via a secondary channel
CN106803921A (zh) * 2017-03-20 2017-06-06 深圳市丰巨泰科电子有限公司 基于ar技术的即时音视频通信方法及装置
WO2019017885A1 (en) * 2017-07-17 2019-01-24 Nokia Solutions And Networks Oy NETWORK-ASSISTED INCREASE IN REALITY
CN110324284A (zh) * 2018-03-30 2019-10-11 华为技术有限公司 接入ims的方法和通信装置

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9113030B2 (en) * 2013-07-25 2015-08-18 Verizon Patent And Licensing Inc. Multimedia-enhanced emergency call systems
US20230007706A1 (en) * 2019-12-03 2023-01-05 Telefonaktiebolaget Lm Ericsson (Publ) First Network Node, Second Wireless Device and Methods Performed Therein

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110221962A1 (en) * 2010-03-10 2011-09-15 Microsoft Corporation Augmented reality via a secondary channel
CN106803921A (zh) * 2017-03-20 2017-06-06 深圳市丰巨泰科电子有限公司 基于ar技术的即时音视频通信方法及装置
WO2019017885A1 (en) * 2017-07-17 2019-01-24 Nokia Solutions And Networks Oy NETWORK-ASSISTED INCREASE IN REALITY
CN110324284A (zh) * 2018-03-30 2019-10-11 华为技术有限公司 接入ims的方法和通信装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4304166A4 *

Also Published As

Publication number Publication date
EP4304166A4 (en) 2024-05-29
EP4304166A1 (en) 2024-01-10
CN115134335A (zh) 2022-09-30
JP2024511504A (ja) 2024-03-13
US20240022619A1 (en) 2024-01-18

Similar Documents

Publication Publication Date Title
WO2022206626A1 (zh) 一种增强现实通信的方法、装置及系统
US10778731B2 (en) Communications methods, apparatus and systems for conserving media resource function resources
US9300696B2 (en) Method of sharing one or more media in a session between terminals
WO2023071915A1 (zh) 业务设置方法和装置、存储介质及电子设备
US20130282820A1 (en) Method and System for an Optimized Multimedia Communications System
EP2991318B1 (en) Hybrid cloud architecture for media communications
KR20220164777A (ko) 라이브 업링크 스트리밍 프레임워크(flus) 및 5g 애플리케이션 기능(af)을 사용한 네트워크 기반 미디어 처리(nbmp) 배포
WO2009033392A1 (fr) Procédé et système de multimédia à la demande, terminal multimédia, système multimédia et serveur mandataire de service
WO2020001610A1 (zh) 一种视频拼接方法及装置
US20240022632A1 (en) Augmented reality communication method, apparatus, and system
US20240292363A1 (en) Information transmission method and apparatus
WO2017197989A1 (zh) 一种信令架构及其实现呼叫处理的方法和装置
US10771510B2 (en) IMS application control protocol
CN104994067A (zh) Sip网络访问rtsp监控网络的系统及方法
WO2011015111A1 (zh) 一种实时流协议代理转发数据的方法、装置及系统
da Silva et al. Cross-layer multiuser session control for optimized communications on SDN-based cloud platforms
WO2023005316A1 (zh) 通信方法、信令控制网元、媒体控制网元及通信系统
US8606243B2 (en) Mobile network system and guidance message providing method
WO2023151431A1 (zh) 一种通信方法、装置及通信系统
EP4236326A1 (en) A method, an apparatus and a computer program product for transmission of immersive media
US20240236166A1 (en) Multimedia cloud service method, communication cloud system and computer readable storage medium
WO2023185854A1 (zh) 一种信息处理方法、装置、通信设备和存储介质
WO2024032213A1 (zh) 通信方法、装置和系统
US20160295249A1 (en) Session Setup Method and Apparatus, and Session Content Delivery Method and Apparatus
CN117014326A (zh) 一种通信方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22778808

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023559102

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 202347066287

Country of ref document: IN

WWE Wipo information: entry into national phase

Ref document number: 2022778808

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2022778808

Country of ref document: EP

Effective date: 20231004

NENP Non-entry into the national phase

Ref country code: DE