WO2021088690A1 - Procédé et appareil de communication basés sur la réalité augmentée - Google Patents

Procédé et appareil de communication basés sur la réalité augmentée Download PDF

Info

Publication number
WO2021088690A1
WO2021088690A1 PCT/CN2020/124167 CN2020124167W WO2021088690A1 WO 2021088690 A1 WO2021088690 A1 WO 2021088690A1 CN 2020124167 W CN2020124167 W CN 2020124167W WO 2021088690 A1 WO2021088690 A1 WO 2021088690A1
Authority
WO
WIPO (PCT)
Prior art keywords
media
terminal device
server
session
media server
Prior art date
Application number
PCT/CN2020/124167
Other languages
English (en)
Chinese (zh)
Inventor
高扬
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2021088690A1 publication Critical patent/WO2021088690A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
    • H04L9/40—Network security protocols
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H04N7/141—Systems for two-way working between two video terminals, e.g. videophone
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/1066—Session management
    • H04L65/1069—Session establishment or de-establishment
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
    • H04N21/4312—Generation of visual interfaces for content selection or interaction; Content or additional data rendering involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
    • H04N21/4788—Supplemental services, e.g. displaying phone caller identification, shopping application communicating with other users, e.g. chatting
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working

Definitions

  • the embodiments of the present application relate to the field of communication technologies, and in particular, to a communication method and device based on augmented reality.
  • Long-term evolution voice bearer (voice over long term evolution, VoLTE) is an end-to-end voice solution based on all-IP conditions on the 4th generation (4G) network. VoLTE makes the connection waiting time shorter when communicating between users, and the voice and video call quality is higher.
  • Augmented reality is a technology that ingeniously integrates virtual information with the real world. It uses a variety of technical methods such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, and sensing, and generates After the virtual information such as text, image, 3D model, music, video, etc. is simulated and simulated, it is applied to the real world, and the two kinds of information complement each other, thus realizing the "enhancement" of the real world. Augmented reality technology can not only effectively reflect the content of the real world, but also promote the display of virtual information content. There is currently no effective way to integrate AR into voice and video calls.
  • the embodiments of the present application provide a communication method and device based on augmented reality, and provide an implementation manner of integrating AR into a voice and video call, thereby improving user experience.
  • an embodiment of the present application provides a communication method based on augmented reality.
  • the method includes: an AR media server receives a first session creation request sent by an application server, and the first session creation request is used to request to create a communication with a first terminal device.
  • a first media session between corresponding first session border controllers (session border controllers, SBC); a first SBC corresponding to a terminal device, in other words, the first SBC to which the first terminal device belongs.
  • the AR media server sends a first session creation response to the application server, where the first session creation response is used to indicate that the first media session is successfully created; further, the AR media server receives the second session creation request sent by the application server.
  • the second session creation request is used to request the creation of a second media session between the second SBC corresponding to the second terminal device; then, the AR media server sends a second session creation response to the application server, and the second session creation response is used to indicate The second media session is successfully created.
  • the media stream channel between the first SBC and the AR media server and between the AR media server and the second SBC is opened.
  • the AR media server performs media enhancement processing, which realizes the integration of AR processing in the video call process and improves user experience.
  • the first session creation response carries the first media description protocol (session description protocol, SDP) information of the AR media server, and the first SDP information includes the information used for the creation between the first SBC and the AR media server The parameters of the media stream channel of the first media session.
  • the AR media server carries the first SDP information of the AR media server in the first session creation response, which is used to negotiate a media stream channel with the first SBC, so as to realize the establishment of a media stream channel between the first SBC and the AR media server.
  • the second session creation response carries the second SDP information of the AR media server, and the second SDP information includes the media stream channel used to create the second media session between the second SBC and the AR media server. parameter.
  • the AR media server carries the second SDP information of the AR media server in the second session creation response, which is used to negotiate a media stream channel with the second SBC, so as to establish a media stream channel between the second SBC and the AR media server.
  • the application server carries the service indication in the second session creation request sent to the AR media server; thus, after the AR media server receives the second session creation request, it can communicate with the first terminal device according to the service indication.
  • the media stream transmitted between the second terminal devices undergoes media enhancement processing.
  • the application server carries an association indication in the second session creation request sent to the AR media server; thus, after the AR media server receives the second session creation request, it can associate the first media session with the first media session according to the association indication.
  • the second media session is a possible design.
  • the above method may further include: the AR media server receives the first media stream sent from the first SBC through the first media session; then, the AR media server performs media enhancement processing on the first media stream to obtain the first media stream. Two media streams; and send the second media stream to the second SBC through the second media session.
  • the above method may further include: the AR media server receives the third media stream from the SBC to which the first terminal device belongs through the first media session, and receives the third media stream from the SBC to which the second terminal device belongs according to the first media session.
  • the second media session sends the fourth media stream; then, the AR media server performs media enhancement processing on the third media stream and the fourth media stream; and sends the media enhancement processed media stream to the first terminal device and/or the second terminal device.
  • the above method may further include: the AR media server receives the fifth media stream sent from the first SBC through the first media session, and receives the virtual model from the application server; The model and the fifth media stream are subjected to media enhancement processing; the media stream after the media enhancement processing is sent to the first terminal device and/or the second terminal device.
  • the AR media server also receives a model operation instruction from the application server.
  • the model operation instruction is used to instruct the AR media server to perform operations on the virtual model; further, the AR media server is working on the virtual model and the fifth
  • the media stream performs media enhancement processing it can be implemented in the following ways: perform processing operations on the virtual model according to the model operation instructions, and perform media enhancement processing on the virtual model and the fifth media stream after the processing operations, such as overlay, or according to the fifth media stream.
  • the media stream performs operations such as rendering the virtual model.
  • an auxiliary transmission channel can be established between the AR media server and the terminal device through the application server.
  • the establishment of an auxiliary transmission channel between the AR media server and the terminal device can be achieved in the following ways: the AR media server receives the establishment request from the application server, the establishment request is used to request the establishment of the auxiliary transmission channel, and the auxiliary transmission channel is used for the first terminal device and the AR
  • the auxiliary media stream is transmitted between the media servers, and the establishment request carries the address of the first terminal device; the AR media server sends a first establishment response to the application server, and the first establishment response carries the address of the AR media server.
  • the AR media server receives the auxiliary media stream sent by the first terminal device according to the channel transmission address of the AR media server.
  • the auxiliary media stream is used to assist the AR media server to perform media enhancement processing.
  • the above method may further include: the AR media server receives the sixth media stream sent from the first SBC; and then the AR media server responds to the auxiliary media stream and the sixth media stream.
  • the media stream undergoes media enhancement processing; and the media stream after the media enhancement processing is sent to the first terminal device and/or the second terminal device.
  • the auxiliary media stream includes one or more of point cloud data, spatial data, user-view video, or virtual model.
  • an embodiment of the present application also provides a communication method based on augmented reality, including: the application server receives a call request from the first terminal device, and sends a first session creation request to the AR media server. For requesting to create a first media session between the session border controller SBC corresponding to the first terminal device; the application server receives the first session creation response sent by the AR media server, and the first session creation response is used to instruct the creation of the first media session Success; after receiving the first session creation response, the application server sends a second session creation request to the AR media server.
  • the second session creation request is used to request the creation of a second media session between the second SBC corresponding to the second terminal device;
  • the application server receives the second session creation response sent by the AR media server, where the second session creation response is used to indicate that the second media session is successfully created.
  • the first session creation response carries the first media description protocol SDP information of the AR media server, and the first SDP information includes the media used to create the first media session between the first SBC and the AR media server Parameters of the streaming channel; the above method further includes: the application server sends the first SDP information of the AR media server to the first SBC.
  • the above-mentioned SDP may include parameters such as address information, media stream type, and supported encoding and decoding formats.
  • the second session creation response carries the second SDP information of the AR media server, and the second SDP information includes the media stream channel used to create the second media session between the second SBC and the AR media server.
  • the above method also includes: the application server sends the second SDP information to the second SBC.
  • the second session creation request carries a service indication
  • the service indication is used to instruct the AR media server to perform media enhancement processing for the media stream transmitted between the first terminal device and the second terminal device.
  • the second session creation request carries an association indication
  • the association indication is used to instruct the AR media server to associate the first media session with the second media session.
  • the application server when the application server receives the AR video call request sent by the first SBC, it triggers the sending of the first session creation request to the AR media server.
  • the application server receives the AR video enhancement request sent by the first terminal device; the application server sends the AR video enhancement request to the first terminal device through the first SBC, and the AR video enhancement request is used to indicate the first terminal device.
  • the terminal device initiates an AR video call, and then when the first terminal device initiates the AR video call, the application server triggers to send the first session creation request to the AR media server.
  • the above method may further include: the application server receives an AR interface operation instruction triggered by the first terminal device; and the application server sends an AR interface operation instruction (such as a model operation instruction) to the AR media server.
  • the application server receives a model operation instruction triggered by the first terminal device; the application server sends a model operation instruction to the AR media server, and the model operation instruction is used to instruct the AR media server to perform an operation on the received virtual model.
  • the above method may further include: the application server receives an establishment request triggered by the first terminal device, the establishment request is used to request the establishment of an auxiliary transmission channel, and the auxiliary transmission channel is used between the first terminal device and the AR media server.
  • the auxiliary media stream is transmitted between the establishment request and the address of the first terminal device; the application server sends the establishment request to the AR media server; the application server receives the establishment response sent by the AR media server, and the establishment response carries the address of the AR media server;
  • the establishment response is sent to the first terminal device.
  • an embodiment of the present application also provides an augmented reality-based communication method, which may include: a first terminal device receives an augmented reality AR media stream from a first session boundary controller SBC corresponding to the first terminal device, The AR media stream includes the video of the AR interface displayed on the first terminal device; the first terminal device sends an AR interface operation instruction to the application server in response to an operation triggered by the user on the displayed AR interface, and the AR interface operation instruction is used to indicate the AR interface Media enhancement processing performed by the media stream; the first terminal device receives the enhanced AR media stream sent by the first SBC.
  • the user can operate on the AR interface, and then display the enhanced AR interface to provide convenience for the user.
  • an auxiliary transmission channel is established between the first terminal device and the AR media server, and the method further includes: the first terminal device sends the auxiliary media stream to the AR media server through the auxiliary transmission channel, and the auxiliary media stream is used for AR The media server performs media enhancement processing.
  • the first terminal device and the AR media server may establish an auxiliary transmission channel in the following manner: before sending the auxiliary media stream to the AR media server through the auxiliary transmission channel, the first terminal device sends an establishment request to the application server, The establishment request is used to request the establishment of an auxiliary transmission channel, and the establishment request carries the address of the first terminal device; the first terminal device receives the establishment response sent by the application server, and the establishment response carries the address of the AR media server.
  • the first terminal device sends the auxiliary media stream to the AR media server through the auxiliary transmission channel, including: the first terminal device sends the auxiliary media server to the AR media server according to the address of the first terminal device and the address of the AR media server Media stream.
  • the auxiliary media stream includes one or more of point cloud video stream, spatial video data, user-view video, or virtual model.
  • an embodiment of the present application also provides a device, which is applied to an AR media server, and the beneficial effects can be referred to the description of the first aspect, which will not be repeated here.
  • the device has the function of realizing the behavior in the method example of the first aspect described above.
  • the function can be realized by hardware, or by hardware executing corresponding software.
  • the hardware or software includes one or more modules corresponding to the above-mentioned functions.
  • the structure of the device may include a sending unit, a receiving unit, and a processing unit. These units can perform the corresponding functions in the above-mentioned method example of the first aspect. For details, please refer to the detailed description in the method example, here Do not repeat it.
  • an embodiment of the present application also provides a device, which is applied to an application server, and the beneficial effects can be referred to the description of the second aspect, which will not be repeated here.
  • the device has the function of realizing the behavior in the method example of the second aspect.
  • the function can be realized by hardware, or by hardware executing corresponding software.
  • the hardware or software includes one or more modules corresponding to the above-mentioned functions.
  • the structure of the device includes a receiving unit, a processing unit, and a sending unit. These units can perform the corresponding functions in the method example of the second aspect. For details, please refer to the detailed description in the method example. Do repeats.
  • the embodiments of the present application also provide a device, which is applied to the terminal device, and the beneficial effects can be referred to the description of the third aspect, which will not be repeated here.
  • the device has the function of realizing the behavior in the method example of the third aspect.
  • the function can be realized by hardware, or by hardware executing corresponding software.
  • the hardware or software includes one or more modules corresponding to the above-mentioned functions.
  • the structure of the device includes a sending unit, a receiving unit, and a processing unit. These units can perform the corresponding functions in the method example of the third aspect. For details, please refer to the detailed description in the method example. Do repeats.
  • an embodiment of the present application also provides a device, which is applied to the AR media server, and the beneficial effects can be referred to the description of the first aspect, which will not be repeated here.
  • the structure of the communication device includes a processor and a communication interface, and may also include a memory.
  • the processor is configured to support the AR media server to perform the corresponding functions in the above-mentioned method of the first aspect through the communication interface.
  • the memory is coupled with the processor, and it stores program instructions and data necessary for the communication device.
  • an embodiment of the present application also provides a device, which is applied to the application server, and the beneficial effects can be referred to the description of the second aspect, which will not be repeated here.
  • the structure of the device includes a processor and a communication interface, and may also include a memory.
  • the processor is configured to support the application server to perform the corresponding function in the above-mentioned second aspect method through the communication interface.
  • the memory is coupled with the processor, and it stores program instructions and data necessary for the device.
  • an embodiment of the present application also provides a device, which is applied to the first terminal device, and the beneficial effects can be referred to the description of the third aspect, which is not repeated here.
  • the structure of the apparatus includes a processor and a communication interface, and may also include a memory.
  • the processor is configured to support the terminal device to perform the corresponding function in the method of the third aspect through the communication interface.
  • the memory is coupled with the processor, and it stores program instructions and data necessary for the device.
  • an embodiment of the present application also provides a device, which is applied to an SBC (first SBC or second SBC).
  • the structure of the communication device includes a processor and a communication interface, and may also include a memory.
  • the processor is configured to support the SBC to perform the corresponding function of the first SBC or the second SBC in any of the above aspects through the communication interface.
  • the memory is coupled with the processor, and it stores program instructions and data necessary for the communication device.
  • the present application also provides a computer-readable storage medium having instructions stored in the computer-readable storage medium, which when run on a computer, cause the computer to execute the methods described in the above aspects.
  • this application also provides a computer program product containing instructions, which when run on a computer, causes the computer to execute the methods described in the above aspects.
  • the present application also provides a computer chip connected to a memory, and the chip is configured to read and execute a software program stored in the memory, and execute the methods described in the foregoing aspects.
  • FIG. 1 is a schematic diagram of a possible AR communication system architecture in an embodiment of this application
  • FIG. 2 is a schematic diagram of another possible AR communication system architecture in an embodiment of this application.
  • FIG. 3 is a schematic diagram of another possible AR communication system architecture in an embodiment of this application.
  • FIG. 4 is a schematic diagram of a display interface of a possible terminal device in an embodiment of this application.
  • FIG. 5 is a schematic diagram of another possible AR communication system architecture in an embodiment of this application.
  • FIG. 6 is a schematic diagram of another possible AR communication system architecture in an embodiment of this application.
  • FIG. 7 is a schematic diagram of input and output of an AR media server in an embodiment of the application.
  • Example 8 is a schematic diagram of the input and output of the AR media server in Example 1 of the embodiments of the application;
  • FIG. 9 is a schematic diagram of the input and output of the AR media server in Example 2 of the embodiment of the application.
  • FIG. 10 is a schematic diagram of input and output of an AR media server in Example 3 of an embodiment of the application.
  • FIG. 11 is a schematic diagram of input and output of another AR media server in Example 3 of an embodiment of the application.
  • FIG. 12 is a schematic diagram of the input and output of the AR media server in Example 4 of the embodiment of the application;
  • FIG. 13 is a schematic flowchart of a possible AR-based communication method in an embodiment of this application.
  • FIG. 14A is a schematic flowchart of another possible AR-based communication method in an embodiment of this application.
  • 14B is a schematic flowchart of another possible AR-based communication method in an embodiment of this application.
  • 15 is a schematic flowchart of another possible AR-based communication method in an embodiment of this application.
  • FIG. 16 is a schematic diagram of a method for a terminal device to trigger an AR video enhancement process in an embodiment of the application
  • FIG. 17 is a schematic flowchart of establishing an auxiliary transmission channel between a terminal device and an AR media server in an embodiment of the application;
  • Figures 18-22 are schematic diagrams of the structure of the device provided by this application.
  • the present application provides an AR-based communication method and device, and provides an implementation method of integrating AR into voice and video calls, thereby improving user experience.
  • Voice and video calls can be but not limited to using VoLTE, and can also be applied to voice and video calls provided by future technologies.
  • the communication system includes one or more session border controllers (SBC) and one or more AR media servers.
  • the AR media server may also be called an AR media enabler (AR media enabler).
  • AR media enabler Two terminal devices can conduct voice and video calls through the communication system, and during the voice and video calls, the AR media enabler performs media enhancement processing on the media streams generated during the voice and video calls.
  • AR media enabler's specific and strong image processing functions and data calculation functions can use AR technology to perform logical operations, screen rendering, virtual scene synthesis and other operations on the received media stream.
  • the AR media server can be deployed in the form of a container service.
  • the AR media server can also be implemented by one or a virtual machine.
  • the AR media server can also include one or a processor, or be implemented by one or more computers, such as a super multi-core computer, a computer deployed with a graphics processing unit (GPU) cluster, a large distributed computer, and hardware resources Pooled cluster computers and so on.
  • SBC is used to manage or control the session of the terminal device.
  • the SBC includes a signaling plane function and a media plane function. For example, it can be used to receive media streams from terminal devices under its management, and send the media streams received from the terminal devices to the AR media server.
  • the AR media server is used to perform media enhancement processing on the received upstream media stream to obtain a downstream video stream.
  • the downstream video stream can be sent by the AR media server to the corresponding terminal device through the SBC.
  • the terminal device may be a device equipped with a camera and a video call function.
  • the terminal device may be a wearable device (such as an electronic watch), and the terminal device may also be a device such as a mobile phone or a tablet computer.
  • the embodiments of the present application do not impose special restrictions on the specific form of the terminal device.
  • two SBCs are taken as an example, namely, a first SBC and a second SBC.
  • the first SBC is used to manage the first terminal device
  • the second SBC is used to manage the second terminal device.
  • different terminal devices can also have the same SBC to manage.
  • the third terminal device described in FIG. 1 is managed by the first SBC.
  • the first SBC is used to receive the first media stream from the first terminal device, and send the received first media stream to the AR media server, and then
  • the AR media server performs media enhancement processing on the received upstream media stream, and the upstream media stream includes the first media stream.
  • the AR media server performs media enhancement processing on the upstream media stream to obtain the downstream media stream
  • the AR media server sends the downstream media stream to the second SBC
  • the second SBC sends it to the second terminal device.
  • the communication system may also include an application server.
  • the application server is used to establish an AR video call triggered by the terminal device. For example, taking the AR video call between the first terminal device and the second terminal device as an example, the application server receives an AR interface operation instruction from the first terminal device, and The AR interface operation instructions are sent to the AR media server, and the AR interface operation instructions are used to instruct the user to perform operations on the AR interface displayed by the first terminal device; and the AR media server is specifically used to perform the received uplink media according to the AR interface operation instructions. Streaming for media enhancement processing.
  • the application server may include a media plug-in service function, and may also be referred to as a plug-in server.
  • the application server also includes application service functions.
  • the media plug-in service function is used to interact with the terminal device, receive the AR interface operation instruction triggered from the terminal device, and send the AR interface operation instruction to the application service function.
  • the application service function is used to interact with the AR media server, and send the AR interface operation instructions sent by the media plug-in service function to the AR media server.
  • the AR media server performs media enhancement processing on the received upstream media stream, it performs media enhancement processing on the received upstream media stream according to the AR interface operation instruction.
  • media plug-in service function and AS can be deployed separately or combined during deployment.
  • the media plug-in service function and AS can be implemented through one device, or through one or more virtual machines.
  • the AR interface operation instruction may be an instruction used to indicate the processing method, which does not have a great requirement for real-time performance, such as a beauty operation.
  • the AR media server performs an operation on each video frame in the received media stream according to the AR interface operation instruction. Perform beauty processing on the face.
  • the AR interface operation instructions may also include instructions for instructing the user to operate in real time, such as model operation instructions for instructing a rotating model, a zooming model, and so on.
  • model operation instructions for instructing a rotating model, a zooming model, and so on are referred to as non-real-time operation instructions
  • instructions for instructing users to operate in real time are referred to as real-time operation instructions.
  • the application server can be deployed at the central node in the system.
  • the first SBC is deployed at the first edge node in the system
  • the AR media server is deployed at the central node in the system.
  • the second SBC is deployed at the second edge node of the system.
  • the edge node is closer to the users of the terminal device, and provides edge computing services, forwarding services, etc. for these users, reducing response delay and bandwidth cost, and reducing the pressure on the central node.
  • the central node and the edge node may be deployed on the cloud.
  • the central node may be referred to as the central cloud
  • the edge node may be referred to as the edge cloud.
  • the edge node may also be a mobile edge computing (Mobile Edge Computing, MEC) node.
  • the central cloud can also deploy the IP Multimedia Subsystem (IMS) core network (core).
  • the IMS core may include a call session control function (CSCF) and a home subscriber server (home subscriber server, HSS).
  • CSCF is the call control center of IMS core, which implements user access, authentication, session routing, and service triggering functions on the IP transmission platform.
  • the CSCF may include one of the serving-call session control function (S-CSCF), proxy-CSCF (Proxy CSCF, P-CSCF), query-CSCF (Interrogating-CSCF, I-CSCF), or Multiple.
  • HSS is used to record the user's subscription data (such as user information, business data).
  • the SBC provides boundary control functions between the access network and the IMS core network and between the IMS core network, and can provide functions such as access control, quality of service control, and firewall traversal.
  • an AR control (may be referred to as an end-side Plugin) can be deployed in the terminal device.
  • the AR control is used for message interaction with the media plug-in service function on the network side.
  • the AR control can also establish an auxiliary transmission channel with the AR media enabler.
  • the auxiliary transmission channel is used for the first terminal device to send the auxiliary media stream to the AR media enabler.
  • the auxiliary media stream may include one or more of point cloud data, spatial data (may also be referred to as spatial pose data), user-view video, or virtual model.
  • Point cloud data refers to data recorded in the form of points. Each point can include spatial location information, as well as color information or reflection intensity information. Spatial data can also be called geometric data.
  • the virtual model may include one or more of a virtual portrait model, a virtual object model, and material images (such as stickers, cartoon avatars, etc.), or a virtual animation model.
  • the user's perspective video may be a video captured by the user through the rear camera of the terminal device, or a video captured by the front camera of the terminal device.
  • the terminal device in the embodiment of the present application may establish different auxiliary transmission channels according to the type of the auxiliary media stream to be transmitted. For example, when point cloud data needs to be transmitted, auxiliary transmission channel 1 is established, and when AR spatial data needs to be transmitted, auxiliary transmission channel 2 is established.
  • the auxiliary transmission channel 2 used to transmit AR spatial data may be referred to as an Action channel, or other names may also be used, which is not limited in the embodiment of the present application.
  • the terminal device can also transmit different types of auxiliary media streams through an auxiliary transmission channel.
  • the user interface of the AR control may be used as a floating window superimposed on the VoLTE call interface during the AR video call of the terminal device.
  • Window 402 displays the image of the user at the opposite end of the call
  • window 401 displays the image of the user at the end of the call
  • window 403 is the user interface of the AR control.
  • the AR media server is deployed in the first edge node of the system (such as The first edge cloud).
  • the two terminal devices that make a call are deployed in the edge cloud respectively corresponding to the AR media server.
  • the AR media server deployed in the first edge cloud is called the first AR media server
  • the AR media server deployed in the second edge cloud is called the second AR media server.
  • the media stream processed by the first AR media server for media enhancement processing is first sent to the second AR media server if it needs to be sent to the second terminal device, and then sent by the second AR media server through the second SBC To the second terminal device.
  • edge nodes corresponding to different terminal devices may be the same or different.
  • the central nodes corresponding to different terminal devices may be the same or different.
  • the AR media server corresponding to different terminal devices may be different or the same.
  • the CSCFs corresponding to different terminals may be the same or different.
  • the AR media server performs media enhancement processing on the received media stream.
  • the media stream input by the AR media server is referred to as an upstream media stream
  • the output media stream is referred to as a downstream media stream.
  • the upstream media stream received by the AR media server may include the first media stream from the first terminal device.
  • the upstream media stream may also include one or more of the following: auxiliary media stream and virtual model.
  • the input of the AR media server may also include an AR interface operation instruction, and the AR interface operation instruction may instruct the user to perform an operation on the AR interface displayed by the first terminal device.
  • the output of the AR media server may include the downstream media stream of the first terminal device, and/or the downstream media stream of the second terminal device.
  • the auxiliary media stream may be sent by the first terminal device to the AR media server through the auxiliary transmission channel.
  • the auxiliary media stream may include one or more of point cloud data, spatial data, user-view video, or virtual model.
  • the virtual model is generated by the terminal device and sent to the AR media server through the auxiliary transmission channel.
  • the terminal device may not have the ability to generate a virtual model.
  • the virtual model can be generated by the application server and sent to the AR media. server.
  • the input and output of the AR media server may be different.
  • composition and flow direction of the media stream of the AR media server will be exemplarily described below in conjunction with application scenarios. Take an AR video call between the first terminal device of user 1 and the second terminal device of user 2 as an example.
  • Example 1 One-way AR enhancement scenario. For example, scenes such as beauty, stickers, super scores, and expression-driven calls. This scenario can be applied to the situation where both ends of the call support AR. It can also be applied to the situation where one side supports AR.
  • the first terminal device needs to perform AR processing during a video call with the second terminal device.
  • the input of the AR media server includes the first media stream of the first terminal device.
  • the first media stream is sent by the first terminal device to the AR media server through the first SBC.
  • the first media stream may include a video collected by the first terminal device through a camera, and may also include a voice collected by a microphone.
  • the AR media server performs media enhancement processing on the first media stream and then outputs the downstream media stream of the second terminal device.
  • the first terminal device of user 1 may also display the image of user 1 itself on the basis of displaying the image of user 2, such as window 401 and window 402 shown in FIG. 4.
  • the AR media server When the first media stream is sent to the second terminal device after performing media enhancement processing, it may also be sent to the first terminal device.
  • the AR control of the first terminal device sends the beauty operation instruction to the application server, and the application server sends the beauty operation to the AR media server.
  • the media server performs a beauty operation on the face included in the video image in the received first media stream.
  • a material library is deployed in the application service function in the application server.
  • the material library can include various materials, such as different styles of stickers, and emoticon avatars with different expressions (such as cute cats, funny faces), or virtual portrait models of different styles, and so on.
  • the input of the AR media server also includes the material images from the application server.
  • the AR control of the first terminal device sends an AR interface operation instruction to the application server in response to the prop (such as a virtual portrait) selected by the user, and the AR interface operation instruction is used to indicate the virtual portrait selected by the user.
  • the application server After the application server receives the AR interface operation instruction, it can send the virtual portrait model in the material library to the AR media server, and the AR media server obtains the facial expressions, actions and other data of user 1 from the received first media stream to the virtual portrait model Perform rendering, and send the rendered media stream to the second terminal device.
  • Example 1 may be applicable to an architecture where the AR media server is deployed on a central node, and may also be applicable to an architecture where the AR media server is deployed on an edge node.
  • the AR media server such as the first AR media server
  • the second AR media server performs media enhancement processing on the media
  • the stream is first sent to the AR media server (second AR media server) corresponding to the second terminal device, and the second AR media server sends the stream to the second terminal device through the second SBC.
  • Example 2 Operating interactive call scenarios, such as advertising and distance education. This scenario can be applied to the situation where both ends of the call support AR. It can also be applied to the situation where one side supports AR.
  • the input of the AR media server includes the first media stream of the first terminal device.
  • the first media stream is sent by the first terminal device to the AR media server through the first SBC.
  • the input of the AR media server also includes real-time operation instructions, such as model rotation, model movement or model scaling, space annotation and other operations.
  • the real-time operation instruction may be generated by the user 1 operation, that is, sent by the first terminal device to the AR media server through the application server.
  • the real-time operation instruction may be generated by the operation of the user 2, that is, sent by the second terminal device through the application server.
  • the AR media server may include at least two media processing instances, taking two as an example, media processing instance 1 and media processing instance 2, respectively.
  • Media processing example 1 is used to perform media enhancement processing on the first media stream of the first terminal device.
  • the input of media processing example 1 can include the first media stream and real-time operation instructions.
  • Media processing example 1 enhances the media stream after media processing. It is sent to the first terminal device through the first SBC.
  • Media processing instance 2 is used to perform media enhancement processing on the second media stream of the second terminal device.
  • the input of media processing instance 2 may include the second media stream and real-time operation instructions, and the media stream after the media enhancement processing is passed through the second media stream.
  • the SBC is sent to the second terminal device.
  • the real-time operation instruction input on the media processing instance 1 and the real-time operation instruction input on the media processing instance 2 may be the same, for example, from the first terminal device or from the second terminal device.
  • the real-time operation instructions input on the media processing instance 1 and the real-time operation instructions input on the media processing instance 2 can also be the same.
  • the real-time operation instructions input on the media processing instance 1 come from the first terminal device, and the real-time operation input on the media processing instance 2
  • the indication comes from the second terminal device.
  • the first AR media server may perform media enhancement processing on the first media stream of the first terminal device
  • the second AR media server may perform media enhancement processing on the second media stream of the second terminal device. deal with.
  • the input of the first AR media server may include a first media stream and a real-time operation instruction, and the media stream after the media enhancement processing of the first AR media server is sent to the first terminal device through the first SBC.
  • the input of the second AR media server may include a second media stream and a real-time operation instruction, and the media stream after the media enhancement processing of the second AR media server is sent to the first terminal device through the first SBC.
  • the real-time operation instruction input on the first AR media server and the real-time operation instruction input by the second AR media server may be the same, for example, the real-time operation instruction comes from the first terminal device or from the second terminal device.
  • the real-time operation instructions input on the first AR media server and the real-time operation instructions input by the second AR media server may also be different.
  • the real-time operation instructions input on the first AR media server come from the first terminal device, and the real-time operation instructions input on the second AR media server The real-time operation instructions come from the second terminal device.
  • the input of the AR media server may also include a virtual model, and the virtual model may be sent by the application server to the AR media server. It should be noted that the virtual model may not be transmitted in real time, but can be transmitted once by the application server. In scenarios where the terminal device provides a virtual model, the terminal device may send the virtual model to the AR media server through the application server, or the AR control on the terminal device may send the virtual model to the AR media server through the auxiliary transmission channel.
  • the housing provider corresponds to the second terminal device
  • the house buyer is the first terminal device.
  • the first terminal device sends the first media stream (as a background stream) of the house-buying user to the AR media server through the first SBC.
  • the first media stream can be collected by a rear camera on the first terminal device.
  • the AR plug-in of the first terminal device sends the model operation instruction of the house buyer to operate the second-hand house model to the AR media server through the application server.
  • the AR media server obtains the spatial pose data of the buyer’s perspective from the first media stream, renders the second-hand house model according to the spatial pose data of the buyer’s perspective, and superimposes the rendered second-hand house model with the background stream of the buyer and sends it to the first Terminal Equipment.
  • the second media stream for the second terminal device may adopt a similar manner to the processing manner for the first media stream, and the description will not be repeated here.
  • the spatial pose data may be sent by the AR control of the first terminal device to the AR media server through the auxiliary transmission channel.
  • Example 3 image interactive call scene, such as AR holographic call scene, etc.
  • both ends of the call support AR.
  • the AR media server can deploy at least two media processing instances. Taking two as examples, see FIG. 10, which are media processing instance 1 and media processing instance 2, respectively.
  • the input and output of media processing example 1 and the input and output of media processing example 2 are shown in FIG. 10.
  • the input of media processing instance 1 includes the first media stream and the second auxiliary media stream.
  • the first media stream can be sent by the first terminal device to media processing instance 1 through the first SBC
  • the second auxiliary media stream can be the second terminal device.
  • the AR control on the above is sent to the media processing instance 1 through the auxiliary transmission channel.
  • the input of media processing instance 2 includes the second media stream and the first auxiliary media stream.
  • the second media stream can be sent by the second terminal device to media processing instance 2 through the second SBC, and the first auxiliary media stream can be the first terminal device.
  • the AR control on the above is sent to the media processing instance 2 through the auxiliary transmission channel.
  • the first media stream and the first auxiliary media stream may be collected by the first terminal device through the front camera and the rear camera respectively.
  • the second media stream and the second auxiliary media stream may be the front camera of the second terminal device, respectively. And the rear camera is collected.
  • the first media stream includes the environment image of the user 1 of the first terminal device
  • the first auxiliary media stream includes the portrait image of the user 1
  • the second media stream includes the location of the user 2 of the second terminal device.
  • the second auxiliary media stream includes the portrait image of user 2.
  • the first auxiliary media stream of user 1 is input to media processing instance 2, and media processing instance 2 obtains real-time expression and action data of user 1 from the first auxiliary media stream of user 1, and drives the virtual model of user 1;
  • the second media stream of 2 is used as the background stream, and the spatial pose data of user 2’s perspective is obtained according to the background stream, and the virtual model of user 1 is rendered according to the spatial pose data of user 2’s perspective, and superimposed with the second media stream as Downstream video stream of the second terminal device.
  • the second auxiliary media stream of user 2 is input to media processing instance 1, and media processing instance 1 obtains real-time expression and action data of user 2 from the second auxiliary media stream of user 2 to drive the virtual model of user 2; media processing instance 1 Use the first media stream of user 1 as the background stream, obtain the spatial pose data of user 1’s perspective according to the background stream, and render the virtual model of user 2 based on the spatial pose data of user 1’s perspective, and superimpose it with the first media stream.
  • the downstream video stream of the first terminal device As the downstream video stream of the first terminal device.
  • the input and output of the first AR media server and the second AR media server are shown in FIG. 11.
  • the processing methods of the first AR media server and the second AR media server are similar to the processing methods of the aforementioned media processing example 1 and media processing example 2, and will not be repeated here.
  • Example 4 virtual and real superimposed call scenarios, such as remote guidance, etc. This scenario can be applied to the situation where both ends of the call support AR. It can also be applied to the situation where one side supports AR.
  • the input of the AR media server includes the first media stream of the first terminal device and the auxiliary media stream (including point cloud data) of the first terminal device.
  • a depth camera may be configured on the first terminal device to obtain point cloud data, and the point cloud data is used to generate a depth map of the shooting picture, such as a red (R) green (G) blue (B)-depth (D) image.
  • the first media stream of the first terminal device is input to the AR media server, and the first media stream is used as the background stream.
  • the AR media server recognizes the spatial position of the object from the background stream with higher accuracy based on the point cloud data, and after recognizing the object , Superimpose a virtual model, or logo, etc. on the background stream, and the output of the AR media server is used as the downlink video stream of the first terminal device and the second terminal device.
  • FIG. 13 is a schematic flowchart of the AR-based communication method provided by this embodiment of the present application. Take the first terminal device triggering the AR video enhancement process to the second terminal device as an example.
  • the first terminal device triggers a call request to the application server through the first SBC.
  • the application server sends a first session creation request to the AR media server.
  • the first session creation request is used to request the creation of a first media session between the first SBC corresponding to the first terminal device.
  • the first session creation request carries the SDP information of the first SBC.
  • the first session creation request may be an INVITE message.
  • the AR media server When receiving the first session creation request, the AR media server sends a first session creation response to the application server.
  • the first session creation response is used to indicate that the first media session is successfully created.
  • the first session creation response carries first media description protocol SDP information of the AR media server, and the first SDP information is used to describe the media used to create the first media session between the first SBC and the AR media server. The parameters of the flow channel.
  • the first session creation response may be 200 OK.
  • the AR media server receives the second session creation request sent by the application server.
  • the second session creation request is used to request the creation of a second media session with the second SBC.
  • the second session creation request may be an INVITE message.
  • the second session creation request may carry a service indication.
  • the service indication is used to indicate the media processing and media flow direction required for this session.
  • the service indication may be a service identification (ServiceID).
  • ServiceID service identification
  • the service indication may also be called the AR service indication.
  • the content indicated by the service indication in different application scenarios is different, that is, in different application scenarios, the media enhancement processing provided by the AR media server is different, and the flow direction of the media stream may also be different.
  • the second session creation request carries an association indication.
  • the association indication may be indicated by the call identifier (for example, CallID) of the second session creation request.
  • the association indication is used to associate the first media session with the second media session.
  • an association relationship between the media stream channel between the first SBC and the AR media server and the media stream channel between the AR media server is established. It can also be said that the media stream of the first terminal device forwarded by the first SBC needs to pass through the AR media server before reaching the second SBC to which the second terminal device belongs.
  • the AR media server sends a second session creation response to the application server.
  • the second session creation response is used to indicate that the second media session is successfully created.
  • the second session creation response carries the second SDP information of the AR media server.
  • the second media description protocol SDP information is used to describe the parameters of the media stream channel used to create the second media session between the second SBC and the AR media server.
  • the first session creation response may be a 183 message.
  • the association relationship between the first interface and the second interface on the AR media server may be established.
  • the first interface is for receiving the media stream sent by the first SBC
  • the second interface is for sending the media stream to the second SBC.
  • the first interface and the second interface may be physical interfaces or physical sub-interfaces, and may also be logical interfaces or logical sub-interfaces.
  • the application server sends a call request to the second terminal device through the second SBC.
  • the application server may bring the second SDP of the AR media server to the second SBC in the call request.
  • the first case is that the AR media server is introduced in the VoLTE call process, that is, the AR media server and the AR media server need to be established during the call establishment process.
  • Media conversation between SBCs For example, the first terminal device initiates the AR video enhancement process when the original call is established, and the first case can be adopted.
  • the second situation is that the original call does not need to perform AR media enhancement (for example, the original call is only an audio call), and the AR media enhancement process is triggered during the call.
  • FIG. 14A uses the same AR media server at both ends of the call as an example.
  • the first terminal device sends a call request 1 to the first SBC.
  • the call request 1 carries media description protocol (session description protocol, SDP) information of the first terminal device.
  • SDP session description protocol
  • the call request 1 may, but is not limited to, use a session initiation protocol (session initiation protocol, SIP), and may also use other types of transmission protocols, which is not limited in this application.
  • SIP session initiation protocol
  • SDP is used for media plane negotiation between two session entities, and a consensus is reached. It belongs to the signaling language family and can be described in the form of text (character). SDP can include one or more of the following: session ID, session version, session time, the IP and port of the local transmission media stream, the description information of the media stream (such as media type, transmission protocol, media format, etc.) One or more of), etc.
  • the SDP information of the first terminal device is used to describe the parameters of the media stream channel that creates the media session between the first terminal device and the first SBC.
  • the first SBC After receiving the call request 1, the first SBC replaces the SDP information of the first terminal device in the call request 1 with the SDP information of the first SBC to obtain the call request 2, and sends the call request 2 to the S-CSCF.
  • S1403 After receiving the call request 2, the S-CSCF forwards the call request 2 to the application server.
  • the S-CSCF determines that the first terminal device has subscribed to the AR media enhancement service according to the subscription data of the first terminal device, and then forwards the call request 2 to the application server.
  • the application server is used to provide AS media enhancement services.
  • the application server replaces the SDP information of the first SBC in the call request 2 with the second SDP information of the AR media server to obtain the call request 3, and sends the call request 3 to the S-CSCF.
  • the S-CSCF forwards the call request 3 to the second SBC.
  • the second SBC may determine that the previous hop of the media stream channel is the AR media server according to the second SDP information of the AR media server.
  • the second SBC replaces the second SDP information of the AR media server in the call request 3 with the SDP information of the second SBC to obtain the call request 4, and sends the call request 4 to the second terminal device.
  • the second terminal device sends a call response 4 (corresponding to the call request 4) to the second SBC, and the call response 4 may carry the SDP information of the second terminal device.
  • the second SBC After receiving the call response 4, the second SBC sends the call response 3 (corresponding to the call request 3) to the S-CSCF.
  • the call response 3 may carry the SDP information of the second SBC.
  • S1413 After receiving the call response 3, the S-CSCF forwards the call response 3 to the application server.
  • the application server After receiving the call response 3, the application server sends the SDP information of the second SBC to the AR media server. After receiving the SDP information of the second SBC, the AR media server may determine that the next hop of the media stream tunnel is the second SBC.
  • the application server sends a call response 2 (a response corresponding to the call request 2) to the S-CSCF.
  • the call response 2 can carry the second SDP information of the AR media enabler.
  • S1416 The S-CSCF forwards the call response 2 to the first SBC.
  • the first SBC After receiving the call response 2, the first SBC sends the call response 1 to the first terminal device.
  • the call response 1 carries the SDP information of the first SBC.
  • call response 1 to call response 4 may adopt the 183 message type.
  • FIG. 14B is a schematic diagram of the flow of the AR-based communication method provided in this embodiment of the application, taking the first terminal device triggering the AR video enhancement flow to the second terminal device as an example.
  • Figure 14B takes as an example the two ends of the call corresponding to different AR media servers, and the two ends of the call corresponding to different application servers.
  • the first application server sends a session creation request 1 to the first AR media server.
  • the session creation request 1 is used to request the creation of a first media session between the first SBC corresponding to the first terminal device.
  • the session creation request 1 carries the SDP information of the first SBC.
  • the first session creation request may be an INVITE message.
  • Session creation response 1 is used to indicate that the first media session is successfully created.
  • the session creation response 1 carries the first media description protocol SDP information of the first AR media server, and the first SDP information is used to describe the media circulation for creating the first media session between the first SBC and the first AR media server The parameters of the road.
  • the first session creation response may be 200 OK.
  • Session creation request 2 is used to request the creation of a second media session with the second SBC.
  • the session creation request 2 may be an INVITE message.
  • the session creation request 2 may carry a service indication.
  • the second session creation request carries the first association indication.
  • the first association indication may be indicated by the call identification (for example, CallID) of the session creation request 2.
  • the first association indication is used to associate the first media session with the second media session.
  • S1407a The first AR media server sends a session creation response 2 to the first application server.
  • Session creation response 2 is used to indicate that the second media session is successfully created.
  • the second session creation response carries the second SDP information of the AR media server.
  • the second media description protocol SDP information is used to describe the parameters of the media stream channel used to create the second media session between the second SBC and the first AR media server.
  • S1408a The first application server replaces the SDP information of the first SBC in the call request 2 with the second SDP information of the first AR media server to obtain the call request 3, and sends the call request 3 to the S-CSCF1.
  • S-CSCF1 forwards call request 3 to S-CSCF2.
  • the S-CSCF2 forwards the call request 3 to the second application server.
  • the second application server sends a session creation request 3 to the second AR media server.
  • the session creation request 3 is used to request the creation of a third media session with the first AR media server.
  • the session creation request 3 carries the second SDP information of the first AR media server.
  • the second AR media server When receiving the session creation request 3, the second AR media server sends a session creation response 3 to the second application server.
  • the session creation response 3 is used to indicate that the third media session is successfully created.
  • the session creation response 3 carries the first SDP information of the second AR media server, and the first SDP information of the second AR media server is used to describe the creation of a third AR media server between the first AR media server and the second AR media server. The parameters of the media stream channel of the media session.
  • the second AR media server receives the session creation request 4 sent by the second application server.
  • the session creation request 4 is used to request the creation of a fourth media session with the second SBC.
  • the session creation request 4 may be an INVITE message.
  • the session creation request 4 may carry a service indication.
  • the second session creation request carries a second association indication.
  • the second association indication may be indicated by the call identification (for example, CallID) of the session creation request 4.
  • the second association indication is used to associate the third media session with the fourth media session.
  • the second AR media server sends a session creation response 4 to the second application server.
  • the session creation response 4 is used to indicate that the fourth media session is successfully created.
  • the session creation response 4 carries the second SDP information of the second AR media server.
  • the second SDP information of the second AR media server is used to describe the parameters of the media stream channel for establishing the fourth media session between the second SBC and the second AR media server.
  • S1415a The second application server replaces the second SDP information of the first AR media server in the call request 3 with the second SDP information of the second AR media server to obtain the call request 4, and sends it to the S-CSCF2.
  • the S-CSCF2 forwards the call request 4 to the second SBC.
  • the second SBC may determine that the previous hop of the media stream channel is the second AR media server according to the second SDP information of the second AR media server.
  • the second SBC replaces the second SDP information of the second AR media server in the call request 4 with the SDP information of the second SBC to obtain the call request 5, and sends the call request 5 to the second terminal device.
  • the second terminal device sends a call response 5 to the second SBC, and the call response 5 may carry the SDP information of the second terminal device.
  • the second SBC After receiving the call response 5, the second SBC sends the call response 4 to the S-CSCF2.
  • the call response 4 may carry the SDP information of the second SBC.
  • S1420a After receiving the call response 4, the S-CSCF2 forwards the call response 4 to the second application server.
  • the second application server After receiving the call response 4, the second application server sends the SDP information of the second SBC to the second AR media server. After receiving the SDP information of the second SBC, the second AR media server may determine that the next hop of the media stream tunnel is the second SBC.
  • the second application server sends a call response 3 to the S-CSCF2.
  • the call response 3 may carry the first SDP information of the second AR media enabler.
  • S-CSCF2 sends a call response 3 to S-CSCF1.
  • S-CSCF1 sends a call response 3 to the first application server.
  • the first application server sends the first SDP information of the second AR media enabler to the first AR media server. After receiving the first SDP information of the second AR media enabler, the first AR media enabler may determine that the next hop of the media stream tunnel is the second AR media enabler.
  • the first application server sends a call response 2 to the first SBC, and the call response 2 carries the first SDP information of the first AR media server.
  • the first SBC After receiving the call response 2, the first SBC sends the call response 1 to the first terminal device.
  • the call response 1 carries the first SDP information of the first SBC.
  • call response 1 to call response 4 may adopt the 183 message type.
  • the VoLTE call is established, and the media stream does not pass through the AR media server.
  • the AR media enhancement process may be triggered by the first terminal device or the second terminal device.
  • the following takes the first terminal device to trigger the AR media enhancement process through the AR control as an example.
  • Figure 14A takes an example where both ends of a call correspond to the same AR media server or an AR media server deployed at one end of the call.
  • the application server sends an AR video call re-invite 1 to the S-CSCF.
  • the AR video call re-request 1 is used to instruct the first terminal device to initiate an AR video call.
  • the AR video call re-request 1 may carry the identification information of the first terminal device, such as the SIP address or Uniform Resource Locator (URL) of the first terminal device.
  • the S-CSCF forwards the AR video call re-request 1 to the first SBC.
  • the AR video call re-request may be Re-INVITE.
  • the S-CSCF may determine that the SBC to which the first terminal device belongs is the first SBC according to the identification information of the first terminal device.
  • the first SBC sends an AR video call re-request 2 to the first terminal device.
  • the first terminal device sends an AR video call response 2 to the first SBC, and the AR call request 2 carries media description protocol (session description protocol, SDP) information of the first terminal device.
  • media description protocol session description protocol, SDP
  • the first SBC After receiving the AR video call response 2, the first SBC sends the AR video call response 1 to the S-CSCF.
  • the AR video call response 1 carries the SDP information of the first SBC.
  • S1507 After receiving the AR video call response 1, the S-CSCF forwards the AR video call response 1 to the application server.
  • the AR video call response 1 and the AR video call response 2 may use a 200 OK message.
  • the application server sends an AR video call repeat request 3 to the S-CSCF, and the AR video call repeat request 3 carries the second SDP information of the AR media server.
  • the S-CSCF forwards the AR video call re-request 3 to the second SBC.
  • the second SBC may determine that the previous hop of the media stream channel is the AR media server according to the second SDP information of the AR media server.
  • the second SBC replaces the second SDP information of the AR media server in the AR video call repeat request 3 with the SDP information of the second SBC to obtain the AR video call repeat request 4, and sends the AR video call repeat request 4 to the second terminal equipment.
  • the second terminal device sends an AR video call response 4 to the second SBC, and the AR video call response 4 may carry SDP information of the second terminal device.
  • the second SBC After receiving the AR video call response 4, the second SBC sends the AR video call response 3 to the S-CSCF.
  • the AR video call response 3 may carry the SDP information of the second SBC.
  • S1517 After receiving the AR video call response 3, the S-CSCF forwards the AR video call response 3 to the application server.
  • the application server After receiving the AR video call response 3, the application server sends the SDP information of the second SBC to the AR media server. After receiving the SDP information of the second SBC, the AR media server may determine that the next hop of the media stream tunnel is the second SBC.
  • the AR video call response 3 and the AR video call response 4 may adopt a 200 OK message.
  • the application server sends an AR video call confirmation 1 to the S-CSCF.
  • the AR video call confirmation 1 may carry the second SDP information of the AR media server.
  • S1520 The S-CSCF forwards the AR video call confirmation 1 to the first SBC.
  • the first SBC After receiving the AR video call confirmation 1, the first SBC sends the AR video call confirmation 2 to the first terminal device.
  • the AR video call confirmation 2 carries the SDP information of the first SBC.
  • the AR video call confirmation 1 and the AR video call confirmation 2 may use an acknowledgment (ACK) message.
  • ACK acknowledgment
  • the AR control is deployed on the first terminal device.
  • the AR video enhancement request triggered by the AR control on the first terminal device can be implemented through the following process, as shown in FIG. 16.
  • the AR control is started.
  • the first terminal device can pull up the AR control by calling a broadcast event.
  • the user interface of the AR control can be superimposed on the call interface as a floating window, as shown in Figure 4, for example.
  • the user interface of the AR control may include an AR enhancement start button, and the AR control receives the first operation of the start button by the user and triggers the AR video enhancement request.
  • the AR control establishes a communication connection with the media plug-in service function in the application server through a UX or UI interface.
  • S1602 The AR control sends the AR video enhancement request to the media plug-in service function.
  • the media plug-in service function sends the AR video enhancement request to the application service function.
  • S1604 The application service function triggers the AR video enhancement process. For example, execute S1502.
  • the first terminal device and the second terminal device have established an AR video enhanced call process, and the process of establishing an auxiliary media channel between the first terminal device and the AR media server is used for description.
  • the AR control of the first terminal device determines that the auxiliary media stream needs to be transmitted, it initiates an auxiliary transmission channel establishment request. For example, if the user triggers the opening of the depth camera for obtaining point cloud data through the AR control, it is determined that the auxiliary media stream needs to be transmitted. For another example, if the application used to generate AR spatial data is triggered by the AR control, it is determined that the auxiliary media stream needs to be transmitted.
  • the AR control sends an establishment request to the media plug-in service function in the application server, carrying the address used to send the auxiliary media stream on the first terminal device.
  • the media plug-in service function sends the establishment request to the application service function.
  • S1704 The application service function sends the establishment request to the AR media server.
  • the AR media server sends an establishment response to the application service function.
  • the establishment response may carry the address used to receive the auxiliary media stream on the AR media server.
  • S1706 The application service function sends the establishment response to the media plug-in service function.
  • the media plug-in service function forwards the establishment response to the AR control of the first terminal device. Furthermore, the auxiliary transmission channel between the AR control and the AR media server is established. The first end of the auxiliary transmission channel is the AR control, and the end of the auxiliary transmission channel is the AR media server. Furthermore, the AR control obtains the auxiliary media stream, and sends the auxiliary media stream to the AR media server according to the address used for sending the auxiliary media stream on the first terminal device and the address used for receiving the auxiliary media stream on the AR media server.
  • an embodiment of the present application provides an apparatus 1800, which is specifically used to implement the method executed by the AR media server in the foregoing method embodiment.
  • the structure of the apparatus 1800 is shown in FIG. 18 and includes a receiving unit. 1801, a sending unit 1802, and a processing unit 1803.
  • the receiving unit 1801 is configured to receive a first session creation request sent by an application server, where the first session creation request is used to request the creation of a first media session between the first session border controller SBC corresponding to the first terminal device ;
  • the sending unit 1802 is configured to send a first session creation response to the application server, where the first session creation response is used to indicate that the first media session is successfully created;
  • the receiving unit 1801 is further configured to receive a second session creation request sent by the application server, where the second session creation request is used to request the creation of a second media session between a second SBC corresponding to the second terminal device ;
  • the sending unit 1802 is further configured to send a second session creation response to the application server, where the second session creation response is used to indicate that the second media session is successfully created.
  • the first session creation response carries first media description protocol SDP information of the AR media server, and the first SDP information includes information used for the first SBC and the AR The parameters of the media stream channel used to create the first media session between the media servers.
  • the second session creation response carries second SDP information of the AR media server, and the second SDP information includes information used for the relationship between the second SBC and the AR media server.
  • the parameters of the media stream channel used to create the second media session are not limited to the first and the second SDP information.
  • the receiving unit 1801 in the AR media server is also used to receive the media stream from the first terminal device and/or the second terminal device, and the processing unit 1803 performs processing on the media stream from the first terminal device and/or The media stream of the second terminal device performs media enhancement processing.
  • the second session creation request carries a service indication; further, the processing unit 1803 performs media enhancement processing on the media stream transmitted between the first terminal device and the second terminal device according to the service indication.
  • the second session creation request received by the receiving unit 1801 carries an association indication; further, the processing unit 1803 associates the first media session with the second media session according to the association indication.
  • Media conversation
  • the receiving unit 1801 is further configured to receive a first media stream sent from the first SBC through the first media session; then, the processing unit 1803 performs the processing on the first media stream. Perform media enhancement processing to obtain a second media stream; finally, the sending unit 1802 sends the second media stream to the second SBC through the second media session.
  • the receiving unit 1801 receives the third media stream sent by the SBC to which the first terminal device belongs through the first media session, and receives the third media stream from the SBC to which the second terminal device belongs according to the The second media session sends a fourth media stream; then, the processing unit 1803 performs media enhancement processing on the third media stream and the fourth media stream; finally, the sending unit 1802 sends the media to the first terminal device and/or the The second terminal device sends the media stream after media enhancement processing.
  • the receiving unit 1801 receives the fifth media stream sent from the first SBC through the first media session, and receives the virtual model from the application server; then, the processing unit 1803 The virtual model and the fifth media stream are subjected to media enhancement processing; finally, the sending unit 1804 sends the media stream after the media enhancement processing to the first terminal device and/or the second terminal device.
  • the receiving unit 1801 receives a model operation instruction from an application server, where the model operation instruction is used to instruct the AR media server to perform an operation on the virtual model;
  • the model operation instruction is used to instruct the AR media server to perform an operation on the virtual model;
  • media enhancement processing on the model and the fifth media stream it can be implemented in the following manner: performing a processing operation on the virtual model according to the model operation instruction, and performing processing operations on the virtual model and the fifth media stream after the processing operation Perform media enhancement processing.
  • the AR media server may establish an auxiliary transmission channel between the application server and the terminal device.
  • the receiving unit 1801 receives an establishment request from an application server, the establishment request is used to request the establishment of an auxiliary transmission channel, and the auxiliary transmission channel is used to transmit an auxiliary media stream between the first terminal device and the AR media server.
  • the establishment request carries the address of the first terminal device; then the sending unit 1802 sends a first establishment response to the application server, and the first establishment response carries the address of the AR media server.
  • the receiving unit 1801 is further configured to receive the auxiliary media stream sent by the first terminal device according to the channel transmission address of the AR media server.
  • the receiving unit 1801 receives the sixth media stream sent from the first SBC; then the processing unit 1803 performs media enhancement processing on the auxiliary media stream and the sixth media stream; finally, the sending unit 1802 sends the The first terminal device and/or the second terminal device sends the media stream after media enhancement processing.
  • the auxiliary media stream includes one or more of point cloud data, spatial data, user-view video, or virtual model.
  • the embodiment of the present application provides an apparatus 1900, which is specifically configured to implement the method executed by the application server in the foregoing embodiment.
  • the structure of the device 1900 is shown in FIG.
  • the receiving unit 1901 is configured to receive a call request from the first terminal device.
  • the sending unit 1902 is configured to send a first session creation request to the AR media server, where the first session creation request is used to request the creation of a first media session between the session border controller SBC corresponding to the first terminal device;
  • the receiving unit 1901 is further configured to receive a first session creation response sent by the AR media server, where the first session creation response is used to indicate that the first media session is successfully created;
  • the sending unit 1902 is further configured to send a second session creation request to the AR media server, where the second session creation request is used to request the creation of a second media session between a second SBC corresponding to the second terminal device ;
  • the receiving unit 1901 is further configured to receive a second session creation response sent by the AR media server, where the second session creation response is used to indicate that the second media session is successfully created.
  • the apparatus 1900 may further include a processing unit 1903, configured to generate a session creation request and a session creation response.
  • the first session creation response carries first media description protocol SDP information of the AR media server, and the first SDP information includes information used for the first SBC and the AR The parameters of the media stream channel for establishing the first media session between the media servers; the sending unit 1902 is further configured to send the first SDP information of the AR media server to the first SBC.
  • the second session creation response carries second SDP information of the AR media server, and the second SDP information includes information used for the relationship between the second SBC and the AR media server.
  • the parameters of the media stream channel used to create the second media session; the sending unit 1902 is further configured to send the second SDP information to the second SBC.
  • the second session creation request carries a service indication
  • the service indication is used to instruct the AR media server to address the media transmitted between the first terminal device and the second terminal device.
  • Media enhancement processing required for streaming is used to instruct the AR media server to address the media transmitted between the first terminal device and the second terminal device.
  • the second session creation request carries an association indication
  • the association indication is used to instruct the AR media server to associate the first media session with the second media session.
  • the receiving unit 1901 is further configured to receive the AR video call request sent by the first SBC, and then the sending unit 1902 executes to send the first session creation request to the AR media server.
  • the receiving unit 1901 receives the AR video enhancement request sent by the first terminal device, and then the sending unit 1902 sends the AR video enhancement request to the first terminal device through the first SBC,
  • the AR video enhancement request is used to instruct the first terminal device to initiate an AR video call.
  • the sending unit 1902 executes sending the first session creation request to the AR media server.
  • the receiving unit 1901 is further configured to receive a model operation instruction triggered by the first terminal device; the sending unit 1902 is further configured to send the model operation instruction to the AR media server, and the model operation instruction is used for To instruct the AR media server to perform operations on the received virtual model.
  • the receiving unit 1901 is further configured to receive an establishment request triggered by the first terminal device, where the establishment request is used to request the establishment of an auxiliary transmission channel, and the auxiliary transmission channel is used for the first terminal device.
  • An auxiliary media stream is transmitted between a terminal device and the AR media server, and the establishment request carries the address of the first terminal device;
  • the sending unit 1902 is further configured to send the establishment request to the AR media server;
  • the receiving unit 1901 receives an establishment response sent by the AR media server, and the establishment response carries the address of the AR media server; finally, the sending unit 1902 sends the establishment response to the first terminal device.
  • the embodiment of the present application provides an apparatus 2000, which is specifically configured to implement the method executed by the terminal device (such as the first terminal device or the second terminal device) in the foregoing embodiment.
  • the structure of the device 2000 is shown in FIG. 20, and includes a receiving unit 2001, a sending unit 2002, and a processing unit 2003.
  • the receiving unit 2001 receives an AR media stream from a first SBC corresponding to a first terminal device, where the AR media stream includes a video of an AR interface displayed on the first terminal device;
  • the processing unit 2003 in response to an operation triggered by the user on the displayed AR interface, instructs the sending unit 2002 to send an AR interface operation instruction to the application server, where the AR interface operation instruction is used to indicate media enhancement processing performed on the AR media stream;
  • the receiving unit 2001 is further configured to receive the enhanced AR media stream sent by the first SBC.
  • an auxiliary transmission channel is established between the first terminal device and the AR media server, and the sending unit 2002 sends the auxiliary media stream to the AR media server through the auxiliary transmission channel.
  • the auxiliary media stream is used by the AR media server to perform media enhancement processing.
  • the sending unit 2002 sends an establishment request to the application server, the establishment request is used to request the establishment of an auxiliary transmission channel, and the establishment request carries the address of the first terminal device; the receiving unit 2001 To receive an establishment response sent by the application server, where the establishment response carries the address of the AR media server. Then the sending unit 2001 executes sending the auxiliary media stream to the AR media server through the auxiliary transmission channel.
  • the sending unit 2002 may send the auxiliary media stream to the AR media server according to the address of the first terminal device and the address of the AR media server.
  • the division of units in the embodiments of this application is illustrative, and is only a logical function division. In actual implementation, there may be other division methods.
  • the functional units in the various embodiments of this application can be integrated into one processing unit. In the device, it can also exist alone physically, or two or more units can be integrated into one module.
  • the above-mentioned integrated unit can be realized in the form of hardware or software function module.
  • the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solution of the present application essentially or the part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , Including several instructions to make a terminal device (which may be a personal computer, a mobile phone, or a network device, etc.) or a processor (processor) execute all or part of the steps of the method in each embodiment of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disk and other media that can store program code .
  • the AR media server, the application server, and the terminal device may all be presented in the form of dividing various functional modules in an integrated manner.
  • the "module” here may refer to a specific ASIC, a circuit, a processor and memory that executes one or more software or firmware programs, an integrated logic circuit, and/or other devices that can provide the above-mentioned functions.
  • the SBC, CSCF, AR media server or application server may adopt the form shown in FIG. 21.
  • the apparatus 2100 shown in FIG. 21 includes at least one processor 2101 and a communication interface 2102.
  • a memory 2103 may also be included.
  • the processor 2101 in FIG. 21 can call the computer execution instructions stored in the memory 2103, so that the AR media server can execute any of the foregoing.
  • the processor 2101 may communicate with other devices through the communication interface 2102. For example, the processor 2101 receives the session creation request (such as the first session creation request, the second session creation request) sent by the application server through the communication interface 2102, and sends the session creation response (such as the first session creation response, the second session creation request) to the application server. Session creation response).
  • the session creation request such as the first session creation request, the second session creation request
  • the session creation response such as the first session creation response, the second session creation request
  • the memory 2103 stores computer-executable instructions for realizing the functions of the sending unit, receiving unit, and processing unit in FIG. 18, and the functions/implementation processes of the sending unit, receiving unit, and processing unit in FIG.
  • the processor 2101 in 21 calls the computer execution instructions stored in the memory 2103, and implements it in combination with the communication interface 2102.
  • the processor 2101 in FIG. 21 can call the computer execution instructions stored in the memory 2103, so that the application server can execute any of the above The method executed by the application server in the method embodiment.
  • the processor 2101 may communicate with other devices through the communication interface 2102. For example, the processor 2101 sends a session creation request (such as a first session creation request, a second session creation request) to the AR media server through the communication interface 2102, and receives a session creation response (such as a first session creation response) sent by the AR media server. , The second session creation response).
  • a session creation request such as a first session creation request, a second session creation request
  • a session creation response such as a first session creation response
  • the memory 2103 stores computer-executable instructions for realizing the functions of the sending unit, receiving unit, and processing unit in FIG. 19, and the functions/implementation processes of the sending unit, receiving unit, and processing unit in FIG.
  • the processor 2101 in 21 calls the computer execution instructions stored in the memory 2103, and implements it in combination with the communication interface 2102.
  • the processor 2101 in FIG. 21 may call the computer execution instructions stored in the memory 2103, so that the SBC can execute any of the foregoing method implementations.
  • the processor 2101 in FIG. 21 can call the computer to execute instructions stored in the memory 2103, so that the CSCF can execute any of the foregoing method implementations.
  • the specific connection medium between the foregoing processor 2101 and the memory 2103 is not limited in the embodiment of the present application.
  • the memory 2103 and the processor 2101 are connected through a bus 2104 in the figure.
  • the bus 2104 is represented by a thick line in the figure. Is limited.
  • the bus 2104 can be divided into an address bus, a data bus, a control bus, and so on. For ease of presentation, only one thick line is used to represent in FIG. 21, but it does not mean that there is only one bus or one type of bus.
  • the terminal device may adopt the form shown in FIG. 22.
  • the apparatus 2200 shown in FIG. 22 includes at least one processor 2201, a communication interface 2202, and optionally a memory 2203.
  • the device 2200 further includes a display 2204.
  • the device may also include a sensor 2205 for detecting the user's operation on the AR interface.
  • the processor 2201 may have the function of sending and receiving data and can communicate with other devices.
  • the processor 2201 may send a media stream to the SBC, or may receive a media stream from the SBC, as shown in the device shown in FIG. 22
  • an independent data transceiver module can also be set, such as a communication interface 2202 for sending and receiving data; when the processor 2201 communicates with other devices, it can perform data transmission through the communication interface 2202, as in the embodiment of the present application,
  • the device 2201 may send a media stream to the SBC through the communication interface 2202, and may also receive a media stream from the SBC through the communication interface 2202.
  • the processor 2201 in FIG. 22 can call the computer execution instructions stored in the memory 2203, so that the terminal device can execute the method described in any of the foregoing method embodiments.
  • the method executed by the terminal device (such as the first terminal device).
  • the memory 2203 stores computer-executable instructions for realizing the functions of the sending unit, receiving unit, and processing unit in FIG. 20.
  • the processor 2201 in 22 calls the computer execution instructions stored in the memory 2203 to implement.
  • the memory 2203 stores computer-executable instructions for implementing the function of the display unit in FIG. 20, and the function/implementation process of the display unit in FIG. 6 can be executed by calling the computer stored in the memory 2203 by the processor 2201 in FIG. 22
  • the function/implementation process of the sending unit and the receiving unit in FIG. 20 can be realized through the communication interface 2202 in FIG. 22.
  • the processor 2201 when the processor 2201 performs the function of the display unit, such as an operation involving displaying an image, such as displaying an AR interface, the processor 2201 may display an image through the display 2204 in the device 2200.
  • the processor 2201 when the processor 2201 performs the function of the display unit, it may also display an image through a display in another device, such as sending a display instruction to the other device to instruct to display the image.
  • the embodiment of the present application does not limit the specific connection medium between the processor 2201, the memory 2203, the communication interface 2202, the display 2204, and the sensor 2205.
  • the embodiment of the present application is connected by a bus 2206 as an example in the figure, and the bus 2206 is represented by a thick line in the figure.
  • the connection mode between other components is only for schematic illustration and is not to be taken as a limitation.
  • the bus 2206 can be divided into an address bus, a data bus, a control bus, and so on. For ease of representation, only one thick line is used to represent in FIG. 22, but it does not mean that there is only one bus or one type of bus.
  • the embodiments of the present application also provide a computer storage medium, the storage medium stores a software program, and the software program can implement any one or more of the above when read and executed by one or more processors.
  • the embodiment provides a method executed by a service platform or an edge computing device.
  • the computer storage medium may include: U disk, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk and other media that can store program codes.
  • the embodiments of the present application also provide a chip including a processor, which is used to implement the function of the AR media server or terminal device or the function of the application server involved in any one or more of the above embodiments.
  • the chip further includes a memory, and the memory is used for necessary program instructions and data to be executed by the processor.
  • the chip can be composed of a chip, or it can include a chip and other discrete devices.
  • system and “network” in this article are often used interchangeably in this article.
  • the term “and/or” in this article is only an association relationship describing the associated objects, which means that there can be three relationships, for example, A and/or B, which can mean: A alone exists, A and B exist at the same time, exist alone B these three situations.
  • the character “/” in this text generally indicates that the associated objects before and after are in an "or” relationship.
  • the term “at least one” referred to in this application refers to one, or more than one, that includes one, two, three and more; “multiple” refers to two, or more than two, that is, two, Three and more.
  • At least one item (a) or similar expressions refers to any combination of these items, including any combination of a single item (a) or a plurality of items (a).
  • at least one item (a) of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple .
  • B corresponding to A means that B is associated with A, and B can be determined according to A.
  • determining B based on A does not mean that B is determined only based on A, and B can also be determined based on A and/or other information.
  • the terms "including” and “having” in the embodiments, claims and drawings of the present application are not exclusive. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the listed steps or modules, and may also include unlisted steps or modules.
  • the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (digital signal processors, DSP), and application-specific integrated circuits. (application specific integrated circuit, ASIC), field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.
  • the general-purpose processor may be a microprocessor or any conventional processor.
  • the method steps in the embodiments of the present application can be implemented by hardware, or can be implemented by a processor executing software instructions.
  • Software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (programmable ROM) , PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically erasable programmable read-only memory (electrically EPROM, EEPROM), register, hard disk, mobile hard disk, CD-ROM or well-known in the art Any other form of storage medium.
  • An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium.
  • the storage medium may also be an integral part of the processor.
  • the processor and the storage medium may be located in the ASIC.
  • the ASIC can be located in a network device or a terminal device.
  • the processor and the storage medium may also exist as discrete components in the network device or the terminal device.
  • the above-mentioned embodiments it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof.
  • software it can be implemented in the form of a computer program product in whole or in part.
  • the computer program product includes one or more computer programs or instructions.
  • the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.
  • the computer program or instruction may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium.
  • the computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server integrating one or more available media.
  • the usable medium may be a magnetic medium, such as a floppy disk, a hard disk, and a magnetic tape; it may also be an optical medium, such as a DVD; and it may also be a semiconductor medium, such as a solid state disk (SSD).

Landscapes

  • Engineering & Computer Science (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Security & Cryptography (AREA)
  • Business, Economics & Management (AREA)
  • General Business, Economics & Management (AREA)
  • Telephonic Communication Services (AREA)

Abstract

L'invention concerne un procédé et un appareil de communication basés sur la réalité augmentée (AR). La présente demande concerne un mode de mise en œuvre d'intégration d'AR dans des appels vocaux et vidéo pour améliorer l'expérience utilisateur. Un serveur multimédia AR est déployé dans un système de communication et un canal de flux multimédia est formé entre le serveur multimédia AR et un SBC auquel appartiennent des dispositifs terminaux sur les deux côtés d'un appel, de sorte que le flux multimédia transmis entre les dispositifs terminaux des deux côtés de l'appel soit envoyé par le SBC au serveur multimédia AR, puis le serveur multimédia AR exécute une amélioration multimédia, ce qui permet de réaliser l'intégration du traitement AR dans le processus d'appel vidéo.
PCT/CN2020/124167 2019-11-08 2020-10-27 Procédé et appareil de communication basés sur la réalité augmentée WO2021088690A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201911089921.9A CN112788274A (zh) 2019-11-08 2019-11-08 一种基于增强现实的通信方法及装置
CN201911089921.9 2019-11-08

Publications (1)

Publication Number Publication Date
WO2021088690A1 true WO2021088690A1 (fr) 2021-05-14

Family

ID=75748550

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/124167 WO2021088690A1 (fr) 2019-11-08 2020-10-27 Procédé et appareil de communication basés sur la réalité augmentée

Country Status (2)

Country Link
CN (1) CN112788274A (fr)
WO (1) WO2021088690A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114760602A (zh) * 2022-04-18 2022-07-15 中国电信股份有限公司 全息通信方法、装置、系统及计算机可读存储介质
WO2024109582A1 (fr) * 2022-11-22 2024-05-30 中兴通讯股份有限公司 Procédé de traitement multimédia, serveur multimédia et support d'enregistrement

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115941761A (zh) * 2021-08-04 2023-04-07 中国移动通信有限公司研究院 一种通信、数据通道的建立方法、设备及存储介质
WO2024050744A1 (fr) * 2022-09-08 2024-03-14 Zte Corporation Systèmes et procédés de communication de réalité augmentée sur la base d'un canal de données

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101971618A (zh) * 2008-03-14 2011-02-09 阿尔卡特朗讯 实现移动终端的丰富视频的方法
CN102281293A (zh) * 2011-08-01 2011-12-14 中兴通讯股份有限公司 传输控制协议类型会话媒体流的传输方法及系统
US20170039774A1 (en) * 2014-04-14 2017-02-09 Tremolant Inc. Augmented Reality Communications
JP6126519B2 (ja) * 2013-11-29 2017-05-10 日本電信電話株式会社 空間投影装置、空間投影方法、空間投影プログラム及び記録媒体
CN106803921A (zh) * 2017-03-20 2017-06-06 深圳市丰巨泰科电子有限公司 基于ar技术的即时音视频通信方法及装置
CN206323408U (zh) * 2017-01-14 2017-07-11 国家电网公司 一种基于ims的融合视频通信系统
CN110266992A (zh) * 2019-06-24 2019-09-20 苏芯物联技术(南京)有限公司 一种基于增强现实的远程视频交互系统以及方法

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104158806B (zh) * 2014-08-13 2018-02-23 大唐移动通信设备有限公司 会话连接建立方法及设备、会话边界控制网元
CN108377355A (zh) * 2016-11-28 2018-08-07 中兴通讯股份有限公司 一种视频数据处理方法、装置及设备
CN109740476B (zh) * 2018-12-25 2021-08-20 北京琳云信息科技有限责任公司 即时通讯方法、装置和服务器

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101971618A (zh) * 2008-03-14 2011-02-09 阿尔卡特朗讯 实现移动终端的丰富视频的方法
CN102281293A (zh) * 2011-08-01 2011-12-14 中兴通讯股份有限公司 传输控制协议类型会话媒体流的传输方法及系统
JP6126519B2 (ja) * 2013-11-29 2017-05-10 日本電信電話株式会社 空間投影装置、空間投影方法、空間投影プログラム及び記録媒体
US20170039774A1 (en) * 2014-04-14 2017-02-09 Tremolant Inc. Augmented Reality Communications
CN206323408U (zh) * 2017-01-14 2017-07-11 国家电网公司 一种基于ims的融合视频通信系统
CN106803921A (zh) * 2017-03-20 2017-06-06 深圳市丰巨泰科电子有限公司 基于ar技术的即时音视频通信方法及装置
CN110266992A (zh) * 2019-06-24 2019-09-20 苏芯物联技术(南京)有限公司 一种基于增强现实的远程视频交互系统以及方法

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114760602A (zh) * 2022-04-18 2022-07-15 中国电信股份有限公司 全息通信方法、装置、系统及计算机可读存储介质
CN114760602B (zh) * 2022-04-18 2024-03-19 中国电信股份有限公司 全息通信方法、装置、系统及计算机可读存储介质
WO2024109582A1 (fr) * 2022-11-22 2024-05-30 中兴通讯股份有限公司 Procédé de traitement multimédia, serveur multimédia et support d'enregistrement

Also Published As

Publication number Publication date
CN112788274A (zh) 2021-05-11

Similar Documents

Publication Publication Date Title
WO2021088690A1 (fr) Procédé et appareil de communication basés sur la réalité augmentée
WO2021088691A1 (fr) Système de communication à réalité augmentée (ar) et procédé de communication à base d'ar
US11490033B2 (en) Video generating method, apparatus, electronic device and computer storage medium
US11895426B2 (en) Method and apparatus for capturing video, electronic device and computer-readable storage medium
WO2019086037A1 (fr) Procédé de traitement de substance vidéo, procédé de synthèse vidéo, dispositif terminal et support de stockage
EP2288104B1 (fr) Décomposition et recomposition flexible de flux de conférence multimédia utilisant les informations de contrôle en temps réel
WO2021104181A1 (fr) Procédé et appareil de communication basée sur la réalité augmentée
US8849900B2 (en) Method and system supporting mobile coalitions
WO2021185302A1 (fr) Procédé de diffusion en continu en direct et de configuration basé sur des téléphones en nuage, et appareil et système associés
US10104153B2 (en) Method and device for processing information
WO2010003332A1 (fr) Procédé, équipement et système destinés à réaliser une vidéoconférence
CN113163230A (zh) 视频消息生成方法、装置、电子设备及存储介质
CN105872832A (zh) 视频通话方法和系统以及显示装置
CN112929680A (zh) 直播间图像渲染方法、装置、计算机设备及存储介质
CN112533053B (zh) 直播互动方法、装置、电子设备及存储介质
JP2015527818A (ja) ビデオ会議環境のためのビデオ表示変更
WO2022022580A1 (fr) Procédé et dispositif d'interaction de diffusion en direct de réseau
CN112995692B (zh) 互动数据处理方法、装置、设备及介质
WO2021088973A1 (fr) Procédé et appareil d'affichage de flux continu en direct, dispositif électronique et support de stockage lisible
CN115941876A (zh) 音视频会议实现方法、装置、存储介质及计算机设备
CN115052186B (zh) 投屏方法及相关设备
CN115037978B (zh) 投屏方法及相关设备
WO2022206624A1 (fr) Procédé, appareil et système de communication à réalité augmentée
CN117729188B (zh) 一种基于WebRTC的水务视频采集系统及方法
CN114189488B (zh) 一种消息共享方法及服务器

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20885172

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20885172

Country of ref document: EP

Kind code of ref document: A1