WO2021104181A1 - Procédé et appareil de communication basée sur la réalité augmentée - Google Patents

Procédé et appareil de communication basée sur la réalité augmentée Download PDF

Info

Publication number
WO2021104181A1
WO2021104181A1 PCT/CN2020/130657 CN2020130657W WO2021104181A1 WO 2021104181 A1 WO2021104181 A1 WO 2021104181A1 CN 2020130657 W CN2020130657 W CN 2020130657W WO 2021104181 A1 WO2021104181 A1 WO 2021104181A1
Authority
WO
WIPO (PCT)
Prior art keywords
terminal device
sbc
media stream
data
call
Prior art date
Application number
PCT/CN2020/130657
Other languages
English (en)
Chinese (zh)
Inventor
高扬
左俊
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2021104181A1 publication Critical patent/WO2021104181A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/1066—Session management
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/1066—Session management
    • H04L65/1069—Session establishment or de-establishment
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/131—Protocols for games, networked simulations or virtual reality
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H04N7/141—Systems for two-way working between two video terminals, e.g. videophone
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/10—Architectures or entities
    • H04L65/1016—IP multimedia subsystem [IMS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/1066—Session management
    • H04L65/1083—In-session procedures
    • H04L65/1089—In-session procedures by adding media; by removing media

Definitions

  • the embodiments of the present application relate to the field of communication technology, and in particular, to a communication method and device based on augmented reality.
  • Long Term Evolution Voice Bearer Voice over long term evolution, VoLTE
  • 4G 4th generation
  • VoLTE makes the connection waiting time shorter when communicating between users, and the voice and video call quality is higher.
  • Augmented reality is a technology that ingeniously integrates virtual information with the real world. It uses a variety of technical methods such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, and sensing, and generates After the virtual information such as text, image, three-dimensional model, music, video, etc. is simulated and simulated, it is applied to the real world, and the two kinds of information complement each other, thereby achieving "enhancement" of the real world.
  • Augmented reality technology can not only effectively reflect the content of the real world, but also promote the display of virtual information content. There is currently no effective way to integrate AR into voice and video calls.
  • the embodiments of the present application provide a communication method and device based on augmented reality, and provide an implementation manner of integrating AR into a voice and video call, thereby improving user experience.
  • an embodiment of the present application provides an augmented reality-based communication method.
  • the method includes: the first terminal device determines the AR service type of this augmented reality AR call, and mediates the media stream of the AR call according to the AR service type.
  • the enhanced processing results in an enhanced AR media stream.
  • the determination of the AR service type of the AR call by the first terminal device can be implemented in any manner: one way is: the first terminal device responds to the AR service type selected by the user. Another way is that the first terminal device determines the default service type.
  • the default service type the default one among multiple service types. It may also be that different types of terminal devices correspond to different service types, and the default service type is the service type corresponding to the type of terminal device.
  • the media stream of the AR call may be generated by the first terminal device, or may be generated by the second terminal device that performs a call with the first terminal device.
  • the media stream of the AR call is generated by the first terminal device, and after obtaining the enhanced AR media stream, the first terminal device sends the enhanced AR media stream to the first session border controller (session border controller, SBC).
  • the media stream of the AR call is generated by the second terminal device, and the first terminal device may display the enhanced AR media stream to the user.
  • the first terminal device may also send the enhanced AR media stream to the first SBC, which in turn sends the enhanced AR media stream to the second terminal device, and displays the enhanced AR media stream to the user.
  • the first terminal device when initiating an AR call process to the second terminal device, sends the first terminal device to the AR server before performing media enhancement processing on the media stream of the AR call according to the AR service type to obtain the enhanced AR media stream.
  • a service request the first service request carries the AR service type; then the first terminal device receives the first service response sent by the AR server, and the first service response carries AR data.
  • the AR data is data required to execute the service corresponding to the AR service type.
  • the first terminal device performs media enhancement processing on the media stream of the AR call according to the AR service type to obtain the enhanced AR media stream, which can be implemented in the following manner: according to the AR service type, the AR data is used to perform media enhancement on the media stream of the AR call The enhanced AR media stream is processed.
  • the AR server sends AR to the terminal device when the first terminal device initiates an AR call to the second terminal device, so that there is no need to configure AR data on the terminal device, reducing the resource occupation of the terminal device.
  • the first terminal device before sending the first service request to the AR server, the first terminal device sends the second service request to the AR server; the first terminal device receives the second service response from the AR server, and the second service response is in the second service response.
  • the first terminal device determines the AR service type of this augmented reality AR call, the first terminal device responds to the AR service type selected by the user from the multiple service types.
  • the AR server sends multiple service types for the user to select to the first terminal device, and the user can select the service type according to requirements to provide convenience for the user.
  • the above method further includes: the first terminal device transmits auxiliary data between the first terminal device and the second terminal device through the data transmission channel, and the auxiliary data includes point cloud data stream, spatial video data, user perspective video, or user Operate one or more of the operation data generated by the AR interface.
  • a data transmission channel is established between the first terminal device and the second terminal device.
  • the above method may further include: the first terminal device sends the first operation data to the second terminal device through the data transmission channel, and the first operation data is transmitted by the user of the first terminal device to the first terminal device. Data generated by operations on the AR interface. Therefore, the second terminal device can perform media enhancement processing according to the first operation data and the media stream of the AR call.
  • the first terminal device receives the operation data generated by the user operating the AR interface from the second terminal device through the data transmission channel, so that the first terminal device performs media enhancement processing on the media stream of the AR call according to the operation data.
  • the AR data is a virtual model
  • a data transmission channel is established between the first terminal device and the second terminal device through the first SBC
  • the above method may further include: the first terminal device receives the data through the data transmission channel Second operating data from the second terminal device, the second operating data is data generated by the user of the second terminal device operating on the virtual model; the first terminal device uses AR data to perform media enhancement processing on the media stream of the first terminal device
  • Obtaining the enhanced AR media stream includes: the first terminal device renders the virtual model according to the second operation data, and performs media enhancement processing on the media stream of the AR call according to the rendered virtual model to obtain the enhanced AR media stream.
  • the second terminal device sends the operating data to the first terminal device, and the first terminal device renders the virtual model according to the operating data, and performs media enhancement processing on the media stream of the AR call according to the rendered virtual model to obtain enhanced AR Media streaming. Furthermore, the first terminal device is displayed to the user, which can reduce the transmission delay.
  • the first terminal device before the first terminal device sends the enhanced AR media stream to the first session boundary controller SBC corresponding to the first terminal device, the first terminal device sends a call request to the first SBC, and the call request carries the first Media description protocol (session description protocol, SDP) information of the terminal device, the SDP information of the first terminal device includes the parameters used by the first terminal device to create a data transmission channel; then, the first terminal device receives the call response sent from the first SBC, The call response carries the SDP information of the first SBC, and the SDP information of the first SBC includes parameters used by the first SBC to create a data transmission channel.
  • SDP session description protocol
  • the first terminal device before the first terminal device sends the enhanced AR media stream to the first SBC corresponding to the first terminal device, the first terminal device sends a call request to the first SBC according to the AR service type, and the call request carries the first SBC.
  • the SDP information of the terminal device includes the parameters of the media stream channel used by the first terminal device to create an AR media session between the first terminal device and the second terminal device, and the SDP information of the first terminal device is based on The AR service type is determined; the first terminal device receives the call response sent from the first SBC, the call response carries the SDP information of the first SBC, and the SDP information of the first SBC includes the first SBC used to create the first terminal device and the second terminal device The parameters of the media stream channel between AR media sessions.
  • the media stream channel supporting AR call is established through the SDP information that conforms to the AR service type, so that both parties in the call can integrate AR processing in the video call process.
  • the first terminal device sends the enhanced AR media stream to the first session boundary controller SBC corresponding to the first terminal device. This can be implemented in the following manner.
  • the first terminal device sends the enhanced AR media stream to the first SBC through the media stream channel. Send enhanced AR media stream.
  • the AR data includes a virtual model.
  • Media enhancement is performed on the media stream of the AR call according to the AR service type to obtain the enhanced AR media stream.
  • the first terminal device responds to the first terminal device
  • the user s operations on the virtual model perform media enhancement processing on the media stream of the AR call according to the operated virtual model to obtain an enhanced AR media stream.
  • the media stream direction of the AR call transmitted by the first terminal device and the second terminal device is unidirectional.
  • one-way can indicate that only the first terminal device sends the media stream to the second terminal device.
  • one-way can instruct the first terminal device to send the media stream to the second terminal device for enhancement processing, while the media stream sent by the second terminal device to the first terminal device does not perform enhancement processing.
  • the AR data includes a virtual model
  • the above method also includes that the first terminal device sends the virtual model to the edge computing node; thus, the first terminal device mediates the media stream of the AR call according to the AR service type.
  • the enhanced AR media stream is obtained by the enhancement processing, it can be implemented in the following manner: the first terminal device responds to the operation of the user of the first terminal device on the virtual model, and sends the third operation data to the edge computing node, and the third operation data is transferred from the first terminal device to the edge computing node.
  • a data transmission channel is established between the first terminal device and the second SBC corresponding to the second terminal device through the first SBC.
  • the first terminal device may send auxiliary data to the second SBC through the data transmission channel, for the second SBC to perform media enhancement processing on the media stream from the first terminal device according to the auxiliary data, and then perform media enhancement processing on the media stream after the media enhancement processing.
  • the auxiliary data may include one or more of point cloud data stream, spatial video data, user-view video, or operation data generated by the user operating the AR interface.
  • the first terminal device sends the fourth operation data to the second SBC through the data transmission channel
  • the fourth operation data is data generated by the user of the first terminal device operating the AR interface on the first terminal device.
  • the first terminal device sends the media stream of the first terminal device to the second SBC through the media stream channel, and further, the second SBC performs media enhancement processing on the media stream of the first terminal device according to the fourth operation data, and then enhances the media The processed media stream is sent to the second terminal device.
  • an embodiment of the present application also provides a communication method based on augmented reality, including: a first SBC receives a first call request sent by a first terminal device, the first call request carries SDP information of the first terminal device, and The SDP information of a terminal device includes parameters used by the first terminal device to create a data transmission channel and parameters used to create a media stream channel; the first SBC sends a first call response to the first terminal device, and the first call response carries the SBC SDP information.
  • the SDP information of the first SBC includes parameters used by the first SBC to create a data transmission channel and parameters used to create a media stream channel.
  • the data transmission channel is used to transmit the auxiliary data of the AR call between the first terminal device and the second terminal device;
  • the auxiliary data includes point cloud data stream, spatial video data, user view video, or user operation AR One or more of the operation data generated by the interface.
  • the above method may further include: the first SBC sends a second call request to the second SBC corresponding to the second terminal device, the second call request carries SDP information of the first SBC; the first SBC receives the second SBC 2.
  • the second call response sent by the SBC, the second call request carries SDP information of the second SBC, and the SDP information of the second SBC includes parameters used by the second SBC to create a data transmission channel and parameters used to create a media stream channel.
  • an embodiment of the present application provides an augmented reality-based communication method, including: a second SBC receives a second call request sent by a first SBC, the second call request carries SDP information of the first SBC, and The SDP information includes the parameters used by the first SBC to create the data transmission channel and the parameters used to create the media stream channel; then, the second SBC sends a second call response to the first SBC, and the second call response carries the first call response of the second SBC. SDP information.
  • the first SDP information of the second SBC includes parameters used by the second SBC to create a data transmission channel and parameters used to create a media stream channel.
  • the method before the second SBC sends the second call response to the first SBC, the method further includes: the second SBC sends a third call request to the second terminal device, and the third call request carries the second SBC One SDP information.
  • the first SDP information of the second SBC includes the parameters used by the second SBC to create the data transmission channel and the parameters used to create the media stream channel;
  • the second SBC receives the third call response sent by the second terminal device,
  • the third call response carries the SDP information of the second terminal device, and the SDP information of the second terminal device includes parameters used by the second terminal device to create a data transmission channel and parameters used to create a media stream channel.
  • the second SBC before the second SBC sends the second call response to the first SBC, when the second SBC determines that the second terminal device does not support AR, the second SBC sends a third call request to the second terminal device.
  • the third call request carries the second SDP information of the second SBC, and the second SDP information of the second SBC only includes the parameters used by the second SBC to create the media stream channel; then, the second SBC receives the third SDP sent by the second terminal device.
  • the third call response carries the SDP information of the second terminal device, and the SDP information of the second SBC includes the parameters used by the second terminal device to create the media stream channel.
  • the above method may further include: the second SBC receives the AR media stream from the first terminal device through the media stream channel, and receives the auxiliary data from the first terminal device through the data transmission channel; the auxiliary data includes One or more of point cloud data stream, spatial video data, user perspective video, or operation data generated by the user operating the AR interface; further, the second SBC performs media enhancement processing on the AR media stream according to the auxiliary data to obtain enhanced AR Media stream; Finally, the second SBC sends an enhanced AR media stream to the second terminal device.
  • an embodiment of the present application provides an augmented reality-based communication method, including: an AR server receives a first service request sent by a first terminal device, the first service request carries an AR service type; The first service response is sent, and the first service response carries AR data.
  • the AR server before receiving the first service request sent by the first terminal device, the AR server receives the second service request sent by the first terminal device; the second service response sent by the AR server to the first terminal device is 2.
  • the service response carries multiple service types for the user to select, and the AR service type is selected from multiple service types.
  • the above method may further include: the AR server receives a third service request sent by the second SBC corresponding to the second terminal device, the third service request carries the AR service type; the AR server sends the first service request to the second SBC The third service response, the third service response carries AR data.
  • an embodiment of the present application also provides a device, which is applied to a terminal device, and the beneficial effects can be referred to the description of the first aspect and will not be repeated here.
  • the device has the function of realizing the behavior in the method embodiment of the first aspect described above.
  • the function can be realized by hardware, or by hardware executing corresponding software.
  • the hardware or software includes one or more modules corresponding to the above-mentioned functions.
  • the structure of the device may include a sending unit, a receiving unit, and a processing unit. These units can perform the corresponding functions in the method embodiment of the first aspect described above. For details, refer to the detailed description in the method embodiment. Do not repeat them here.
  • the embodiments of the present application also provide a device, which is applied to the SBC, and the beneficial effects can be referred to the descriptions of the second and third aspects, which will not be repeated here.
  • the device has the function of realizing the behavior in the method embodiment of the second aspect or the third aspect.
  • the function can be realized by hardware, or by hardware executing corresponding software.
  • the hardware or software includes one or more modules corresponding to the above-mentioned functions.
  • the structure of the device includes a receiving unit, a processing unit, and a sending unit. These units can perform the corresponding functions in the above-mentioned method in the second aspect or in the embodiments of the third aspect. For details, please refer to the detailed description in the method example. Description, I won’t repeat it here.
  • the embodiments of the present application also provide a device, which is applied to the AR server, and the beneficial effects can be referred to the description of the fourth aspect, which will not be repeated here.
  • the device has the function of realizing the behavior in the method embodiment of the fourth aspect.
  • the function can be realized by hardware, or by hardware executing corresponding software.
  • the hardware or software includes one or more modules corresponding to the above-mentioned functions.
  • the structure of the device includes a sending unit, a receiving unit, and a processing unit. These units can perform the corresponding functions in the method embodiment of the fourth aspect. For details, please refer to the detailed description in the method embodiment. I won’t repeat it here.
  • an embodiment of the present application also provides a device, which is applied to the terminal device, and the beneficial effects can be referred to the description of the first aspect and will not be repeated here.
  • the structure of the communication device includes a processor and a communication interface, and may also include a memory.
  • the processor is configured to support the terminal device to execute the corresponding function of the first terminal device of the method of the first aspect through the communication interface.
  • the memory is coupled with the processor, and it stores program instructions and data necessary for the communication device.
  • the embodiments of the present application also provide a device, which is applied to the SBC (first SBC or second SBC), and the beneficial effects can be referred to the description of the second aspect or the third aspect.
  • the structure of the device includes a processor and a communication interface, and may also include a memory.
  • the processor is configured to support the application server to perform the corresponding function of the first SBC in the method of the second aspect through the communication interface, or to perform the first SBC.
  • the corresponding function of the second SBC in the three-way method.
  • the memory is coupled with the processor, and it stores program instructions and data necessary for the device.
  • an embodiment of the present application also provides a device, which is applied to the AR server, and the beneficial effects can be referred to the description of the fourth aspect, which is not repeated here.
  • the structure of the apparatus includes a processor and a communication interface, and may also include a memory.
  • the processor is configured to support the terminal device to perform the corresponding function in the above-mentioned fourth aspect method through the communication interface.
  • the memory is coupled with the processor, and it stores program instructions and data necessary for the device.
  • the present application also provides a computer-readable storage medium having instructions stored in the computer-readable storage medium, which when run on a computer, cause the computer to execute the methods described in the above aspects.
  • this application also provides a computer program product containing instructions, which when run on a computer, causes the computer to execute the methods described in the above aspects.
  • the present application also provides a computer chip connected to a memory, and the chip is used to read and execute a software program stored in the memory, and execute the methods described in the foregoing aspects.
  • FIG. 1 is a schematic diagram of a possible AR communication system architecture in an embodiment of this application
  • FIG. 2 is a schematic diagram of another possible AR communication system architecture in an embodiment of this application.
  • 3-7 are schematic diagrams of the flow of the AR-based communication method in the embodiments of this application.
  • Figures 8-12 are schematic diagrams of the structure of the provided device.
  • the present application provides an AR-based communication method and device, and provides an implementation manner of integrating AR into voice and video calls, thereby improving user experience.
  • Voice and video calls can but are not limited to using VoLTE, and can also be applied to voice and video calls provided by future technologies.
  • the communication system includes one or more session border controllers (SBC), AR server, application server, and multiple terminal devices.
  • SBC session border controllers
  • FIG. 1 two SBCs are taken as an example, namely a first SBC and a second SBC.
  • the first SBC is used to manage the first terminal device
  • the second SBC is used to manage the second terminal device.
  • different terminal devices can also have the same SBC to manage.
  • the third terminal device described in FIG. 1 is managed by the first SBC.
  • Two terminal devices can make AR calls through the communication system.
  • the terminal device in the embodiment of the present application may have AR media enhancement processing functions, such as image processing functions and data calculation functions, and can use AR technology to perform operations such as logical operations, screen rendering, or virtual scene synthesis on received media streams.
  • SBC is used to manage or control the session of the terminal device.
  • the first terminal device may perform media enhancement processing on the media stream on the first terminal device according to the AR service type selected by the user to obtain an enhanced AR media stream, and then send the enhanced AR media to the second terminal device through the first SBC flow.
  • the SBC includes a signaling plane function, for example, it can be used to receive a media stream from a terminal device it manages, and send the media stream received from the terminal device to another terminal device.
  • the SBC may also have image processing functions and data calculation functions.
  • the terminal device in the embodiment of the present application may be a device equipped with a camera and a video call function.
  • the terminal device may be a wearable device (such as an electronic watch), and the terminal device may also be a device such as a mobile phone or a tablet computer.
  • the embodiments of the present application do not impose special restrictions on the specific form of the terminal device.
  • the embodiments of the present application can be applied to a scenario in which terminal devices of both parties in a call support AR media enhancement processing. It can also be applied to scenarios where one party supports AR media enhancement processing. For the convenience of description, terminal devices that support AR media enhancement processing will be referred to as enhanced terminal devices in the following.
  • the AR server and the application server can be deployed separately or together, which is not specifically limited in this application.
  • the AR server is used for signaling interaction with terminal equipment.
  • the AR server can deploy a material library to provide materials for terminal devices to make AR calls.
  • the material library may include various materials, such as stickers of different styles, facial avatars with different expressions (such as cute cats, funny faces), or virtual portrait models of different styles, scene images of different styles, and so on.
  • the application server is used to process media service requests and provide related services such as voice and video calls.
  • IP multimedia system IP multimedia subsystem, IMS
  • the IMS core may include a call session control function (CSCF) and a home subscriber server (home subscriber server, HSS).
  • CSCF is the call control center of IMS core, which implements user access, authentication, session routing, and service triggering functions on the IP transmission platform.
  • the CSCF may include one of the serving-call session control function (S-CSCF), proxy-CSCF (Proxy CSCF, P-CSCF), query-CSCF (Interrogating-CSCF, I-CSCF), or Multiple.
  • the HSS is used to record the user's subscription data (such as user information, business data).
  • the SBC is also used to provide boundary control functions between the access network and the IMS core network and between the IMS core network, and can provide functions such as access control, quality of service control, and firewall traversal.
  • the CSCF to which different terminal devices belong may be the same or different.
  • the AS to which different terminal devices belong may be the same or different. For example, different terminal devices have different attributions, and the CSCF and AS to which they belong may be different.
  • the SBC may also support the establishment of a data transmission channel for the terminal devices of the two parties in the conversation.
  • the data transmission channel is used to transmit auxiliary data between terminal devices.
  • the auxiliary data may include one or more of point cloud data, spatial data (may also be referred to as spatial pose data), user perspective video, or virtual model.
  • Point cloud data refers to data recorded in the form of points. Each point can include spatial location information, as well as color information or reflection intensity information.
  • Spatial data can also be called geometric data. It is used to represent the position, shape, size distribution and other information of an object. It is a quantitative description of things and phenomena with positioning significance in the current world.
  • the virtual model may include one or more of a virtual portrait model, a virtual object model, and material images (such as stickers, cartoon avatars, etc.), or a virtual animation model.
  • the user's perspective video may be a video captured by the user through the rear camera of the terminal device, or a video captured by the front camera of the terminal device.
  • the SBC of the terminal device can support the establishment of a data transmission channel between the enhanced terminal device and the enhanced terminal device.
  • the data transmission channel is used to communicate between the enhanced terminal device and the non-enhanced terminal device.
  • Auxiliary data is transmitted between, and then SBC replaces non-enhanced terminal equipment to perform media enhancement processing.
  • an application server (AS), an AR server, and an IMS core can be deployed in a central node in the system.
  • the first SBC is deployed at the first edge node in the system
  • the AR media server is deployed at the central node in the system.
  • the second SBC is deployed at the second edge node of the system.
  • the edge node is closer to the users of the terminal device, and provides these users with edge computing services, forwarding services, etc., reducing response delay and bandwidth costs, and reducing the pressure on the central node.
  • the central node and the edge node may be deployed on the cloud.
  • the central node may be referred to as the central cloud
  • the edge node may be referred to as the edge cloud.
  • edge nodes corresponding to different terminal devices may be the same or different.
  • central nodes corresponding to different terminal devices may be the same or different.
  • FIG. 3 is a schematic flowchart of an AR-based communication method provided in an embodiment of this application. Take the first terminal device triggering the AR video enhancement process to the second terminal device as an example.
  • the first terminal device may display an interface for selecting an AR service type, and different service types may correspond to different scenarios.
  • one-way AR enhanced scenes such as scenes such as beauty, stickers, super scores, and expression-driven calls.
  • Another example is operating interactive call scenarios, such as advertising and distance education.
  • Another example is an image interactive call scene, such as an AR holographic call scene.
  • virtual and real overlapping call scenes such as remote guidance, etc.
  • S301 The first terminal device determines the AR service type of this AR call.
  • the determination of the AR service type of the AR call by the first terminal device can be implemented in any manner: one way is: the first terminal device responds to the AR service type selected by the user. Another way is that the first terminal device determines the default service type.
  • the default service type the default one among multiple service types. It may also be that different types of terminal devices correspond to different service types, and the default service type is the service type corresponding to the type of terminal device.
  • S302 Perform media enhancement processing on the media stream of the AR call according to the AR service type to obtain an enhanced AR media stream.
  • the media stream of the first terminal device may be generated by the first terminal device, for example, collected by a camera of the first terminal device. Or it can also be received from the second SBC of the second terminal device, that is, generated by the second terminal device, and sent to the first terminal device.
  • the media stream of the first terminal device is generated by the first terminal device, S303, the first terminal device sends the enhanced AR media stream to the first SBC corresponding to the first terminal device.
  • the first SBC sends the enhanced AR media stream to the second SBC.
  • the second SBC sends the enhanced AR media stream to the second terminal device.
  • the type of service available for the user to select can be configured in the terminal device according to the application scenario of the terminal device. In addition, it can also be sent by the AR server to the terminal device.
  • the media stream of the AR call may be generated by the second terminal device and sent to the first terminal device.
  • the first terminal device performs media enhancement processing on the media stream from the second terminal device according to the AR data, and then displays it to the user.
  • the first terminal device and the second terminal device are both enhanced terminal devices, and the first terminal device triggers the AR video to the second terminal device.
  • the enhancement process Take the enhancement process as an example.
  • the first terminal device initiates an AR call to the second terminal device, and sends a service request 1 to the AR server.
  • the AR server After receiving the service request 1, the AR server sends a service response 1 to the first terminal device, and the service response 1 carries multiple service types for the user to select.
  • the first terminal device After the first terminal device receives multiple service types for the user to select, it can be displayed to the user, so that the user can select one service type from the multiple service types as the service type of this call.
  • the user The selected service type is called the AR service type.
  • the service type may be configured on the terminal device.
  • the first terminal device may display to the user multiple service types that can be selected by the user. In this case, the above S401 and S402 are no longer executed. Initiate an AR call on the first terminal device, and execute S403.
  • the first terminal device In response to the AR service type selected by the user, the first terminal device sends a service request 2 to the AR server, and the service request 2 carries the AR service type.
  • the AR server sends a service response 2 to the first terminal device.
  • the service response 2 may carry AR data, and the AR data is data required to execute the service corresponding to the AR service type, such as model data, scene data, and so on.
  • the first terminal device sends a call request 1 to the first SBC.
  • the call request carries the media description protocol (session description protocol, SDP) information of the first terminal device.
  • SDP information of the first terminal device may include parameters used to create a media stream channel on the first terminal device.
  • Media streams can include audio streams and video streams.
  • the SDP information of the first terminal device may also include parameters used to create a data transmission channel on the first terminal device.
  • SDP includes parameters such as address information, media stream type, and supported codec format.
  • Media stream types can include video (video stream), audio (audio stream), and datachannel (auxiliary data).
  • the SDP can include parameters such as address information corresponding to different media stream types and supported codec formats. For example, in this case, an m-line description can be added to the SDP to describe the data transmission channel.
  • the call request 1 may, but is not limited to, use a session initiation protocol (session initiation protocol, SIP), and may also use other types of transmission protocols, which is not limited in this application.
  • SIP session initiation protocol
  • the first SBC After receiving the call request 1, the first SBC replaces the SDP information of the first terminal device in the call request 1 with the SDP information of the first SBC to obtain the call request 2, and sends the call request 2 to the S-CSCF.
  • the SDP information of the first SBC includes the parameters used to create the media stream channel on the first SBC.
  • the SDP information of the first terminal device includes a parameter for creating a data transmission channel
  • the SDP information of the first SBC includes a parameter for creating a data transmission channel.
  • S407 After receiving the call request 2, the S-CSCF forwards the call request 2 to the application server.
  • S408 The application server sends the call request 2 to the S-CSCF.
  • the S-CSCF forwards the call request 2 to the second SBC.
  • the second SBC may determine that the previous hop of the media stream channel is the first SBC according to the SDP information of the first SBC.
  • the S-CSCF and AS to which the first terminal device belongs and the S-CSCF and AS to which the second terminal device belongs are the same S-CSCF as an example.
  • the second SBC replaces the SDP information of the first SBC in the call request 2 with the SDP information of the second SBC to obtain the call request 3, and sends the call request 3 to the second terminal device.
  • the SDP information of the second SBC includes the parameters used to create the media stream channel on the second SBC.
  • the SDP information of the second SBC includes the parameters for creating the data transmission channel.
  • S411 The second terminal device sends a service request 3 to the AR server.
  • the AR server sends a service response 3 to the second terminal device.
  • the service response 3 can carry AR data.
  • the second terminal device sends a call response 3 to the second SBC, and the call response 3 may carry the SDP information of the second terminal device.
  • the SDP information of the second terminal device includes the parameters used to create the media stream channel on the second terminal device.
  • the SDP information of the second terminal device includes the parameters for creating the data transmission channel.
  • the second SBC After receiving the call response 3, the second SBC sends the call response 2 to the S-CSCF.
  • the call response 3 may carry the SDP information of the second SBC.
  • S415 After receiving the call response 2, the S-CSCF forwards the call response 2 to the application server.
  • the application server After receiving the call response 2, the application server forwards the call response 2 to the S-CSCF.
  • the call response 2 can carry the SDP information of the second SBC.
  • S417 The S-CSCF forwards the call response 2 to the first SBC.
  • the first SBC After receiving the call response 2, the first SBC sends the call response 1 to the first terminal device.
  • the call response 1 carries the SDP information of the first SBC.
  • call response 1 to call response 4 may use the 183 message type.
  • the media stream can be transmitted between the first terminal device and the second terminal device through the media stream channel.
  • the first terminal device and the second terminal device are both enhanced terminal device architectures and can be applied to various application scenarios. The following exemplarily describes the processing and trend of media streams in several scenarios.
  • Example 1 One-way AR enhancement scenario. For example, scenes such as beauty, stickers, super scores, and expression-driven calls.
  • the first terminal device needs to perform AR processing during a video call with the second terminal device.
  • the first terminal device performs media enhancement processing on the media stream of the first terminal device to obtain an enhanced AR media stream.
  • the first terminal device performs media enhancement processing on the video collected by the camera on the first terminal device.
  • beautification when the user of the first terminal device triggers the beautification operation, the first terminal device performs the beautification operation on the face included in the video image in the media stream generated by the first terminal device, and performs the beautification process on the face
  • the media stream is sent to the first SBC through the media stream channel. Then the first SBC is sent to the second SBC, and then the second SBC is sent to the second terminal device.
  • the second terminal device may perform the beauty operation on the face included in the video image in the received media stream generated by the second terminal device, and perform the beauty processing
  • the latter media stream is sent to the second SBC through the media stream channel, and then forwarded by the second SBC to the first SBC, and finally forwarded by the first SBC to the first terminal device.
  • Example 2 Operating interactive call scenarios, such as advertising and distance education. Take the first terminal device of user 1 that needs to perform AR processing during a video call with the second terminal device of user 2 as an example.
  • the first terminal device performs media enhancement processing on the media stream of the first terminal device according to the operation data to obtain an enhanced AR media stream, and then sends the enhanced AR media stream to the second terminal through the first SBC and the second SBC Device and displayed to the user of the first terminal device.
  • the operating data is data generated by the user on the first terminal device operating on the AR interface of the first terminal device.
  • the operation data may be data generated for the operation of the virtual model.
  • the user of the first terminal device is referred to as user 1 and the user of the second terminal device is referred to as user 2.
  • the housing provider corresponds to the second terminal device
  • the house buyer is the first terminal device.
  • the first terminal device uses the media stream of the house buyer as the background stream.
  • the media stream of the house buyer can be collected by the rear camera on the first terminal device.
  • the first terminal device obtains the spatial pose data of the buyer's perspective from the media stream of the buyer, and renders the second-hand house model according to the spatial pose data of the buyer's perspective and the operation data of the second-hand house model operated by the buyer.
  • the house model is superimposed on the background stream of the house buyer to enhance the AR media stream and displayed to the house buyer.
  • the first terminal device also sends the enhanced AR media stream to the second terminal device through the first SBC and the second SBC.
  • the second terminal device displays the enhanced AR media stream to the buyer.
  • the first terminal device may send the operation data of user 1 to the second terminal device through the data transmission channel, and send the media stream generated by the first terminal device to the second terminal device through the media stream channel.
  • the second terminal device performs media enhancement processing on the media stream generated by the first terminal device according to the operation data of the user 1 to obtain an enhanced AR media stream, which is displayed to the user 2.
  • the second terminal device may send the operation data of user 2 to the first terminal device through the data transmission channel, and send the media stream generated by the second terminal device to the first terminal device through the media stream channel.
  • the device performs media enhancement processing on the media stream generated by the second terminal device according to the operation data of user 1 to obtain an enhanced AR media stream, which is displayed to user 1.
  • this method has a smaller time delay.
  • the buyer can operate the second-hand house model, and the buyer's second terminal device will operate the second-hand house model through the data transmission channel.
  • the operating data is sent to the first terminal device, and then the first terminal device renders the second-hand house model according to the spatial pose data from the perspective of the purchaser and the operation data of the second-hand house model operated by the purchaser, and compares the rendered second-hand house model with that of the purchaser
  • the background stream is superimposed to obtain an enhanced AR media stream and displayed to the home buyer.
  • the first terminal device also sends the enhanced AR media stream to the second terminal device through the first SBC and the second SBC.
  • the second terminal device displays the enhanced AR media stream to the buyer.
  • Example 3 image interactive call scene, such as AR holographic call scene, etc.
  • the first terminal device acquires a first media stream and first auxiliary data.
  • the first media stream is an image of the environment where the user 1 of the first terminal device is located, and the first auxiliary data includes a portrait image of the user 1.
  • the second terminal device acquires a second media stream and second auxiliary data.
  • the second media stream is an image of the environment where the user 2 of the second terminal device is located, and the second auxiliary data includes a portrait image of the user 2.
  • the first terminal device sends the first auxiliary data to the second terminal device through the data transmission channel, and the second terminal device obtains the real-time expression and action data of the user 1 from the first auxiliary data of the user 1, and drives the virtual model of the user 1;
  • Use the second media stream of user 2 as the background stream obtain the spatial pose data of user 2’s perspective according to the background stream, and render the virtual model of user 1 according to the spatial pose data of user 2’s perspective, and superimpose it with the second media stream Show to user 2.
  • the second terminal device sends the second auxiliary data to the first terminal device through the data transmission channel, and the first terminal device obtains the real-time expression and action data of the user 2 from the second auxiliary data of the user 2 to drive the virtual model of the user 2;
  • the first terminal device uses the first media stream of user 1 as the background stream, obtains the spatial pose data of user 1’s perspective according to the background stream, and renders the virtual model of user 2 according to the spatial pose data of user 1’s perspective, which is consistent with the first The media stream is superimposed and displayed to user 1.
  • Example 4 virtual and real superimposed call scenarios, such as remote guidance, etc.
  • the first terminal device obtains auxiliary data (such as point cloud data) and the first media stream.
  • a depth camera may be configured on the first terminal device to obtain point cloud data, and the point cloud data is used to generate a depth map of the shooting picture, such as a red (R) green (G) blue (B)-depth (D) image.
  • the first terminal device recognizes the spatial position of the object from the background stream with higher accuracy. After recognizing the object, superimposes a virtual model or logo on the background stream to obtain an enhanced AR media stream, and then enhances the AR media stream.
  • the AR media stream is displayed to the user of the first terminal device and sent to the second terminal device through the media stream channel.
  • the first terminal device is an enhanced terminal device
  • the second terminal device is a non-enhanced terminal device
  • the first terminal device communicates to the second terminal device.
  • the first terminal device sends a call request 1 to the first SBC.
  • the call request carries the media description protocol (session description protocol, SDP) information of the first terminal device.
  • the SDP information of the first terminal device may include parameters used to create a media stream channel on the first terminal device.
  • Media streams can include audio streams and video streams.
  • the call request 1 may, but is not limited to, use a session initiation protocol (session initiation protocol, SIP), and may also use other types of transmission protocols, which is not limited in this application.
  • SIP session initiation protocol
  • S506-S510 please refer to S406-S410, which will not be repeated here.
  • the media stream can be transmitted between the first terminal device and the second terminal device through the media stream channel.
  • the architecture in which the first terminal device is an enhanced terminal device and the second terminal device is a non-enhanced terminal device can be applied to a variety of application scenarios. The following exemplarily describes the processing and trend of media streams in several scenarios.
  • Example 5 one-way AR enhancement scenario.
  • scenes such as beauty, stickers, super scores, and expression-driven calls.
  • the first terminal device needs to perform AR processing during a video call with the second terminal device.
  • the first terminal device performs media enhancement processing on the media stream of the first terminal device to obtain an enhanced AR media stream.
  • the first terminal device performs media enhancement processing on the video collected by the camera on the first terminal device.
  • beautification when the user of the first terminal device triggers the beautification operation, the first terminal device performs the beautification operation on the face included in the video image in the media stream generated by the first terminal device, and performs the beautification process on the face
  • the media stream is sent to the first SBC through the media stream channel. Then the first SBC is sent to the second SBC, and then the second SBC is sent to the second terminal device.
  • the second terminal device does not support media enhancement processing, and media enhancement processing is not performed on the media stream of the second terminal device.
  • Example 6 operating interactive call scenarios, such as advertising, distance education, etc. Take the first terminal device of user 1 that needs to perform AR processing during a video call with the second terminal device of user 2 as an example.
  • the first terminal device performs media enhancement processing on the media stream of the first terminal device according to the operating data to obtain an enhanced AR media stream, and then sends the enhanced AR media stream to the second terminal device through the first SBC and the second SBC, and displays it to the first terminal device.
  • the operating data is data generated by the user on the first terminal device operating on the AR interface of the first terminal device.
  • the operation data may be data generated for the operation of the virtual model.
  • the user of the first terminal device is referred to as user 1 and the user of the second terminal device is referred to as user 2.
  • Example 7 virtual and real superimposed call scenarios, such as remote guidance. Similar to the solution adopted in Example 4, please refer to the description of Example 4, which will not be repeated here.
  • the first terminal device is an enhanced terminal device
  • the second terminal device is a non-enhanced terminal device
  • the first terminal device communicates to the second terminal device.
  • the second SBC to which the second terminal device belongs supports media enhancement.
  • the establishment of a data transmission channel may be supported between the first terminal device and the second SBC.
  • S601-S609 please refer to S401-S409, which will not be repeated here.
  • the second SBC determines that the second terminal device does not support media enhancement processing (that is, does not have media enhancement processing capabilities), and sends a service request 3 to the AR server.
  • the AR server sends a service response 3 to the second SBC, and the service response 3 carries AR data.
  • the second SBC determines that the second terminal device does not support media enhancement processing, and replaces the SDP information of the first SBC in the call request 2 with the second SDP information of the second SBC to obtain the call request 3, and sends the call request 3 to The second terminal device.
  • the second SDP information of the second SBC includes the parameters used to create the media stream channel on the second SBC.
  • the SDP information of the first SBC includes the parameters for creating the data transmission channel
  • the second SDP information of the second SBC also does not include the parameters for creating the data transmission channel.
  • the second terminal device sends a call response 3 to the second SBC, and the call response 3 may carry the SDP information of the second terminal device.
  • the SDP information of the second terminal device includes the parameters used to create the media stream channel on the second terminal device.
  • the second SBC After receiving the call response 3, the second SBC sends the call response 2 to the S-CSCF.
  • the call response 3 may carry the first SDP information of the second SBC.
  • the SDP information of the second SBC includes the parameters used to create the media stream channel on the second SBC.
  • the SDP information of the second SBC includes the parameters for creating the data transmission channel.
  • the media stream can be transmitted between the first terminal device and the second terminal device through the media stream channel.
  • the first terminal device is an enhanced terminal device
  • the second terminal device is an architecture of a non-enhanced terminal device, which can be applied to a variety of application scenarios. The following exemplarily describes the processing and trend of media streams in several scenarios.
  • Example 8 One-way AR enhancement scenario. For example, scenes such as beauty, stickers, super scores, and expression-driven calls. See Example 5, which will not be repeated here.
  • Example 9 operating interactive call scenarios, such as advertising, distance education, etc. Take the first terminal device of user 1 that needs to perform AR processing during a video call with the second terminal device of user 2 as an example.
  • the first terminal device performs media enhancement processing on the media stream of the first terminal device according to the operation data to obtain an enhanced AR media stream, and then sends the enhanced AR media stream to the second terminal through the first SBC and the second SBC Device and displayed to the user of the first terminal device.
  • the operating data is data generated by the user on the first terminal device operating on the AR interface of the first terminal device.
  • the second terminal device is a non-enhanced terminal device, that is, it does not support users to perform operations.
  • the first terminal device may send the operation data of user 1 to the second SBC through the data transmission channel, and send the media stream generated by the first terminal device to the second SBC through the media stream channel.
  • the SBC performs media enhancement processing on the media stream generated by the first terminal device according to the operation data of the user 1 to obtain an enhanced AR media stream, which is displayed to the user 2.
  • this method has a smaller time delay.
  • Example 10 virtual and real superimposed call scenarios, such as remote guidance, etc. See Example 4, which will not be repeated here.
  • FIG. 7 a schematic flow chart of the AR-based communication method provided by this embodiment of the application, with the first terminal device being an enhanced terminal device, the second terminal device being a non-enhanced terminal device, and the first terminal device reporting to the second terminal device Take the device triggering the AR video enhancement process as an example.
  • transferring the function of the rendering model from the terminal device to the MEC can reduce the power consumption of the terminal device and reduce the processing resources occupied by the terminal device.
  • the first terminal device sends a service request 4 to the MEC, and the service request 4 carries a virtual model.
  • S718 The MEC sends a service response 4 to the first terminal device.
  • the media stream can be transmitted between the first terminal device and the second terminal device through the media stream channel.
  • the architecture in which the first terminal device is an enhanced terminal device and the second terminal device is a non-enhanced terminal device can be applied to a variety of application scenarios. The following exemplarily describes the processing and trend of media streams in several scenarios.
  • Example 11 one-way AR enhancement scenario. See Example 5, which will not be repeated here.
  • Example 12 operating interactive call scenarios, such as advertising and distance education. Take the first terminal device of user 1 that needs to perform AR processing during a video call with the second terminal device of user 2 as an example.
  • the first terminal device sends the operation data to the MEC, and the operation data is data generated by the user on the first terminal device operating on the AR interface of the first terminal device.
  • MEC renders the virtual model according to the operating data, and then sends the rendered virtual model to the first terminal device.
  • the first terminal device superimposes the rendered virtual model and the media stream generated by the first terminal device to obtain an enhanced AR media stream , And then the enhanced AR media stream is sent to the second terminal device through the first SBC and the second SBC, and displayed to the user of the first terminal device.
  • Example 13 virtual and real superimposed call scenarios, such as remote guidance. Similar to the solution adopted in Example 4, please refer to the description of Example 4, which will not be repeated here.
  • the embodiment of the present application provides an apparatus 800, which is specifically configured to implement the method executed by the terminal device (such as the first terminal device or the second terminal device) in the above method embodiment.
  • the apparatus 800 As shown in FIG. 8, it includes a processing unit 801, a sending unit 802, a receiving unit 803, and a display unit 804.
  • the processing unit 801 is configured to determine the AR service type of this augmented reality AR call, and perform media enhancement processing on the media stream of the AR call according to the AR service type to obtain an enhanced AR media stream;
  • the sending unit 802 is configured to send an enhanced AR media stream to the first session boundary controller SBC.
  • the display unit 804 is used to display an enhanced AR media stream.
  • the sending unit 802 is further configured to: before the processing unit 801 performs media enhancement processing on the media stream of the AR call according to the AR service type to obtain an enhanced AR media stream, the first terminal device When initiating an AR call process to the second terminal device, sending a first service request to the AR server, where the first service request carries the AR service type;
  • the receiving unit 803 is configured to receive a first service response sent by the AR server, where the first service response carries AR data;
  • the processing unit 801 when performing media enhancement processing on the media stream of the AR call according to the AR service type to obtain an enhanced AR media stream, is specifically configured to use the AR data to communicate the media of the AR call according to the AR service type.
  • the stream undergoes media enhancement processing to obtain an enhanced AR media stream.
  • the sending unit 802 is further configured to send a second service request to the AR server before sending the first service request to the AR server;
  • the receiving unit 803 is further configured to receive a second service response from the AR server, where the second service response carries multiple service types for the user to select; the AR service type is selected from the multiple service types Selected.
  • the receiving unit 803 is further configured to receive auxiliary data from the second terminal device through a data transmission channel, and the auxiliary data includes point cloud data stream, spatial video data, and user perspective. One or more of the video or the operation data generated by the user operating the AR interface.
  • the sending unit 802 is further configured to send auxiliary data to the second terminal device through a data transmission channel.
  • the sending unit 802 is configured to send first operation data to the second terminal device through the data transmission channel, and the first operation data is transmitted by the user of the first terminal device to the AR on the first terminal device.
  • the AR data is a virtual model
  • the data transmission channel is established between the first terminal device and the second terminal device through the first SBC
  • the receiving unit 803. It is further configured to receive second operation data from a second terminal device through the data transmission channel, the second operation data being data generated by a user of the second terminal device operating on the virtual model;
  • the unit 801 is specifically configured to render the virtual model according to the second operation data when performing media enhancement processing on the media stream of the first terminal device using the AR data to obtain an enhanced AR media stream, and according to The rendered virtual model performs media enhancement processing on the media stream of the first terminal device to obtain an enhanced AR media stream.
  • the sending unit 802 is further configured to send a call request to the first SBC before sending the enhanced AR media stream to the first session border controller SBC corresponding to the first terminal device.
  • the request carries the media description protocol SDP information of the first terminal device, and the SDP information of the first terminal device includes the parameters used by the first terminal device to create the data transmission channel;
  • the receiving unit 803 is further configured to A call response sent from a first SBC is received, the call response carries the SDP information of the first SBC, and the SDP information of the first SBC includes the parameters used by the first SBC to create the data transmission channel.
  • the sending unit 802 is further configured to send the enhanced AR media stream to the first SBC according to the AR service type before sending the enhanced AR media stream to the first session border controller SBC corresponding to the first terminal device.
  • Send a call request the call request carries the SDP information of the first terminal device, and the SDP information of the first terminal device includes the first terminal device used to create a connection between the first terminal device and the second terminal device Parameters of the media stream channel of the AR media session, and the SDP information of the first terminal device is determined according to the AR service type;
  • the receiving unit 803 is further configured to receive a call response sent from a first SBC, where the call response carries SDP information of the first SBC, and the SDP information of the first SBC includes the The parameters of the media stream channel of the AR media session between the first terminal device and the second terminal device.
  • the sending unit 802 when the sending unit 802 sends an enhanced AR media stream to the first SBC corresponding to the first terminal device, it is specifically configured to send the media stream to the first SBC through the media stream channel.
  • Enhanced AR media streaming when the sending unit 802 sends an enhanced AR media stream to the first SBC corresponding to the first terminal device, it is specifically configured to send the media stream to the first SBC through the media stream channel.
  • the AR data includes a virtual model
  • the processing unit 801 when the processing unit 801 performs media enhancement processing on the media stream of the AR call according to the AR service type to obtain an enhanced AR media stream, it is specifically configured to respond to the The user of the first terminal device performs the media enhancement process on the media stream of the AR call according to the operation of the virtual model on the virtual model to obtain the enhanced AR media stream.
  • the AR data includes a virtual model
  • the sending unit 802 is further configured to send the virtual model to an edge computing node
  • the processing unit 801 is further configured to respond to the first The operation of the user of the terminal device on the virtual model instructs the sending unit 802 to send third operating data to the edge computing node, the third operating data being generated by the user of the first terminal device operating on the virtual model
  • the receiving unit 803 is also configured to receive an enhanced virtual model obtained after the edge computing device renders the virtual model according to the third operation data; the processing unit is specifically configured to transfer the media of the AR call The stream and the enhanced virtual model are superimposed to obtain the enhanced AR media stream.
  • an embodiment of the present application provides an apparatus 900, which is specifically configured to implement the method executed by the SBC (such as the first SBC or the second SBC) in the above method embodiment.
  • the structure of the apparatus 900 is as follows As shown in FIG. 9, a receiving unit 901 and a sending unit 902 are included.
  • the receiving unit 901 is configured to receive a first call request sent by a first terminal device, where the first call request carries SDP information of the first terminal device, and the SDP information of the first terminal device includes information used by the first terminal device.
  • the sending unit 902 is configured to send a first call response to the first terminal device, where the first call response carries SDP information of the SBC, and the SDP information of the first SBC includes that the first SBC is used to create The parameters of the data transmission channel and the parameters used to create the media stream channel.
  • the data transmission channel is used to transmit auxiliary data for AR calls between the first terminal device and the second terminal device;
  • the auxiliary data includes point cloud data stream, spatial video data, and user One or more items of perspective video or operation data generated by the user operating the AR interface.
  • the sending unit 902 is further configured to send a second call request to a second SBC corresponding to a second terminal device, where the second call request carries SDP information of the first SBC;
  • the receiving unit 901 is further configured to receive a second call response sent by the second SBC, where the second call request carries SDP information of the second SBC, and the SDP information of the second SBC includes the first Second, the SBC is used to create the parameters of the data transmission channel, and the parameters used to create the media stream channel.
  • the receiving unit 901 is configured to receive a second call request sent by a first SBC, where the second call request carries SDP information of the first SBC, and the SDP information of the first SBC includes that the first SBC is used to create data transmission The parameters of the channel and the parameters used to create the media stream channel;
  • the sending unit 902 is configured to send a second call response to the first SBC, where the second call response carries the first SDP information of the second SBC, and the first SDP information of the second SBC includes the second SBC The parameters used to create the data transmission channel and the parameters used to create the media stream channel.
  • the sending unit 902 before the sending unit 902 sends the second call response to the first SBC, it is further configured to send a third call request to the second terminal device, and the third call request carries the first call response.
  • the first SDP information of the SBC where the first SDP information of the second SBC includes parameters used by the second SBC to create a data transmission channel and parameters used to create a media stream channel;
  • the receiving unit 901 is further configured to receive a third call response sent by the second terminal device, where the third call response carries the SDP information of the second terminal device, and the SDP information of the second terminal device includes all
  • the second terminal device is used to create the parameters of the data transmission channel, and the parameters used to create the media stream channel.
  • a processing unit 903 may also be included. Before sending the second call response to the first SBC, the sending unit 902 is further configured to send a third call request to the second terminal device when the processing unit 903 determines that the second terminal device does not support AR.
  • the third call request carries second SDP information of the second SBC, and the second SDP information of the second SBC only includes parameters used by the second SBC to create a media stream channel;
  • the receiving unit 901 is further configured to receive a third call response sent by the second terminal device, where the third call response carries the SDP information of the second terminal device, and the SDP information of the second SBC includes all
  • the second terminal device is used to create parameters of the media stream channel.
  • the receiving unit is further configured to receive the AR media stream from the first terminal device through the media stream channel, and receive auxiliary data from the first terminal device through the data transmission channel;
  • the auxiliary data includes one or more of point cloud data stream, spatial video data, user-view video, or operation data generated by the user operating the AR interface;
  • the processing unit 903 is further configured to perform media enhancement processing on the AR media stream according to the auxiliary data to obtain an enhanced AR media stream;
  • the sending unit 902 is further configured to send the enhanced AR media stream to the second terminal device.
  • an embodiment of the application provides a device 1000, which is specifically used to implement the method executed by the AR server in the above method embodiment.
  • the structure of the device 1000 is shown in FIG. 10 and includes a receiving unit 1001. , Sending unit 1002.
  • the receiving unit 1001 is configured to receive a first service request sent by a first terminal device, where the first service request carries an AR service type;
  • the sending unit 1002 is configured to send a first service response to the first terminal device, where the first service response carries AR data.
  • the receiving unit 1001 is further configured to receive a second service request sent by the first terminal device before receiving the first service request sent by the first terminal device;
  • the sending unit 1002 is further configured to send a second service response to the first terminal device, where the second service response carries multiple service types for the user to select, and the AR service type is selected from the multiple service types. Selected in.
  • the receiving unit 1001 is further configured to receive a third service request sent by a second SBC corresponding to the second terminal device, where the third service request carries the AR service type;
  • the sending unit 1002 is further configured to send a third service response to the second SBC, where the third service response carries the AR data.
  • the division of units in the embodiments of this application is illustrative, and is only a logical function division. In actual implementation, there may be other division methods.
  • the functional units in the various embodiments of this application can be integrated into one processing unit. In the device, it can also exist alone physically, or two or more units can be integrated into one module.
  • the above-mentioned integrated unit can be realized in the form of hardware or software function module.
  • the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solution of the present application essentially or the part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , Including several instructions to enable a terminal device (which may be a personal computer, a mobile phone, or a network device, etc.) or a processor to execute all or part of the steps of the method in each embodiment of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disk and other media that can store program code .
  • the AR server, the application server, the terminal device, the SBC, and the CSCF may all be presented in the form of dividing various functional modules in an integrated manner.
  • the "module” here can refer to a specific ASIC, circuit, processor and memory that executes one or more software or firmware programs, integrated logic circuit, and/or other devices that can provide the above-mentioned functions.
  • the SBC, CSCF, AR server or application server may adopt the form shown in FIG. 11.
  • the apparatus 1100 shown in FIG. 11 includes at least one processor 1101 and a communication interface 1102.
  • a memory 1103 may also be included.
  • the processor 1101 in FIG. 11 can execute instructions by calling a computer stored in the memory 1103, so that the SBC can execute any of the foregoing method embodiments.
  • the processor 1101 may communicate with other devices through the communication interface 1102. For example, the processor 1101 receives a media stream sent by a terminal device through the communication interface 1102, or sends a media stream to the terminal device.
  • the memory 1103 stores computer-executable instructions for realizing the functions of the sending unit, the receiving unit, and the processing unit in FIG. 9.
  • the processor 1101 in 11 calls the computer execution instructions stored in the memory 1103, and implements it in combination with the communication interface 1102.
  • the processor 1101 in FIG. 11 can call the computer to execute instructions stored in the memory 1103, so that the AR server can execute any of the above The method executed by the AR server in the method embodiment.
  • the processor 1101 may communicate with other devices through the communication interface 1102. For example, the processor 1101 receives a service request from a terminal device through the communication interface 1102, and sends a service response to the terminal device.
  • the memory 1103 stores computer-executable instructions for realizing the functions of the sending unit and the receiving unit in FIG. 10, and the functions/implementation processes of the sending unit and the receiving unit in FIG. 10 can be implemented by the processor 1101 in FIG. 11
  • the computer execution instructions stored in the memory 1103 are called and implemented in combination with the communication interface 1102.
  • the processor 1101 in FIG. 11 can call the computer execution instructions stored in the memory 1103, so that the application server can execute any of the above The method executed by the application server in the method embodiment.
  • the processor 1101 in FIG. 11 can execute instructions by calling a computer stored in the memory 1103, so that the CSCF can execute any of the foregoing method implementations.
  • the specific connection medium between the foregoing processor 1101 and the memory 1103 is not limited in the embodiment of the present application.
  • the memory 1103 and the processor 1101 are connected through a bus 1104 in the figure.
  • the bus 1104 is represented by a thick line in the figure. Is limited.
  • the bus 1104 can be divided into an address bus, a data bus, a control bus, and so on. For ease of presentation, only one thick line is used to represent in FIG. 11, but it does not mean that there is only one bus or one type of bus.
  • the terminal device may adopt the form shown in FIG. 12.
  • the apparatus 1200 shown in FIG. 12 includes at least one processor 1201, a communication interface 1202, and optionally a memory 1203.
  • the device 1200 further includes a display 1204.
  • the device may also include a sensor 1205 for detecting the user's operation on the AR interface.
  • the processor 1201 may have the function of sending and receiving data and can communicate with other devices.
  • the processor 1201 may send a media stream to the SBC, or may receive a media stream from the SBC, as shown in the device shown in FIG. 12
  • an independent data transceiver module may also be set, such as a communication interface 1202 for sending and receiving data; when the processor 1201 communicates with other devices, it may perform data transmission through the communication interface 1202, as in the embodiment of the present application,
  • the device 1201 may send a media stream to the SBC through the communication interface 1202, and may also receive a media stream from the SBC through the communication interface 1202.
  • the processor 1201 in FIG. 12 can call the computer execution instructions stored in the memory 1203, so that the terminal device can execute the method described in any of the foregoing method embodiments.
  • a method executed by a terminal device (such as a first terminal device or a second terminal device).
  • the memory 1203 stores computer-executable instructions for realizing the functions of the sending unit, receiving unit, processing unit, and display unit in FIG. 8, and the functions of the sending unit, receiving unit, processing unit, and display unit in FIG.
  • the implementation process can all be implemented by the processor 1201 in FIG. 12 calling a computer execution instruction stored in the memory 1203.
  • the memory 1203 stores computer execution instructions for implementing the functions of the processing unit in FIG. 8, and the functions/implementation process of the processing unit in FIG. 6 can be executed by the processor 1201 in FIG. 12 calling a computer stored in the memory 1203.
  • the function/implementation process of the sending unit and the receiving unit in FIG. 8 can be realized through the communication interface 1202 in FIG. 12.
  • the processor 1201 when the processor 1201 performs the function of the display unit, such as an operation involving displaying an image, such as displaying an AR interface, the processor 1201 may display an image through the display 1204 in the device 1200.
  • the processor 1201 when the processor 1201 performs the function of the display unit, it may also display an image through a display in another device, such as sending a display instruction to the other device to instruct to display the image.
  • the embodiment of the present application does not limit the specific connection medium between the processor 1201, the memory 1203, the communication interface 1202, the display 1204, and the sensor 1205.
  • the embodiment of the present application is connected through a bus 1206 as an example in the figure, and the bus 1206 is represented by a thick line in the figure.
  • the connection mode between other components is only for schematic description and is not to be taken as a limitation.
  • the bus 1206 can be divided into an address bus, a data bus, a control bus, and so on. For ease of presentation, only one thick line is used in FIG. 12 to represent it, but it does not mean that there is only one bus or one type of bus.
  • the embodiments of the present application also provide a computer storage medium, the storage medium stores a software program, and the software program can implement any one or more of the above when read and executed by one or more processors.
  • the embodiment provides a method executed by a service platform or an edge computing device.
  • the computer storage medium may include: U disk, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk and other media that can store program codes.
  • the embodiments of the present application also provide a chip, which includes a processor, configured to implement the functions of the AR server or the terminal device or the SBC server involved in any one or more of the above embodiments.
  • the chip further includes a memory, and the memory is used for necessary program instructions and data to be executed by the processor.
  • the chip can be composed of a chip, or it can include a chip and other discrete devices.
  • system and “network” in this article are often used interchangeably in this article.
  • the term “and/or” in this article is only an association relationship that describes the associated objects, which means that there can be three kinds of relationships, for example, A and/or B, which can mean: A alone exists, A and B exist at the same time, exist alone B these three situations.
  • the character “/” in this text generally indicates that the associated objects before and after are in an "or” relationship.
  • at least one referred to in this application refers to one, or more than one, that includes one, two, three and more; “multiple” refers to two, or more than two, that is, two, Three and more.
  • B corresponding to A means that B is associated with A, and B can be determined according to A.
  • determining B based on A does not mean that B is determined only based on A, and B can also be determined based on A and/or other information.
  • the terms "including” and “having” in the embodiments of the present application, claims and drawings are not exclusive. For example, a process, method, system, product, or device that includes a series of steps or modules is not limited to the listed steps or modules, and may also include unlisted steps or modules.
  • the processor in the embodiments of the present application may be a central processing unit (central processing unit, CPU), or other general-purpose processors, digital signal processors (digital signal processors, DSP), and application-specific integrated circuits. (application specific integrated circuit, ASIC), field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof.
  • the general-purpose processor may be a microprocessor or any conventional processor.
  • the method steps in the embodiments of the present application can be implemented by hardware, and can also be implemented by a processor executing software instructions.
  • Software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (programmable ROM) , PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically erasable programmable read-only memory (electrically EPROM, EEPROM), register, hard disk, mobile hard disk, CD-ROM or well-known in the art Any other form of storage medium.
  • An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium.
  • the storage medium may also be an integral part of the processor.
  • the processor and the storage medium may be located in the ASIC.
  • the ASIC can be located in a network device or a terminal device.
  • the processor and the storage medium may also exist as discrete components in the network device or the terminal device.
  • the above embodiments it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof.
  • software it can be implemented in the form of a computer program product in whole or in part.
  • the computer program product includes one or more computer programs or instructions.
  • the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.
  • the computer program or instruction may be stored in a computer readable storage medium or transmitted through the computer readable storage medium.
  • the computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server integrating one or more available media.
  • the usable medium may be a magnetic medium, such as a floppy disk, a hard disk, and a magnetic tape; it may also be an optical medium, such as a DVD; and it may also be a semiconductor medium, such as a solid state disk (SSD).

Landscapes

  • Engineering & Computer Science (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Business, Economics & Management (AREA)
  • General Business, Economics & Management (AREA)
  • Telephonic Communication Services (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Abstract

La présente invention concerne un procédé et un appareil de communication basée sur la réalité augmentée (RA), qui fournissent un moyen d'implémentation pour intégrer la RA dans des appels vocaux et vidéo, ce qui permet d'améliorer l'expérience utilisateur. Des dispositifs terminaux prennent en charge un traitement d'amélioration RA, qui implémente un traitement d'amélioration multimédia sur des flux multimédias d'appels en RA, et réalise en outre l'intégration d'un traitement de RA dans un processus d'appel vidéo. De plus, un canal de transmission de données peut être établi entre des dispositifs terminaux améliorés au moyen de SBC, et est utilisé pour transmettre certaines données auxiliaires. Quand les dispositifs terminaux ne prennent pas en charge la RA, un traitement d'amélioration multimédia peut également être effectué par les SBC. Les SBC peuvent également établir un canal de transmission de données avec les dispositifs terminaux, le canal étant utilisé pour recevoir des données auxiliaires en provenance des dispositifs terminaux, et un traitement d'amélioration multimédia est ensuite effectué sur les flux multimédias des appels en RA sur la base des données auxiliaires.
PCT/CN2020/130657 2019-11-29 2020-11-20 Procédé et appareil de communication basée sur la réalité augmentée WO2021104181A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201911207574.5 2019-11-29
CN201911207574.5A CN112887258B (zh) 2019-11-29 2019-11-29 一种基于增强现实的通信方法及装置

Publications (1)

Publication Number Publication Date
WO2021104181A1 true WO2021104181A1 (fr) 2021-06-03

Family

ID=76039196

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/130657 WO2021104181A1 (fr) 2019-11-29 2020-11-20 Procédé et appareil de communication basée sur la réalité augmentée

Country Status (2)

Country Link
CN (1) CN112887258B (fr)
WO (1) WO2021104181A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115919103A (zh) * 2022-10-28 2023-04-07 慕思健康睡眠股份有限公司 基于智能床垫的通话方法、智能床垫及存储介质

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116633905A (zh) * 2022-02-10 2023-08-22 华为技术有限公司 一种通信方法、装置及通信系统
WO2024050744A1 (fr) * 2022-09-08 2024-03-14 Zte Corporation Systèmes et procédés de communication de réalité augmentée sur la base d'un canal de données

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2009036782A1 (fr) * 2007-09-18 2009-03-26 Vrmedia S.R.L. Dispositif et procédé de traitement d'informations pour une assistance technique à distance
CN104158806A (zh) * 2014-08-13 2014-11-19 大唐移动通信设备有限公司 会话连接建立方法及设备、会话边界控制网元
CN106803921A (zh) * 2017-03-20 2017-06-06 深圳市丰巨泰科电子有限公司 基于ar技术的即时音视频通信方法及装置
US20190068653A1 (en) * 2017-08-31 2019-02-28 Metaswitch Networks Ltd. Media bypass

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1560368A1 (fr) * 2004-01-30 2005-08-03 France Telecom Procédé d'établissement d'une session multimédia entre un équipement appelant et un équipement appelé d'un réseau du type à sous domaine multimédia et système de communication mettant en oeuvre ce procédé
EP2223464A4 (fr) * 2007-12-20 2015-08-26 Optis Wireless Technology Llc Procédé permettant d'établir une connexion multimédia locale dans un système de communication
FR2928805B1 (fr) * 2008-03-14 2012-06-01 Alcatel Lucent Procede permettant la mise en oeuvre de la video enrichie sur les terminaux mobiles.
CN102281293B (zh) * 2011-08-01 2017-04-05 中兴通讯股份有限公司 传输控制协议类型会话媒体流的传输方法及系统
US9990770B2 (en) * 2011-12-20 2018-06-05 Intel Corporation User-to-user communication enhancement with augmented reality
WO2015123774A1 (fr) * 2014-02-18 2015-08-27 Sulon Technologies Inc. Système et procédé pour des applications de réalité augmentée et de réalité virtuelle
CN107395551A (zh) * 2016-05-16 2017-11-24 中兴通讯股份有限公司 一种信令架构及其实现呼叫处理的方法和装置
DE102016118647B4 (de) * 2016-09-30 2018-12-06 Deutsche Telekom Ag Augmented-Reality-Kommunikationssystem und Augmented-Reality-Interaktionsvorrichtung
CN108377355A (zh) * 2016-11-28 2018-08-07 中兴通讯股份有限公司 一种视频数据处理方法、装置及设备
CN107222468B (zh) * 2017-05-22 2020-12-18 北京邮电大学 增强现实处理方法、终端、云端服务器和边缘服务器
CN109802932B (zh) * 2017-11-17 2022-04-12 华为技术有限公司 一种直播通信的方法和装置

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2009036782A1 (fr) * 2007-09-18 2009-03-26 Vrmedia S.R.L. Dispositif et procédé de traitement d'informations pour une assistance technique à distance
CN104158806A (zh) * 2014-08-13 2014-11-19 大唐移动通信设备有限公司 会话连接建立方法及设备、会话边界控制网元
CN106803921A (zh) * 2017-03-20 2017-06-06 深圳市丰巨泰科电子有限公司 基于ar技术的即时音视频通信方法及装置
US20190068653A1 (en) * 2017-08-31 2019-02-28 Metaswitch Networks Ltd. Media bypass

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115919103A (zh) * 2022-10-28 2023-04-07 慕思健康睡眠股份有限公司 基于智能床垫的通话方法、智能床垫及存储介质

Also Published As

Publication number Publication date
CN112887258A (zh) 2021-06-01
CN112887258B (zh) 2022-12-27

Similar Documents

Publication Publication Date Title
WO2021104181A1 (fr) Procédé et appareil de communication basée sur la réalité augmentée
US11895426B2 (en) Method and apparatus for capturing video, electronic device and computer-readable storage medium
WO2021088690A1 (fr) Procédé et appareil de communication basés sur la réalité augmentée
WO2021088691A1 (fr) Système de communication à réalité augmentée (ar) et procédé de communication à base d'ar
TWI732240B (zh) 視頻檔案的生成方法、裝置及儲存媒體
WO2021023055A1 (fr) Procédé d'appel vidéo
WO2020215818A1 (fr) Procédé et système d'appel et appareil associé
EP4246957A1 (fr) Procédé, système et dispositif électronique de photographie
WO2017125050A1 (fr) Procédé, dispositif et support de stockage pour le traitement d'image
WO2014008841A1 (fr) Procédé et appareil de traitement d'image
WO2018086548A1 (fr) Procédé et appareil d'affichage d'interface
EP4221193A1 (fr) Appareil et procédé d'appel vidéo
WO2010003332A1 (fr) Procédé, équipement et système destinés à réaliser une vidéoconférence
CN110448902A (zh) 对外部设备进行虚拟化映射与控制的方法、装置与系统
WO2022127670A1 (fr) Procédé et système d'appel, et dispositif associé
WO2022111599A1 (fr) Procédé et appareil d'interaction d'appel, dispositif et support de stockage
CN116758201B (zh) 三维场景的渲染处理方法、设备、系统及计算机存储介质
US20240022619A1 (en) Augmented reality communication method, apparatus, and system
EP4152756A1 (fr) Procédé de recommandation de dispositif et dispositif électronique
WO2022151687A1 (fr) Procédé et appareil de génération d'image photographique de groupe, dispositif, support de stockage, programme informatique et produit
CN111367598B (zh) 动作指令的处理方法、装置、电子设备及计算机可读存储介质
WO2023067940A1 (fr) Dispositif de traitement d'informations, système de traitement d'informations et procédé de traitement d'informations
WO2023125105A1 (fr) Système et procédé d'acquisition d'image d'écran entre dispositifs
WO2024016103A1 (fr) Procédé et appareil d'affichage d'images
WO2023000662A1 (fr) Procédé de distribution de contenu multimédia et dispositif

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20893340

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20893340

Country of ref document: EP

Kind code of ref document: A1