CN114945213A

CN114945213A - Holographic communication system and method for synchronizing multiple data channels

Info

Publication number: CN114945213A
Application number: CN202210509737.0A
Authority: CN
Inventors: 彭莉; 蒋玉香; 袁晓志; 马金兰; 朱晓洁
Original assignee: China Telecom Corp Ltd
Current assignee: China Telecom Corp Ltd
Priority date: 2022-05-11
Filing date: 2022-05-11
Publication date: 2022-08-26

Abstract

The embodiment of the invention provides a holographic communication system and a method for synchronizing multiple data channels, wherein the system comprises: the system comprises a service control device, a media computing device and a plurality of terminals, wherein the service control device comprises a media computing resource scheduling module, and the media computing device comprises a plurality of media computing modules; the terminal sends a call request to the service control equipment, the media computing resource scheduling module distributes target media computing power modules for each media based on each media type identifier in the call request, and the identifier information of each target media computing power module is sent to the terminal; the terminal establishes media channels with the target media computing power modules respectively according to the identification information and sends the collected media data to the corresponding target media computing power modules through the media channels; the media calculation module processes the media data to obtain target media data and sends the target media data to the terminal; the terminal integrates and presents the target media data, and processing of the multidimensional media data in the holographic communication process is achieved.

Description

Holographic communication system and method for synchronizing multiple data channels

Technical Field

The invention relates to the technical field of mobile communication, in particular to a holographic communication system and a method for synchronizing multiple data channels.

Background

AR (Augmented Reality)/VR (Virtual Reality) is one of the currently important applications of 5G (5th Generation Mobile Communication Technology) networks. It is expected that in the near future, information interaction forms will evolve from AR/VR to high fidelity XR (Extended Reality), even information interaction based on holographic communications. The holographic technique can reproduce any objects including human bodies, equipment, buildings, natural landscapes and the like in a hundred percent three-dimensional way. Holographic communication organically combines the transmission capability, information interaction capability and holographic technology of a communication network, utilizes the holographic display technology to capture images of people and surrounding objects at remote positions, transmits holographic data through the network, uses laser beam projection at a terminal, projects real-time dynamic stereoscopic images in a hologram mode, and can provide a completely immersive holographic interaction experience for 6G (6th generation mobile communication systems) users at any time and any place.

In the process of holographic communication, how to process multidimensional media data supporting video, audio, holographic data and the like becomes a technical problem to be solved urgently.

Disclosure of Invention

The embodiment of the invention aims to provide a holographic communication system and a holographic communication method for synchronizing multiple data channels so as to realize processing of multidimensional media data in the holographic communication process. The specific technical scheme is as follows:

in a first aspect, an embodiment of the present invention provides a holographic communication system with multiple data channels synchronized, where the system includes: the system comprises service control equipment, media computing equipment and a plurality of terminals, wherein the service control equipment comprises a media computing resource scheduling module, the media computing equipment comprises a plurality of media computing modules, and the media computing modules are used for realizing preset service functions;

the terminal is used for sending a call request to the service control equipment, wherein the call request comprises media type identifiers corresponding to various media;

a media computing resource scheduling module in the service control device, configured to allocate a corresponding target media computing power module to each media based on each media type identifier in the call request, and send identifier information of each target media computing power module to the terminal;

the terminal is used for respectively establishing media channels with the target media computing power modules according to the identification information of the target media computing power modules and sending the collected media data to the corresponding target media computing power modules through the established media channels;

the media computing module is used for processing the received media data to obtain target media data, and sending the target media data to the terminal through a media channel between the media computing module and the terminal;

and the terminal is used for receiving each target media data and integrating and presenting each target media data.

Optionally, the media type identifier is used to identify a type of the media, the type of the media includes at least one of an audio type, a video type, a holographic data type, a data text type, and a haptic data type;

the media computing power resource scheduling module is specifically configured to allocate a corresponding target media computing power module to each media based on each media type identifier in the call request and the function and performance of each media computing power module currently in an idle state.

Optionally, the terminal includes a calling terminal and a called terminal;

the calling terminal is used for sending a call request to the service control equipment;

the media computing resource scheduling module is specifically configured to send identification information of each target media computing module to the calling terminal and the called terminal respectively;

the calling terminal is specifically used for establishing a first media channel with each target media computing power module according to the identification information of each target media computing power module, and sending the collected media data to the corresponding target media computing power module through each established first media channel;

the media calculation module is specifically configured to receive the media data sent by the calling terminal, process the media data to obtain target media data, and send the target media data to the called terminal through a second media channel between the target media data and the called terminal;

the called terminal is used for respectively establishing a second media channel with each target media computing power module according to the identification information of each target media computing power module, receiving each target media data through each established second media channel, and integrating and presenting each target media data.

Optionally, the calling terminal is further configured to collect initial media data, and divide the initial media data into media data corresponding to each first media channel according to the type of the media data processed by the target media computing power module corresponding to each first media channel;

the calling terminal comprises a first media channel synchronization module,

and the first media channel synchronization module is used for adding corresponding data acquisition time stamps to the media data corresponding to each first media channel.

Optionally, the media effort module includes a synchronization control sub-module,

the synchronization control sub-module is configured to extract a timestamp included in the media data when it is determined that the media data needs to be subjected to service processing, insert the extracted timestamp into the media data after the media computation module performs service processing on the media data, obtain target media data, and send the target media data to the called terminal through a second media channel between the target media data and the called terminal.

Optionally, the media force module is specifically configured to:

and receiving the media data sent by the calling terminal, and sending the media data serving as target media data to the called terminal through a second media channel between the calling terminal and the called terminal under the condition that the media data needs to be transparently transmitted.

Optionally, the called terminal comprises a second media channel synchronization module,

the second media channel synchronization module is configured to, for each received target media data, extract a timestamp in each target media data;

and the called terminal is used for selecting the target media data with the same timestamp from all the target media data to integrate and presenting the integrated result.

Optionally, the service control device further comprises a session management module,

the calling terminal is specifically configured to send a call request to a session management module of the service control device;

the session management module is used for forwarding the call request to the media computing power resource scheduling module;

the media computing resource scheduling module is specifically configured to send identification information of each target media computing module to the session management module;

and the session management module is used for respectively forwarding the identification information of each target media computing power module to the calling terminal and the called terminal.

In a second aspect, an embodiment of the present invention provides a holographic communication method for synchronizing multiple data channels, which is applied to a terminal in a holographic communication system for synchronizing multiple data channels, where the holographic communication system for synchronizing multiple data channels further includes a server, the server includes a media computing resource scheduling module and multiple media computing modules, and the media computing modules are used to implement a preset service function, and the method includes:

sending a call request to a server, wherein the call request comprises media type identifiers corresponding to various media;

receiving identification information of a target media computing power module distributed for each media by a media computing power resource scheduling module of the server based on each media type identification in the call request;

establishing a media channel with each target media computing power module of the server side according to the identification information of each target media computing power module;

and sending the collected media data to the corresponding target media computing power module through each established media channel.

Optionally, the method further comprises:

acquiring initial media data, and dividing the initial media data into media data corresponding to each media channel according to the type of the media data processed by a target media computing power module corresponding to each media channel;

and adding corresponding data acquisition time stamps to the media data corresponding to each media channel.

Optionally, the method further comprises:

receiving target media data obtained after the media data are processed by each target media computing module;

and integrating and presenting the target media data.

Optionally, the integrating and presenting each of the target media data includes:

extracting a time stamp in each target media data aiming at each received target media data;

and selecting target media data with the same timestamp from the target media data for integration, and presenting an integration result.

In a third aspect, an embodiment of the present invention provides a holographic communication method for synchronizing multiple data channels, which is applied to a server in a holographic communication system for synchronizing multiple data channels, where the holographic communication system for synchronizing multiple data channels further includes a terminal, the server includes a media computing resource scheduling module and multiple media computing modules, and the media computing modules are used to implement a preset service function, and the method includes:

the server receives a call request sent by a terminal, wherein the call request comprises media type identifiers corresponding to various media;

a media computing resource scheduling module of the server allocates a corresponding target media computing module for each media based on each media type identifier in the call request;

the media computing resource scheduling module sends the identification information of each target media computing module to the terminal;

a media computing module of the server receives media data sent by the terminal and processes the media data to obtain target media data;

and the media calculation module sends the target media data to the terminal through a media channel between the media calculation module and the terminal.

the media computing power resource scheduling module of the server allocates a corresponding target media computing power module for each media based on each media type identifier in the call request, and the method comprises the following steps:

and the media computing resource scheduling module of the server allocates a corresponding target media computing module for each media based on each media type identifier in the call request and the function and performance of each media computing module in an idle state at present.

Optionally, the receiving, by the media force module of the server, the media data sent by the terminal, and processing the media data to obtain target media data includes:

and the media computing module of the server receives the media data sent by the terminal, extracts the timestamp contained in the media data under the condition that the media data needs to be subjected to business processing, performs business processing on the media data, and inserts the extracted timestamp into the processed media data to obtain target media data.

and the media computing module of the server receives the media data sent by the terminal, and takes the media data as target media data under the condition that the media data needs to be transparently transmitted.

Optionally, the server further includes a session management module,

the method for receiving the call request sent by the terminal by the server includes:

a session management module of the server receives a call request sent by the terminal and forwards the call request to the media computing resource scheduling module;

the media computing resource scheduling module sends the identification information of each target media computing module to the terminal, and the method comprises the following steps:

the media computing power resource scheduling module sends the identification information of each target media computing power module to the session management module;

and the session management module forwards the identification information of each target media computing power module to the terminal.

In a fourth aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor and the communication interface complete communication between the memory and the processor through the communication bus;

a memory for storing a computer program;

a processor for implementing the method steps of any of the first aspect when executing a program stored in the memory.

In a fifth aspect, the present invention provides a computer-readable storage medium, in which a computer program is stored, and the computer program, when executed by a processor, implements the method steps of any one of the first aspect.

Embodiments of the present invention also provide a computer program product comprising instructions, which when run on a computer, cause the computer to perform the method steps according to any of the first aspect described above.

The embodiment of the invention has the following beneficial effects:

in the holographic communication system and method for synchronizing multiple data channels provided by the embodiments of the present invention, the media computing resource scheduling module may allocate a corresponding target media computing module to each media included in the call request, and then each target media computing module may perform synchronous processing of multiple data channels on the received media data, and further, the terminal may integrate and present each target media data, thereby implementing distributed processing and data synchronous processing of multidimensional media data in the holographic communication process.

Of course, not all of the advantages described above need to be achieved at the same time in the practice of any one product or method of the invention.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other embodiments can be obtained by referring to these drawings.

FIG. 1 is a schematic diagram of a multi-data channel synchronous holographic communication system according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of another multi-data channel synchronized holographic communication system according to an embodiment of the present invention;

FIG. 3 is a schematic signaling interaction diagram of multi-data channel synchronous holographic communication according to an embodiment of the present invention;

FIG. 4 is a flow chart of a holographic communication method for multiple data channel synchronization according to an embodiment of the present invention;

FIG. 5 is a flow chart of another method for holographic communication with multiple data channel synchronization according to an embodiment of the present invention;

fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived from the embodiments of the present invention by those skilled in the art based on the description, are within the scope of the present invention.

In the related art, the holographic communication service is a dedicated application customized based on the user's requirement, and specifically, when the user needs to use the holographic communication service, the user can only customize and develop through a system integration business, but cannot apply for and open as the voice and video services, so that the user can use the holographic communication service, which causes the defects that the existing holographic communication service has a limited application range, the customized and developed system integration business easily causes high service deployment cost, low equipment utilization rate and the like, and how to process multidimensional media data supporting video, audio, holographic data and the like in the holographic communication process becomes a technical problem to be solved urgently.

In order to implement distributed processing and data synchronization processing of multidimensional media data in a holographic communication process, an embodiment of the present invention provides a holographic communication system and method for synchronizing multiple data channels, where the holographic communication system for synchronizing multiple data channels includes: the system comprises a service control device, a media computing device and a plurality of terminals, wherein the service control device comprises a media computing resource scheduling module, the media computing device comprises a plurality of media computing modules, and the media computing modules are used for realizing preset service functions; a terminal sends a call request to service control equipment, wherein the call request comprises media type identifiers corresponding to various media; a media computing resource scheduling module in the service control equipment allocates corresponding target media computing power modules for each media based on each media type identifier in the call request, and sends the identifier information of each target media computing power module to the terminal; the terminal establishes media channels with the target media computing power modules respectively according to the identification information of the target media computing power modules, and sends the collected media data to the corresponding target media computing power modules through the established media channels; the media calculation module processes the received media data to obtain target media data, and sends the target media data to the terminal through a media channel between the target media data and the terminal; and the terminal receives each target media data and integrates and presents each target media data.

In the holographic communication system with multiple synchronous data channels provided by the embodiment of the invention, the media computing resource scheduling module can allocate corresponding target media computing modules to each media contained in the call request, and then each target media computing module can perform synchronous processing on multiple data channels on the received media data, and further, the terminal can integrate and present each target media data, so that distributed processing and data synchronous processing of multi-dimensional media data in the holographic communication process are realized.

Referring to fig. 1, a holographic communication system 100 for multiple data channel synchronization according to an embodiment of the present invention includes: the system comprises a service control device 120, a media computing device 130 and a plurality of terminals 110, wherein the service control device 120 comprises a media computing resource scheduling module 121, the media computing device 130 comprises a plurality of media computing modules 131, and the media computing modules 131 are used for realizing preset service functions.

In an example, the service control device 120 and the media computing device 130 may be separate devices, or may belong to the same server device. The terminal may be a holographic device or the like capable of acquiring and/or presenting holographic data. The media computing resource scheduling module 121 and the media computing resource module 131 may be separate devices, or different units, processes, and the like on the devices. The preset service function may be rendering, transcoding, encoding and decoding of media data, and the like.

The terminal 110 is configured to send a call request to the service control device 120, where the call request includes media type identifiers corresponding to the media.

The terminal 110 may be a calling terminal or a called terminal, and when the call request is sent, the terminal 110 may be the calling terminal. The call request may include a media type identifier corresponding to each media corresponding to the call, and the media type identifier may be used to identify the type of the media. In some embodiments, the type of media may include at least one of an audio type, a video type, a holographic data type, a datatext type, and a haptic data type, although other types may be included.

The media computing power resource scheduling module 121 in the service control device 120 is configured to allocate a corresponding target media computing power module to each media based on each media type identifier in the call request, and send identifier information of each target media computing power module to the terminal 110.

The service control device 120 receives a call request sent by the terminal 110, and the media computing resource scheduling module 121 in the service control device 120 may allocate a corresponding target media computing module to each media based on each media type identifier in the call request. In some embodiments, the media computing resource scheduling module 121 may allocate a corresponding target media computing power module to each media based on each media type identifier in the call request and the function and performance of each media computing power module currently in an idle state.

One media computing module can realize one service function and can also realize multiple service functions, namely, one media computing module can process one type of media data and can also process multiple types of media data simultaneously. For example, the media force module for encoding and decoding can encode and decode media data of a haptic data type, and the media force module that can simultaneously process a plurality of types of media data can render, encode and decode media data of a video type or a holographic data type, and the like. The media computing module may be, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a process, or the like.

The functions and performances of different media computing power modules may be the same or different, and the media computing power resource scheduling module 121 may allocate corresponding target media computing power modules to each media one by one according to each media type having computing power demand in the call request and the functions and performances of each media computing power module currently in an idle state. For example, the function of the media force computing module may be a business function or the like that the media force computing module can implement, and the performance of the media force computing module may be the operating speed, the storage space, or the like of the media force computing module.

The media calculation resource scheduling module 121 may feed back, to the terminal 110, identification information of the target media calculation module allocated for the call after allocating the corresponding target media calculation module to each media in the call one by one, where in one example, the identification information may be coding information capable of identifying the target media calculation module, or address information of the target media calculation module.

And the terminal 110 is configured to establish a media channel with each target media computing power module according to the identification information of each target media computing power module, and send the acquired media data to the corresponding target media computing power module through each established media channel.

In an example, the terminal 110 receives address information of each target media computing power module fed back by the media computing power resource scheduling module 121, and establishes media channels with each target media computing power module one by one according to each address information.

The terminal 110 may collect initial media data through a camera, a microphone, and the like, divide the collected initial media data according to the type of the media data processed by each target media computing power module to obtain each media data, and send each divided media data to the corresponding target media computing power module through each established media channel. The initial media data collected by the terminal 110 is the media data that needs to perform holographic communication.

Illustratively, a target media computing power module processes media data of a holographic data type, and the terminal 110 sends the media data of the holographic data type divided from the collected initial media data to the target media computing power module through a media channel between the terminal and the target media computing power module; another target media power module processes media data of the haptic data type, the terminal 110 transmits the media data of the haptic data type divided from the collected initial media data to the another target media power module through a media channel with the another target media power module, and so on.

The media force module 131 is configured to process the received media data to obtain target media data, and send the target media data to the terminal 110 through a media channel between the terminal 110 and the target media data.

The media calculation module 131 receives the media data sent by the terminal 110 through the media channel with the terminal 110, processes the received media data, for example, performs rendering, encoding and decoding, transcoding, or transparent transmission on the received media data to obtain target media data, and sends the target media data to the terminal 110 through the media channel with the terminal 110.

The types of media contained in the call request may include at least one of an audio type, a video type, a holographic data type, a data text type, and a haptic data type, that is, the call request is directed to multidimensional media data, the media calculation resource scheduling module 121 allocates corresponding target media calculation modules to each media contained in the call request one by one, and each target media calculation module may perform multi-data channel synchronous processing on the received media data, thereby implementing distributed processing of the multidimensional media data.

And the terminal 110 is configured to receive each target media data, and integrate and present each target media data.

The terminal 110 may act as a called terminal when receiving the target media data. The terminal 110 receives each target media data through the media channel between the terminal and each target media computing module, and then can synchronously integrate each target media data, and present the integrated result, so as to implement data synchronization processing of the multidimensional media data.

In the holographic communication system with multiple synchronous data channels provided by the embodiment of the invention, the media computing resource scheduling module can allocate corresponding target media computing modules to each media contained in the call request, and then each target media computing module can perform synchronous processing on multiple data channels on the received media data, and further, the terminal can integrate and present each target media data, so that distributed processing and data synchronous processing of multi-dimensional media data in the holographic communication process are realized. The type of the media contained in the call request can comprise at least one of an audio type, a video type, a holographic data type, a data text type and a tactile data type, so that the multi-data-channel synchronous holographic communication of the embodiment of the invention can provide service requirements for supporting multi-dimensional media data holographic communication services such as audio, video, holographic, tactile and data text for 6G users, customized development is not needed, the difficulty in deployment and popularization of the holographic communication services is reduced, and the development of the holographic communication services is promoted.

In some embodiments, as shown in fig. 2, the terminal 110 may include a calling terminal 111 and a called terminal 112.

Correspondingly, the sending of the call request from the terminal 110 to the service control device 120 may specifically be: the calling terminal 111 sends a call request to the service control device 120.

The media calculation resource scheduling module 121 in the service control device 120 sends the identification information of each target media calculation module to the terminal 110, and specifically may be: the media calculation resource scheduling module 121 sends the identification information of each target media calculation module to the calling terminal 111 and the called terminal 112, respectively.

The terminal 110 establishes media channels with the target media force computing modules according to the identification information of the target media force computing modules, and sends the collected media data to the corresponding target media force computing modules through the established media channels, which may specifically be: the calling terminal 111 receives the identification information of each target media computing power module, establishes first media channels with each target media computing power module one by one according to the identification information of each target media computing power module, and sends the collected media data to the corresponding target media computing power module through each established first media channel.

The called terminal 112 receives the identification information of each target media computing power module, and establishes a second media channel with each target media computing power module one by one according to the identification information of each target media computing power module.

The media calculation module 131 processes the received media data to obtain target media data, and sends the target media data to the terminal through a media channel between the terminal and the media calculation module, and specifically, the media calculation module may: the media calculation module 131 receives the media data sent by the calling terminal 111, processes the media data to obtain target media data, and sends the target media data to the called terminal 112 through a second media channel between the target media data and the called terminal 112.

The receiving, integrating and presenting of each target media data by the terminal 110 may specifically be: the called terminal 112 receives each target media data through each second media channel established between the called terminal and each target media computing module, and integrates and presents each target media data.

In some embodiments, the calling terminal 111 may be further configured to collect initial media data, and divide the initial media data into media data corresponding to each first media channel according to a type of the media data processed by the target media computing power module corresponding to each first media channel.

The calling terminal 111 may include a first media channel synchronization module, where the first media channel synchronization module may be configured to add a corresponding data acquisition timestamp to media data corresponding to each first media channel.

The calling terminal 111 may collect initial media data through a camera, a microphone, and the like, and for the collected initial media data, divide the initial media data into media data corresponding to each first media channel according to the type of media data processed by the target media computation module corresponding to each established first media channel. The first media channel synchronization module in the calling terminal 111 may add a corresponding data acquisition timestamp to the media data corresponding to each first media channel, so that the media data sent to the corresponding target media computing power module through each established first media channel all carries the timestamp when the media data is acquired, so that each target media computing power module can perform synchronization processing on the corresponding media data.

The calling terminal 111 can divide the media data of the collected initial media data, and can process or not process the divided media data according to the performance of the calling terminal 111, and correspondingly, the media data sent to each target media computing power module by the calling terminal 111 can also carry an identifier of whether the media data needs to be processed and what kind of processing needs to be performed, so that the media computing power module can process the received media data as soon as possible.

In some embodiments, the media strength module 131 may include a synchronization control sub-module, and the synchronization control sub-module may be configured to, in a case that it is determined that the media data needs to be subjected to service processing, extract a timestamp included in the media data, insert the extracted timestamp after the media strength module performs service processing on the media data, obtain target media data, and send the target media data to the called terminal 112 through a second media channel between the target media data and the called terminal 112.

The calling terminal may not process the divided media data, or partially process the divided media data, and correspondingly, the media force calculation module needs to process the media data, after the media force calculation module 131 receives the media data sent by the calling terminal 111, it may first determine whether the media data needs to be subjected to service processing, and under the condition that the media data needs to be subjected to service processing, the synchronization control sub-module may extract a timestamp included in the media data before the media force calculation module 131 performs service processing on the received media data, and insert the extracted timestamp after the media force calculation module 131 completes service processing on the media data, so as to obtain target media data, where the target media data carries a timestamp (the timestamp carried in the target media data is the timestamp corresponding to the target media data during acquisition), further, the target media data is sent to the called terminal 112 through a second media channel between the called terminal 112 and the target media data, so that the called terminal 112 can perform time synchronization processing on each target media data according to the timestamp carried in the target media data. In one example, the business process may be rendering, encoding and decoding, transcoding, etc. of data, and the synchronization control sub-module may be a unit or process in the media force module, etc.

Illustratively, a media computation module 131 receives media data of a holographic data type, and in a case that it is determined that the media data needs to be rendered, the synchronization control sub-module extracts a timestamp included in the media data before the media computation module 131 renders the media data, and after the media computation module 131 finishes rendering the media data, the synchronization control sub-module inserts the extracted timestamp into the rendered media data, and sends the rendered media data carrying the timestamp to the called terminal 112 through a second media channel between the media computation module 131 and the called terminal 112. The other media calculation module 131 receives the media data of the type of the tactile data, and when it is determined that the media data needs to be transcoded, the synchronization control sub-module extracts the timestamp contained in the media data before the other media calculation module 131 transcodes the media data, and after the other media calculation module 131 completes transcoding the media data, the synchronization control sub-module inserts the extracted timestamp into the transcoded media data, and sends the transcoded media data carrying the timestamp to the called terminal 112 through a second media channel between the media data and the called terminal 112.

In some embodiments, the synchronization control sub-module may be further configured to, when it is determined that the media data needs to be subjected to service processing, extract a timestamp included in the media data, perform service processing on the media data, and insert the extracted timestamp into the processed media data to obtain the target media data.

In some embodiments, after receiving the media data sent by the calling terminal 111, the media force module 131 sends the media data as target media data to the called terminal 112 through a second media channel with the called terminal 112 in a case that it is determined that the media data needs to be transmitted through.

The calling terminal 111 may process the divided media data, and accordingly, the media data sent to the media computation module does not need to be processed, or part of the divided media data does not need to be processed, and at this time, only the media computation module needs to transmit the media data to the called terminal 112, and the media computation module 131 may send the media data to the called terminal 112 through a second media channel between the media computation module and the called terminal 112 as target media data when determining that the media data needs to be transmitted.

In some embodiments, the called terminal 112 may include a second media channel synchronization module, where the second media channel synchronization module may be configured to extract a timestamp in each target media data for each received target media data, and then the called terminal 112 may select target media data with the same timestamp in each target media data to integrate, and present an integration result.

The first media channel synchronization module and the second media channel synchronization module may be units, processes, and the like in the terminal. The second media channel synchronization module may extract a timestamp in each target media data for each target media data received by the called terminal 112, so that the called terminal 112 can select the target media data with the same timestamp in each target media data for integration, thereby implementing synchronous processing of each target media data and further performing synchronous presentation on an integration result.

In some embodiments, the service control device 120 may further include a session management module 122, as shown in fig. 2, the calling terminal 111 may send a call request to the session management module 122 of the service control device 120, where the call request may include media type identifiers corresponding to the media. The session management module 122 forwards the received call request to the media effort resource scheduling module 121. The media force calculation resource scheduling module 121 allocates corresponding target media force calculation modules to each media one by one based on each media type identifier in the call request and the function and performance of each media force calculation module currently in an idle state, and sends the identifier information of each target media force calculation module to the session management module 122. The session management module 122 forwards the identification information of each target media computing power module to the calling terminal 111 and the called terminal 112, and the calling terminal 111 establishes a first media channel with each target media computing power module according to the identification information of each target media computing power module, and sends the collected media data to the corresponding target media computing power module through each established first media channel. The called terminal 112 establishes a second media channel with each target media force computing module according to the identification information of each target media force computing module. The media calculation module 131 receives the media data sent by the calling terminal 111, processes the media data to obtain target media data, and sends the target media data to the called terminal 112 through a second media channel between the target media data and the called terminal 112. The called terminal 112 receives each target media data through each established second media channel, and integrates and presents each target media data.

In some embodiments, the Service control device 120 may further include a data warehouse module, a QOS (Quality of Service) rule engine, a sensitive media security management module, and the like, and specifically, the functions and implementations of the data warehouse module, the QOS rule engine, and the sensitive media security management module may refer to the existing holographic communication infrastructure, which is not described herein again in the embodiments of the present invention. The embodiment of the invention can realize holographic communication by the mode of the system embodiment on the basis of the existing holographic communication infrastructure, can share the holographic communication service infrastructure basic module, provides multi-dimensional media distributed processing and synchronous service realization for users by utilizing the holographic communication infrastructure of an operator network, reduces the service deployment cost and improves the service deployment range to the maximum extent.

Referring to fig. 3, fig. 3 is a schematic signaling interaction diagram of multi-data channel synchronous holographic communication according to an embodiment of the present invention, where the multi-data channel synchronous holographic communication process may include:

the calling terminal may send a call request to the session management module, where the call request may include media type identifiers corresponding to the media. Wherein the media type identifier is used to identify a type of the media, and the type of the media may include at least one of an audio type, a video type, a holographic data type, a datatext type, and a haptic data type.

After receiving the call request, the session management module forwards the call request to the media computing resource scheduling module.

The media computing resource scheduling module distributes corresponding target media computing power modules for each media one by one based on each media type identification in the call request and the function and performance of each media computing power module in the idle state at present, and sends the identification information of each target media computing power module to the session management module. The identification information of each target media computing power module may be address information of each target media computing power module, and the like.

And the session management module receives the identification information of each target media computing power module and respectively forwards the identification information of each target media computing power module to the calling terminal and the called terminal.

And the calling terminal establishes a first media channel with each target media computing power module respectively according to the identification information of each target media computing power module.

And the called terminal establishes a second media channel with each target media force computing module respectively according to the identification information of each target media force computing module.

The calling terminal collects initial media data and divides the initial media data into media data corresponding to each first media channel according to the type of the media data processed by the target media computing power module corresponding to each first media channel. And a first media channel synchronization module in the calling terminal adds corresponding data acquisition time stamps to the media data corresponding to each first media channel. And the calling terminal sends the media data added with the timestamp to the corresponding target media computing power module through each established first media channel.

The media force calculation module receives the media data, under the condition that the media data need to be subjected to business processing, a synchronous control submodule in the media force calculation module extracts a time stamp contained in the media data, and inserts the extracted time stamp after the media force calculation module performs business processing on the media data to obtain target media data. And the media force calculation module takes the media data as the target media data under the condition of determining that the media data needs to be transmitted transparently. And further sending the target media data to the called terminal through a second media channel between the target media data and the called terminal. The business process may include, but is not limited to, rendering, transcoding, or codec, etc.

And a second media channel synchronization module in the called terminal takes out the time stamp in each target media data according to each received target media data, so that the called terminal can select the target media data with the same time stamp from each target media data to integrate, and present the integrated result.

In some embodiments, the service control device and the media computing device may belong to the same server device or the like. Corresponding to the holographic communication system with multiple data channels synchronized provided in the foregoing embodiment, an embodiment of the present invention further provides a holographic communication method with multiple data channels synchronized, which is applied to a terminal in the holographic communication system with multiple data channels synchronized, where the holographic communication system with multiple data channels synchronized may further include a server, and the server may include a media computing resource scheduling module and multiple media computing modules, and the media computing modules are used to implement preset service functions, as shown in fig. 4, and the method may include:

s401, sending a call request to a server, wherein the call request comprises media type identifiers corresponding to various media;

s402, receiving identification information of a target media computing power module distributed for each media by a media computing power resource scheduling module of a server based on each media type identification in a call request;

s403, respectively establishing media channels with each target media force computing module of the server side according to the identification information of each target media force computing module;

s404, sending the collected media data to the corresponding target media calculation module through each established media channel.

In some embodiments, the method may further include:

and receiving target media data obtained after the media data are processed by each target media computing module, and integrating and presenting each target media data.

In some embodiments, the integrating and presenting the target media data includes:

extracting a timestamp in each target media data aiming at each received target media data;

In some embodiments, the service control device and the media computing device may belong to the same server device or the like. Corresponding to the holographic communication system with multiple data channels synchronized provided in the foregoing embodiment, an embodiment of the present invention further provides a holographic communication method with multiple data channels synchronized, which is applied to a server in the holographic communication system with multiple data channels synchronized, where the holographic communication system with multiple data channels synchronized may further include a terminal, the server may include a media computing resource scheduling module and multiple media computing modules, and the media computing modules are used to implement a preset service function, as shown in fig. 5, the method may include:

s501, a server receives a call request sent by a terminal, wherein the call request comprises media type identifiers corresponding to various media;

s502, a media computing resource scheduling module of the server allocates a corresponding target media computing module for each media based on each media type identifier in the call request;

s503, the media computing resource scheduling module sends the identification information of each target media computing module to the terminal;

s504, a media calculation module of the server receives media data sent by the terminal and processes the media data to obtain target media data;

and S505, the media calculation module sends the target media data to the terminal through a media channel between the media calculation module and the terminal.

In some embodiments, the media type identifier is used to identify a type of the media, the type of the media including at least one of an audio type, a video type, a holographic data type, a datatext type, and a haptic data type;

and the media computing resource scheduling module of the server allocates corresponding target media computing module for each media based on each media type identification in the call request and the function and performance of each media computing module in the idle state at present.

In some embodiments, the receiving, by the media calculation module of the server, the media data sent by the terminal, and processing the media data to obtain target media data includes:

and a media computing module of the server receives media data sent by the terminal, extracts a timestamp contained in the media data under the condition that the media data need to be subjected to business processing, performs business processing on the media data, and inserts the extracted timestamp into the processed media data to obtain target media data.

In some embodiments, the receiving, by the media force module of the server, the media data sent by the terminal, and processing the media data to obtain the target media data includes:

and the media computing module of the server receives the media data sent by the terminal, and takes the media data as target media data under the condition that the media data needs to be transmitted.

In some embodiments, the server further includes a session management module,

the server receives a call request sent by the terminal, and the call request comprises the following steps:

a session management module of a server receives a call request sent by a terminal and forwards the call request to a media computing resource scheduling module;

the media computing resource scheduling module sends the identification information of each target media computing module to the session management module;

and the session management module forwards the identification information of each target media calculation module to the terminal.

An embodiment of the present invention further provides an electronic device, as shown in fig. 6, including a processor 601, a communication interface 602, a memory 603, and a communication bus 604, where the processor 601, the communication interface 602, and the memory 603 complete mutual communication through the communication bus 604,

a memory 603 for storing a computer program;

the processor 601 is configured to implement the steps of any one of the above-mentioned holographic communication methods with multiple data channels synchronized when executing the program stored in the memory 603, so as to achieve the same technical effect.

The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this is not intended to represent only one bus or type of bus.

The communication interface is used for communication between the electronic equipment and other equipment.

The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.

The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.

In yet another embodiment of the present invention, a computer-readable storage medium is further provided, in which a computer program is stored, and the computer program, when executed by a processor, implements the steps of any one of the above-mentioned multiple data channel synchronized holographic communication methods.

In yet another embodiment provided by the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform any one of the above described embodiments of the holographic communication method for multiple data channel synchronization.

In the above embodiments, all or part of the implementation may be realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.

It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.

All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the method embodiment, since it is substantially similar to the system embodiment, the description is simple, and the relevant points can be referred to the partial description of the method embodiment.

The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.

Claims

1. A holographic communication system with multiple data channel synchronization, the system comprising: the system comprises service control equipment, media computing equipment and a plurality of terminals, wherein the service control equipment comprises a media computing resource scheduling module, the media computing equipment comprises a plurality of media computing modules, and the media computing modules are used for realizing preset service functions;

a media computing resource scheduling module in the service control device, configured to allocate a corresponding target media computing module to each media based on each media type identifier in the call request, and send identifier information of each target media computing module to the terminal;

2. The system of claim 1, wherein the media type identifier identifies a type of the media, the type of the media comprising at least one of an audio type, a video type, a holographic data type, a datatext type, and a haptic data type;

the media computing resource scheduling module is specifically configured to allocate a corresponding target media computing module to each media based on each media type identifier in the call request and the function and performance of each media computing module currently in an idle state.

3. The system according to claim 1 or 2, wherein the terminals comprise a calling terminal and a called terminal;

the media computing power resource scheduling module is specifically configured to send identification information of each target media computing power module to the calling terminal and the called terminal respectively;

the media calculation module is specifically configured to receive the media data sent by the calling terminal, process the media data to obtain target media data, and send the target media data to the called terminal through a second media channel between the media calculation module and the called terminal;

4. The system of claim 3, wherein the calling terminal is further configured to collect initial media data, and divide the initial media data into media data corresponding to each first media channel according to a type of the media data processed by the target media computing power module corresponding to each first media channel;

the calling terminal comprises a first media channel synchronization module,

the first media channel synchronization module is configured to add a corresponding data acquisition timestamp to media data corresponding to each first media channel.

5. The system of claim 4, wherein the media effort module comprises a synchronization control sub-module,

6. The system of claim 4, wherein the media effort module is specifically configured to:

7. The system of claim 5, wherein the called terminal comprises a second media channel synchronization module,

the second media channel synchronization module is configured to extract a timestamp in each received target media data according to each received target media data;

8. The system of claim 3, wherein the traffic control device further comprises a session management module,

the session management module is used for forwarding the call request to the media computing resource scheduling module;

9. A holographic communication method for synchronizing multiple data channels is characterized in that the holographic communication method is applied to a terminal in a holographic communication system for synchronizing multiple data channels, the holographic communication system for synchronizing multiple data channels further comprises a server, the server comprises a media computing resource scheduling module and a plurality of media computing modules, the media computing modules are used for realizing preset service functions, and the method comprises the following steps:

respectively establishing a media channel with each target media computing power module of the server side according to the identification information of each target media computing power module;

10. The method of claim 9, further comprising:

11. The method of claim 10, further comprising:

and integrating and presenting the target media data.

12. The method of claim 11, wherein the integrating and presenting each of the target media data comprises:

extracting a time stamp in each received target media data;

13. A holographic communication method for synchronizing multiple data channels is characterized in that the method is applied to a server side in a holographic communication system for synchronizing multiple data channels, the holographic communication system for synchronizing multiple data channels further comprises a terminal, the server side comprises a media computing power resource scheduling module and a plurality of media computing power modules, the media computing power modules are used for realizing preset service functions, and the method comprises the following steps:

14. The method of claim 13, wherein the media type identifier identifies a type of the media, and wherein the type of the media comprises at least one of an audio type, a video type, a holographic data type, a data text type, and a haptic data type;

the media computing resource scheduling module of the server allocates a corresponding target media computing module for each media based on each media type identifier in the call request, and the method comprises the following steps:

15. The method of claim 13, wherein the receiving, by the media force module of the server, the media data sent by the terminal, and processing the media data to obtain the target media data comprises:

16. The method of claim 13, wherein the receiving, by the media force module of the server, the media data sent by the terminal, and processing the media data to obtain the target media data comprises:

17. The method of claim 13, wherein the server further comprises a session management module,

18. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;

a memory for storing a computer program;

a processor for implementing the method steps of any of claims 9 to 17 when executing a program stored in the memory.

19. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method steps of any of the claims 9-17.