CN112532945A

CN112532945A - Multi-type media service fusion system

Info

Publication number: CN112532945A
Application number: CN202011388248.1A
Authority: CN
Inventors: 苏鹏; 林雪; 杨鹏; 董章晔; 代守群
Original assignee: Beijing Tonghe Shiyi Telecommunication Science And Technology Research Institute Co ltd; Data communication science and technology research institute; XINGTANG COMMUNICATION TECHNOLOGY CO LTD
Current assignee: Beijing Tonghe Shiyi Telecommunication Science And Technology Research Institute Co ltd; Data communication science and technology research institute; XINGTANG COMMUNICATION TECHNOLOGY CO LTD
Priority date: 2020-12-01
Filing date: 2020-12-01
Publication date: 2021-03-19
Anticipated expiration: 2040-12-01
Also published as: CN112532945B

Abstract

The invention relates to a multi-type media service fusion system, belongs to the technical field of multimedia communication, and solves the problems that the existing multi-type media service fusion needs a large amount of manual intervention, the efficiency is low, and the service mode is inflexible. The system comprises a service control module, a service control module and a service management module, wherein the service control module is used for receiving a service request of a service control client and sending a control instruction; the signaling media fusion module is used for completing service control, completing adaptation and connection establishment among different types of terminal equipment, organizing and managing media streams in a release channel and subscription channel mode, and fusing and communicating different types of media services; the basic service module is used for uniformly managing system resources and providing basic services required by system operation; and the resource management and scheduling module is used for managing and scheduling the media service resources. The system provides basic service of business fusion in a cloud mode, manual intervention is not needed, the efficiency is high, the application mode is rich, the applicability is strong, and the utilization rate of resources is improved.

Description

Multi-type media service fusion system

Technical Field

The invention relates to the technical field of multimedia communication, in particular to a multi-type media service fusion system.

Background

At present, users of army, government, industry and the like have established a plurality of sets of video conference, video monitoring and video command systems which are mutually independent, use independent technical systems respectively and cannot perform system fusion and information sharing. With the updating iteration of the technology and the continuous development of the user requirements, a unified platform capable of fusing various services such as video conferences, video monitoring, video commanding and the like is established on the basis of the cloud computing technology and the audio and video technology, and has become a necessary trend of the audio and video service development. The video cloud platform multi-type service fusion technology is designed and realizes multi-type media service fusion standards, realizes unified access of various audio and video system resources, integrates and shares the resources, and realizes technical innovation and application mode innovation in the field of audio and video services. Different video services respectively use different media negotiation protocols to realize media access.

The traditional video conference field mainly takes an H.323 protocol as a main part; the field of video command mainly takes SIP protocol as a main part; the video monitoring field mainly takes SIP, RTSP and other protocols as main. Because different media negotiation protocols have different negotiation processes and media capability expression modes, different negotiation processes need to be unified in the process of fusing media services.

Firstly, in the process of fusing multi-type media services, because negotiation protocols, coding protocols and network adaptability mechanisms of different types of media are different, coding parameters and network adaptability mechanisms of media streams need to be manually set in the process of fusing, the method cannot flexibly cope with the scene of fusing the various different media streams, and has low efficiency and poor flexibility; secondly, the system resources cannot be automatically managed and scheduled, and the utilization rate of the system resources is low; and thirdly, only a media forwarding function is provided, a video multi-picture function comprising different types of media streams cannot be provided, the application mode is few, and the applicability is poor.

Disclosure of Invention

In view of the foregoing analysis, the present invention aims to provide a multi-type media service fusion system, so as to solve the problems of low efficiency, poor flexibility, poor applicability and low resource utilization rate that the existing multimedia fusion technology requires manual intervention.

The invention provides a multi-type media service fusion system, comprising:

the service control module is used for receiving a service request of a service control client and sending a control instruction according to the service request;

the signaling media fusion module is used for completing service control according to the control instruction and completing adaptation among different types of terminal equipment so as to establish connection among the different types of terminal equipment;

the basic service module is used for uniformly managing system resources and providing basic services required by system operation;

and the resource management and scheduling module is used for managing and scheduling the media service resources.

Further, the service control module includes:

the video conference sub-module is used for sending a video conference control instruction to the signaling media fusion module according to the video conference request sent by the service control client;

the video command submodule is used for sending a video command control instruction to the signaling media fusion module according to a video command request sent by the service control client;

and the system management submodule is used for carrying out unified management on the terminal equipment and the system information.

Further, the signaling media fusion module includes:

a fusion control sub-module, a media negotiation sub-module and a channel management sub-module;

the fusion control sub-module includes:

the signaling control unit is used for accessing different types of terminal equipment according to the session establishment instruction in the control instruction and establishing connection;

the channel control unit is used for establishing a media channel subscription relation among different types of terminal equipment according to a channel configuration instruction in the control instruction;

and the resource control unit is used for calling corresponding media service resources according to the resource starting instruction in the control instruction, wherein the media service resources comprise audio processing service resources, video processing service resources and network adaptive adaptation resources.

Further, the media negotiation submodule comprises an h.323 negotiation unit, an SIP negotiation unit and an RTSP negotiation unit;

the H.323 negotiation unit is used for completing the bidirectional access of an H.323 conference terminal in the terminal equipment;

the SIP negotiation unit is used for completing the bidirectional access of an SIP command terminal in the terminal equipment;

and the RTSP negotiation unit is used for completing the one-way access of the monitoring camera equipment in the terminal equipment.

Furthermore, the signaling media fusion module further comprises an audio processing sub-module, a video processing sub-module, a network adaptability sub-module and a media transceiving sub-module.

Further, the channel management sub-module is configured to complete media stream fusion between terminal devices with different media protocols, and includes:

the media stream comprises an audio stream and a video stream;

the at least one audio distribution channel is used for receiving and decoding the audio stream sent by the terminal equipment of the sender and carrying out parameter configuration;

the at least one audio subscription channel is used for reading the decoded audio stream, mixing the audio stream, carrying out audio coding and network adaptability packaging on the audio stream after mixing according to the media parameter and the network adaptability parameter of the terminal equipment of the receiving party, and sending the audio stream to the terminal equipment of the receiving party;

at least one video distribution channel, which is used for receiving and decoding the video stream sent by the sender terminal equipment and carrying out parameter configuration;

at least one video subscription channel, configured to read the decoded video stream, perform video coding and network adaptability encapsulation on the video stream according to a media parameter and a network adaptability parameter of a receiving-side terminal device, and send the video stream to the receiving-side terminal device;

and the at least one multi-picture channel is used for reading the video streams issued by the video issuing channels, synthesizing the multi-video streams, and carrying out video coding and network adaptive packaging on the synthesized video streams according to the media parameters and the network adaptive parameters of the terminal equipment of the receiving party.

Further, the network adaptability sub-module comprises:

the first unit is used for carrying out retransmission algorithm adaptation on packet loss occurring in the process of media stream fusion;

the second unit is used for carrying out algorithm adaptation on a forward error correction mechanism appearing in the process of media stream fusion;

the third unit is used for carrying out algorithm adaptation on the network jitter resistance occurring in the media stream fusion process;

a fourth unit, configured to perform algorithm adaptation on a media stream sending policy occurring in the media stream fusion process;

and the fifth unit is used for dynamically adjusting the resolution, the code rate and the frame rate of the video stream according to the network environment in the process of merging the media streams so as to ensure the quality of the audio stream.

Further, the audio processing sub-module includes:

the audio coding unit is used for completing audio coding according to the channel audio coding protocol parameter configuration;

the audio decoding unit is used for finishing audio decoding according to the channel audio coding protocol parameter configuration;

the audio mixing unit is used for mixing the audio of multiple paths of audio streams;

and the audio and video synchronization unit is used for completing audio and video synchronization according to the timestamp information of the media stream.

Further, the video processing sub-module includes:

the video coding unit is used for completing video coding according to the channel video coding protocol parameter configuration;

the video decoding unit is used for finishing video decoding according to the channel video decoding protocol parameter configuration;

a picture composition unit for picture composition of the multi-channel video stream;

Further, the resource management and scheduling module encapsulates media service resources into a Docker container by building a kubernets cluster environment, and deploys the media service resources on the kubernets cluster environment to manage and schedule the media service resources.

Compared with the prior art, the invention can realize at least one of the following beneficial effects:

1. according to the multi-type media service fusion system provided by the invention, media service resources are scheduled and managed in a cloud mode, so that media negotiation adaptation, media coding adaptation and network adaptability adaptation among different types of terminal equipment in the multi-type media fusion process are automatically completed, the utilization rate of the media service resources is improved, meanwhile, the efficiency, flexibility and applicability of service fusion are also improved, and the defects of manpower consumption, low efficiency and poor applicability of traditional manual setting are overcome.

2. The invention provides a multi-type media service fusion system, which completes the adaptation and the establishment of the connection relation among different types of terminal equipment through a signaling media fusion module, organizes and manages media streams in the form of a publishing channel and a subscribing channel, leads the different types of media services to be fused and communicated, namely realizes the media stream fusion among the terminal equipment with different media protocols by establishing the subscribing relation between the publishing channel and the subscribing channel of the media streams, and can synthesize a plurality of video streams by utilizing a multi-picture channel, thereby providing the video multi-picture function, being capable of providing a plurality of application modes and being suitable for different application scenes.

3. The system for fusing the multi-type media services provided by the invention provides system resource management services through Kubernets, wherein the Kubernets comprise services of load balancing, automatic capacity expansion and capacity reduction, automatic fault triggering and the like, the cross-platform deployment capacity and the transverse capacity expansion capacity of the system can be improved, and the resource utilization rate and the availability of the system are improved.

In the invention, the technical schemes can be combined with each other to realize more preferable combination schemes. Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and drawings.

Drawings

The drawings are only for purposes of illustrating particular embodiments and are not to be construed as limiting the invention, wherein like reference numerals are used to designate like parts throughout.

Fig. 1 is a schematic diagram of a multi-type media service convergence system according to an embodiment of the present invention;

fig. 2 is a schematic diagram of interfaces inside the convergence system and between the convergence system and the terminal device and the service control client according to the embodiment of the present invention;

FIG. 3 is a diagram illustrating a media negotiation fusion process between different types of terminal devices according to an embodiment of the present invention;

FIG. 4 is a diagram illustrating a process of establishing media channels between different types of terminal devices according to an embodiment of the present invention;

fig. 5 is a schematic diagram illustrating a process of multi-picture media fusion according to an embodiment of the present invention.

Detailed Description

The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate preferred embodiments of the invention and together with the description, serve to explain the principles of the invention and not to limit the scope of the invention.

For a better illustration of the invention, the following terms of art will now be described:

PCM: pulse Code Modulation is an abbreviation for Pulse Code Modulation, which is one of encoding methods for digital communication. The main process is to sample the analog signals of voice, image, etc. at regular intervals to make them discretize, at the same time, to round the sampled values by hierarchical unit to make them be rounded and quantized, and at the same time, to express the amplitude of sampled pulse by a group of binary codes.

YUV: a color coding method is shown, applied in each video processing component. YUV is represented by one "luminance" component (equivalent to gray scale) and two "chrominance" components (blue projection) and (red projection).

H.323: a communication control protocol specified by the ITU is represented for providing multimedia services in a packet switched network. Call control is an important component among others that can be used to establish point-to-point media sessions and multipoint-to-multipoint media conferences.

SIP (Session Initiation Protocol): is a multimedia communication protocol established by the IETF (Internet Engineering Task Force). It is a text-based application-layer control protocol for creating, modifying and releasing sessions of one or more participants. The SIP is an IP voice session control protocol originated from the Internet, and has the characteristics of flexibility, easiness in implementation, convenience in expansion and the like.

Rtsp (real Time Streaming protocol): is an application layer protocol which is proposed by Real Network and Netscape together and how to effectively transmit streaming media data on an IP Network. RTSP provides control over streaming media such as pause, fast forward, etc., without itself transferring data, and functions as a remote control of a streaming server. The server side can choose to use TCP or UDP to transmit network data.

H.264: h.264 is a new generation digital video compression format following MPEG4, commonly proposed by the international organization for standardization (ISO) and the International Telecommunication Union (ITU), and is one of the video codec technical standards named by the ITU-T under the name of the h.26x series.

The invention discloses a system for fusing multi-type media services. The fusion system can integrate various video streams and multimedia data, provide video conference, video monitoring and mobile image real-time conference service, facilitate clear information grasping, flexibly adjust strategies, rapidly command decisions and the like; and the system can control video monitoring, clustering and video conference systems in a centralized way, provide a visual unified management platform, and uniformly manage resources, services, authorities and the like. As shown in fig. 1. The system comprises:

and the service control module is used for receiving the service request of the service control client and sending a control instruction according to the service request. And the signaling media fusion module is used for finishing service control according to the control instruction, finishing adaptation among different types of terminal equipment, establishing connection among the different types of terminal equipment, organizing and managing media streams in a publishing channel and a subscribing channel mode, and fusing and communicating different types of media services. And the basic service module is used for uniformly managing system resources and providing basic services required by system operation. And the resource management and scheduling module is used for managing and scheduling the media service resources.

The service request can be a video conference request, a video monitoring request or a video command request. The completion of adaptation among different types of terminal devices means that media negotiation adaptation, media coding adaptation and network adaptability adaptation among terminal devices of different protocol types are completed in the process of multi-type media service fusion. The media service resources comprise audio processing service resources, video processing service resources, network adaptive adaptation resources and channel resources.

Preferably, the basic service module comprises basic function modules such as a database, a resource management service, an RPC service, a file service, a log service and the like. The database is used for providing a persistent storage function for the system; the resource management service module is used for uniformly managing channel resources, audio processing service resources, video processing service resources and network adaptability service resources of the system and realizing dynamic allocation of media service resources of the system; RPC service, which is used to provide support for communication between each module or each unit in the system; the file service is used for providing file storage service for the system; log service: for providing a logging service for the system.

Preferably, the service control module mainly implements a service function and is responsible for interacting with the service control client. The user sends service request to the service control module through the service control client to complete various application service operations. The service control module specifically comprises:

and the video conference submodule is used for receiving and analyzing the video conference request sent by the service control client, generating a corresponding video conference control instruction and sending the corresponding video conference control instruction to the signaling media fusion module.

The video command submodule is used for receiving and analyzing a video command request sent by the service control client, generating a corresponding video command control instruction and sending the corresponding video command control instruction to the signaling media fusion module;

Specifically, the video conference sub-module can provide services such as holding multiple groups of conferences, deciding meetings, dynamically dividing screens, applying for speech by terminals, performing double-flow and remote WEB management, can provide several conference modes such as a talkback speech mode, a free discussion mode and voice excitation, and has the cooperation functions of document sharing, desktop sharing and the like. The video conference sub-module can control access to a traditional video conference terminal, and can also control access to a monitoring camera, a mobile command terminal and the like. And provides the function of viewing the monitoring image and the image of the mobile command terminal in the video conference by the image viewing mode.

The video command sub-module can provide video command functions, including command calling, image viewing, special command, override command, cooperative command, take-over command and the like. In the command service, except for carrying out the command service between the traditional command terminal and the coder-decoder, the video conference terminal and the monitoring camera can be accessed through the signaling media fusion service, and the two-way access of the video conference terminal and the one-way access of the monitoring camera are realized. In addition, images and monitoring images of the conference terminal can be checked through the command terminal, and bidirectional audio and video communication can be carried out with the conference terminal.

And the system management submodule is used for managing and configuring basic data such as users, organizations, authorities and the like of the system, and information such as system video conference parameters, command service groups and the like. The video conference parameters comprise conference participant information and conference audio/video media parameter information. The command service group information comprises command group member information, a coder/decoder, a command terminal and the like. The system management submodule can perform registration authentication and state monitoring on a video conference terminal, a command terminal and a monitoring camera of the access system, and unified management is achieved.

Preferably, the system further comprises a video monitoring sub-module, the control principle is the same as that of the video conference sub-module and the video command sub-module, and the control principle is not repeated here, and the system can specifically provide services such as video monitoring, audio monitoring, electronic maps and video recording.

The signaling media fusion module is mainly used for completing the work of media negotiation adaptation, media coding adaptation, network adaptability adaptation and the like among different types of terminal equipment.

Preferably, the signaling media fusion module includes a fusion control sub-module, a media negotiation sub-module and a channel management sub-module.

Specifically, the fusion control submodule includes a signaling control unit, a channel control unit, and a resource control unit.

The signaling control unit is used for establishing connection with a corresponding video conference terminal, a monitoring camera, a mobile device or other terminal devices through the media negotiation submodule according to a session establishment instruction in the control instruction. Specifically, the channel control unit is configured to establish a media channel subscription relationship between different types of terminal devices through the channel management submodule according to a channel configuration instruction in the control instruction.

The resource control unit is used for calling corresponding media service resources from the resource management and scheduling module through the resource management service in the basic service module according to the resource starting instruction in the control instruction; illustratively, a channel service resource, an audio processing service resource, a video processing service resource, or a network adaptive adaptation resource, etc. is invoked.

The media negotiation submodule establishes media negotiation session with various types of terminal equipment by introducing H.323, SIP and RTSP protocol stacks, and can be compatible with various types of terminal equipment. Because each media negotiation protocol has different negotiation processes and expression modes of media capability sets, in the process of accessing terminal devices of different protocols, compatibility adaptation of the media capability sets needs to be completed, capability set differences of different types of media negotiation signaling are shielded, and a uniform media negotiation flow is established, so that uniform negotiation access of the terminal devices of different protocols is realized.

Preferably, the media negotiation submodule includes an h.323 negotiation unit, an SIP negotiation unit and an RTSP negotiation unit. The H.323 negotiation unit is used for completing the bidirectional access of an H.323 conference terminal in the terminal equipment; the SIP negotiation unit is used for completing the bidirectional access of an SIP command terminal in the terminal equipment; and the RTSP negotiation unit is used for completing the one-way access of the monitoring camera equipment in the terminal equipment. The bidirectional access means that the accessed terminal equipment can send media stream data and can also receive the media stream data, and the unidirectional access means that the accessed terminal equipment can only send the media stream data.

Preferably, the signaling media fusion module further comprises an audio processing sub-module, a video processing sub-module, a network adaptability sub-module and a media transceiving sub-module.

The media receiving and sending sub-module mainly comprises an asynchronous receiving and sending unit, and is used for realizing asynchronous receiving and sending of media streams, namely realizing media stream interaction between the terminal equipment and the fusion system.

The audio processing submodule is used for completing media audio stream adaptation among terminal devices with different protocols, mainly comprises audio coding, audio decoding, audio mixing, audio and video synchronization, media packaging, media unpacking and the like, and supports audio coding protocols such as G.711, G.729A, G.722.1C, AAC-LC, AAC-LD and the like.

Preferably, the audio processing sub-module comprises:

and the audio coding unit is used for completing audio coding according to the channel audio coding protocol parameter configuration.

And the audio decoding unit is used for finishing audio decoding according to the channel audio coding protocol parameter configuration.

And the audio mixing unit is used for mixing the multi-path PCM audio stream.

The media packaging unit is used for providing audio packaging service according to different packaging rules.

And the media unpacking unit is used for providing audio unpacking service according to different packing rules.

The video processing submodule is used for completing the adaptation of media video streams among terminal devices with different protocols, mainly comprises video coding, video decoding, picture composition, video forwarding, media packaging, media unpacking, I frame processing and audio and video synchronization, supports video coding protocols such as H.264, H.265, VP8 and VP9, and covers multiple resolutions such as QCIF, CIF, D1, 4CIF, 720P, 1080P, 1920x1200 and 4K.

Preferably, the video processing sub-module comprises:

and the video coding unit is used for finishing video coding according to the channel video coding protocol parameter configuration.

And the video decoding unit is used for finishing video decoding according to the channel video decoding protocol parameter configuration.

And the picture synthesis unit is used for carrying out picture synthesis on the multi-channel YUV video streams.

And the video forwarding unit is used for directly finishing video forwarding based on the channel subscription relationship among the terminal devices with the same protocol type.

The media packaging unit is used for providing video packaging service according to different packaging rules.

And the media unpacking unit provides audio unpacking service according to different packing rules.

And the I frame processing unit is used for receiving an I frame processing request of the terminal equipment when the subscription relationship of the video subscription channel is changed or network abnormity occurs.

Because different media protocols have different network adaptability mechanisms, aiming at the network adaptability adaptation requirement, the network adaptability sub-module provides functions of NACK, FEC, Jitter Buffer, smooth transmission, dynamic bandwidth adjustment and the like, and solves the problems of network adaptability adaptation in the fusion process of different media protocols and the problems of time delay, Jitter, packet loss and the like in the real-time transmission process of audio and video data under extreme network conditions.

Preferably, the network adaptation sub-module comprises:

and the first unit (NACK unit) is used for carrying out retransmission algorithm adaptation on lost packets such as single packet retransmission, combined retransmission, multiple retransmission and the like in the process of media stream fusion.

And a second unit (FEC unit) for performing algorithm adaptation on the forward error correction mechanism occurring in the media stream fusion process.

And the third unit (Jitter Buffer unit) is used for carrying out algorithm adaptation on the network Jitter resistance occurring in the media stream fusion process.

And the fourth unit (smooth sending unit) is used for carrying out algorithm adaptation on the media stream sending strategy in the media stream fusion process so as to effectively ensure that the fusion media has a smooth sending mechanism.

And a fifth unit (dynamic bandwidth adjustment unit) for dynamically adjusting the resolution, the bit rate and the frame rate of the video stream according to the network environment in the process of media stream fusion so as to ensure the quality of the audio stream.

The fusion system realizes the compatibility processing of media streams among different types of terminal equipment by calling media basic services such as an audio processing submodule, a video processing submodule, a network adaptability submodule and the like, and realizes the fusion access of the different types of media streams. The media channels may be divided into a publish channel and a subscribe channel according to the media stream direction, where the publish channel and the subscribe channel may be in a one-to-one, one-to-many, many-to-one, or many-to-many relationship. The media channels may be divided into audio channels and video channels according to the type of media streaming protocol. In addition, the system also comprises a multi-picture channel which is used for synthesizing multiple pictures of the image and realizing the multi-picture function of the fusion system.

Preferably, the channel management sub-module is configured to complete media stream fusion between terminal devices with different media protocols, and includes:

wherein the media stream comprises an audio stream and a video stream.

And the at least one audio distribution channel is used for receiving the audio stream sent by the terminal equipment of the sender, decoding the received media stream into a PCM audio stream (namely the original audio stream of the terminal equipment of the sender) through the audio processing submodule, and matching parameters such as an audio sampling rate, a code rate and a sound channel.

And the audio subscription channel is used for reading the PCM audio stream issued by the single-channel or multi-channel audio issuing channel, mixing the PCM audio stream through an audio mixing unit in the audio processing sub-module when reading the multi-channel PCM audio stream, and carrying out audio coding and network adaptive packaging on the mixed PCM audio stream according to the media parameter and the network adaptive parameter of the terminal equipment of the receiving party through the audio processing sub-module and the network adaptive sub-module so as to enable the packaged PCM audio stream to accord with the media protocol of the terminal equipment of the receiving party and send the packaged PCM audio stream to the terminal equipment of the receiving party.

And the at least one video publishing channel is used for receiving the video stream sent by the terminal equipment of the sending party, decoding the received video stream into a YUV video stream (namely the original video stream sent by the terminal equipment of the sending party) through the video processing submodule, and matching parameters such as video resolution, code rate, frame rate and the like.

And the video subscription channel is used for reading the YUV video stream issued by the video issuing channel, performing video coding and network adaptability packaging on the YUV video stream through the video processing submodule and the network adaptability submodule according to the media parameter and the network adaptability parameter of the receiving party terminal equipment, so that the packaged YUV video stream conforms to the media protocol of the receiving party terminal equipment and is sent to the receiving party terminal equipment.

The video processing subunit and the network adaptability unit are used for carrying out video coding and network adaptability packaging on the synthesized YUV video stream according to the media parameters and the network adaptability parameters of the receiving terminal equipment, so that the packaged YUV video stream conforms to the media protocol of the receiving terminal equipment to meet a multi-picture service request, and then the multi-picture service request is sent to the receiving terminal equipment through the video subscription channel.

The obtaining of the media parameter and the network adaptability parameter is provided through a media capability negotiation process, and is specifically described in embodiment 1, embodiment 2, and embodiment 3 below.

The resource management and scheduling module performs unified management on media resources required by the terminal equipment, including computing resources, network resources and storage resources, and adds a basic service module on the basis to support different types of business services. Meanwhile, video and audio resources of a video conference, video monitoring and video command scheduling are uniformly integrated into a platform resource with video content as a standard, a resource list is formed, operations of receiving, caching, processing, forwarding and the like of the platform resource are standardized, and the unified management and scheduling are facilitated.

Specifically, the resource management and scheduling module selects kubernets as container cloud management software, builds a kubernets local cluster environment, packages basic media services such as audio mixing, media forwarding, transcoding, code stream error correction and the like into a Docker container, and deploys the Docker container on the cluster environment, so that management and scheduling of media service resources are achieved. Kubernets provides management services including load balancing, automatic capacity expansion and reduction, automatic fault triggering and the like. The load balancing service can ensure that the load of the basic service module is uniformly expanded to a variable number of cluster nodes so as to improve the utilization rate of resources. The automatic capacity expansion and reduction and the automatic triggering of faults can improve the stability and the safety of the fusion system. Compared with the application directly running on the physical server, Kubernets has great advantages in resource allocation, illustratively, when more paths of media streams need transcoding at the same time, most physical resources may need to be occupied, and a distributed architecture can be adopted by using the Kubernets, so that better scalability is realized, and the resource utilization rate of the physical server is improved.

The fusion system provided by the invention realizes service fusion, media fusion and resource containerization management through the mutual cooperation of the modules, wherein the interfaces among the modules are shown as figure 2. Specifically, the interface a is used for implementing service interaction between the service control client and the service control module in the convergence system. And the interface B is used for controlling the service of the service control module through fusing the control sub-modules to realize specific video conference, video command and other services. And the interface C is used for controlling the media negotiation submodule to establish a media negotiation session by the fusion control submodule. And the interface D is used for the media negotiation submodule to finish the media fusion negotiation of the video conference terminal equipment, the monitoring camera and the mobile command equipment. And the interface E is used for realizing the interaction of the media stream between the fusion system and the video conference terminal equipment, the monitoring camera and the mobile command equipment by the media receiving and transmitting sub-module. And the interface F is used for the media receiving and sending sub-module and the channel management sub-module to carry out media stream interaction. And the interface G is used for setting the media channel parameters by the media negotiation submodule according to the negotiation result. And the interface H is used for controlling the channel management submodule to establish and manage the media channel by the channel control unit and establishing a channel subscription relation. And the interface I is used for the resource control unit to apply for the media service resource through the resource management service in the basic service module. And the interface J is used for calling the audio processing submodule by the channel management submodule to realize the fusion processing of the audio stream. And the interface K is used for calling the network adaptability submodule by the channel management submodule to realize the network adaptability fusion processing of the media stream. And the interface L is used for calling the video processing submodule by the channel management submodule to realize the fusion processing of the video stream. And the interface M is used for controlling the resource management and scheduling module by the resource management service to realize containerization management of the system basic service and the media basic service. And the interface N is used for packaging and operating the media basic service by the resource management and scheduling module in a container form and providing the media basic service capability for the system.

Example 1

A specific media negotiation fusion process between different types of terminal devices will now be described through the following embodiments, as shown in fig. 3.

In order to access various types of terminal equipment of a video conference, a video monitoring and command scheduling system, the fusion system introduces standard media negotiation submodules such as H.323, SIP and RTSP, and is responsible for call establishment, media capability negotiation, logical channel establishment and the like with the terminal equipment of the corresponding type. The fusion system obtains the capability of receiving and sending the media stream of all types of terminal equipment through the process, and provides a basis for parameter configuration of a subsequent media channel.

Specifically, arrows 1-1 indicate: the video conference sub-module receives the user request and initiates a conference holding application to the signaling control unit. Arrows 1-2 indicate: the signaling control unit sends a call establishment request to the H.323 negotiation unit, and calls the H.323 terminal equipment to enter a meeting, namely, establishes connection. Arrows 1-3 indicate: and the audio subscription channel returns the result to the channel control submodule. Arrows 1-4 indicate: the channel control submodule applies for an audio distribution channel for the H.323 terminal equipment. Arrows 1-5 indicate: the H.323 negotiation unit sends the platform capability set to the H.323 terminal equipment, and ensures that the sent multimedia data can be received and coded by the H.323 terminal equipment. Arrows 1-6 indicate: and the H.323 terminal equipment feeds back the platform capability set and responds to the H.323 negotiation unit. Arrows 1-7 indicate: h.323 terminal equipment sends terminal capability set to H.323 negotiation unit, ensures that the multimedia data sent can be received and coded by the fusion system. Arrows 1-8 indicate: and the H.323 negotiation unit feeds back the terminal capability set and responds to the H.323 terminal equipment. Arrows 1-9 indicate: the H.323 negotiation unit sends a master-slave determination instruction to the H.323 terminal equipment, and is used for avoiding a collision phenomenon in a signaling process. Arrows 1-10 indicate: h.323 terminal equipment feeds back a master-slave determination instruction and responds to the H.323 negotiation unit. Arrows 1-11 indicate: h.323 terminal equipment sends a master-slave determination instruction to an H.323 negotiation unit, and is used for avoiding a collision phenomenon in a signaling process. Arrows 1-12 indicate: the H.323 negotiation unit feeds back the master-slave determination instruction and responds to the H.323 terminal equipment. Arrows 1-13 indicate: the H.323 negotiation unit sends a command of opening a logical channel to the H.323 terminal equipment for transmitting the media data. Arrows 1-14 indicate: h.323 terminal equipment feeds back a command of opening a logical channel and responds to the H.323 negotiation unit. Arrows 1-15 indicate: h.323 terminal equipment sends a command of opening a logical channel to an H.323 negotiation unit for transmitting media data. Arrows 1-16 indicate: the H.323 negotiation unit feeds back the instruction of opening the logical channel and responds to the H.323 terminal equipment. Arrows 1-17 indicate: h.323 negotiation unit feeds back the call establishment result to the signaling control unit. Arrows 1-18 indicate: the signaling control unit sends a call establishment request to the SIP negotiation unit, and the SIP terminal equipment is called to enter a meeting, namely, a connection relation is established. Arrows 1-19 indicate: the SIP negotiation unit sends an INVITE request to the SIP terminal device. Arrows 1-20 indicate: the SIP terminal device feeds back a response message (100Trying) in the call processing to the SIP negotiation unit. Arrows 1-21 indicate: after the SIP terminal equipment rings, the Ringing information (180Ringing) is sent to the SIP negotiation unit. Arrows 1-22 indicate: the SIP terminal device sends a call success indication (200OK) to the SIP negotiation unit. Arrows 1-23 indicate: the SIP negotiation unit feeds back a call establishment result to the signaling control unit. Arrows 1-24 indicate: the signaling control unit sends a call establishment request to the RTSP negotiation unit to call the RTSP terminal equipment to enter the session. Arrows 1-25 indicate: the RTSP negotiation unit sends an OPTION request to the RTSP terminal device, inquiring about available methods. Arrows 1-26 indicate: and the RTSP terminal equipment feeds back an OPTION response to the RTSP negotiation unit. Arrows 1-27 indicate: the RTSP negotiation unit sends a DESCRIBE request to the RTSP terminal device to request to obtain the media initialization description information that the terminal device can provide. Arrows 1-28 indicate: the RTSP terminal device feeds back the DESCRIBE response, i.e. feeds back the media initialization description information to the RTSP negotiation unit. Arrows 1-29 indicate: the RTSP negotiation unit sends a SETUP request to the RTSP terminal equipment, sets the attribute of the session and the transmission mode, and reminds the establishment of the session. Arrows 1-30 indicate: the RTSP terminal device feeds back a SETUP response to the RTSP negotiation unit, containing the session identifier and session related information. Arrows 1-31 indicate: the RTSP negotiation unit feeds back a call establishment result to the signaling control unit. Arrows 1-32 indicate: and the signaling control unit feeds back the conference holding result to the video conference submodule.

Example 2

The process of establishing media channels between different types of terminal devices will now be described through the following embodiments, as shown in fig. 4.

The fusion system allocates audio and video channels for each terminal device establishing a call, and each audio and video channel is provided with a receiving and sending buffer area for caching a real-time media stream, so that the realization of various optimization mechanisms is guaranteed. The audio and video channels are divided into subscription channels and publication channels according to the direction of the media stream. The subscription channels logically correspond to the send buffers and the publication channels logically correspond to the receive buffers. The media parameters of the audio and video channels are provided by a media capability negotiation process, such as an audio codec protocol, a video codec protocol, and specific encoding parameters.

The fusion system sets a channel control submodule to control the establishment and destruction of a media channel, the direction of a media data stream and the application of container resources required by the channel to a resource management service in a basic service module, such as audio decoding, audio encoding, audio mixing, video encoding, picture synthesis and the like. The specific parameters used by the audio and video processing submodule refer to the media parameters of the media channel.

Specifically, arrows 1-1 indicate: h.323 terminal equipment initiates a channel establishment application to the channel control submodule. Arrows 1-2 indicate: and the channel control submodule applies for an audio subscription channel for the H.323 terminal equipment. Arrows 1-3 indicate: and the audio subscription channel returns the result to the channel control submodule. Arrows 1-4 indicate: the channel control submodule applies for an audio distribution channel for the H.323 terminal equipment. Arrows 1-5 indicate: the audio distribution channel returns the result to the channel control submodule. Arrows 1-6 indicate: and the channel control submodule applies for a video subscription channel for the H.323 terminal equipment. Arrows 1-7 indicate: and the video subscription channel returns the result to the channel control submodule. Arrows 1-8 indicate: and the channel control submodule applies for a video release channel for the H.323 terminal equipment. Arrows 1-9 indicate: and the video distribution channel returns the result to the channel control submodule. Arrows 1-10 indicate: the channel control submodule applies for a multi-picture channel. Arrows 1-11 indicate: the multi-picture channel returns the result to the channel control submodule. Arrows 1-12 indicate: the channel control submodule applies for computing resources and network resources to the resource control unit. Arrows 1-13 indicate: the resource control module applies for computing resources and network resources from the resource management service in the system base service. Arrows 1-14 indicate: the resource management service applies for computing resources and network resources from a resource management and scheduling module (i.e., a container cloud). Arrows 1-15 indicate: and feeding back a result to the resource management service by the container cloud, wherein the feedback result is a resource address, and updating the existing container resource by container cloud management software. Arrows 1-16 indicate: the resource management service feeds back a result to the resource control unit, and the feedback result is a resource address. Arrows 1-17 indicate: the resource control unit feeds back a result to the channel control submodule, the feedback result is a resource address, and the H.323 terminal equipment directly calls the container resource at the resource address when needing media processing. Arrows 1-18 indicate: and the channel control submodule feeds back a channel application result to the H.323 terminal equipment. Arrow 2-1 indicates: and the SIP terminal equipment initiates a channel establishment application to the channel control submodule. Arrows 2-2 indicate: and the channel control submodule applies for an audio subscription channel for the SIP terminal equipment. Arrows 2-3 indicate: and the audio subscription channel returns the result to the channel control submodule. Arrows 2-4 indicate: and the channel control submodule applies for an audio publishing channel for the SIP terminal equipment. Arrows 2-5 indicate: the audio distribution channel returns the result to the channel control submodule. Arrows 2-6 indicate: and the channel control submodule applies for a video subscription channel for the SIP terminal equipment. Arrows 2-7 indicate: and the video subscription channel returns the result to the channel control submodule. Arrows 2-8 indicate: and the channel control submodule applies for a video publishing channel for the SIP terminal equipment. Arrows 2-9 indicate: and the video distribution channel returns the result to the channel control submodule. Arrows 2-10 indicate: the channel control submodule applies for computing resources and network resources to the resource control unit. Arrows 2-11 indicate: the resource control unit applies for computing resources and network resources from the resource management service in the basic service module. Arrows 2-12 indicate: the resource management service applies for computing resources and network resources from the container cloud. Arrows 2-13 indicate: and feeding back a result to the resource management service by the container cloud, wherein the feedback result is a resource address, and updating the existing container resource by container cloud management software. Arrows 2-14 indicate: the resource management service feeds back a result to the resource control unit, and the feedback result is a resource address. Arrows 2-15 indicate: the resource control unit feeds back a result to the channel control submodule, the feedback result is a resource address, and the SIP terminal equipment directly calls the container resource corresponding to the resource address when needing media processing. Arrows 2-16 indicate: and the channel control submodule feeds back a channel application result to the SIP terminal equipment. Arrow 3-1 indicates: the RTSP terminal equipment initiates a channel establishment application to the channel control submodule. Arrows 3-2 indicate: and the channel control submodule applies for an audio release channel for the RTSP terminal equipment. Arrows 3-3 indicate: the audio distribution channel returns the result to the channel control submodule. Arrows 3-4 indicate: and the channel control submodule applies for a video release channel for the RTSP terminal equipment. Arrows 3-5 indicate: and the video distribution channel returns the result to the channel control submodule. Arrows 3-6 indicate: the channel control submodule applies for computing resources and network resources to the resource control unit.

Arrows 3-7 indicate: the resource control unit applies for computing resources and network resources from the resource management service in the basic service module. Arrows 3-8 indicate: the resource management service applies for computing resources and network resources from the container cloud. Arrows 3-9 indicate: and feeding back a result to the resource management service by the container cloud, wherein the feedback result is a resource address, and updating the existing container resource by container cloud management software. Arrows 3-10 indicate: the resource management service feeds back a result to the resource control unit, and the feedback result is a resource address. Arrows 3-11 indicate: the resource control unit feeds back a result to the channel control submodule, and the feedback result is that the H.323 terminal equipment directly calls the container resource corresponding to the resource address when the resource address needs media processing. Arrows 3-12 indicate: and the channel control sub-module feeds back the channel application result to the RTSP terminal equipment.

Example 3

The specific process of multi-picture media fusion will now be specifically described by the following embodiments. The fusion system supports the multi-picture broadcast composed of multi-type media to multi-type media terminal equipment, and has the core principle of controlling the media data stream transmission directions of a video subscription channel and a publishing channel among the multi-type terminal equipment, setting the multi-picture channel and fusing the multi-type media data. The specific process is as follows: firstly, all video publishing channels mapped by a video subscription channel perform network adaptability adaptation, decoding and the like of media data according to media parameters of the channel to obtain an original media data stream of the channel; then, the multi-picture channel reads the original media data streams of all video publishing channels mapped by the video subscription channel, and carries out picture synthesis and coding of media data, network adaptability mechanism and audio-video synchronization mechanism addition and the like according to the media parameters of the video subscription channel, thereby obtaining the media data streams meeting the requirements of corresponding terminal equipment; and finally, the video subscription channel reads the fusion media data stream in the multi-picture channel.

Illustratively, the service scenario is that the h.323 terminal device watches a converged media picture of the h.323, SIP, and RTSP terminal devices, and a specific process is as shown in fig. 5.

Arrows 1-1 indicate: h.323 terminal equipment sends video data stream to asynchronous receiving and sending unit. Arrows 1-2 indicate: and the media asynchronous receiving and transmitting unit sends the video data stream to the H.323 video distribution channel according to the source address of the terminal equipment. Arrows 1-3 indicate: the channel management submodule controls the H.323 video subscription channel and sets the publishing channels corresponding to the channel, namely the H.323 video publishing channel, the SIP video publishing channel and the RTSP video publishing channel. Arrows 1-4 indicate: and the H.323 video subscription channel feeds back a result to the channel management submodule. Arrow 2-1 indicates: the SIP terminal equipment sends the video data stream to the asynchronous receiving and sending unit. Arrows 2-2 indicate: and the asynchronous receiving and sending unit sends the video data stream to the SIP video publishing channel according to the source address of the terminal equipment. Arrow 3-1 indicates: and the RTSP terminal equipment sends the video data stream to the asynchronous receiving and sending module. Arrows 3-2 indicate: and the asynchronous receiving and sending module sends the video data stream to the RTSP video publishing channel according to the source address of the equipment terminal. Arrow 4-1 indicates: and the H.323 terminal equipment applies for watching the mixed multi-picture of the H.323 terminal equipment, the SIP terminal equipment and the RTSP terminal equipment to the channel management module. Arrows 4-2 indicate: and the channel management submodule determines a publishing channel and a subscribing channel according to the signaling, and sends a control instruction to the H.323 video publishing channel to instruct the H.323 video sending channel to start processing the media data. Arrows 4-3 indicate: and the H.323 video sending channel sends a control instruction to the corresponding network adaptability module according to the channel media parameters and directly calls the allocated container resources.

Arrows 4-4 indicate: the network adaptability sub-module unpacks the media data stream of the H.323 video distribution channel according to the corresponding mechanism in the channel media parameters, and sends a control instruction to the video decoding unit according to the video decoding parameters. Arrows 4-5 indicate: the video decoding unit directly calls the allocated container resources, decodes the media data stream into an original media stream according to the video decoding parameters, and feeds back the result to the network adaptability submodule. Arrows 4-6 indicate: and the network adaptability sub-module feeds back the result to the H.323 video distribution channel. Arrows 4-7 indicate: and the H.323 video release channel feeds back the decoding result of the media stream to the channel control submodule. Arrows 4-8 indicate: and the channel management submodule sends a control instruction to the SIP video publishing channel to indicate the SIP video publishing channel to start processing the media data. Arrows 4-9 indicate: and the SIP video publishing channel sends a control instruction to the corresponding network adaptability sub-module according to the channel media parameter and directly calls the allocated container resource. Arrows 4-10 indicate: and the network adaptive submodule unpacks the media data stream of the SIP video publishing channel and sends a control instruction to the video decoding unit according to the video decoding parameters. Arrows 4-11 indicate: the video decoding unit directly calls the allocated container resources, decodes the media data stream into an original media stream according to the video decoding parameters, and feeds back the result to the network adaptability submodule. Arrows 4-12 indicate: and the network adaptability sub-module feeds back the result to the SIP video publishing channel. Arrows 4-13 indicate: and the SIP video publishing channel feeds back the decoding result of the media stream to the channel control submodule. Arrows 4-14 indicate: and the channel management submodule sends a control instruction to the RTSP video issuing channel and indicates the RTSP video sending channel to start processing the media data. Arrows 4-15 indicate: and the RTSP video issuing channel sends a control instruction to the corresponding network adaptive submodule according to the channel media parameters and directly calls the allocated container resources. Arrows 4-16 indicate: and the network adaptive submodule unpacks the media data stream of the RTSP video release channel and sends a control instruction to the video decoding unit according to the video decoding parameters. Arrows 4-17 indicate: the video decoding unit directly calls the allocated container resources, decodes the media data stream into an original media stream according to the video decoding parameters, and feeds back the result to the network adaptability submodule. Arrows 4-18 indicate: and the network adaptability sub-module feeds back the result to the RTSP video release channel. Arrows 4-19 indicate: and the RTSP video release channel feeds back the media stream decoding result to the channel control submodule. Arrows 4-20 indicate: the h.323 video subscription channel sends a request to the multivision channel to merge the original media streams. Arrows 4-21 indicate: the multi-screen channel responds to the request of the h.323 video subscription channel. Arrows 4-22 indicate: the h.323 video distribution channel sends a notification to the multi-picture channel, which reads the data frames from the h.323 video distribution channel. Arrows 4-23 indicate: the SIP video publishing channel sends a notification to the multi-picture channel, and the multi-picture channel reads the data frame from the SIP video publishing channel. Arrows 4-24 indicate: the RTSP video publishing channel sends a notification to the multi-picture channel, and the multi-picture channel reads the data frame from the RTSP video publishing channel. Arrows 4-25 indicate: and the multi-picture channel fusion reads the data frames from the three video distribution channels, sends a control instruction to the picture synthesis unit and synthesizes the three video pictures. Arrows 4-26 indicate: the picture synthesizing unit sends a control instruction to the corresponding video coding unit according to the channel media parameters, and directly calls the allocated container resources, and the coding principle is that the video coding unit can be accepted and played by a user, namely H.323 terminal equipment. Arrows 4-27 indicate: the video coding unit sends a control command to the audio and video synchronization unit, and an audio and video synchronization mechanism is added. Arrows 4-28 indicate: and the audio and video synchronization unit feeds back the result to the video coding unit. Arrows 4-29 indicate: the video coding unit sends an instruction to the network adaptability sub-module to add a network adaptability mechanism. Arrows 4-30 indicate: the network adaptation sub-module feeds back the result to the h.323 video subscription channel, indicating that the media data stream conforming to the channel is acquired. Arrows 4-31 indicate: and feeding back a result to the channel management submodule by the H.323 video subscription channel to indicate that the channel subscription is successful. Arrows 4-32 indicate: and the channel management submodule feeds back a result to the H.323 terminal equipment to indicate that the multi-picture acquisition is successful. Arrow 5-1 indicates: the channel management submodule sends a control instruction to the H.323 video subscription channel to instruct to directly transmit the media data stream to the asynchronous receiving and transmitting unit. Arrows 5-2 indicate: and the H.323 video subscription channel sends the media data stream to the asynchronous transceiving unit. Arrows 5-3 indicate: and the asynchronous receiving and sending unit sends the media data stream to the H.323 terminal equipment for playing.

Compared with the prior art, the multi-type media service fusion system provided by the invention firstly schedules and manages media service resources in a clouding mode, so that media negotiation adaptation, media coding adaptation and network adaptability adaptation among different types of terminal equipment in the multi-type media fusion process are automatically completed, the utilization rate of the media service resources is improved, meanwhile, the efficiency, flexibility and applicability of service fusion are also improved, and the defects of manpower consumption, low efficiency and poor applicability of traditional manual setting are overcome. Secondly, the invention completes the adaptation and the establishment of the connection relation among different types of terminal equipment through a signaling media fusion module, organizes and manages the media stream in the form of a publishing channel and a subscribing channel, so that different types of media services are fused and communicated, namely, the media stream fusion among the terminal equipment with different media protocols is realized by establishing the subscribing relation between the publishing channel and the subscribing channel of the media stream, and a plurality of video streams can be synthesized by utilizing a multi-picture channel, thereby providing the video multi-picture function, providing a plurality of application modes and being suitable for different application scenes. Finally, the invention provides system resource management services through Kubernets, including services of load balancing, automatic capacity expansion and capacity reduction, automatic fault triggering and the like, can improve the cross-platform deployment capacity and the transverse capacity expansion capacity of the system, and improve the resource utilization rate and the availability of the system.

Those skilled in the art will appreciate that all or part of the flow of the method implementing the above embodiments may be implemented by a computer program, which is stored in a computer readable storage medium, to instruct related hardware. The computer readable storage medium is a magnetic disk, an optical disk, a read-only memory or a random access memory.

The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention.

Claims

1. A multi-type media service convergence system, comprising:

2. The multi-type media service convergence system of claim 1, wherein the service control module comprises:

3. The multi-type media service convergence system of claim 1, wherein the signaling media convergence module comprises:

the fusion control sub-module includes:

4. The multi-type media service convergence system of claim 3, wherein the media negotiation sub-module comprises an h.323 negotiation unit, an SIP negotiation unit and an RTSP negotiation unit;

5. The multi-type media service convergence system of claim 3, wherein the signaling media convergence module further comprises an audio processing sub-module, a video processing sub-module, a network adaptability sub-module and a media transceiving sub-module.

6. The multi-type media service convergence system of claim 5, wherein the channel management sub-module is configured to perform media stream convergence between terminal devices with different media protocols, and comprises:

the media stream comprises an audio stream and a video stream;

7. The multi-type media service convergence system of claim 6, wherein the network adaptation sub-module comprises:

8. The multi-type media service convergence system of claim 6, wherein the audio processing sub-module comprises:

9. The multi-type media service convergence system of claim 6, wherein the video processing sub-module comprises:

10. The multi-type media service convergence system of any one of claims 1 to 9, wherein the resource management and scheduling module encapsulates media service resources into a Docker container by building a kubernets cluster environment, and deploys the media service resources on the kubernets cluster environment to manage and schedule the media service resources.