CN117294684A - Media service arrangement method, device, media server and storage medium - Google Patents
Media service arrangement method, device, media server and storage medium Download PDFInfo
- Publication number
- CN117294684A CN117294684A CN202210692888.4A CN202210692888A CN117294684A CN 117294684 A CN117294684 A CN 117294684A CN 202210692888 A CN202210692888 A CN 202210692888A CN 117294684 A CN117294684 A CN 117294684A
- Authority
- CN
- China
- Prior art keywords
- media
- service
- stream
- different
- scene
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 78
- 238000012545 processing Methods 0.000 claims abstract description 173
- 230000008569 process Effects 0.000 claims abstract description 32
- 238000004891 communication Methods 0.000 claims abstract description 20
- 230000002194 synthesizing effect Effects 0.000 claims description 16
- 230000006870 function Effects 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 6
- 230000003993 interaction Effects 0.000 claims description 4
- 230000000694 effects Effects 0.000 abstract description 5
- 238000010586 diagram Methods 0.000 description 16
- 230000001419 dependent effect Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000008602 contraction Effects 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 239000002131 composite material Substances 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000002955 isolation Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000012163 sequencing technique Methods 0.000 description 2
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005315 distribution function Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/60—Software deployment
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/40—Support for services or applications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/60—Network streaming of media packets
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The application relates to a media service arrangement method, a device, a media server and a storage medium, and relates to the technical field of audio and video communication. The media service orchestration method comprises: acquiring the occupation demands of different service scenes on resources on a media server; determining media processing components required by different service scenes from the multiple media processing components according to the occupation requirements; and arranging media processing components required by different service scenes to form media services corresponding to the different service scenes, so as to process the services corresponding to the different service scenes through the media services. Therefore, the media processing component can be flexibly arranged according to different service scenes to obtain the media service applicable to the different service scenes, so that the effects of meeting the requirements of the different service scenes and expanding the service scene use range are achieved.
Description
Technical Field
The present invention relates to the field of audio and video communications technologies, and in particular, to a media service arrangement method, a device, a media server, and a storage medium.
Background
With the further development of the fifth generation mobile communication technology (5 th generation mobile networks, abbreviated as 5G), the types of devices capable of performing audio and video communication, such as 5G terminals and Extended Reality (XR) terminals, are increasing, which results in more and more complex service scenarios of audio and video communication.
Currently, audio and video communication generally adopts a multipoint fusion conference unit (Multipoint Conferencing Unit, abbreviated as MCU) scheme and a routing forwarding unit (Selective Forwarding Unit, abbreviated as SFU) scheme. The MCU scheme comprises a star structure formed by a server and a plurality of terminals, each terminal sends audio and video streams to be shared to the server, the server mixes the audio and video streams of all the terminals in the same room, and finally generates a mixed audio and video stream and sends the mixed audio and video stream to each terminal; the SFU scheme also comprises a server and a plurality of terminals, but is different from the MCU scheme in that the SFU scheme does not mix audio and video, and after receiving the audio and video stream shared by a certain terminal, the SFU scheme directly forwards the audio and video stream to other terminals in the room, like a media stream routing repeater. Because the SFU scheme and the MCU scheme are realized based on specific service scenes, the requirements of different service scenes cannot be met, and the service scenes are limited in use.
Disclosure of Invention
The application provides a media service arrangement method, a device, a media server and a storage medium, which are used for solving the problem that the service scene is limited to use because the prior art is realized based on specific service scenes and the requirements of different service scenes cannot be met.
In a first aspect, the present application provides a media service orchestration method applied to a media server, where the media server includes a plurality of media processing components, and different media processing components are used to implement different processing capabilities in a business processing process; the method comprises the following steps:
acquiring the occupation demands of different service scenes on the resources on the media server;
determining media processing components required by different service scenes from the plurality of media processing components according to the occupation requirements;
and arranging media processing components required by different service scenes to form media services corresponding to the different service scenes, so as to process the services corresponding to the different service scenes through the media services.
Optionally, the resources on the media server include port resources, memory resources and processor resources, and the media processing component includes a media access layer component dependent on the port resources, a media forwarding layer component dependent on the memory resources and a media processing layer component dependent on the processor resources.
Optionally, the media access layer component includes a network processing unit NPU, and the media forwarding layer component includes at least one of a content service bus CSB, a concatenation unit CAS, and a multi-path code stream routing unit SWI; the media processing layer assembly comprises at least one of an audio processing unit APU, a video processing unit VPU and a file processing unit FPU;
the NPU is used for providing media stream access capability and network applicability processing capability, the CSB is used for providing capability of forwarding media streams among any components, the CAS is used for providing capability of data interaction and data synchronization among different media services, the SWI is used for providing capability of outputting the same media streams to a plurality of receiving ends, the APU is used for providing capability of mixing and transcoding audio streams, the VPU is used for providing capability of synthesizing and transcoding video streams, and the FPU is used for providing file reading and file recording capability.
Optionally, the arranging the media processing components required by different service scenes to form media services corresponding to the different service scenes includes:
when the service scene is a scene for forwarding the media stream, arranging the NPU and the CSB or arranging the NPU, the CSB and the SWI to form a first media service; or,
In the case that the service scene is a scene for mixing and/or transcoding an audio stream, arranging the NPU, the CSB and the APU to form a second media service; or,
under the condition that the service scene is a scene for synthesizing and/or transcoding the video stream, the NPU, the CSB and the VPU are arranged to form a third media service; or,
and arranging the NPU, the CSB and the FPU under the condition that the service scene is a scene for playing and recording the file to form a fourth media service.
Optionally, the arranging the media processing components required by different service scenes to form media services corresponding to the different service scenes further includes:
and under the condition that the service scene is a scene of mixing and/or transcoding an audio stream or synthesizing and/or transcoding a video stream and the processing capacity of the APU and/or the VPU is insufficient, the encoding of the CAS is added on the basis of the second media service and/or the third media service, and the cascade connection of a plurality of media services is realized through the CAS.
Optionally, the media services include a media processing service MPS and a video processing service VPS; the cascading mode of the plurality of media services comprises that the MPS is cascaded with the MPS, and that the MPS is cascaded with the VPS.
Optionally, before the arranging the media processing components required by different service scenes to form the media services corresponding to the different service scenes, the method further includes:
registering media processing components except the CSB in media processing components required by a first service scene on the CSB to form global routing information, wherein the first service scene is any service scene, and the global routing information comprises the corresponding relation between a source port and a target port of each media stream in the first service scene;
after the media processing components required by different service scenes are arranged to form the media services corresponding to the different service scenes, the method further comprises the steps of:
under the condition that any media stream in the first service scene is received, determining a destination port of the media stream based on the global routing information and a source port of the media stream;
directly forwarding the media stream to a destination port of the media stream under the condition that the destination port of the media stream and a source port of the media stream are two ports in the same thread;
and forwarding the media stream to a destination port of the media stream in a target thread when the destination port of the media stream and a source port of the media stream are two ports of a cross-thread, wherein the target thread is determined based on the destination port of the media stream.
In a second aspect, the present application further provides a media service orchestration device, applied to a media server, where the media server includes a plurality of media processing components, and different media processing components are used to implement different media processing functions in a business processing process; the device comprises:
the acquisition module is used for acquiring the occupation demands of different service scenes on the resources on the media server;
the first determining module is used for determining media processing components required by different service scenes from the plurality of media processing components according to the occupation requirements;
and the arrangement module is used for arranging the media processing components required by different service scenes to form media services corresponding to the different service scenes so as to process the services corresponding to the different service scenes through the media services.
In a third aspect, the present application further provides an air conditioner control device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;
a memory for storing a computer program;
a processor, configured to implement the steps of the media service orchestration method according to any one of the embodiments of the first aspect, when executing a program stored on a memory.
In a fourth aspect, the present application also provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the media service orchestration method according to any one of the embodiments of the first aspect.
In the embodiment of the application, the occupation requirements of different service scenes on resources on a media server are acquired; determining media processing components required by different service scenes from the plurality of media processing components according to the occupation requirements; and arranging media processing components required by different service scenes to form media services corresponding to the different service scenes, so as to process the services corresponding to the different service scenes through the media services. By the method, the media processing component can be flexibly arranged aiming at different service scenes to obtain the media service applicable to the different service scenes, so that the effects of meeting the requirements of the different service scenes and expanding the service scene use range are achieved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
In order to more clearly illustrate the embodiments of the invention or the technical solutions of the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, and it will be obvious to a person skilled in the art that other drawings can be obtained from these drawings without inventive effort.
Fig. 1 is a flow chart of a media service orchestration method according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a media processing component according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a first media service according to an embodiment of the present disclosure;
FIG. 4 is a second schematic diagram of a first media service according to an embodiment of the present disclosure;
FIG. 5 is a schematic structural diagram of a second media service according to an embodiment of the present disclosure;
FIG. 6 is a schematic structural diagram of a third media service according to an embodiment of the present disclosure;
FIG. 7 is a schematic structural diagram of a fourth media service according to an embodiment of the present disclosure;
FIG. 8 is a schematic diagram of a cascade structure of a plurality of media services according to an embodiment of the present disclosure;
FIG. 9 is a schematic diagram of a media processing service according to an embodiment of the present application;
fig. 10 is a schematic structural diagram of a video processing service according to an embodiment of the present application;
FIG. 11 is a second schematic diagram of a cascade of multiple media services according to an embodiment of the present disclosure;
FIG. 12 is a third schematic diagram of a cascade of multiple media services according to an embodiment of the present disclosure;
fig. 13 is a schematic diagram of a code stream circulation process according to an embodiment of the present application;
FIG. 14 is a schematic diagram of a media service orchestration device according to an embodiment of the present disclosure;
fig. 15 is a schematic structural diagram of a media server according to an embodiment of the present application.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present application based on the embodiments herein.
Referring to fig. 1, fig. 1 is a flow chart of a media service orchestration method according to an embodiment of the present application. As shown in fig. 1, the media service orchestration method is applied to a media server, and the media server comprises a plurality of media processing components, wherein different media processing components are used for realizing different processing capacities in the business processing process; the method may comprise the steps of:
Step 101, acquiring the occupation demands of different service scenes on resources on a media server;
specifically, the media server is deployed based on containerization, and multiple media services can be deployed on the media server, and each media service realizes micro-service and is deployed in a separate container. In particular, each media service may include a plurality of media processing components, with different media processing components being used to implement different processing capabilities in the business process. The different media services adopt small network communication, and generally register service domain names through a micro service Bus (MSB for short), and domain name access is adopted among the micro services. The media service and the external terminal realize media communication by distributing large network IP. The resources on the media server may include port resources, memory resources, processor resources, etc., so that the media processing component on the media server may be divided according to the type of resources on the media server on which the media processing component depends.
102, determining media processing components required by different service scenes from a plurality of media processing components according to the occupation requirements;
because the media processing components on the media server can be divided according to the types of the resources on the media server on which the media processing components depend, after the occupation requirements of different service scenes on the resources on the media server are determined, the media processing components required by the different service scenes can be determined according to the occupation requirements of the different service scenes on the resources on the media server.
And 103, arranging media processing components required by different service scenes to form media services corresponding to the different service scenes, so as to process the services corresponding to the different service scenes through the media services.
After the media processing components required by different service scenes are determined, the media processing components required by the different service scenes can be arranged, so that the media services corresponding to the different service scenes are obtained, and the services corresponding to the different service scenes can be processed through the media services.
In this embodiment, the media processing component may be flexibly arranged for different service scenarios, so as to obtain media services applicable to different service scenarios, thereby achieving the effects of meeting the requirements of different service scenarios and expanding the service scenario application range.
Further, the resources on the media server include port resources, memory resources, and processor resources, and the media processing component includes a media access layer component that depends on the port resources, a media forwarding layer component that depends on the memory resources, and a media processing layer component that depends on the processor resources.
Referring to fig. 2, fig. 2 is a schematic diagram of a media processing component according to an embodiment of the present application. The media processing component may be divided into a media access layer component, a media forwarding layer component, and a media processing layer component, depending on the type of resources on the media server on which it is dependent. The media access layer component mainly consumes port resources and is mainly used for accessing and outputting media streams; the media forwarding layer component mainly consumes memory resources and is mainly used for forwarding the media stream; the media processing layer component mainly consumes processor resources, which are mainly used for processing audio and video, and the resource consumption of video processing is larger than that of audio processing.
In this embodiment, on the basis that the media processing component divides according to the type of the resource on which the media processing component depends, service arrangement can be performed according to the requirement of resource occupation in the service scene, and for the case that only the forwarding of the media stream is required, arrangement from the media access layer component to the media forwarding layer component can be completed only; for audio-video conferences or live broadcasts, the arrangement from the media access layer component to the media forwarding layer component and the media processing layer component can be completed; meanwhile, the audio and video capability isolation can be realized through service arrangement, when the video capability is insufficient, the video processing module can be dynamically expanded, and when the audio capability is insufficient, the audio processing module can be dynamically expanded. Therefore, the resources can be flexibly allocated according to the resource occupation, so that the resource occupation under different service scenes is saved.
Further, with continued reference to fig. 2, the media access layer component includes a network processing unit NPU, and the media forwarding layer component includes at least one of a content service bus CSB, a concatenation unit CAS, and a multi-way code stream routing unit SWI; the media processing layer component comprises at least one of an audio processing unit APU, a video processing unit VPU and a file processing unit FPU;
The system comprises an NPU, a CSB, a CAS, a SWI, an APU, a VPU, a file reading and recording capability, wherein the NPU is used for providing media stream access capability and network applicability processing capability, the CSB is used for providing media stream forwarding capability among any components, the CAS is used for providing data interaction and data synchronization capability among different media services, the SWI is used for providing the capability of outputting the same media stream to a plurality of receiving ends, the APU is used for providing audio stream mixing and transcoding capability, the VPU is used for providing video stream synthesizing and transcoding capability, and the FPU is used for providing file reading and file recording capability.
In particular, a media access layer component may support a real-time transport protocol (Realtime Transport Protocol, abbreviated RTP) and a real-time transport control protocol (Realtime Transport Control Protocol, abbreviated RTCP), which may include a network processing unit (Network Process Unit, abbreviated NPU). The NPU component can simultaneously provide access capability of audio streams and video streams, has network adaptability processing capability of audio and video, and can support various capabilities of automatic packet loss resistance, jitter resistance and the like; and the NPU component can also independently provide audio access and video access capability, and the arrangement is suitable for different types of terminals, such as the arrangement of 1 audio to access capability of an audio terminal, the arrangement of 1 video access capability of a monitoring terminal, the arrangement of 1 audio access capability and a plurality of video access capabilities of a live broadcast terminal, and the like.
The media forwarding layer components can be divided into three components, namely a multipath code stream routing Unit (SWI) for short, a Cascade Unit (CAS for short) and a content service bus (Content Service Bus for short, CSB) according to the forwarding modes. The SWI component can uniformly set an input source, support setting a plurality of output addresses and output the same media stream to a plurality of destination terminals, and is mainly applied to a large-capacity conference scene. The CAS component is mainly responsible for data exchange among different media services and data synchronization during dynamic capacity expansion. The CAS component can establish a media link between media services according to the media IP and adopts an RTP multi-stream scheme to reduce the resource consumption of the media ports. The CSB component is mainly responsible for forwarding the media stream between any media processing components, including NPU components, CAS components, SWI components, media processing layer components, audio processing units (Audio Process Unit, abbreviated as APUs), video processing units (Video Process Unit, abbreviated as VPUs), and file processing units (File Process Unit, abbreviated as FPUs), and any media processing components that may be subsequently extended. When arranging all media processing components, firstly, registration is needed on the CSB component, a forwarding data area is allocated, then, a route is issued on the CSB component, and a forwarding path is confirmed. The CSB component reads the media stream and writes the destination address according to the forwarding path. The CSB component is the infrastructure for the streaming of the internal media processing component media streams.
The media processing layer components may be divided into an APU component, a VPU component, and a FPU component according to audio and video processing capabilities. Wherein the APU component comprises a plurality of audio processing units for mixing, sorting, decoding, encoding and the like. Under MCU architecture, supporting complete audio decoding, sequencing, mixing, encoding and other processes; under SFU architecture, processes such as decoding and sequencing of audio are supported, and coding capacity consumption is reduced. The VPU component includes a plurality of video processing units for decoding, synthesizing, encoding, etc. Under MCU architecture, supporting complete video decoding, synthesizing and encoding flow; under SFU architecture, RTP header modification flow is supported without consuming encoding and decoding capability. The FPU component mainly integrates file reading and file recording functions.
In this embodiment, a plurality of media service components may be provided, and media services under a plurality of complex service scenarios may be flexibly arranged through the media service components, so as to implement customization of audio and video services. Meanwhile, because the audio and video encoding and decoding consume more CPU resources, the dynamic deployment of the APU component and the VPU component is significant for the soft media server.
Further, the step 103 of arranging the media processing components required by different service scenes to form media services corresponding to the different service scenes includes:
Under the condition that the service scene is a scene for forwarding the media stream, the NPU and the CSB are arranged, or the NPU, the CSB and the SWI are arranged, so that a first media service is formed; or,
under the condition that the service scene is a scene for mixing and/or transcoding the audio stream, the NPU, the CSB and the APU are arranged to form a second media service; or,
under the condition that the service scene is a scene for synthesizing and/or transcoding the video stream, the NPU, the CSB and the VPU are arranged to form a third media service; or,
and arranging the NPU, the CSB and the FPU under the condition that the service scene is a scene for playing and recording the file to form a fourth media service.
In an embodiment, media processing components required by different service scenes can be arranged according to different service scenes to form media services corresponding to the different service scenes. Specifically, when the service scenario is a scenario of forwarding a media stream, the NPU and the CSB may be arranged, or the NPU, the CSB, and the SWI may be arranged, to form a first media service. For example, as shown in fig. 3, when user-defined forwarding of the audio and video code stream is required, the uplink audio and video code stream of the terminal 1 may be received and demultiplexed by the NPU component, and demultiplexed into multiple different format code streams, forwarded to the destination port through the CSB component, and finally forwarded to the terminal 2 and the terminal 3 through the NPU component. As shown in fig. 4, when a large-capacity conference is required, a large number of forwarding operations occur due to the batch video/audio forwarding, especially in the case of switching broadcast sources, if the first media service shown in fig. 3 is adopted, the routing relationship between each sending end and each receiving end needs to be re-issued, and if the first media service shown in fig. 4 is adopted, the batch distribution of the code streams can be realized through the SWI component. After receiving the uplink code stream of the terminal 1, the NPU module may forward the uplink code stream to the SWI module by the CSB module, and then forward the uplink code stream to a destination port other than a broadcast source in batches by the SWI module according to a route set in advance, and finally forward the uplink code stream to the terminal 2, the terminal 3 and the terminal 4 by the NPU module. When the audio and video sources of the transmitting end are switched, the quick switching of the broadcast source code stream can be realized only by updating the input source setting of the SWI component.
In particular, when the traffic scenario is a scenario in which audio streams are mixed and/or transcoded, the NPU, CSB, and APU may be orchestrated to form a second media service. For example, as shown in FIG. 5, when an audio conference is desired, audio mixing and transcoding functions may be implemented through the APU component provided by the second media server. The uplink audio code streams of the terminal 1, the terminal 2 and the terminal 3 are received by the NPU component and forwarded to the APU component by the CSB component, the APU component mixes the audio according to the RTP extension header of the code stream or the volume value obtained by decoding calculation, and the audio code streams after mixing are respectively transcoded into audio formats negotiated by each terminal and forwarded to the destination port by the CSB component, and are sent to the terminal 4, the terminal 5 and the terminal 6 by the NPU component.
Specifically, when the service scenario is a scenario in which the video stream is synthesized and/or transcoded, the NPU, CSB, and VPU may be arranged to form a third media service. For example, as shown in fig. 6, when a multi-screen conference is required, the video stream synthesizing and transcoding functions may be implemented through a VPU component provided by the third media server. The uplink video code streams of the terminal 1, the terminal 2 and the terminal 3 are received by the NPU component and forwarded to the VPU component by the CSB component, the VPU component transcodes the video code streams into the adaptive media capability, and transcodes the multipath terminal code streams into the composite pictures with the multi-picture layout, and then forwarded to the destination port by the CSB component, and sent to the terminal 4, the terminal 5 and the terminal 6 by the NPU component.
Specifically, when the service scene is a scene for playing and recording the file, the NPU, the CSB, and the FPU may be arranged to form a fourth media service. For example, as shown in fig. 7, the media service generally stores media files by mounting a disk, and when playing audio and video files is required, the FPU component can read the file content in the disk, forward the file content to the NPU component through the CSB component, and forward the code stream to the terminal 1 through the NPU component.
In this embodiment, the media processing component may be flexibly arranged for different service scenarios, so as to obtain media services applicable to different service scenarios, thereby achieving the effects of meeting the requirements of different service scenarios and expanding the service scenario application range.
Further, the step 103 of arranging the media processing components required by different service scenes to form media services corresponding to the different service scenes further includes:
under the condition that the service scene is a scene of mixing and/or transcoding an audio stream or synthesizing and/or transcoding a video stream and the processing capacity of an APU and/or a VPU is insufficient, the encoding of the CAS is added on the basis of the second media service and/or the third media service, and the cascade connection of a plurality of media services is realized through the CAS.
Specifically, when the service scenario is audio stream mixing and the APU processing capability is insufficient, encoding of CAS may be added on the basis of the second media service, and cascading of multiple media services may be implemented through CAS. For example, as shown in FIG. 8, when the audio mixing and transcoding capabilities provided by the APU component provided by the second media server 1 are insufficient, the CAS component may be encoded in the second media server 1 and the second media server 2, such that the second media server 1 and the second media server 2 may be cascaded by the CAS component for audio mixing and transcoding by the APU component in the second media server 1 and the second media server 2. Similarly, when the service scenario is that the video stream is synthesized and the VPU processing capability is insufficient, encoding of the CAS may be added on the basis of the third media service, and cascading of a plurality of media services may be implemented through the CAS. Therefore, the audio and video capability isolation can be realized through service arrangement, and when the video capability is insufficient, the video processing component can be dynamically expanded; when the audio capability is insufficient, the audio processing component can be dynamically expanded to improve the overall performance of the media server.
Further, the media services include a media processing service MPS and a video processing service VPS; the cascade of the plurality of media services includes MPS and MPS cascade, and MPS and VPS cascade.
Specifically, the structure of the above media processing service (Media Process Service, abbreviated MPS) is shown in fig. 9, and the media processing service MPS may support functions such as video forwarding, audio forwarding, and audio mixing. The code stream is input by the NPU component, and the audio and video are distributed according to the routing information. The audio code stream can be forwarded to the APU component through the CSB component, the APU component sequences, mixes and encodes the code stream, and then forwards the code stream to the NPU component through the CSB component. If the audio capability is insufficient, the MPS resource is increased through dynamic expansion and contraction, and the audio code stream is sent to another MPS for processing. Under the video code stream forwarding scene, the video code stream is accessed through the NPU component, can be sent to the SWI component through the CSB component, and is forwarded to each destination port through the unified routing rule of the SWI component; in the video code stream transcoding scene, the video code stream can be forwarded to the VPS component through the CAS component, so that CPU resource consumption of audio encoding and decoding and video encoding and decoding is separated.
The structure of the video processing service (Video Process Service, simply referred to as VPS) is shown in fig. 10, and the video processing service VPS can support video transcoding and video forwarding functions. The code stream is input by the NPU component or the CAS component, distributed according to the route information, and then the processes of decoding, encoding, synthesizing pictures and the like are completed according to the business arrangement. Video transcoding consumes significant CPU resources, e.g., 16C machines only support 16-way video codecs, so video codec performance is a key factor limiting conference capacity. Compared with the traditional media server, the video processing service realizes the self-adaptive expansion and contraction of CPU resources through a dynamic flicking mechanism under a micro-service architecture, can realize a large-capacity conference, and completes the resource recovery after the conference.
When cascading a plurality of media services, a mode of cascading MPS and MPS can be adopted, or a mode of cascading MPS and VPS can be adopted. For example, as shown in fig. 11, when cross-domain concatenation is required, the CAS component may implement code stream forwarding between multiple MPS, and when a certain MPS has insufficient performance, the CAS component may implement dynamic expansion of performance, so that multiple MPS accommodates more forwarding or transcoding services. Fig. 11 shows a simple scenario of cascading forwarding, where the uplink code stream of the terminal 1 flows to the NPU component of the media processing service MPS1, is forwarded to the cascading port of the CAS component via the SWI component, is transmitted to the cascading port of the CAS component of the media processing service MPS2 through the network, is forwarded to the NPU component by the CSB component, and is further downstream to the terminal 2 and the terminal 3. The CAS component is also applied to cross-domain forwarding of audio code streams, and relates to mixing and forwarding of APU components, which are not described in detail in this embodiment.
As shown in fig. 12, when cascaded video transcoding is required, video compositing and transcoding functions can be implemented by the VPU component. The function serves two scenes, wherein the first is that different terminals carry different video capabilities and can not directly receive the opposite-end code stream, and a media server is required to transcode the opposite-end code stream into the adaptive media capability; and secondly, in the multi-picture conference, the media server is required to transcode the multi-channel terminal code stream into a composite picture with multi-picture layout. Since the video transcoding function belongs to an extensible service, it can be deployed as a video processing service VPS alone to achieve dynamic extension of the service. In fig. 12, the NPU component may receive the uplink code streams from the terminal 1 and the terminal 2, forward the uplink code streams to the cascade port of the CAS component through the CSB component, send the uplink code streams to the cascade port of the CAS component of the opposite video processing service VPS through the network, and forward the uplink code streams to the VPU component for synthesis and transcoding, and send the output video code streams to the cascade ports through the opposite NPU to the terminal 1 and the terminal 2.
Therefore, the method and the device can be compatible with MCU architecture and SFU architecture, and realize flexible arrangement of media services in various service scenes. The method utilizes the micro-service architecture, completes flexible allocation of resources in conference service through dynamic expansion and contraction, has clear component interfaces through decoupling division, can arrange and cover most of main stream service scenes, and has strong service expansibility.
Further, before arranging the media processing components required by different service scenarios in step 103 to form the media services corresponding to the different service scenarios, the method further includes:
registering media processing components except CSB in the media processing components required by the first service scene on the CSB to form global routing information, wherein the first service scene is any service scene, and the global routing information comprises the corresponding relation between a source port and a target port of each media stream in the first service scene;
after the step 103 of arranging the media processing components required by different service scenes to form the media services corresponding to the different service scenes, the method further includes:
under the condition that any media stream in a first service scene is received, determining a destination port of the media stream based on global routing information and a source port of the media stream;
Under the condition that the destination port of the media stream and the source port of the media stream are two ports in the same thread, directly forwarding the media stream to the destination port of the media stream;
and forwarding the media stream to the destination port of the media stream in the target thread under the condition that the destination port of the media stream and the source port of the media stream are two ports of the cross-thread, wherein the target thread is determined based on the destination port of the media stream.
In an embodiment, an NPU-bus-NPU forwarding manner may be adopted to forward the inter-process media stream (i.e. the audio/video code stream), reform the inter-process media stream into the data pointer transfer in the process, so that flexibility of arrangement is maintained, and meanwhile, forwarding efficiency of media is improved. The forwarding of media streams within a process mainly involves the streaming of media data packets between NPU, APU, SWI, VPU and other components, which is done depending on the CSB. NPU, APU, SWI, VPU support setting up routes individually, where a certain Stream (i.e. Stream) in the form of a certain object (i.e. PortID) of the present module is sent to an object (i.e. PortID) of any module, so that the CSB is required to build a global routing table, and aggregate the routing information of each module.
The object of each media processing component registers its own information on the CSB by means of registration, where the registration content may include its own identifier PortID and a callback function pFunc () for processing the media data packet, and stores it in the in-memory database CsbDataArea. When a certain component needs to forward a media data packet to a receiving end of the media data packet, a CSB distribution function is directly called, and the CSB searches global routing information through Stream information and source PortID (i.e. source port in the above) information to which the media data packet belongs, and obtains all destination PortIDs (i.e. destination ports in the above). After obtaining all the destination PortID, forwarding the media data packet in the same thread, and executing the callback function registered by each destination PortID in sequence; for the media data packet forwarding of the cross-thread, after the media data packet is copied, the destination PortID after the cross-thread is recorded in the media data packet, then the copied media data packet is written into a buffer area (such as RingBuffer) special for the cross-thread, the destination thread reads the media data packet in the buffer area through a fixed task driver, and then a callback function registered by each destination PortID is sequentially called according to the destination PortID information in the media data packet, so that the in-process circulation of the media data packet is completed, and the circulation process is shown in figure 13.
In addition, referring to fig. 14, fig. 14 is a schematic structural diagram of a media service orchestration device according to an embodiment of the present application. As shown in fig. 14, the media service orchestration device 1400 is applied to a media server that includes various media processing components for implementing different media processing functions in a business process; the apparatus 1400 includes:
an obtaining module 1401, configured to obtain occupation requirements of different service scenarios on resources on a media server;
a first determining module 1402, configured to determine media processing components required for different service scenarios from among the multiple media processing components according to the occupation requirement;
the arrangement module 1403 is configured to arrange media processing components required by different service scenarios, so as to form media services corresponding to the different service scenarios, so as to process services corresponding to the different service scenarios through the media services.
Further, the resources on the media server include port resources, memory resources, and processor resources, and the media processing component includes a media access layer component that depends on the port resources, a media forwarding layer component that depends on the memory resources, and a media processing layer component that depends on the processor resources.
Further, the media access layer component comprises a network processing unit NPU, and the media forwarding layer component comprises at least one of a content service bus CSB, a concatenation unit CAS, and a multi-path code stream routing unit SWI; the media processing layer component comprises at least one of an audio processing unit APU, a video processing unit VPU and a file processing unit FPU;
the system comprises an NPU, a CSB, a CAS, a SWI, an APU, a VPU, a file reading and recording capability, wherein the NPU is used for providing media stream access capability and network applicability processing capability, the CSB is used for providing media stream forwarding capability among any components, the CAS is used for providing data interaction and data synchronization capability among different media services, the SWI is used for providing the capability of outputting the same media stream to a plurality of receiving ends, the APU is used for providing audio stream mixing and transcoding capability, the VPU is used for providing video stream synthesizing and transcoding capability, and the FPU is used for providing file reading and file recording capability.
Further, orchestration module 1403 includes:
the first arrangement sub-module is used for arranging the NPU and the CSB or arranging the NPU, the CSB and the SWI to form a first media service under the condition that the service scene is a scene for forwarding the media stream; or,
the second arrangement sub-module is used for arranging the NPU, the CSB and the APU to form a second media service under the condition that the service scene is a scene for mixing and/or transcoding the audio stream; or,
The third arrangement sub-module is used for arranging the NPU, the CSB and the VPU to form a third media service under the condition that the service scene is a scene for synthesizing and/or transcoding the video stream; or,
and the fourth arrangement sub-module is used for arranging the NPU, the CSB and the FPU to form a fourth media service under the condition that the service scene is a scene for playing and recording the file.
Further, orchestration module 1403 further comprises:
and the fifth arrangement sub-module is used for increasing the encoding of the CAS on the basis of the second media service and/or the third media service under the condition that the service scene is a scene of mixing and/or transcoding an audio stream or synthesizing and/or transcoding a video stream and the processing capacity of an APU and/or a VPU is insufficient, and realizing cascade connection of a plurality of media services through the CAS.
Further, the media services include a media processing service MPS and a video processing service VPS; the cascade of the plurality of media services includes MPS and MPS cascade, and MPS and VPS cascade.
Further, the apparatus 1400 further comprises:
the registration module is used for registering media processing components except the CSB in the media processing components required by the first service scene to form global routing information, wherein the first service scene is any service scene, and the global routing information comprises the corresponding relation between a source port and a target port of each media stream in the first service scene;
The second determining module is used for determining a destination port of the media stream based on the global routing information and a source port of the media stream under the condition that any media stream in the first service scene is received;
the first forwarding module is used for forwarding the media stream directly to the destination port of the media stream under the condition that the destination port of the media stream and the source port of the media stream are two ports in the same thread;
and the second forwarding module is used for forwarding the media stream to the destination port of the media stream in the target thread when the destination port of the media stream and the source port of the media stream are two ports of the cross-thread, and the target thread is determined based on the destination port of the media stream.
It should be noted that, the media service orchestration device 1400 may implement the steps of the media service orchestration method provided in any one of the foregoing method embodiments, and achieve the same technical effects, which are not described herein.
As shown in fig. 15, embodiments of the present application provide a media server comprising a processor 1511, a communication interface 1512, a memory 1513, and a communication bus 1514, wherein the processor 1511, the communication interface 1512, the memory 1513 communicate with each other via the communication bus 1514,
A memory 1513 for storing a computer program;
in one embodiment of the present application, the processor 1511 is configured to implement the media service orchestration method provided in any one of the foregoing method embodiments when executing the program stored on the memory 1513, where the method includes:
acquiring the occupation demands of different service scenes on resources on a media server;
determining media processing components required by different service scenes from the multiple media processing components according to the occupation requirements;
and arranging media processing components required by different service scenes to form media services corresponding to the different service scenes, so as to process the services corresponding to the different service scenes through the media services.
The present application also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the media service orchestration method provided by any one of the method embodiments described above.
It should be noted that in this document, relational terms such as "first" and "second" and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The foregoing is only a specific embodiment of the invention to enable those skilled in the art to understand or practice the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A media service orchestration method, characterized by being applied to a media server, wherein the media server comprises a plurality of media processing components, and different media processing components are used for realizing different processing capacities in a business processing process; the method comprises the following steps:
acquiring the occupation demands of different service scenes on the resources on the media server;
determining media processing components required by different service scenes from the plurality of media processing components according to the occupation requirements;
and arranging media processing components required by different service scenes to form media services corresponding to the different service scenes, so as to process the services corresponding to the different service scenes through the media services.
2. The method of claim 1, wherein the resources on the media server comprise port resources, memory resources, and processor resources, and the media processing component comprises a media access layer component that relies on the port resources, a media forwarding layer component that relies on the memory resources, and a media processing layer component that relies on the processor resources.
3. The method of claim 2, wherein the media access layer component comprises a network processing unit NPU, and the media forwarding layer component comprises at least one of a content service bus CSB, a concatenation unit CAS, and a multi-way code stream routing unit SWI; the media processing layer assembly comprises at least one of an audio processing unit APU, a video processing unit VPU and a file processing unit FPU;
the NPU is used for providing media stream access capability and network applicability processing capability, the CSB is used for providing capability of forwarding media streams among any components, the CAS is used for providing capability of data interaction and data synchronization among different media services, the SWI is used for providing capability of outputting the same media streams to a plurality of receiving ends, the APU is used for providing capability of mixing and transcoding audio streams, the VPU is used for providing capability of synthesizing and transcoding video streams, and the FPU is used for providing file reading and file recording capability.
4. A method according to claim 3, wherein the arranging the media processing components required for different service scenarios to form the media services corresponding to the different service scenarios comprises:
when the service scene is a scene for forwarding the media stream, arranging the NPU and the CSB or arranging the NPU, the CSB and the SWI to form a first media service; or,
in the case that the service scene is a scene for mixing and/or transcoding an audio stream, arranging the NPU, the CSB and the APU to form a second media service; or,
under the condition that the service scene is a scene for synthesizing and/or transcoding the video stream, the NPU, the CSB and the VPU are arranged to form a third media service; or,
and arranging the NPU, the CSB and the FPU under the condition that the service scene is a scene for playing and recording the file to form a fourth media service.
5. The method of claim 4, wherein the arranging the media processing components required for different service scenarios to form the media services corresponding to the different service scenarios further comprises:
And under the condition that the service scene is a scene of mixing and/or transcoding an audio stream or synthesizing and/or transcoding a video stream and the processing capacity of the APU and/or the VPU is insufficient, the encoding of the CAS is added on the basis of the second media service and/or the third media service, and the cascade connection of a plurality of media services is realized through the CAS.
6. The method according to claim 5, wherein the media services comprise a media processing service MPS and a video processing service VPS; the cascading mode of the plurality of media services comprises that the MPS is cascaded with the MPS, and that the MPS is cascaded with the VPS.
7. The method of claim 3, wherein prior to the orchestrating the media processing components required for the different traffic scenarios to form the media services corresponding to the different traffic scenarios, the method further comprises:
registering media processing components except the CSB in media processing components required by a first service scene on the CSB to form global routing information, wherein the first service scene is any service scene, and the global routing information comprises the corresponding relation between a source port and a target port of each media stream in the first service scene;
After the media processing components required by different service scenes are arranged to form the media services corresponding to the different service scenes, the method further comprises the steps of:
under the condition that any media stream in the first service scene is received, determining a destination port of the media stream based on the global routing information and a source port of the media stream;
directly forwarding the media stream to a destination port of the media stream under the condition that the destination port of the media stream and a source port of the media stream are two ports in the same thread;
and forwarding the media stream to a destination port of the media stream in a target thread when the destination port of the media stream and a source port of the media stream are two ports of a cross-thread, wherein the target thread is determined based on the destination port of the media stream.
8. A media service orchestration device, characterized by being applied to a media server, wherein the media server comprises a plurality of media processing components, and different media processing components are used for realizing different media processing functions in a business processing process; the device comprises:
the acquisition module is used for acquiring the occupation demands of different service scenes on the resources on the media server;
The first determining module is used for determining media processing components required by different service scenes from the plurality of media processing components according to the occupation requirements;
and the arrangement module is used for arranging the media processing components required by different service scenes to form media services corresponding to the different service scenes so as to process the services corresponding to the different service scenes through the media services.
9. The media server is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus;
a memory for storing a computer program;
a processor for implementing the steps of the media service orchestration method according to any one of claims 1-7, when executing a program stored on a memory.
10. A computer readable storage medium having stored thereon a computer program, which when executed by a processor performs the steps of the media service orchestration method according to any one of claims 1-7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210692888.4A CN117294684A (en) | 2022-06-17 | 2022-06-17 | Media service arrangement method, device, media server and storage medium |
| PCT/CN2023/100712 WO2023241693A1 (en) | 2022-06-17 | 2023-06-16 | Media service orchestration method and apparatus, and media server and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210692888.4A CN117294684A (en) | 2022-06-17 | 2022-06-17 | Media service arrangement method, device, media server and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117294684A true CN117294684A (en) | 2023-12-26 |
Family
ID=89192346
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210692888.4A Pending CN117294684A (en) | 2022-06-17 | 2022-06-17 | Media service arrangement method, device, media server and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN117294684A (en) |
| WO (1) | WO2023241693A1 (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117793428A (en) * | 2023-12-27 | 2024-03-29 | 北京中科大洋科技发展股份有限公司 | System and method for driving flow equipment |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102624743A (en) * | 2012-04-09 | 2012-08-01 | 苏州工业园区云视信息技术有限公司 | Resource allocation method of media server |
| US11159990B2 (en) * | 2019-10-29 | 2021-10-26 | At&T Intellectual Property I, L.P. | Microservices coordinator for 5G or other next generation network |
| CN111935321B (en) * | 2020-10-12 | 2021-01-22 | 中国传媒大学 | Converged media micro-service platform based on container cloud |
| CN112751856A (en) * | 2020-12-30 | 2021-05-04 | 湖南快乐阳光互动娱乐传媒有限公司 | Media processing system capable of arranging flow |
| CN113703784A (en) * | 2021-08-25 | 2021-11-26 | 上海哔哩哔哩科技有限公司 | Data processing method and device based on container arrangement |
| CN114187541A (en) * | 2021-10-27 | 2022-03-15 | 福建亿榕信息技术有限公司 | Intelligent video analysis method and storage device for user-defined service scene |
-
2022
- 2022-06-17 CN CN202210692888.4A patent/CN117294684A/en active Pending
-
2023
- 2023-06-16 WO PCT/CN2023/100712 patent/WO2023241693A1/en unknown
Also Published As
| Publication number | Publication date |
|---|---|
| WO2023241693A1 (en) | 2023-12-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9521006B2 (en) | Duplicating digital streams for digital conferencing using switching technologies | |
| CN100583985C (en) | Method, apparatus and system for switching pictures in video service | |
| CN109640029B (en) | Method and device for displaying video stream on wall | |
| US8633962B2 (en) | Video decoder which processes multiple video streams | |
| CN100454820C (en) | MCU cascade system and establishing and communication method for the same | |
| US20080117965A1 (en) | Multiple-Channel Codec and Transcoder Environment for Gateway, Mcu, Broadcast, and Video Storage Applications | |
| CN101370114A (en) | Video and audio processing method, multi-point control unit and video conference system | |
| WO2016180189A1 (en) | Method and apparatus for accessing multimedia conference system | |
| CN109474715B (en) | Resource configuration method and device based on video network | |
| CN109660816B (en) | Information processing method and device | |
| CN110460804B (en) | Conference data transmitting method, system, device and computer readable storage medium | |
| WO2023241693A1 (en) | Media service orchestration method and apparatus, and media server and storage medium | |
| WO2014095671A1 (en) | Method and a device for optimizing large scaled video conferences | |
| CN111478884B (en) | Method, device and storage medium for data transmission among different video networks | |
| CN111431966B (en) | Service request processing method and device, electronic equipment and storage medium | |
| CN113194278A (en) | Conference control method and device and computer readable storage medium | |
| CN111131760B (en) | Video recording method and device | |
| CN101316352B (en) | Method and device for implementing multiple pictures of conference television system, video gateway and implementing method thereof | |
| CN111193767A (en) | Request data sending method and device and clustered server system | |
| CN110737519A (en) | theme switching method and device | |
| US20200329083A1 (en) | Video conference transmission method and apparatus, and mcu | |
| CN108270731A (en) | A kind of data flow transmission method, device and relevant device | |
| CN110519543B (en) | Video telephone dialing method and device | |
| JPH11163939A (en) | Autonomous routing changeover device and video conference system using the same changeover device | |
| CN118200625A (en) | Multimedia processing system, multimedia processing method and related equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication |