CN116801039A - Fusion cascade display method and device, multimedia broadcast control system and electronic device - Google Patents

Fusion cascade display method and device, multimedia broadcast control system and electronic device Download PDF

Info

Publication number
CN116801039A
CN116801039A CN202310732353.XA CN202310732353A CN116801039A CN 116801039 A CN116801039 A CN 116801039A CN 202310732353 A CN202310732353 A CN 202310732353A CN 116801039 A CN116801039 A CN 116801039A
Authority
CN
China
Prior art keywords
video
cascade
fusion
display
path
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310732353.XA
Other languages
Chinese (zh)
Inventor
许丽明
方兴文
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Rockchip Electronics Co Ltd
Original Assignee
Rockchip Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Rockchip Electronics Co Ltd filed Critical Rockchip Electronics Co Ltd
Priority to CN202310732353.XA priority Critical patent/CN116801039A/en
Publication of CN116801039A publication Critical patent/CN116801039A/en
Pending legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs
    • H04N21/44016—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream, rendering scenes according to MPEG-4 scene graphs involving splicing one content stream with another content stream, e.g. for substituting a video clip
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/4104—Peripherals receiving signals from specially adapted client devices
    • H04N21/4122—Peripherals receiving signals from specially adapted client devices additional display device, e.g. video projector
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
    • H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
    • H04N21/43076—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen of the same content streams on multiple devices, e.g. when family members are watching the same movie on different devices
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/436—Interfacing a local distribution network, e.g. communicating with another STB or one or more peripheral devices inside the home
    • H04N21/4363—Adapting the video or multiplex stream to a specific local network, e.g. a IEEE 1394 or Bluetooth® network
    • H04N21/43632—Adapting the video or multiplex stream to a specific local network, e.g. a IEEE 1394 or Bluetooth® network involving a wired protocol, e.g. IEEE 1394
    • H04N21/43635—HDMI

Abstract

The present application relates to the field of image processing technologies, and in particular, to a fusion cascade display method and apparatus, a multimedia broadcast control system, and an electronic apparatus. The fusion cascade display method comprises the following steps: acquiring heterogeneous multi-path video streams from various video sources; splicing the heterogeneous multi-path video streams to obtain spliced video; writing the spliced video into frame buffer equipment; and displaying the spliced video on a display device cascaded with the frame buffer device. According to the embodiment of the disclosure, the multi-channel video streams of various content sources are spliced and fused and then written into the frame buffer device, and the frame buffer device is used as a virtual screen to drive the display device cascaded with the frame buffer device to display, so that the multi-device to multi-device screen casting can be realized in the multimedia broadcasting control system. Compared with the traditional hardware cascade, the fusion cascade display mode has more flexible variability, and can be used for cascade display after complex image processing.

Description

Fusion cascade display method and device, multimedia broadcast control system and electronic device
Technical Field
The present application relates to the field of image processing technologies, and in particular, to a method and an apparatus for fusion cascade display, a multimedia broadcast control system, and an electronic device.
Background
Mobile devices share screen content to large screen devices typically employ wireless screen casting technology. The wireless screen-throwing technology comprises a screen-throwing sending end and a screen-throwing receiving end, and the roles of the two parties are not peer-to-peer, and are received and sent. The screen-throwing sending end and the screen-throwing receiving end need to establish handshake information by a screen-throwing protocol. And the screen-throwing transmitting end encodes the content such as the video and transmits the content to the screen-throwing receiving end through the Wi-Fi network. The screen throwing receiving end decodes after receiving the coding signal, restores the video and plays the video. The main stream screen projection protocol at present comprises an AirPlay protocol of apples, a DLNA protocol of the digital living network alliance and a Miracast protocol of the Wi-Fi alliance.
The screen projection system is a single device to single device (point to point) screen projection, and the multimedia broadcasting control system has multiple video inputs and multiple video outputs, so the screen projection system cannot be applied to the multimedia broadcasting control system.
Disclosure of Invention
The application provides a fusion cascade display method, fusion cascade equipment, a multimedia playing control system and electronic equipment, which can render heterogeneous video streams to a virtual screen, and obviously improve the configuration flexibility of the system.
In a first aspect, a fusion cascade display method is provided. The method comprises the following steps: acquiring heterogeneous multi-path video streams from various video sources; splicing the heterogeneous multi-path video streams to obtain spliced video; writing the spliced video into frame buffer equipment; and displaying the spliced video on a display device cascaded with the frame buffer device.
In some embodiments, splicing the heterogeneous multi-path video stream to obtain a spliced video comprises: respectively demultiplexing and decoding the heterogeneous multi-path video stream; and splicing the decoded multipath videos to obtain the spliced videos.
In some embodiments, demultiplexing and decoding the heterogeneous multi-path video streams, respectively, includes: and respectively decoding the heterogeneous multi-path video streams according to the synchronous beats.
In some embodiments, stitching the decoded multi-path video to obtain the stitched video comprises: and splicing the decoded multi-path video according to fusion cascade configuration information to obtain the spliced video stream, wherein the fusion cascade configuration information comprises information related to the layout of the multi-path video in the splicing and fusion process.
In some embodiments, the method further comprises: starting clock source synchronization and display synchronization according to the fusion cascade configuration information; driving a multi-path decoder according to the clock source synchronization with a synchronous beat so as to decode the heterogeneous multi-path video streams respectively; and rendering the spliced video to the frame buffer device serving as a virtual screen according to the display synchronization.
In some embodiments, displaying the stitched video on a display device cascaded with the frame buffer device comprises: detecting that rendering of the frame buffer device is complete; and notifying the display device to render.
In some embodiments, displaying the stitched video on a display device cascaded with the frame buffer device comprises: and rendering the spliced video on a physical display device or a virtual display device.
In some embodiments, rendering the stitched video on a physical display device or a virtual display device comprises: rendering the spliced video on at least one physical display device and/or at least one virtual display device according to device cascade configuration information, wherein the device cascade configuration information comprises information associated with each device of a cascade of which a total task is divided into a plurality of subtasks.
In a second aspect, a fusion cascade device is provided. The apparatus includes: the video acquisition module is configured to acquire heterogeneous multi-path video streams from various video sources; the video splicing module is configured to splice the heterogeneous multi-path video streams to obtain spliced videos; and the frame buffer device is configured to write the spliced video and drive a display device cascaded with the frame buffer device to display the spliced video.
In some embodiments, the apparatus further comprises: and the multi-path decoder is configured to decode the heterogeneous multi-path video streams respectively according to a hardware synchronous clock, wherein the video splicing module is configured to splice the decoded multi-path video to acquire the spliced video.
In some embodiments, the apparatus further comprises: the display device comprises at least one physical display device and/or at least one virtual display device and is configured to render, post-process, store or forward the stitched video.
In a third aspect, a multimedia broadcast control system is provided. The system comprises: a heterogeneous multi-path video input configured to input heterogeneous multi-path video streams of a plurality of video sources; a demultiplexing component configured to demultiplex the heterogeneous multi-path video using a corresponding demultiplexer; and a fusion cascade device according to any of the preceding claims.
IN some embodiments, the heterogeneous multi-path video input includes at least one of a local video stream input, a video-on-screen video stream input, a DRM video stream input, a network video stream input, an HDMI-IN video stream input, and an IPC video stream input; the multiplexing component comprises at least one of a file demultiplexing module, a screen recording demultiplexing module, a DRM demultiplexing module, a network demultiplexing module, an HDMI-IN demultiplexing module and an IPC demultiplexing module.
In a fourth aspect, an electronic device is provided. The electronic device includes: a memory configured to store a set of instructions; and a processor configured to execute the set of instructions to implement the fusion cascade display method described above.
In some embodiments, the electronic device includes an NVR play device, an online educational set top box, an educational play device, or a meeting play device.
According to the embodiment of the disclosure, the multi-channel video streams of various content sources are spliced and fused and then written into the frame buffer device, and the frame buffer device is used as a virtual screen to drive the display device cascaded with the frame buffer device to display, so that the multi-device to multi-device screen casting can be realized in the multimedia broadcasting control system. Particularly, compared with the traditional hardware cascade, the fusion cascade display mode has more flexible variability, and can be used for performing complex image processing and then cascade so as to be compatible with various types of video input ends; and the heterogeneous video streams are mapped and rendered through the frame buffer device, so that compatibility of various virtual/physical display devices can be realized. In some embodiments, components (such as decoders) with synchronization requirements can be caused to synchronize under the scheduling of synchronization beats, so as to prevent picture jitter and picture tearing, and realize synchronous display. Therefore, the method and the device can obviously improve the flexibility of system configuration and expand the fusion cascade function of the system; furthermore, the video synchronous display of the fusion connection can be realized.
The foregoing summary is merely an overview of the present application, and may be implemented according to the text and the accompanying drawings in order to make it clear to a person skilled in the art that the present application may be implemented, and in order to make the above-mentioned objects and other objects, features and advantages of the present application more easily understood, the following description will be given with reference to the specific embodiments and the accompanying drawings of the present application.
Drawings
The drawings are only for purposes of illustrating the principles, implementations, applications, features, and effects of the present application and are not to be construed as limiting the application.
Fig. 1 is a system configuration diagram showing a fusion cascade device according to an embodiment of the present disclosure;
fig. 2 is a system configuration diagram illustrating a multimedia broadcast control system according to an embodiment of the present disclosure;
fig. 3 is a system configuration diagram illustrating a multimedia broadcast control system according to an embodiment of the present disclosure;
FIG. 4 is a flow diagram illustrating a fusion cascade display method according to an embodiment of the disclosure;
FIG. 5 is a flow diagram illustrating a fusion cascade synchronization display method according to an example of the present disclosure;
FIG. 6 is a method flow diagram illustrating a cascading display of a fusion cascading device according to an example of the present disclosure;
fig. 7 is a block diagram illustrating an electronic device according to an embodiment of the present disclosure.
Detailed Description
In order to describe the possible application scenarios, technical principles, practical embodiments, and the like of the present application in detail, the following description is made with reference to the specific embodiments and the accompanying drawings. The embodiments described herein are only for more clearly illustrating the technical aspects of the present application, and thus are only exemplary and not intended to limit the scope of the present application.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearances of the phrase "in various places in the specification are not necessarily all referring to the same embodiment, nor are they particularly limited to independence or relevance from other embodiments. In principle, in the present application, as long as there is no technical contradiction or conflict, the technical features mentioned in each embodiment may be combined in any manner to form a corresponding implementable technical solution.
Unless defined otherwise, technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application pertains; the use of related terms herein is for the purpose of describing particular embodiments only and is not intended to limit the application.
In the description of the present application, the term "and/or" is a representation for describing a logical relationship between objects, which means that three relationships may exist, for example a and/or B, representing: there are three cases, a, B, and both a and B. In addition, the character "/" herein generally indicates that the front-to-back associated object is an "or" logical relationship.
In the present application, terms such as "first" and "second" are used merely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual number, order, or sequence of such entities or operations.
Without further limitation, the use of the terms "comprising," "including," "having," or other like terms in this specification is intended to cover a non-exclusive inclusion, such that a process, method, or article of manufacture that comprises a list of elements does not include additional elements but may include other elements not expressly listed or inherent to such process, method, or article of manufacture.
In the present application, the expressions "greater than", "less than", "exceeding", etc. are understood to exclude this number; the expressions "above", "below", "within" and the like are understood to include this number. Furthermore, in the description of embodiments of the present application, the meaning of "a plurality of" is two or more (including two), and similarly, the expression "a plurality of" is also to be understood as such, for example, "a plurality of" and the like, unless specifically defined otherwise.
Aiming at the problem that the existing single-device-to-single-device (point-to-point) screen-casting technology cannot be applied to a multimedia broadcasting control system, the application provides a fusion cascade device and a fusion cascade display method based on virtual display, which can be applied to the multimedia broadcasting control system to realize the screen-casting from multiple devices to multiple devices; the flexibility of the system configuration is obviously improved, and the fusion cascade function of the system is expanded; furthermore, the video synchronous display of the fusion connection can be realized.
A detailed implementation of embodiments according to the present disclosure will be described in detail below with reference to exemplary embodiments and in conjunction with the accompanying drawings.
Referring to fig. 1, fig. 1 is a system configuration diagram illustrating a fusion tandem device according to an embodiment of the present disclosure. As shown in fig. 1, the fusion cascade device 100 includes at least a video acquisition module 101, a video stitching module 102, and a frame buffer device 103. The video acquisition module 101, the video stitching module 102 and the frame buffer device 103 are electrically connected in sequence. In particular, the frame buffer device 103 will cascade with the display device 104, and if there are multiple display devices, such as 104a to 104f, the frame buffer device 103 will cascade with the display devices 104a to 104 f.
The video acquisition module 101 is configured to acquire heterogeneous multi-path video streams from a variety of video sources. The heterogeneous multi-path video stream corresponds to heterogeneous multi-path video input equipment, and refers to a set of more than two types of video streams acquired from more than two video input equipment, namely heterogeneous multi-path video streams for short. The heterogeneous multi-path video input device comprises, but is not limited to, an internal memory and an external memory for storing videos, a screen recording device, a Drm video device, a network module, an HDMI-IN device, a UVC video device, an IPC video device and the like.
The video stitching module 102 is configured to stitch the heterogeneous multi-path video streams to obtain stitched video streams.
The frame buffer 103, that is, the frame buffer, may be simply understood as a block of display buffer, and writing data in a specific format into the display buffer means outputting corresponding contents to the display device connected thereto. The frame buffer is configured to write the spliced video and drive a display device cascaded with the frame buffer device to display the spliced video.
The cascade is referred to herein as connecting the frame buffer device with at least one display device to establish a mapping relationship between the frame buffer device to the at least one display device, thereby achieving a fused cascade effect.
It can be understood that the frame buffer device is used as a virtual screen, the input end of the frame buffer device is connected with the video stitching module to realize complex image processing before display, and the output end of the frame buffer device is cascaded with other display devices to realize fusion cascade display, so that the content (mobile phone, set top box, HDMI device, PC, tablet, IPC device and conference device) of various content sources is cascaded and then is pre-stitched (fused) into N frames buffer, and then the N frames buffer is displayed on other devices cascaded with the frame buffer. Therefore, compared with hardware cascade, the fusion cascade of the embodiment can achieve complex image post-processing and render to other display devices, achieves multi-device to multi-device screen projection, and is high in flexibility.
In this embodiment, the display device 104 includes at least one physical display device V0 and/or at least one virtual display device V1, and is configured to render, post-process, store, or forward the stitched video. The cascaded physical display device V0 refers to another fusion cascade device that has established a connection cascade channel. The cascaded virtual display device V1 refers to a virtual component, a virtual display component, or a screen display component, etc., which has established a connection cascade channel. The virtual/physical display device is a device with independent functions, and a user can render, post-process, store or forward the spliced video through the virtual/physical display device. Alternatively, the display device may be a cell phone, a set top box, an HDMI device, a PC, a tablet, an IPC device, a conference device, or the like.
The fusion cascade device of the embodiment can be NVR broadcasting control device, online education set top box, education broadcasting control device, conference broadcasting control device and the like.
The fusion cascade device based on the embodiment can render heterogeneous multipath video streams to the display device cascaded with the heterogeneous multipath video streams through the frame buffer device, and simultaneously supports physical fusion cascade, virtual fusion cascade or virtual-real hybrid fusion cascade, so that the configuration flexibility of the system is obviously improved, and the fusion cascade function of the system is expanded.
In some specific implementations of this embodiment, the video stitching module in the fusion tandem device may be a componentized stitching engine, and the rendering engine may also be componentized. Componentization is the key to software decoupling or component flexibility, i.e., typical tasks are split into fine-grained tasks, and then complex large tasks are completed by assembly of components into large pipeline. The coupling relationship of the rendering engine with the virtual screen, the physical display device, and the virtual display device is a coupling relationship (soft coupling relationship) of the large task-centered organization (profile) component.
In some embodiments of the present embodiment, the fusion cascade device further comprises a multi-way decoder (not shown in fig. 1). The input end of the multi-path decoder is connected with the input end of the heterogeneous multi-path video stream, namely the video acquisition module 101, and the output end of the multi-path decoder is connected with the video splicing module 102. The multi-path decoder is configured to decode the heterogeneous multi-path video streams according to a hardware synchronous clock and then transmit the decoded heterogeneous multi-path video streams to the video splicing module. Correspondingly, the video stitching module is further configured to stitch the decoded multipath videos to obtain the stitched videos.
As a preferred embodiment of the foregoing specific embodiment, all components with synchronization requirements in the fusion cascade device are configured with a hardware synchronization clock, so as to ensure that the components with synchronization requirements can be uniformly scheduled by the system in a proper synchronization beat. In particular, the multi-way decoder is configured with a hardware synchronous clock. Optionally, the rendering component is also configured with a hardware synchronization clock. Accordingly, the multi-path decoder and the rendering component (if configured with a hardware synchronization clock) will start the hardware synchronization clock according to a fusion cascade configuration (explained in detail below), and decode and render heterogeneous multi-path video streams according to a synchronization beat (if configured with a hardware synchronization clock). Therefore, the embodiment can realize the video synchronous display when the multi-input device is projected to the multi-display device.
Referring to fig. 2, fig. 2 is a system configuration diagram illustrating a multimedia broadcast control system 200 according to an embodiment of the present disclosure. This embodiment is further extended based on the embodiment of fig. 1, and provides a multimedia broadcast control system. As shown in fig. 2, the multimedia broadcast control system 200 of the present embodiment includes heterogeneous multi-path video input, i.e., the illustrated multi-path video input device interface 201. The multimedia broadcast control system 200 also includes a demultiplexing component 202 that specifically includes video multiplexers corresponding to the video access device interfaces. The multimedia broadcast control system 200 also includes the fusion cascade device 100 described in the embodiment of fig. 1 and a separate display device 104 (a-f), which may be multiple. The multiplexed video input device interface 201, the demultiplexing component 202, the multiplexer 105 in the fusion cascade device 100, the video splicing module 102 and the frame buffer device 103 are electrically connected in series in sequence, and the frame buffer device 103 is in cascade with the display device 104 (a-f).
The multi-way video input device interface 201 is configured to obtain heterogeneous multi-way video streams from a variety of video sources.
The demultiplexing component 202 is configured to demultiplex the acquired heterogeneous multi-path video using a corresponding demultiplexer.
The multi-path decoder 105 in the fusion tandem device is configured to decode the demultiplexed heterogeneous multi-path video stream using a corresponding decoder.
The video stitching module 102 in the fusion tandem device is configured to stitch the decoded multiple paths of videos to obtain the stitched videos.
The frame buffer device 103 in the fusion tandem device is configured to write the spliced video and then drive a display device in tandem with the spliced video to display the spliced video stream.
As a specific embodiment, referring to fig. 3, the multiple video input devices include at least one of the following video input devices, but may be plural in number: internal and external memory storing video, typically abstracted into local video streams; the mobile phone/PC/BOX and other screen recording equipment is generally abstracted Cheng Lu screen video stream; a Drm video device comprising a Drm video stream; a network module comprising a network video stream; an HDMI-IN device containing an HDMI-IN video stream; an IPC video device contains an IPC device stream. Accordingly, the multiplexing and demultiplexing component comprises a demultiplexer of at least one of: a file demultiplexer corresponding to the local video stream; a screen recording demultiplexer corresponding to the screen recording video stream; a Drm demultiplexer corresponding to the Drm video stream; a network demultiplexer corresponding to the network video stream; an HDMI-IN demultiplexer corresponding to the HDMI-IN video stream; an IPC demultiplexer corresponding to the IPC device stream. Each demultiplexer is connected to a multiplex decoder. The display devices include physical display devices (HDMI/VGA/EPG display device, MIPI display device, etc.) and/or virtual display devices (another fusion cascade device that establishes a cascade). In addition, the display device also supports video after splicing and supports storage or forwarding, and accordingly, only the video storage and network forwarding module is required to be configured.
The software system in the multimedia broadcasting control system processes the heterogeneous multi-path video stream acquired by heterogeneous multi-path video input and mainly comprises: the heterogeneous video input device is abstracted into heterogeneous video input streams, and the video input streams are connected with corresponding video demultiplexers in series; the video demultiplexer is connected in series with the corresponding video decoder; the multi-path video is subjected to video splicing processing, and the spliced video is output to frame buffer equipment to drive display equipment cascaded with the video to display; the display device comprises at least one physical display device and/or at least one virtual display device and is configured to render, post-process, store or forward the stitched video.
The coupling relationship between the fusion cascade device and the virtual screen (i.e., the frame buffer device), the physical display device, and the virtual display device in this embodiment is a coupling relationship that organizes (configures) components with a large task as a center, i.e., a soft coupling relationship.
Referring to fig. 4, fig. 4 is a flowchart illustrating a fusion cascade display method according to an embodiment of the disclosure. The embodiment provides a fusion cascade display method based on the fusion cascade device described in the embodiment of fig. 1. As shown in fig. 4, the method at least includes the following steps S100 to S400.
In step S100, heterogeneous multi-path video streams are acquired from a variety of video sources.
The heterogeneous multi-path video stream corresponds to various types of heterogeneous video input devices, such as a memory, a screen recording device, a Drm video device, a network module, an HDMI-IN device, a UVC video device, an IPC video device and the like.
In step S200, the heterogeneous multi-path video streams are spliced to obtain a spliced video.
After video streams are respectively acquired from heterogeneous multi-channel video input equipment, each frame of the heterogeneous multi-channel video streams is spliced, and spliced video is acquired. It can be simply understood that the frame videos of the heterogeneous video streams corresponding to the same time are spliced together and displayed together in the same picture.
In step S300, the spliced video is written into a frame buffer device.
The frame buffer is mapped to the process space based on the mapping display card function of the frame buffer, and the spliced video stream is written into the frame buffer, so that the written spliced video stream can be directly reflected to the display device cascaded with the frame buffer, and the effect of inputting multiple input devices to multiple display devices is realized.
In step S400, the spliced video is displayed on a display device cascaded with the frame buffer device.
The number and types of the display devices support flexible configuration according to user requirements. Here, the frame buffer is understood to be an image of the video memory, and the read/write operation of the frame buffer will be directly reflected on the display device cascaded therewith. Therefore, in this document, by configuring the types and the number of the display devices cascaded with the frame buffer device, the frame buffer device can be used as a virtual screen, and the frame buffer device is used for mapping and displaying the frame buffer device into a plurality of display devices cascaded with the frame buffer device, so that the screen from the multi-channel video input device to the multi-channel display device is realized.
In this embodiment, the content (mobile phone, set top box, HDMI device, PC, tablet, IPC device, conference device) of the multiple content sources is pre-spliced (fused) into N blocks of frame buffers, and the frame buffers are used as virtual screens and then cascaded with other devices (such as NVR broadcasting devices). Therefore, fusion cascade display is realized, and the screen-throwing display from the multi-video input equipment to the multi-video output equipment is realized, so that the system can be suitable for a multi-media broadcasting control system. In addition, compared with hardware cascade, the fusion cascade of the embodiment can achieve complex image post-processing and cascade display, has higher flexibility and is better compatible with various types of video input ends.
In some specific implementations of this embodiment, the splicing the heterogeneous multi-path video stream to obtain the spliced video, that is, the step S200, specifically includes the following sub-steps: respectively demultiplexing and decoding the heterogeneous multi-path video stream; and splicing the decoded multipath splicing to obtain the spliced video. The multi-channel video streams with different structures are subjected to demultiplexing processing by adopting corresponding demultiplexers, decoding processing by adopting corresponding decoders, and video splicing processing.
Further, the demultiplexing and decoding the heterogeneous multi-path video streams respectively includes: and respectively decoding the heterogeneous multi-path video streams according to the synchronous beats. The introduction of a synchronization beat will enable multi-channel synchronous decoding.
In some specific implementations of this embodiment, the splicing the decoded multiple paths of video to obtain the spliced video includes: and splicing the decoded multi-path video according to fusion cascade configuration information to obtain the spliced video stream, wherein the fusion cascade configuration information comprises information related to the layout of the multi-path video in the splicing and fusion process.
The fusion cascade configuration information refers to the parameter description of the component and the parameter description of the pipeline. A pipeline is made up of components, which are described by a cascading profile, like pipeline. The fusion cascade configuration is used for configuring how to layout (width and position of rendering, when width and height are not matched, a scaling process is implicitly included) and how to render in the process of splicing and fusing the plurality of display content sources.
Therefore, the fusion cascade display mode of the embodiment supports splicing of multiple paths of videos according to the pre-configuration, so that an ideal splicing layout effect is obtained, and the splicing mode is more various and flexible.
Furthermore, the specific embodiment can ensure synchronous display of the video and solve the problems of picture jitter and picture tearing. Specifically, the step S200 of the fusion cascade display method specifically includes the following substeps.
In step S201, clock source synchronization and display synchronization are started according to the fusion cascade configuration information.
Here, for components with synchronization requirements, such as a video decoding component and a video rendering component. The components will be controlled in synchronization by way of issuing a hardware synchronization signal comprising a hardware clock source signal and a display synchronization signal of the display device.
In step S202, the multi-path decoder is driven in a synchronous beat according to the clock source synchronization to decode the heterogeneous multi-path video streams, respectively.
Here, the synchronization beat, i.e. the synchronization signal event, typically drives the video decoder in the manner of an event callback. Too fast decoding by the video decoder can cause the transition to occupy decoder and DDR resources; video decoder decoding non-uniformity can cause video stuck and frame skip. The present embodiment achieves uniform decoding by a mechanism, i.e., a hardware synchronization event is used to drive a video decoder, thereby achieving synchronous display of video pictures.
In step S203, the spliced video is rendered to the frame buffer device as a virtual screen according to the display synchronization.
As an example of this embodiment, referring to fig. 5, a video synchronization process of a fusion tandem device is provided. First, the hardware synchronization mode of a part of the components is started according to the fusion cascade configuration information. The synchronous beats drive the video decoder to decode. And rendering the multiple paths of videos to a virtual screen, namely a frame buffer device according to the fusion cascade configuration. A frame buffer device rendering virtual screen completion event is detected. The physical display device is notified to render immediately. The virtual display device is notified to render immediately.
As another example of this embodiment, referring to fig. 6, a cascade display method of a fusion cascade device is provided. As shown in fig. 6, the fusion cascade device is connected with the scene a rendering device, the fusion cascade device is connected with the scene B rendering device, the fusion cascade device renders the main scene X, the rendering main scene X may use a stitching engine or an image acceleration engine to perform stitching processing on the image, and the fusion cascade device renders the auxiliary scenes a and B. The fusion cascade device outputs the spliced video image, and the connected virtual/physical display device can perform various processes after receiving the video data, including post-processing, display processing, medium storage, network forwarding or cascade forwarding, etc. And the fusion cascade device outputs the fused video image to the device. The roles of the fusion cascade devices are configured by the cascade configuration files, the fusion cascade devices can be cascaded, and the fusion cascade devices are converted into common rendering devices when in slave roles. The scene rendering device belongs to the coupling relation of large-scale task-centered organization (configuration file) components, namely the soft coupling relation, like the coupling relation of the virtual screen, the physical display device and the virtual display device.
Referring to fig. 7, fig. 7 is a block diagram illustrating an electronic device 700 according to an embodiment of the disclosure. This embodiment provides, based on any of the above embodiments, an electronic device 700 including a memory 701 and a processor 702. The memory 701 is configured to store a set of instructions. The processor 702 is configured to execute the set of instructions to implement the fusion cascade display method described in any of the embodiments of fig. 4-6.
In some embodiments, the electronic device includes, but is not limited to: NVR broadcast control equipment, online educational set top boxes, educational broadcast control equipment, and/or meeting broadcast control equipment, etc.
Finally, it should be noted that, although the embodiments have been described in the text and the drawings, the scope of the application is not limited thereby. The technical scheme generated by replacing or modifying the equivalent structure or equivalent flow by utilizing the content recorded in the text and the drawings of the specification based on the essential idea of the application, and the technical scheme of the embodiment directly or indirectly implemented in other related technical fields are included in the patent protection scope of the application.

Claims (15)

1. A fusion cascade display method, comprising:
acquiring heterogeneous multi-path video streams from various video sources;
splicing the heterogeneous multi-path video streams to obtain spliced video;
writing the spliced video into frame buffer equipment; and
and displaying the spliced video on a display device cascaded with the frame buffer device.
2. The fusion cascade display method of claim 1, wherein splicing the heterogeneous multi-path video streams to obtain a spliced video comprises:
respectively demultiplexing and decoding the heterogeneous multi-path video stream; and
and splicing the decoded multipath videos to obtain the spliced videos.
3. The fusion cascade display method of claim 2, wherein demultiplexing and decoding the heterogeneous multi-path video streams respectively comprises:
and respectively decoding the heterogeneous multi-path video streams according to the synchronous beats.
4. The fusion cascade display method of claim 2, wherein stitching the decoded multi-path video to obtain the stitched video comprises:
and splicing the decoded multi-path video according to fusion cascade configuration information to obtain the spliced video stream, wherein the fusion cascade configuration information comprises information related to the layout of the multi-path video in the splicing and fusion process.
5. The fusion cascade display method of claim 4, further comprising:
starting clock source synchronization and display synchronization according to the fusion cascade configuration information;
driving a multi-path decoder according to the clock source synchronization with a synchronous beat so as to decode the heterogeneous multi-path video streams respectively; and
and rendering the spliced video to the frame buffer device serving as a virtual screen according to the display synchronization.
6. The fusion cascade display method of claim 5, wherein displaying the stitched video on a display device in cascade with the frame buffer device comprises:
detecting that rendering of the frame buffer device is complete; and
and notifying the display equipment to render.
7. The fusion cascade display method of claim 1, wherein displaying the stitched video on a display device in cascade with the frame buffer device comprises:
and rendering the spliced video on a physical display device or a virtual display device.
8. The fusion cascade display method of claim 7, wherein rendering the stitched video on a physical display device or a virtual display device comprises:
rendering the spliced video on at least one physical display device and/or at least one virtual display device according to device cascade configuration information, wherein the device cascade configuration information comprises information associated with each device of a cascade of which a total task is divided into a plurality of subtasks.
9. A fusion cascade device, comprising:
the video acquisition module is configured to acquire heterogeneous multi-path video streams from various video sources;
the video splicing module is configured to splice the heterogeneous multi-path video streams to obtain spliced videos;
and the frame buffer device is configured to write the spliced video and drive a display device cascaded with the frame buffer device to display the spliced video.
10. The fusion cascade device of claim 9, further comprising:
a multi-path decoder configured to decode the heterogeneous multi-path video streams respectively according to a hardware synchronization clock,
the video splicing module is configured to splice the decoded multipath videos to obtain the spliced videos.
11. The fusion cascade device of claim 9, further comprising:
the display device comprises at least one physical display device and/or at least one virtual display device and is configured to render, post-process, store or forward the stitched video.
12. A multimedia broadcast control system, comprising:
a heterogeneous multi-path video input configured to input heterogeneous multi-path video streams of a plurality of video sources;
a demultiplexing component configured to demultiplex the heterogeneous multi-path video using a corresponding demultiplexer; and
the fusion cascade device according to any of claims 9 to 11.
13. The multimedia broadcast control system of claim 12, wherein,
the heterogeneous multi-path video input comprises at least one of a local video stream input, a screen video stream input, a DRM video stream input, a network video stream input, an HDMI-IN video stream input and an IPC video stream input;
the multiplexing component comprises at least one of a file demultiplexing module, a screen recording demultiplexing module, a DRM demultiplexing module, a network demultiplexing module, an HDMI-IN demultiplexing module and an IPC demultiplexing module.
14. An electronic device, comprising:
a memory configured to store a set of instructions; and
a processor configured to execute the set of instructions to implement the fusion cascade display method according to any of claims 1 to 8.
15. The electronic device of claim 14, wherein the electronic device comprises an NVR broadcast device, an online educational set top box, an educational broadcast device, or a meeting broadcast device.
CN202310732353.XA 2023-06-20 2023-06-20 Fusion cascade display method and device, multimedia broadcast control system and electronic device Pending CN116801039A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310732353.XA CN116801039A (en) 2023-06-20 2023-06-20 Fusion cascade display method and device, multimedia broadcast control system and electronic device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310732353.XA CN116801039A (en) 2023-06-20 2023-06-20 Fusion cascade display method and device, multimedia broadcast control system and electronic device

Publications (1)

Publication Number Publication Date
CN116801039A true CN116801039A (en) 2023-09-22

Family

ID=88044635

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310732353.XA Pending CN116801039A (en) 2023-06-20 2023-06-20 Fusion cascade display method and device, multimedia broadcast control system and electronic device

Country Status (1)

Country Link
CN (1) CN116801039A (en)

Similar Documents

Publication Publication Date Title
US9992545B2 (en) Method for play synchronization and device using the same
CN100382566C (en) Video data processing apparatus
US7823080B2 (en) Information processing apparatus, screen display method, screen display program, and recording medium having screen display program recorded therein
KR101339535B1 (en) Contents reproduction device and recording medium
RU2604987C2 (en) Electronic device, method for transmitting stereoscopic image information from electronic device, and method for receiving stereoscopic image information by electronic device
JP6500890B2 (en) Transmission apparatus, transmission method, reception apparatus and reception method
TW200531537A (en) Method and apparatus to communicate graphics overlay information
CN102387332A (en) Transmission system
US20080152319A1 (en) Apparatus for processing multimedia stream and method for transmitting multimedia stream
KR101250721B1 (en) Transitioning Between Two High Resolution Video Sources
CN110996168A (en) Video terminal equipment and image synthesis method
CN103647906A (en) Multipath audio and video synthetic method for embedded hard-disk video recorder
KR101513413B1 (en) Android-based Device of providing matrix-type UHD video
CN112312040B (en) Video processor and display system
CN101137057A (en) Image device possible to reverse play alleviating time delay and method thereof
CN116801039A (en) Fusion cascade display method and device, multimedia broadcast control system and electronic device
CN101883228B (en) Equipment and method for reproducing captions
CN102595162A (en) Image generation device, method and program
CN106254928A (en) A kind of picture caption presentation method and device
CN114189730B (en) Video recording method, video playback method and device
US8750672B2 (en) Playback method and apparatus
JP2009159313A (en) Data transmission device, data reception device, control method of data transmission device, data transmission device control program, and computer-readable recording medium recording the program
JP4968946B2 (en) Information processing apparatus, video display apparatus, and program
CN117651186A (en) Display device, video seamless switching method, and storage medium
JP2009044392A (en) Video reproducing method and video reproducing device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination