CN110784758B - Screen projection processing method and device, electronic equipment and computer program medium - Google Patents

Screen projection processing method and device, electronic equipment and computer program medium Download PDF

Info

Publication number
CN110784758B
CN110784758B CN201910779831.6A CN201910779831A CN110784758B CN 110784758 B CN110784758 B CN 110784758B CN 201910779831 A CN201910779831 A CN 201910779831A CN 110784758 B CN110784758 B CN 110784758B
Authority
CN
China
Prior art keywords
video
frame data
screen
projected
video frame
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910779831.6A
Other languages
Chinese (zh)
Other versions
CN110784758A (en
Inventor
王乾
晏家红
孙炜
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to CN201910779831.6A priority Critical patent/CN110784758B/en
Publication of CN110784758A publication Critical patent/CN110784758A/en
Application granted granted Critical
Publication of CN110784758B publication Critical patent/CN110784758B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
    • H04N21/4341—Demultiplexing of audio and video streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/4104—Peripherals receiving signals from specially adapted client devices
    • H04N21/4122—Peripherals receiving signals from specially adapted client devices additional display device, e.g. video projector
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/436—Interfacing a local distribution network, e.g. communicating with another STB or one or more peripheral devices inside the home
    • H04N21/43615—Interfacing a Home Network, e.g. for connecting the client to a plurality of peripherals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/439—Processing of audio elementary streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44016—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving splicing one content stream with another content stream, e.g. for substituting a video clip
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Controls And Circuits For Display Device (AREA)

Abstract

The application provides a screen projection processing method and device. The screen projection processing method comprises the steps of responding to a screen projection request through screen projection equipment, and acquiring a transmission protocol type of a video file to be projected; if the transmission protocol type of the video file to be screen-projected is not matched with the transmission protocol type of the video screen-projected, converting the video file to be screen-projected according to the transmission protocol type of the video screen-projected to obtain a converted video stream; and sending the converted video stream to the screen projection equipment so that the screen projection equipment performs screen projection display based on the converted video stream. The technical scheme provided by the application effectively solves the problem that the screen projection of the video stream data acquired by the transmission protocol of the video screen projection specified in the non-DLNA protocol cannot be performed through the screen projection equipment.

Description

Screen projection processing method and device, electronic equipment and computer program medium
Technical Field
The application relates to the technical field of screen projection, in particular to a screen projection processing method and device.
Background
The terminal device can project the Network television box in the local area Network based on a protocol (DLNA) specified by the Digital Living Network Alliance. Because a transmission Protocol adopting video screen projection is specified to be a hypertext Transfer Protocol (HTTP) in the current DLNA Protocol, and video stream data of live broadcast or recorded broadcast on a network, which is required to be screen-projected by a terminal device, is acquired by using other protocols other than the HTTP Protocol, the terminal device cannot project the video stream data acquired by using the transmission Protocol adopting video screen projection specified in the non-DLNA Protocol through the screen projection device.
Disclosure of Invention
One objective of the present application is to solve the problem that video stream data obtained by using a video screen-casting transmission protocol specified in a non-DLNA protocol cannot be subjected to screen casting by a screen casting device.
According to an aspect of the embodiments of the present application, there is provided a screen projection processing method, including: responding to a screen projection request through screen projection equipment, and acquiring a transmission protocol type of a video file to be projected; if the transmission protocol type of the video file to be screen-projected is not matched with the transmission protocol type of the video screen-projected, converting the video file to be screen-projected according to the transmission protocol type of the video screen-projected to obtain a converted video stream; and sending the converted video stream to the screen projection equipment so that the screen projection equipment performs screen projection display based on the converted video stream.
According to an aspect of the embodiments of the present application, there is provided a screen projection processing apparatus, including: the device comprises an acquisition unit, a display unit and a display unit, wherein the acquisition unit is used for responding to a screen projection request of screen projection equipment and acquiring the transmission protocol type of a video file to be projected; the first conversion unit is used for converting the video file to be screen-projected according to the transmission protocol type of the video screen-projected if the transmission protocol type of the video file to be screen-projected is not matched with the transmission protocol type of the video screen-projected, so as to obtain a converted video stream; and the sending unit is used for sending the converted video stream to the screen projection equipment so that the screen projection equipment performs screen projection display based on the converted video stream.
In an embodiment of the application, based on the foregoing scheme, the first conversion unit is configured to demultiplex the video file to be screened to obtain demultiplexed video frame data and demultiplexed audio data; decoding the demultiplexed video frame data to obtain decoded video frame data, and decoding the demultiplexed audio data to obtain decoded audio data; respectively encoding the decoded video frame data and the decoded audio data according to the transmission protocol type of the video screen projection to obtain first video frame data and first audio data; and multiplexing the first video frame data and the first audio data according to the transmission protocol type of the video screen projection to obtain a converted video stream.
In an embodiment of the application, based on the foregoing scheme, if the video file to be screened includes at least two paths of video files, the first conversion unit is configured to perform demultiplexing on the at least two paths of video files respectively to obtain at least two paths of demultiplexed video frame data and at least two paths of demultiplexed audio data; decoding the at least two paths of demultiplexed video frame data to obtain at least two paths of decoded video frame data, and decoding the at least two paths of demultiplexed audio data to obtain at least two paths of decoded audio data; splicing the at least two paths of decoded video frame data to obtain second video frame data, and fusing the at least two paths of decoded audio data to obtain second audio data; according to the transmission protocol type of the video screen projection, encoding the second video frame data to obtain third video frame data, and encoding the second audio data to obtain third audio data; and multiplexing the third video frame data and the third audio data according to the transmission protocol type of the video screen projection to obtain a converted video stream.
In an embodiment of the application, based on the foregoing scheme, the first conversion unit is configured to perform splicing processing on video frames at the same time in the at least two paths of decoded video frame data according to a time sequence order, so as to obtain the second video frame data.
In an embodiment of the application, based on the foregoing scheme, the first conversion unit is configured to perform fusion processing on the at least two paths of decoded audio data according to a time sequence order to obtain the second audio data.
In an embodiment of the application, based on the foregoing scheme, the screen-casting processing device further includes a second conversion unit, configured to convert the text file into picture frame data to be screen-cast if the text file to be screen-cast and displayed is acquired, and convert the picture frame data to be screen-cast and the video file to be screen-cast according to a transmission protocol type of the video screen-casting to obtain a converted video stream.
In an embodiment of the application, based on the foregoing scheme, the second conversion unit is configured to convert the text file into at least one frame of picture frame data as the picture frame data to be screened.
In an embodiment of the application, based on the foregoing scheme, the second conversion unit is configured to demultiplex the video file to be screened to obtain demultiplexed video frame data and demultiplexed audio data; decoding the demultiplexed video frame data to obtain decoded video frame data, and decoding the demultiplexed audio data to obtain decoded audio data; splicing the picture frame data to be projected and the decoded video frame data to obtain fourth video frame data; according to the transmission protocol type of the video screen projection, encoding the fourth video frame data to obtain fifth video frame data, and encoding the decoded audio data to obtain first audio data; and multiplexing the fifth video frame data and the first audio data according to the transmission protocol type of the video screen projection to obtain a converted video stream.
In an embodiment of the application, based on the foregoing scheme, the second conversion unit is configured to, if it is detected that the picture format of the picture frame data to be screen-projected does not match the picture format of the decoded video frame data, convert the picture frame data to be screen-projected according to the picture format of the decoded video frame data to obtain converted picture frame data; and splicing the converted picture frame data and the decoded video frame data to obtain fourth video frame data.
According to an aspect of an embodiment of the present application, there is provided an electronic device including:
a memory storing computer readable instructions;
a processor reading computer readable instructions stored by the memory to perform the method as described above.
According to an aspect of embodiments of the present application, there is provided a computer program medium having stored thereon computer readable instructions which, when executed by a processor of a computer, cause the computer to perform the method as described above.
In the embodiment of the application, when the screen is projected through the screen projection equipment, the transmission protocol type of the video file to be projected can be obtained, the transmission protocol type of the video file to be projected is matched with the transmission protocol type of the video screen, if the transmission protocol type of the video file to be projected is not matched with the transmission protocol type of the video screen, the video file to be projected is converted according to the transmission protocol type of the video screen, a converted video stream is obtained, video stream data capable of being transmitted through the transmission protocol type of the video screen projection specified by the DLNA protocol is obtained, the converted video stream is sent to the screen projection equipment, so that the screen projection equipment performs screen projection display based on the converted video stream, and further, the terminal equipment can project the video stream data obtained by the transmission protocol of the video screen projection not specified in the DLNA protocol, the applicability of screen projection display of the video stream data by adopting the DLNA protocol is improved.
Other features and advantages of the present application will be apparent from the following detailed description, or may be learned by practice of the application.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
The above and other objects, features and advantages of the present application will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings.
Fig. 1 is a schematic diagram illustrating a system architecture to which a screen projection processing method according to an exemplary embodiment of the present application is applied.
Fig. 2 shows a flowchart of a screen projection processing method according to an example embodiment of the present application.
Fig. 3 shows a detailed flowchart of step S220 of the screen projection processing method according to an example embodiment of the present application.
Fig. 4 shows a detailed flowchart of step S220 of the screen projection processing method according to an example embodiment of the present application.
Fig. 5 shows a flowchart of a screen projection processing method according to another example embodiment of the present application.
Fig. 6 shows a detailed flowchart of step S221 of the screen projection processing method according to another exemplary embodiment of the present application.
Fig. 7 shows a flowchart of a screen projection processing method according to another example embodiment of the present application.
Fig. 8 shows a detailed flowchart of step S530 of a screen projection processing method according to another example embodiment of the present application.
Fig. 9 shows a data interaction diagram for projecting a single non-HTTP video stream data according to an example embodiment of the present application.
Fig. 10 shows a data interaction diagram for projecting two non-HTTP video streams according to an example embodiment of the present application.
Fig. 11 is a block diagram of a projection processing apparatus according to an example embodiment of the present application.
Fig. 12 illustrates a hardware block diagram of an electronic device according to an example embodiment of the present application.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The drawings are merely schematic illustrations of the present application and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repetitive description will be omitted.
Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more example embodiments. In the following description, numerous specific details are provided to give a thorough understanding of example embodiments of the present application. One skilled in the relevant art will recognize, however, that the subject matter of the present application can be practiced without one or more of the specific details, or with other methods, components, steps, and so forth.
Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.
Fig. 1 shows a schematic diagram of a system architecture applied according to an embodiment of the present application. The system architecture includes a terminal device 101, a screen projection device 102 connected to the terminal device 101, and a server 103 connected to the terminal device 101.
The terminal device 101 may be one of a smart phone, a tablet computer, and a portable computer, and the user terminal 101 may perform screen projection control on the screen projection device 102 in the same lan as the terminal device 101 based on a protocol (DLNA) specified by the Digital home Network Alliance. The screen projecting device 102 may be an electronic display device that outputs an image, for example, a device such as a smart television, and the screen projecting device 102 may further include a network television box connected to the smart television, where data interaction is performed between the terminal device 101 and the network television box, so as to implement screen projecting control on the smart television, and the terminal device 101 may specifically perform data interaction with the network television box through the installed screen projecting application.
The server 103 has a software-oriented server and a hardware-oriented server. From the viewpoint of hardware, the server 103 is a physically present server, and may be a server constituted by one physical computer or a cluster server constituted by a plurality of physical computers. From a software perspective, the server 103 refers to computer software with server-side functions and running server-side software. It may be a virtual server that occupies only a part of one computer, or may be a virtual cluster server that occupies a part or all of a plurality of computers, respectively. Particularly in a cloud environment, the application of the virtual cluster server is common. Thus, the overall system architecture can be viewed as being constructed from numerous nodes and connecting channels. In terms of "hardware," the architecture is built from a myriad of hardware servers and other digital computing device terminals (e.g., terminal device 101, etc.) and intermediate connection devices (e.g., network wires, routers, etc.). In terms of "soft," the framework is constructed from numerous running server-side software and client-side software (e.g., applications on terminal device 101) and their interconnections. The screen-casting application on the terminal device 101 receives the video file to be screen-cast sent by the server 103, and obtains the transmission protocol type of the video file to be screen-cast according to the video file to be screen-cast, that is, obtains the transmission protocol type of the video file to be screen-cast transmitted between the terminal device 101 and the server 103. The terminal device 101 matches the transmission protocol type of the video file for screen projection with the transmission protocol type of the video screen projection, if the transmission protocol type of the video file for screen projection is not matched with the transmission protocol type of the video screen projection, it is determined that the protocol format of the video file for screen projection is not the transmission protocol type of the video screen projection specified by the current DLNA protocol, the terminal device 101 converts the video file for screen projection according to the protocol format of the video screen projection, and the converted video stream is obtained and sent to the screen projection device 102, so that the screen projection device 102 performs screen projection display based on the converted video stream.
Therefore, according to the technical scheme of the embodiment of the application, even when the protocol format of the video file to be screen-projected, which is acquired by the terminal device 101, is not matched with the protocol format of the screen-projection transmission, the terminal device 101 can convert the video file to be screen-projected according to the protocol format of the screen-projection transmission to obtain the converted video stream, so that the screen-projection device 102 performs screen-projection display based on the converted video stream, and further the terminal device 101 can project the video stream data, which is acquired by using the transmission protocol of the video screen-projection specified in the non-DLNA protocol, and the applicability of the DLNA protocol to performing screen-projection display on the video stream data is improved.
It should be noted that the screen projection processing method provided by the embodiment of the present invention is generally executed by the terminal device 101, and accordingly, the screen projection processing apparatus is generally disposed in the terminal device 101.
Some of the solutions of the embodiments of the present application can be embodied based on a system architecture as shown in fig. 1 or a system architecture that is a variation thereof.
Referring to fig. 2, fig. 2 shows a flowchart of a screen projection processing method according to an example embodiment of the present application, the screen projection processing method being executed by a terminal apparatus 101 shown in fig. 1, the screen projection control process including:
step S210, in response to a request for screen projection by the screen projection device, obtaining a transmission protocol type of the video file to be screen projected.
Step S220, if the transmission protocol type of the video file to be screen-projected is not matched with the transmission protocol type of the video screen-projected, converting the video file to be screen-projected according to the transmission protocol type of the video screen-projected to obtain a converted video stream.
Step S230, sending the converted video stream to the screen projection device, so that the screen projection device performs screen projection display based on the converted video stream.
These steps are described in detail below.
In step S210, the request for screen projection by the screen projection device refers to a request initiated by a user operating the terminal device to display a screen projection of the video file by the screen projection device, where the request may be triggered by the user operating a virtual button provided in the terminal device for video screen projection. It should be noted that when the terminal device needs to project a screen through the screen projecting device, the terminal device needs to be connected with the screen projecting device first, and if the terminal device can be connected with the screen projecting device in the same local area network, the terminal device can control the screen projecting device to project a screen of a video file to be projected.
The terminal device performs data interaction with the screen projection device specifically through multiple protocols of a Digital Living Network Alliance (DLNA), wherein the transmission Protocol type of the video screen projection specified by the DLNA Protocol is a hypertext Transfer Protocol (HTTP), and when the terminal device controls the screen projection device to project a video file, the terminal device transmits the video file to be projected to the screen projection device through the HTTP.
The video file to be projected is a video file played in the terminal device, and the video file may be a locally stored video file or a video file originated from other devices other than the terminal device, such as a live video or a recorded video originated from a remote streaming media server. The transmission Protocol type of the video file to be screened refers to a transmission Protocol type adopted when the terminal device obtains the video file to be screened, for example, a HTTP Protocol, a Real Time Messaging Protocol (RTMP) Protocol, or a hypertext Transfer Protocol over Secure Socket Layer (HTTPs) Protocol. It should be noted that, the network live broadcast video generally adopts the RTMP protocol, and the network recorded broadcast video generally adopts the HTTPS protocol.
In an embodiment of the application, when a video file to be subjected to screen projection is a live webcast video or a recorded webcast video from a remote streaming media server, a terminal device sends a request for acquiring a play address of the live webcast video or the recorded webcast video to be subjected to screen projection to the remote streaming media server, and the remote streaming media server feeds the remote play address of the live webcast video or the recorded webcast video back to the terminal device according to the request, so that the terminal device can acquire the corresponding live webcast video or the recorded webcast video based on the remote play address of the live webcast video or the recorded webcast video.
When determining the transmission protocol type of the video file to be screened, the terminal device may determine the transmission protocol type of the video file to be screened according to the corresponding attribute information of the video file to be screened, and of course, may also determine the transmission protocol type of the video file to be screened according to the corresponding relationship between the source server type of the received video file to be screened and the transmission protocol type of the video file to be screened, which is not limited herein.
In step S220, when the terminal device projects the video file to be projected through the screen projecting device, the terminal device needs to compare the transmission protocol type of the video file to be projected with the transmission protocol type of the video screen projecting, so as to determine whether the transmission protocol type of the video file to be projected is matched with the transmission protocol type of the video screen projecting.
If the transmission protocol type of the video file to be screen-cast is not matched with the transmission protocol type of the video screen-cast, it is indicated that the video file to be screen-cast is video stream data obtained by adopting a video screen-cast transmission protocol specified in a non-DLNA protocol, the terminal device converts the video file to be screen-cast according to the video screen-cast transmission protocol type to obtain a converted video stream, and the converted video stream is the video stream data which can be transmitted by the video screen-cast transmission protocol type. It can be understood that if the transmission protocol type of the video file to be screen-projected is consistent with the transmission protocol type of the video screen-projected, the video file to be screen-projected does not need to be converted according to the transmission protocol type of the video screen-projected, and the video file to be screen-projected can be directly transmitted to the screen-projecting device through the transmission protocol type of the video screen-projected, so that the video file to be screen-projected can be conveniently screen-projected and displayed through the screen-projecting device.
Referring to fig. 3, fig. 3 shows a detailed flowchart of step S220 of the screen projection processing method in an exemplary embodiment of the present application, and in this embodiment, step S220 may include the following steps:
and step S310, carrying out demultiplexing processing on the video file to be projected to obtain demultiplexed video frame data and demultiplexed audio data.
Step S320, decoding the demultiplexed video frame data to obtain decoded video frame data, and decoding the demultiplexed audio data to obtain decoded audio data.
And step S330, respectively encoding the decoded video frame data and the decoded audio data according to the protocol format of the screen projection transmission to obtain first video frame data and first audio data.
Step S340, multiplexing the first video frame data and the first audio data according to the protocol format of the screen projection transmission to obtain a converted video stream.
These steps are described in detail below.
In step S310, when the terminal device converts the video file to be screened according to the transmission protocol type of the video screen defined by the DLNA protocol, the terminal device needs to demultiplex the video file to be screened to obtain demultiplexed video frame data and demultiplexed audio data, so as to achieve the purpose of separating the video frame data and the audio data in the video file to be screened, so as to recode the video file to be screened.
In step S320, in order to facilitate re-encoding of the video frame data and the audio data, the terminal device further needs to decode the demultiplexed video frame data to obtain decoded video frame data, and decode the demultiplexed audio data to obtain decoded audio data, so as to obtain video frame data that can be re-encoded and audio data that can be re-encoded, so as to re-encode the decoded video frame data and the decoded audio data according to the transmission protocol type of the video screen.
In step S330, the terminal device encodes the decoded video frame data according to the transmission protocol type of video screen projection specified by the DLNA protocol to obtain first video frame data, where the first video frame data is the video frame data obtained by encoding the decoded video frame data according to the transmission protocol type of video screen projection specified by the DLNA protocol; and the terminal equipment also encodes the decoded audio data according to the transmission protocol type of the video screen projection specified by the DLNA protocol to obtain first audio data, wherein the first video frame data is the audio data obtained by encoding the decoded audio data according to the transmission protocol type of the video screen projection specified by the DLNA protocol.
In step S340, the terminal device multiplexes the first video frame data and the first audio data according to the transmission protocol type of video screen projection specified by the DLNA protocol, so as to implement encapsulation of the first video frame data and the first audio data to obtain a converted video stream, where the converted video stream is a video file that can be transmitted according to the transmission protocol type of video screen projection specified by the DLNA protocol.
Referring to fig. 4, fig. 4 is a detailed flowchart illustrating step S220 of a screen projection processing method in an exemplary embodiment of the present application, and in this embodiment, if the video file to be screen projected includes at least two video files, step S220 may include the following steps:
step S410, the at least two paths of video files are respectively subjected to demultiplexing processing to obtain at least two paths of demultiplexed video frame data and at least two paths of demultiplexed audio data;
step S420, decoding the at least two demultiplexed video frame data to obtain at least two decoded video frame data, and decoding the at least two demultiplexed audio data to obtain at least two decoded audio data;
step S430, splicing the at least two paths of decoded video frame data to obtain second video frame data, and fusing the at least two paths of decoded audio data to obtain second audio data;
step S440, according to the protocol format of the screen projection transmission, encoding the second video frame data to obtain third video frame data, and encoding the second audio data to obtain third audio data;
step S450, multiplexing the third video frame data and the third audio data according to the protocol format of the screen projection transmission to obtain a converted video stream.
These steps are described in detail below.
In step S410, since the current DLNA protocol only supports screen projection of one video file, if the video file to be screen-projected includes at least two video files, the at least two video files need to be spliced to obtain one video file, so as to implement screen projection display of the at least two video files. Specifically, the terminal device needs to perform demultiplexing on at least two paths of video files respectively to obtain at least two paths of demultiplexed video frame data and at least two paths of demultiplexed audio data, and then separates the video frame data and the audio data in the at least two paths of video files, so as to implement splicing processing on the at least two paths of video files.
In step S420, in order to implement the splicing processing on the at least two paths of video files, it is further required to decode the at least two paths of demultiplexed video frame data to obtain at least two paths of decoded video frame data, and decode the at least two paths of demultiplexed audio data to obtain at least two paths of decoded audio data, where the decoded video frame data is generally video frame data in a YUV format.
In step S430, the terminal device performs a splicing process on the at least two decoded video frame data to obtain second video frame data, where the second video frame data is video frame data obtained by splicing the at least two decoded video frame data; the terminal equipment further performs fusion processing on the at least two paths of decoded audio data to obtain second audio data, and the second audio data is the audio data obtained by performing fusion processing on the at least two paths of decoded audio data.
In an embodiment of the present application, the splicing the at least two decoded video frame data to obtain a second video frame data includes:
and splicing the video frames at the same time in the at least two paths of decoded video frame data according to a time sequence to obtain second video frame data.
In an embodiment of the present application, when the terminal device performs the splicing processing on the at least two decoded video frame data, the terminal device performs the splicing processing on the video frames at the same time in the at least two decoded video frame data according to the time sequence order of the video frames in the decoded video frame data to obtain the spliced multi-frame video frame data, which is used as the second video frame data.
In an embodiment of the application, the fusing the at least two decoded audio data to obtain second audio data includes:
and performing fusion processing on the at least two paths of decoded audio data according to a time sequence order to obtain the second audio data.
In an embodiment of the application, when the terminal device performs fusion processing on the at least two decoded audio data, the at least two decoded audio data are subjected to fusion processing according to a time sequence order, and the fused audio data are obtained as second audio data, wherein audios at the same time in the at least two decoded audio data are fused.
In step S440, since the transmission protocol types of the at least two paths of video files are not matched with the transmission protocol type of the video screen projection, in order to enable the screen projection device to perform screen projection display on the at least two paths of video files, for the second video frame data, the terminal device further performs encoding processing on the second video frame data according to the transmission protocol type of the video screen projection specified by the DLNA protocol to obtain third video frame data; and for the second audio data, the terminal equipment further encodes the second audio data according to the transmission protocol type of the video screen projection specified by the DLNA protocol to obtain third audio data.
In step S450, the terminal device performs multiplexing processing on the third video frame data and the third audio data according to the transmission protocol type of video screen projection specified by the DLNA protocol, so as to implement encapsulation of the third video frame data and the third audio data to obtain a converted video stream, where the converted video stream is a video stream that can be transmitted according to the transmission protocol type of video screen projection specified by the DLNA protocol.
The technical solution provided in this embodiment enables, when a video file to be subjected to screen projection is a transmission protocol that adopts a video screen projection specified in a non-DLNA protocol and the video file to be subjected to screen projection includes at least two video files, to obtain spliced video frame data by splicing data of video parts in the at least two video files, and to fuse data of audio parts in the at least two video files to obtain fused audio data, and to convert the spliced video frame data and the fused audio data by using a transmission protocol type of the video screen projection specified in the DLNA protocol to obtain a converted video stream, thereby generating video stream data that can be transmitted by using the transmission protocol type of the video screen projection specified in the DLNA protocol and one video stream data that conforms to the specification of the DLNA protocol, and further enabling a terminal device to obtain multiple channels of video stream data obtained by using the transmission protocol of the video screen projection specified in the non-DLNA protocol via a screen projection device The screen projection of the video streaming data improves the applicability of screen projection display of the video streaming data by adopting a DLNA protocol. Compared with the prior art, under the condition that the multi-channel videos need to be displayed on a screen, a mode of displaying only one of the multi-channel videos or a mode of simply stacking the multi-channel videos is adopted, the method for splicing the multi-channel videos provided by the embodiment can ensure that the multi-channel videos needing to be displayed on the screen can be displayed on the screen completely, the loss of video content cannot be caused, and the effect of displaying the multi-channel videos on the screen through a DLNA protocol is improved.
Still referring to fig. 2, in step S230, the terminal device sends the converted video stream to the screen projection device through the transmission protocol type of the video screen projection specified by the DLNA protocol, so that the screen projection device performs screen projection display based on the converted video stream.
In an embodiment of the application, after obtaining the converted video stream, the terminal device sends a local streaming media playing address of the converted video stream to the screen projection device. The screen projection equipment sends a data acquisition request to the terminal equipment according to the local streaming media playing address of the converted video stream, and then requests the converted video stream to the terminal equipment, and after receiving the data acquisition request of the screen projection equipment, the terminal equipment sends the converted video stream to the screen projection equipment, so that the screen projection equipment performs screen projection display based on the converted video stream.
It can be seen from the above that, when the terminal device performs screen projection through the screen projection device, the terminal device obtains the transmission protocol type of the video file to be projected, matches the transmission protocol type of the video file to be projected with the transmission protocol type of the video screen projection, if the transmission protocol type of the video file to be projected does not match with the transmission protocol type of the video screen projection, converts the video file to be projected according to the transmission protocol type of the video screen projection to obtain a converted video stream, obtains video stream data that can be transmitted through the transmission protocol type of the video screen projection specified by the DLNA protocol, and sends the converted video stream to the screen projection device, so that the screen projection device performs screen projection display based on the converted video stream, and further the terminal device can project the video stream data obtained by using the transmission protocol of the video screen projection not specified by the DLNA protocol, the applicability of screen projection display of the video stream data by adopting the DLNA protocol is improved.
Referring to fig. 5, fig. 5 shows a flowchart of a screen projection processing method according to an exemplary embodiment of the present application, and in this embodiment, the screen projection processing method further includes:
step S221, if a text file needing screen projection display is obtained, converting the text file into picture frame data of a screen to be projected, and converting the picture frame data of the screen to be projected and the video file of the screen to be projected according to the transmission protocol type of the video screen to obtain a converted video stream.
In step S221, if the transmission protocol type of the video file to be screen-projected does not match the transmission protocol type of the video screen-projected specified by the DLNA protocol, and the terminal device further obtains the text file to be screen-projected and displayed, because the text file cannot be directly displayed in the video, the terminal device needs to convert the text file into the picture frame data to be screen-projected, so as to convert the picture frame data to be screen-projected and the video file to be screen-projected according to the transmission protocol type of the video screen-projected specified by the DLNA protocol, and obtain the converted video stream.
In an embodiment of the present application, the converting the text file into picture frame data to be screened includes:
and converting the text file into at least one frame of picture frame data to be used as the picture frame data to be projected.
When the terminal equipment converts the text file into the picture frame data to be screen-projected, a canvas with a preset size is established first, and the content in the text file is drawn into the established drawing to convert the text file into at least one frame of picture frame data, wherein the picture frame data is the picture frame data to be screen-projected, the number of the picture frames can be determined according to the content size of the text file, in other words, the number of the picture frames and the content size of the text file have a corresponding relationship.
Referring to fig. 6, fig. 6 is a detailed flowchart illustrating step S221 of a screen projection processing method in an exemplary embodiment of the present application, where in this embodiment, if a text file that needs to be displayed on a screen projection is acquired, the text file is converted into picture frame data to be projected on the screen, and the step S221 of converting the picture frame data to be projected on the screen and the video file to be projected on the screen according to a transmission protocol type of the video screen projection to obtain a converted video stream includes:
step S2211, if a request for screen-casting display of the text file is detected, determining whether the text file required to be screen-cast and displayed is acquired;
step S2212, if a text file needing screen projection display is obtained, converting the text file into picture frame data of a screen to be projected, and converting the picture frame data of the screen to be projected and the video file of the screen to be projected according to the transmission protocol type of the video screen to obtain a converted video stream.
In step S2211, the request for screen-casting and displaying the text file is a request initiated by the user operating the terminal device to screen-cast and display the text file through the screen-casting device, and the request may be triggered by the user operating a virtual button provided in the terminal device to screen-cast the text file as well. In a screen-casting scene of a video file of live network broadcast or recorded network broadcast, the text file can be a text file of a chat area contained in the live network broadcast or the recorded network broadcast. The user can be enabled to control the screen projection display of the text file according to the requirement in a self-defined mode through the request.
In step S2212, when detecting the request for screen-projecting display of the text file and acquiring the text file to be screen-projected and displayed, the terminal device converts the text file into picture frame data to be screen-projected, and converts the picture frame data to be screen-projected and the video file to be screen-projected according to the transmission protocol type of video screen projection to obtain a converted video stream, so as to implement screen-projection display of the content of the text file and the content of the video file together.
Referring to fig. 7, fig. 7 is a flowchart illustrating a screen projection processing method in an example embodiment of the present application, and in this embodiment, the converting, according to a protocol format of the screen projection transmission, the picture frame data to be screen projected and the video file to be screen projected to obtain a converted video stream includes:
step S510, performing demultiplexing on the video file to be screened to obtain demultiplexed video frame data and demultiplexed audio data.
Step S520, decoding the demultiplexed video frame data to obtain decoded video frame data, and decoding the demultiplexed audio data to obtain decoded audio data.
Step S530, splicing the picture frame data to be projected and the decoded video frame data to obtain fourth video frame data.
And step S540, according to the transmission protocol type of the video screen projection, encoding the fourth video frame data to obtain fifth video frame data, and encoding the decoded audio data to obtain first audio data.
And step S550, multiplexing the fifth video frame data and the first audio data according to the transmission protocol type of the video screen projection to obtain a converted video stream.
These steps are described in detail below.
In this embodiment, please refer to the contents of step S310 and step S320 in the foregoing embodiment for the contents of step S510 and step S520, which are not repeated herein.
In step S530, in order to implement the screen-projection display on the text file, the picture frame data to be screen-projected and the decoded video frame data may be spliced to obtain fourth video frame data, where the fourth video frame data is used as the video frame data obtained by splicing the picture frame data corresponding to the text file and the video frame data corresponding to the video file.
Referring to fig. 8, fig. 8 is a detailed flowchart illustrating step S530 of a screen projection processing method in an exemplary embodiment of the present application, where in this embodiment, the step S530 of performing a splicing process on the to-be-projected-screen picture frame data and the decoded video frame data to obtain fourth video frame data includes:
step S5301, if it is detected that the picture format of the picture frame data to be screen-projected does not match the picture format of the decoded video frame data, converting the picture frame data to be screen-projected according to the picture format of the decoded video frame data to obtain converted picture frame data;
step S5302, performing a splicing process on the converted picture frame data and the decoded video frame data to obtain fourth video frame data.
In step S5301, when the terminal device splices the picture frame data to be screen-projected and the decoded video frame data to obtain fourth video frame data, it needs to detect whether the picture format of the picture frame data to be screen-projected matches the picture format of the decoded video frame data, that is, when it needs to ensure that the picture format of the picture frame data to be screen-projected and the picture format of the decoded video frame data are consistent, the picture frame data and the decoded video frame data can be spliced. And if the picture format of the picture frame data to be projected is detected not to be matched with the picture format of the decoded video frame data, converting the picture frame data to be projected according to the picture format of the decoded video frame data to obtain converted picture frame data consistent with the picture format of the picture frame data to be projected so as to be spliced according to the converted picture frame data and the decoded video frame data.
In step S5302, the terminal device performs a splicing process on the converted picture frame data and the decoded video frame data to obtain fourth video frame data.
Still referring to fig. 7, in step S540, in order to enable the screen projection device to perform screen projection display on the fourth video frame data obtained through the splicing processing, for the fourth video frame data obtained through the splicing processing, the terminal device performs encoding processing on the fourth video frame data according to the transmission protocol type of the video screen projection specified in the DLNA protocol to obtain fifth video frame data, where the fifth video frame data is the video frame data obtained by encoding the fourth video frame data according to the transmission protocol type of the video screen projection specified in the DLNA protocol; and for the decoded audio data, the terminal equipment further encodes the decoded audio data according to the video screen-casting transmission protocol type specified in the DLNA protocol to obtain first audio data, wherein the first video frame data is the audio data obtained by encoding the decoded audio data according to the video screen-casting transmission protocol type specified in the DLNA protocol.
In step S550, the terminal device multiplexes the fifth video frame data and the first audio data according to the transmission protocol type of video screen projection specified by the DLNA protocol, so as to implement encapsulation of the fifth video frame data and the first audio data to obtain a converted video stream, where the converted video stream is a video stream that can be transmitted according to the transmission protocol type of video screen projection specified by the DLNA protocol.
The technical solution provided in this embodiment enables, when a text file needs to be screen-projected and displayed, the text file to be converted into picture frame data to be screen-projected, and the data of a video portion in the video file and the picture frame data to be screen-projected are spliced to obtain a splicing result, according to the type of transmission protocol of the video screen, the splicing result is encoded to obtain an encoding result of the video portion, and the decoded audio data is encoded to obtain an encoding result of the audio portion, according to the type of transmission protocol of the video screen, the encoding result of the video portion and the encoding result of the audio portion are multiplexed to obtain a converted video stream, and the converted video stream is sent to the screen projection device, so that the screen projection device performs screen projection and display based on the converted video stream, thereby achieving the purpose of increasing screen projection and display of the text file while performing screen projection, the applicability of screen projection display of the video stream data by adopting the DLNA protocol is improved.
Referring to fig. 9, fig. 9 shows a data interaction diagram for screen casting of a single non-HTTP video stream data according to an example embodiment of the present application.
As shown in fig. 9, the local streaming media module may be an application installed in the terminal device for playing a video stream, and the remote streaming media server may be a server storing a non-HTTP video stream data source. If the non-HTTP video stream data to be projected is not converted according to the transmission protocol type of the video projection screen to obtain the converted video stream, the terminal equipment cannot display the non-HTTP video stream data on the projection screen. And after the non-HTTP video file to be projected is converted according to the type of the transmission protocol of the video projection screen to obtain the converted video stream, the terminal equipment can project the video stream data acquired by the transmission protocol which is not the video projection screen specified in the DLNA protocol.
Specifically, a schematic diagram of data interaction for projecting a single-channel non-HTTP video stream data according to an exemplary embodiment of the present application is shown in fig. 9, and may include the following steps:
step S601, the terminal device sends a request for obtaining a play address to the remote streaming server. The request is used for acquiring a playing address of the single-channel non-HTTP video stream data to be projected from the remote streaming media server.
Step S602, the remote streaming media server feeds back the remote play address to the terminal device. And the remote streaming media server feeds back the remote playing address of the single-path non-HTTP video stream data to the terminal equipment according to the request.
Step S603, the terminal device controls the local streaming media module to convert the remote play address of the single-channel non-HTTP video streaming data, so as to obtain the converted local streaming media play address.
Step S604, the local streaming media module transmits the converted local streaming media playing address to the terminal device.
And step S605, the terminal equipment sends a screen casting request to the screen casting equipment according to the converted local streaming media playing address.
Step S606, the screen projection equipment requests a single-channel video stream from the terminal equipment according to the converted local streaming media playing address so as to perform screen projection display according to the single-channel video stream.
Step S607, the terminal device requests the remote streaming media server for the single non-HTTP video stream data according to the play address of the single non-HTTP video stream data.
Step S608, the remote streaming media server pushes the non-HTTP video stream data to the terminal device according to the play address of the single non-HTTP video stream data.
And step S609, the terminal equipment converts the single-channel non-HTTP video stream data to be projected according to the transmission protocol type of the video projection screen to obtain the converted video stream. The converted video stream is the video stream which can be transmitted through the transmission protocol type of the video screen projection specified by the DLNA protocol, and then screen projection display is conveniently carried out through screen projection equipment.
And step S610, the terminal equipment sends the converted video stream to the screen projection equipment, so that the screen projection equipment performs screen projection display according to the converted video stream.
In the technical solution of the embodiment of the present application, when the terminal device performs screen projection on the single-channel non-HTTP video stream data through the screen projection device, the single-channel non-HTTP video stream data may be converted according to a transmission protocol type of the video screen projection to obtain a converted video stream, so as to obtain video stream data that can be transmitted according to the transmission protocol type of the video screen projection specified by the DLNA protocol, and the converted video stream is sent to the screen projection device, so that the screen projection device performs screen projection display based on the converted video stream, and thus the terminal device may perform screen projection on the single-channel video stream data obtained by using the transmission protocol of the video screen projection specified by the non-DLNA protocol, and the applicability of screen projection display on the single-channel video stream data by using the DLNA protocol is improved.
Referring to fig. 10, fig. 10 shows a data interaction diagram for projecting two non-HTTP video streams according to an example embodiment of the present application.
As shown in fig. 10, the local streaming media module may be specifically an application installed in the terminal device for playing a video stream, and the remote streaming media server may be a server storing a non-HTTP video stream data source. If the two paths of non-HTTP video stream data to be screen-projected are not spliced and are not converted according to the transmission protocol type of the video screen projection to obtain the converted video stream, the terminal equipment cannot screen-project and display the two paths of non-HTTP video stream data to be screen-projected. And after the two paths of non-HTTP video stream data to be projected are spliced and converted according to the type of the transmission protocol of the video projection screen to obtain the converted video stream, the terminal equipment can project the two paths of video stream data acquired by the transmission protocol of the video projection screen specified in the non-DLNA protocol through the screen projecting equipment, so that the terminal equipment can project the video stream data acquired by the transmission protocol of the video projection screen specified in the non-DLNA protocol.
Specifically, a schematic data interaction diagram for projecting two paths of non-HTTP video stream data according to an exemplary embodiment of the present application is shown in fig. 10, and may include the following steps:
step S701, the terminal device sends a request for obtaining a remote play address to the remote streaming server. The request is used for acquiring remote playing addresses of two paths of non-HTTP video stream data to be projected from a remote streaming media server.
Step S702, the remote play addresses of the two paths of non-HTTP video stream data fed back by the remote streaming media server are sent to the terminal equipment.
Step S703, the terminal device controls the local streaming media module to convert the address list corresponding to the remote play address of the two non-HTTP video stream data, so as to obtain the converted one-way local streaming media play address.
Step S704, the local streaming media module transmits the converted one-way local streaming media playing address to the terminal device.
Step S705, the terminal device sends a screen casting request to the screen casting device according to the converted single-path local streaming media playing address.
Step S706, the screen projection device requests the video stream from the terminal device according to the converted single-path local streaming media playing address, so as to perform screen projection display according to the video stream.
Step S707, the terminal device requests the first non-HTTP video stream data from the remote streaming media server according to the remote play address of the first non-HTTP video stream data.
Step S708, the remote streaming media server pushes the first non-HTTP video stream data to the terminal device according to the remote play address of the first non-HTTP video stream data.
Step S709, the terminal device requests the second path of non-HTTP video stream data from the remote streaming media server according to the remote playing address of the second path of non-HTTP video stream data.
Step S710, the remote streaming media server pushes the second path of non-HTTP video stream data to the terminal device according to the remote playing address of the second path of non-HTTP video stream data.
Step S711, the terminal device decodes the first non-HTTP video stream data to be projected according to the transmission protocol type of the video projection, to obtain a decoded first video stream.
And step S712, the terminal device decodes the second path of non-HTTP video stream data to be projected according to the transmission protocol type of the video projection screen to obtain a decoded second path of video stream.
Step S713, the terminal device splices the decoded first path of video stream and the decoded second path of video stream to obtain a spliced path of video stream, and converts the spliced path of video stream according to the transmission protocol type of the video screen projection to obtain a converted video stream. The converted video stream is the video stream which can be transmitted through the transmission protocol type of the video screen projection specified by the DLNA protocol, and then screen projection display is conveniently carried out through screen projection equipment.
And step S712, the terminal device sends the converted video stream to the screen projection device, so that the screen projection device performs screen projection display according to the converted video stream.
In the technical scheme of the embodiment of the application, when the terminal device projects the two paths of non-HTTP video stream data through the screen projecting device, the two paths of non-HTTP video stream data to be projected can be spliced, and the two paths of non-HTTP video stream data are not converted according to the type of the transmission protocol of the video screen projection to obtain the converted video stream, so that the terminal device can project the multiple paths of video stream data obtained by the transmission protocol of the video screen projection specified in the non-DLNA protocol through the screen projecting device, and the applicability of the video stream data to be projected and displayed by the DLNA protocol is improved. Compared with the prior art, under the condition that two paths of videos need to be projected and displayed, a mode of displaying only one path of videos or a mode of simply stacking two paths of videos is adopted, the method for splicing two paths of videos provided by the embodiment can ensure that two paths of videos needing to be projected can be completely projected and displayed, the loss of video content cannot be caused, and the effect of projecting and displaying two paths of videos is improved.
Referring to fig. 11, fig. 11 is a block diagram illustrating a screen projection processing method according to an exemplary embodiment of the present application, the screen projection processing apparatus 800 is mainly executed by the terminal device 101 shown in fig. 1, and the screen projection control apparatus 800 includes an acquisition unit 810, a first conversion unit 820, and a transmission unit 830:
the obtaining unit 810 is configured to obtain a transmission protocol type of a video file to be screen-projected in response to a request for screen projection by a screen projection device.
A first converting unit 820, configured to, if the transmission protocol type of the video file to be screen-projected is not matched with the transmission protocol type of the video screen-projected, convert the video file to be screen-projected according to the transmission protocol type of the video screen-projected, so as to obtain a converted video stream.
A sending unit 830, configured to send the converted video stream to the screen projection device, so that the screen projection device performs screen projection display based on the converted video stream.
In an embodiment of the present application, the first conversion unit 820 is configured to demultiplex the video file to be screened to obtain demultiplexed video frame data and demultiplexed audio data; decoding the demultiplexed video frame data to obtain decoded video frame data, and decoding the demultiplexed audio data to obtain decoded audio data; respectively encoding the decoded video frame data and the decoded audio data according to the transmission protocol type of the video screen projection to obtain first video frame data and first audio data; and multiplexing the first video frame data and the first audio data according to the transmission protocol type of the video screen projection to obtain a converted video stream.
In an embodiment of the application, if the video file to be projected on the screen includes at least two video files, the first conversion unit 820 is configured to perform demultiplexing on the at least two video files respectively to obtain at least two demultiplexed video frame data and at least two demultiplexed audio data; decoding the at least two paths of demultiplexed video frame data to obtain at least two paths of decoded video frame data, and decoding the at least two paths of demultiplexed audio data to obtain at least two paths of decoded audio data; splicing the at least two paths of decoded video frame data to obtain second video frame data, and fusing the at least two paths of decoded audio data to obtain second audio data; according to the transmission protocol type of the video screen projection, encoding the second video frame data to obtain third video frame data, and encoding the second audio data to obtain third audio data; and multiplexing the third video frame data and the third audio data according to the transmission protocol type of the video screen projection to obtain a converted video stream.
In an embodiment of the present application, the first converting unit 820 is configured to perform splicing processing on video frames at the same time in the at least two paths of decoded video frame data according to a time sequence order, so as to obtain the second video frame data.
In an embodiment of the present application, the first converting unit 820 is configured to perform fusion processing on the at least two paths of decoded audio data according to a time sequence order to obtain the second audio data.
In an embodiment of the application, the screen-casting processing apparatus 800 further includes a second converting unit, configured to, if a text file that needs to be displayed on a screen is acquired, convert the text file into picture frame data to be screen-cast, and convert the picture frame data to be screen-cast and the video file to be screen-cast according to a transmission protocol type of the video screen to obtain a converted video stream.
In one embodiment of the present application, the second conversion unit is configured to convert the text file into at least one frame of picture frame data as the picture frame data to be projected.
In an embodiment of the application, the second conversion unit is configured to demultiplex the video file to be screened to obtain demultiplexed video frame data and demultiplexed audio data; decoding the demultiplexed video frame data to obtain decoded video frame data, and decoding the demultiplexed audio data to obtain decoded audio data; splicing the picture frame data to be projected and the decoded video frame data to obtain fourth video frame data; according to the transmission protocol type of the video screen projection, encoding the fourth video frame data to obtain fifth video frame data, and encoding the decoded audio data to obtain first audio data; and multiplexing the fifth video frame data and the first audio data according to the transmission protocol type of the video screen projection to obtain a converted video stream.
In an embodiment of the application, the second conversion unit is configured to, if it is detected that a picture format of the picture frame data to be screen-projected does not match a picture format of the decoded video frame data, convert the picture frame data to be screen-projected according to the picture format of the decoded video frame data to obtain converted picture frame data; and splicing the converted picture frame data and the decoded video frame data to obtain fourth video frame data.
Referring to fig. 12, fig. 12 shows a hardware structure diagram of an electronic device according to an example embodiment of the present application, and the screen projection processing method according to the embodiment of the present application may be implemented by the electronic device shown in fig. 12, which may specifically be the terminal device 101 shown in fig. 1, and the following description is given by taking the terminal device 101 as an example with reference to fig. 12, where the terminal device 101 shown in fig. 12 is merely an example, and should not bring any limitation to the functions and the use range of the embodiment of the present application.
As shown in fig. 12, the terminal apparatus 101 is represented in the form of a general-purpose computing apparatus. The components of terminal device 101 may include, but are not limited to: the at least one processing unit 910, the at least one memory unit 920, and a bus 830 that couples various system components including the memory unit 920 and the processing unit 910.
Wherein the storage unit stores program code, which can be executed by the processing unit 910, to cause the processing unit 910 to perform the steps according to various exemplary embodiments of the present application described in the description part of the above exemplary methods of the present specification. For example, the processing unit 910 may perform various steps as shown in fig. 2 to 8.
The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM)9201 and/or a cache memory unit 9202, and may further include a read only memory unit (ROM) 9203.
Storage unit 920 may also include a program/utility 9204 having a set (at least one) of program modules 9205, such program modules 9205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
Bus 930 can be any of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
Terminal device 101 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the terminal device 101, and/or with any devices (e.g., router, modem, etc.) that enable the terminal device 101 to communicate with one or more other computing devices. Such communication may occur via input/output (I/O) interface 950. Also, the terminal device 101 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the internet) via the network adapter 960. As shown in fig. 12, the network adapter 960 communicates with the other modules of the terminal device 101 through the bus 930. It should be understood that although not shown in fig. 12, other hardware and/or software modules may be used in conjunction with terminal device 101, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to make a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) execute the method according to the embodiments of the present application.
In an exemplary embodiment of the present application, there is also provided a computer program medium having stored thereon computer readable instructions which, when executed by a processor of a computer, cause the computer to perform the method described in the above method embodiment section.
According to an embodiment of the present application, there is also provided a program product for implementing the method in the above method embodiment, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
A computer readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the application. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.
Moreover, although the steps of the methods herein are depicted in the drawings in a particular order, this does not require or imply that the steps must be performed in this particular order, or that all of the depicted steps must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions, etc.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.
Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.

Claims (20)

1. A screen projection processing method is characterized by comprising the following steps:
responding to a screen projection request through screen projection equipment, and acquiring a transmission protocol type of a video file to be projected;
if the transmission protocol type of the video file to be screen-projected is not matched with the transmission protocol type of the video screen-projected specified by the digital living network alliance DLNA, converting the video file to be screen-projected according to the transmission protocol type of the video screen-projected to obtain a converted video stream;
and sending the converted video stream to the screen projection equipment so that the screen projection equipment performs screen projection display based on the converted video stream.
2. The screen-projection processing method of claim 1, wherein the converting the video file to be screen-projected according to the transmission protocol type of the video screen projection to obtain a converted video stream comprises:
demultiplexing the video file to be projected to obtain video frame data subjected to demultiplexing and audio data subjected to demultiplexing;
decoding the demultiplexed video frame data to obtain decoded video frame data, and decoding the demultiplexed audio data to obtain decoded audio data;
respectively encoding the decoded video frame data and the decoded audio data according to the transmission protocol type of the video screen projection to obtain first video frame data and first audio data;
and multiplexing the first video frame data and the first audio data according to the transmission protocol type of the video screen projection to obtain a converted video stream.
3. The screen-projection processing method of claim 1, wherein if the video file to be screen-projected includes at least two video files, converting the video file to be screen-projected according to a transmission protocol type of the video screen-projection to obtain a converted video stream, including:
the at least two paths of video files are respectively subjected to demultiplexing processing to obtain at least two paths of demultiplexed video frame data and at least two paths of demultiplexed audio data;
decoding the at least two paths of demultiplexed video frame data to obtain at least two paths of decoded video frame data, and decoding the at least two paths of demultiplexed audio data to obtain at least two paths of decoded audio data;
splicing the at least two paths of decoded video frame data to obtain second video frame data, and fusing the at least two paths of decoded audio data to obtain second audio data;
according to the transmission protocol type of the video screen projection, encoding the second video frame data to obtain third video frame data, and encoding the second audio data to obtain third audio data;
and multiplexing the third video frame data and the third audio data according to the transmission protocol type of the video screen projection to obtain a converted video stream.
4. The screen-projection processing method of claim 3, wherein the splicing the at least two paths of decoded video frame data to obtain second video frame data comprises:
and splicing the video frames at the same time in the at least two paths of decoded video frame data according to a time sequence to obtain second video frame data.
5. The screen-projection processing method of claim 3, wherein the fusing the at least two decoded audio data to obtain second audio data comprises:
and performing fusion processing on the at least two paths of decoded audio data according to a time sequence order to obtain the second audio data.
6. The screen-projection processing method according to claim 1, further comprising:
and if the text file needing screen projection display is acquired, converting the text file into picture frame data of a screen to be projected, and converting the picture frame data of the screen to be projected and the video file of the screen to be projected according to the transmission protocol type of the video screen to obtain a converted video stream.
7. The screen-projection processing method of claim 6, wherein the converting the text file into the picture frame data to be screen-projected comprises:
and converting the text file into at least one frame of picture frame data to be used as the picture frame data to be projected.
8. The screen-projection processing method of claim 6, wherein the converting the picture frame data to be screen-projected and the video file to be screen-projected according to the transmission protocol type of the video screen-projection to obtain a converted video stream comprises:
demultiplexing the video file to be projected to obtain video frame data subjected to demultiplexing and audio data subjected to demultiplexing;
decoding the demultiplexed video frame data to obtain decoded video frame data, and decoding the demultiplexed audio data to obtain decoded audio data;
splicing the picture frame data to be projected and the decoded video frame data to obtain fourth video frame data;
according to the transmission protocol type of the video screen projection, encoding the fourth video frame data to obtain fifth video frame data, and encoding the decoded audio data to obtain first audio data;
and multiplexing the fifth video frame data and the first audio data according to the transmission protocol type of the video screen projection to obtain a converted video stream.
9. The screen projection processing method of claim 8, wherein the splicing processing of the picture frame data to be screen projected and the decoded video frame data to obtain fourth video frame data comprises:
if the picture format of the picture frame data to be screen-projected is detected not to be matched with the picture format of the decoded video frame data, converting the picture frame data to be screen-projected according to the picture format of the decoded video frame data to obtain converted picture frame data;
and splicing the converted picture frame data and the decoded video frame data to obtain fourth video frame data.
10. A screen projection processing apparatus, comprising:
the device comprises an acquisition unit, a display unit and a display unit, wherein the acquisition unit is used for responding to a screen projection request of screen projection equipment and acquiring the transmission protocol type of a video file to be projected;
the first conversion unit is used for converting the video file to be screen-projected according to the transmission protocol type of the video screen-projected if the transmission protocol type of the video file to be screen-projected is not matched with the transmission protocol type of the video screen-projected specified by the digital living network alliance DLNA, so as to obtain a converted video stream;
and the sending unit is used for sending the converted video stream to the screen projection equipment so that the screen projection equipment performs screen projection display based on the converted video stream.
11. The apparatus according to claim 10, wherein the first conversion unit is configured to perform demultiplexing on the video file to be projected to obtain demultiplexed video frame data and demultiplexed audio data; decoding the demultiplexed video frame data to obtain decoded video frame data, and decoding the demultiplexed audio data to obtain decoded audio data; respectively encoding the decoded video frame data and the decoded audio data according to the transmission protocol type of the video screen projection to obtain first video frame data and first audio data; and multiplexing the first video frame data and the first audio data according to the transmission protocol type of the video screen projection to obtain a converted video stream.
12. The apparatus according to claim 10, wherein if the video file to be projected includes at least two video files, the first converting unit is configured to demultiplex the at least two video files respectively to obtain at least two demultiplexed video frame data and at least two demultiplexed audio data; decoding the at least two paths of demultiplexed video frame data to obtain at least two paths of decoded video frame data, and decoding the at least two paths of demultiplexed audio data to obtain at least two paths of decoded audio data; splicing the at least two paths of decoded video frame data to obtain second video frame data, and fusing the at least two paths of decoded audio data to obtain second audio data; according to the transmission protocol type of the video screen projection, encoding the second video frame data to obtain third video frame data, and encoding the second audio data to obtain third audio data; and multiplexing the third video frame data and the third audio data according to the transmission protocol type of the video screen projection to obtain a converted video stream.
13. The apparatus of claim 12, wherein the first converting unit is configured to perform a splicing process on video frames at the same time in the at least two paths of decoded video frame data according to a time sequence order to obtain the second video frame data.
14. The apparatus as claimed in claim 12, wherein the first converting unit is configured to perform a fusion process on the at least two decoded audio data according to a time sequence order to obtain the second audio data.
15. The apparatus of claim 10, wherein the apparatus further comprises:
and the second conversion unit is used for converting the text file into picture frame data of a screen to be projected if the text file required to be projected and displayed is acquired, and converting the picture frame data of the screen to be projected and the video file of the screen to be projected according to the transmission protocol type of the video screen to acquire a converted video stream.
16. The apparatus of claim 15, wherein a second conversion unit is configured to convert the text file into at least one frame of picture frame data as the picture frame data to be screened.
17. The apparatus according to claim 15, wherein the second conversion unit is configured to perform demultiplexing on the video file to be projected to obtain demultiplexed video frame data and demultiplexed audio data; decoding the demultiplexed video frame data to obtain decoded video frame data, and decoding the demultiplexed audio data to obtain decoded audio data; splicing the picture frame data to be projected and the decoded video frame data to obtain fourth video frame data; according to the transmission protocol type of the video screen projection, encoding the fourth video frame data to obtain fifth video frame data, and encoding the decoded audio data to obtain first audio data; and multiplexing the fifth video frame data and the first audio data according to the transmission protocol type of the video screen projection to obtain a converted video stream.
18. The apparatus of claim 17, wherein the second converting unit is configured to, if it is detected that the picture format of the picture frame data to be projected does not match the picture format of the decoded video frame data, convert the picture frame data to be projected according to the picture format of the decoded video frame data to obtain converted picture frame data; and splicing the converted picture frame data and the decoded video frame data to obtain fourth video frame data.
19. An electronic device, comprising:
a memory storing computer readable instructions;
a processor to read computer readable instructions stored by the memory to perform the method of any of claims 1-9.
20. A computer program medium having computer readable instructions stored thereon, which, when executed by a processor of a computer, cause the computer to perform the method of any one of claims 1-9.
CN201910779831.6A 2019-08-22 2019-08-22 Screen projection processing method and device, electronic equipment and computer program medium Active CN110784758B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910779831.6A CN110784758B (en) 2019-08-22 2019-08-22 Screen projection processing method and device, electronic equipment and computer program medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910779831.6A CN110784758B (en) 2019-08-22 2019-08-22 Screen projection processing method and device, electronic equipment and computer program medium

Publications (2)

Publication Number Publication Date
CN110784758A CN110784758A (en) 2020-02-11
CN110784758B true CN110784758B (en) 2021-11-05

Family

ID=69384013

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910779831.6A Active CN110784758B (en) 2019-08-22 2019-08-22 Screen projection processing method and device, electronic equipment and computer program medium

Country Status (1)

Country Link
CN (1) CN110784758B (en)

Families Citing this family (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111386700A (en) * 2020-02-25 2020-07-07 深圳市乐橙互联有限公司 Multifunctional receiving device and conference system
CN111427527B (en) * 2020-03-20 2023-10-31 海信视像科技股份有限公司 Screen projection method, device, equipment and computer readable storage medium
CN111614654B (en) * 2020-05-18 2021-03-19 江苏澳云软件技术有限公司 Cloud domain network data transmission method and device, cloud platform and cloud platform service system
CN111666056B (en) * 2020-05-28 2024-01-05 阿波罗智联(北京)科技有限公司 Screen switching display method and device, intelligent rearview mirror and readable storage medium
CN111866559A (en) * 2020-07-10 2020-10-30 北京百家视联科技有限公司 Wireless screen projection protocol system and method
CN112004143B (en) * 2020-08-06 2022-07-08 北京奇艺世纪科技有限公司 Information processing method and device of equipment, electronic equipment and storage medium
CN112019900A (en) * 2020-10-19 2020-12-01 深圳乐播科技有限公司 DLNA (digital Living network alliance) -based screen projection method, system, equipment and storage medium
CN112788384A (en) * 2021-02-07 2021-05-11 深圳市大鑫浪电子科技有限公司 Wireless digital television screen projection method and device, computer equipment and storage medium
CN114040232B (en) * 2021-11-01 2023-09-01 北京奇艺世纪科技有限公司 Screen projection system, screen projection method, electronic equipment and storage medium
CN114302202B (en) * 2021-12-22 2023-03-14 深圳创维－Rgb电子有限公司 Audio and video screen projection method, device, equipment, system and storage medium
CN114489549B (en) * 2022-01-30 2023-04-25 深圳创维－Rgb电子有限公司 Screen projection image processing method and device, electronic equipment and storage medium
CN114666641B (en) * 2022-05-25 2022-08-26 舟谱数据技术南京有限公司 Dealer multi-scene television large-screen projection system and method
CN115499695B (en) * 2022-08-16 2024-01-16 珠海全志科技股份有限公司 Smart television control method, smart television control device, advertising machine and medium
CN116456136B (en) * 2023-04-20 2023-12-15 广州市迪士普音响科技有限公司 Equipment screen throwing method and device based on localization

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102143182A (en) * 2011-03-31 2011-08-03 上海盛盟信息技术有限公司 Internet-based digital television multimedia information making, examination and publishing system
CN102325271A (en) * 2011-09-02 2012-01-18 深圳创维－Rgb电子有限公司 Method and device of sending document from mobile terminal to television for sharing and mobile terminal
CN104754402A (en) * 2015-04-08 2015-07-01 好迪龙电子(深圳)有限公司 Multi-unit same-screen displaying method and device
CN105991962A (en) * 2015-02-03 2016-10-05 阿里巴巴集团控股有限公司 Information display method, device and system
CN107809664A (en) * 2017-11-10 2018-03-16 四川长虹电器股份有限公司 The TV for turning HDMI based on wifi throws screen device
CN108399056A (en) * 2018-02-26 2018-08-14 上海康斐信息技术有限公司 It is a kind of wirelessly to throw screen method and wireless router

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102143182A (en) * 2011-03-31 2011-08-03 上海盛盟信息技术有限公司 Internet-based digital television multimedia information making, examination and publishing system
CN102325271A (en) * 2011-09-02 2012-01-18 深圳创维－Rgb电子有限公司 Method and device of sending document from mobile terminal to television for sharing and mobile terminal
CN105991962A (en) * 2015-02-03 2016-10-05 阿里巴巴集团控股有限公司 Information display method, device and system
CN104754402A (en) * 2015-04-08 2015-07-01 好迪龙电子(深圳)有限公司 Multi-unit same-screen displaying method and device
CN107809664A (en) * 2017-11-10 2018-03-16 四川长虹电器股份有限公司 The TV for turning HDMI based on wifi throws screen device
CN108399056A (en) * 2018-02-26 2018-08-14 上海康斐信息技术有限公司 It is a kind of wirelessly to throw screen method and wireless router

Also Published As

Publication number Publication date
CN110784758A (en) 2020-02-11

Similar Documents

Publication Publication Date Title
CN110784758B (en) Screen projection processing method and device, electronic equipment and computer program medium
WO2019205872A1 (en) Video stream processing method and apparatus, computer device and storage medium
KR102520019B1 (en) Speech enhancement for speech recognition applications in broadcast environments
KR20180050961A (en) Method and device for decoding multimedia file
CN112804256B (en) Method, device, medium and equipment for processing track data in multimedia file
CN112261416A (en) Cloud-based video processing method and device, storage medium and electronic equipment
EP3148200B1 (en) Information processing device and method selecting content files based on encoding parallelism type
CN112637661B (en) Video stream switching method and device, computer storage medium and electronic equipment
CN112188277B (en) Screen projection control method and device, electronic equipment and computer program medium
CN111629253A (en) Video processing method and device, computer readable storage medium and electronic equipment
US20170127152A1 (en) Information processing device and information processing method
US11687589B2 (en) Auto-populating image metadata
CN109547839A (en) The dynamic previewing processing method and processing device of live content, medium, electronic equipment
CN115243074A (en) Video stream processing method and device, storage medium and electronic equipment
CN113473165A (en) Live broadcast control system, live broadcast control method, device, medium and equipment
US20180124435A1 (en) Systems and methods for live multimedia information collection, presentation, and standardization
US11157146B2 (en) Display apparatus and control method thereof for providing preview content
CN109948082B (en) Live broadcast information processing method and device, electronic equipment and storage medium
CN111918074A (en) Live video fault early warning method and related equipment
CN115460422B (en) Ultra-high-definition network video and audio professional decoder and video frame synchronization method
US20170163980A1 (en) Information processing device and method
US20120281066A1 (en) Information processing device and information processing method
TW202243481A (en) Point cloud decoding method and device, point cloud encoding method and device, and electronic apparatus
US11265357B2 (en) AV1 codec for real-time video communication
KR20160087225A (en) System for cloud streaming service, method of image cloud streaming service to provide a multi-view screen, and apparatus for the same

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
REG Reference to a national code

Ref country code: HK

Ref legal event code: DE

Ref document number: 40020401

Country of ref document: HK

SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant