CN114040251A - Audio and video playing method, system, storage medium and computer program product - Google Patents
Audio and video playing method, system, storage medium and computer program product Download PDFInfo
- Publication number
- CN114040251A CN114040251A CN202111094908.XA CN202111094908A CN114040251A CN 114040251 A CN114040251 A CN 114040251A CN 202111094908 A CN202111094908 A CN 202111094908A CN 114040251 A CN114040251 A CN 114040251A
- Authority
- CN
- China
- Prior art keywords
- audio
- code stream
- browser
- stream data
- video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 144
- 238000004590 computer program Methods 0.000 title claims abstract description 20
- 238000004891 communication Methods 0.000 claims abstract description 42
- 238000012544 monitoring process Methods 0.000 claims abstract description 5
- 230000008569 process Effects 0.000 claims description 60
- 230000003321 amplification Effects 0.000 claims description 8
- 238000003199 nucleic acid amplification method Methods 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 7
- 238000004806 packaging method and process Methods 0.000 claims description 4
- 238000009434 installation Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 6
- 238000009877 rendering Methods 0.000 description 6
- 230000005540 biological transmission Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 230000005236 sound signal Effects 0.000 description 3
- 230000007547 defect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- VYZAMTAEIAYCRO-UHFFFAOYSA-N Chromium Chemical compound [Cr] VYZAMTAEIAYCRO-UHFFFAOYSA-N 0.000 description 1
- 238000005538 encapsulation Methods 0.000 description 1
- 239000012634 fragment Substances 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
- H04N21/440218—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by transcoding between formats or standards, e.g. from MPEG-2 to MPEG-4
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/439—Processing of audio elementary streams
- H04N21/4398—Processing of audio elementary streams involving reformatting operations of audio signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client
- H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
- H04N21/637—Control signals issued by the client directed to the server or network components
- H04N21/6371—Control signals issued by the client directed to the server or network components directed to network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/8166—Monomedia components thereof involving executable data, e.g. software
- H04N21/8173—End-user applications, e.g. Web browser, game
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The embodiment of the invention provides an audio and video playing method, an audio and video playing system, a storage medium and a computer program product. The method comprises the following steps: monitoring a local port, and receiving a connection request of a browser through the local port; establishing a communication connection with the browser based on the connection request; receiving a playing request instruction of an audio and video signal sent by a browser through communication connection; acquiring a stream taking address of the audio/video signal according to the playing request instruction; establishing network connection with the stream taking address and acquiring code stream data of the audio and video signals from the stream taking address; and transmitting the code stream data to the browser through the communication connection so as to play the audio and video signals by the browser. The scheme can support various mainstream browsers. Meanwhile, the scheme can be realized without the participation of a remote server.
Description
Technical Field
The present invention relates to the field of audio/video playing, and more particularly, to an audio/video playing method, system, storage medium, and computer program product.
Background
Currently, the browser mainly plays audio and video through the following two methods. The first method is to embed an ActiveX control in the browser. And decoding and displaying the audio and video by utilizing the ActiveX control. The second method is to add a background service of the client to transcode various audio and video coding data into an H264 video coding format or an AAC audio coding format. And packaging the audio and video data after transcoding, and converting the audio and video data into a fragment mp4 format. And then the browser plays the packaged audio and Video by using the Video tag for expanding the MSE.
Although the above methods can all realize playing the audio and video on the browser, there still exist many problems. For the first method, it can only be implemented on the browser with the IE engine, e.g., IE11, 360 browser, but not on the browser with the Chrome engine. I.e. the first method cannot be used across browsers. For the second method, the background service needs to perform operations such as audio/video decoding, encoding, and encapsulation, which not only increases the consumption of audio/video encoding resources, but also performs secondary processing on the front-end video stream, resulting in a certain play delay. When the number of playing paths is large, the performance consumption of the background service is also multiplied correspondingly.
Therefore, a new method for playing audio/video is needed to solve the above problems.
Disclosure of Invention
The present invention has been made in view of the above problems. According to an aspect of the present invention, there is provided an audio/video playing method, including: monitoring a local port, and receiving a connection request of a browser through the local port; establishing a communication connection with the browser based on the connection request; receiving a playing request instruction of an audio and video signal sent by a browser through communication connection; acquiring a stream taking address of the audio/video signal according to the playing request instruction; establishing network connection with the stream taking address and acquiring code stream data of the audio and video signals from the stream taking address; and transmitting the code stream data to the browser through the communication connection so as to play the audio and video signals by the browser.
Illustratively, the method further comprises: acquiring an output mode of the audio and video signal according to the playing request instruction; and after the code stream data of the audio and video signals are acquired from the streaming address and before the code stream data are sent to the browser, processing the code stream data according to the output mode to acquire the code stream data conforming to the output mode.
Illustratively, after the code stream data of the audio/video signal is acquired from the self-fetching stream address and before the code stream data is sent to the browser, the method further comprises: for the case that the format of the code stream data is the video data of the H264 coding format, packaging the code stream data into code stream data of the fmp4 format; the playing of the audio and video signals by the browser comprises: and playing the code stream data by using the video label conforming to the media source expansion specification.
Illustratively, after the code stream data of the audio/video signal is acquired from the self-fetching stream address and before the code stream data is sent to the browser, the method further comprises: decoding the code stream data to obtain brightness and chroma information of the code stream data under the condition that the format of the code stream data is video data in a non-H264 coding format; sending the code stream data to the browser, so that the browser plays the audio and video signals, wherein the step of sending the code stream data to the browser comprises the following steps: and sending the brightness and the chrominance information of the code stream data to a browser so as to render the brightness and the chrominance by the browser.
Illustratively, after the code stream data of the audio/video signal is acquired from the self-fetching stream address and before the code stream data is sent to the browser, the method further comprises: and decoding the code stream data into the code stream data modulated by the pulse code in case that the code stream data is audio data.
Illustratively, the method further comprises: acquiring a stream taking protocol of the audio and video signals according to the playing request instruction; the method for acquiring the code stream data of the audio and video signals from the stream taking address comprises the following steps: and acquiring code stream data from the stream-taking address based on the stream-taking protocol.
Illustratively, after establishing a network connection with the streaming address and acquiring the code stream data of the audio/video signal from the streaming address, the method further includes: determining whether code stream data is successfully acquired; and for the condition that the code stream data is not successfully acquired, establishing network connection with the stream taking address and acquiring the code stream data of the audio and video signals from the stream taking address again until the code stream data is successfully acquired.
Illustratively, the method further comprises: receiving a playing control instruction of an audio and video signal sent by a browser through communication connection; and controlling the browser to play the audio and video signals according to the playing control instruction.
Illustratively, the play control instructions include one or more of the following instructions: pause playing instruction, resume playing instruction, video recording instruction and video electronic amplification instruction.
Illustratively, the method further comprises: responding to the operation of breaking the communication connection of the browser, breaking the network connection and stopping obtaining the code stream data of the audio and video signals from the stream taking address; and clearing the cached code stream data.
According to another aspect of the present invention, there is also provided an audio-video playing system, which includes a processor and a memory. Wherein the memory has stored therein computer program instructions for executing the audio-video playback method as described above when the computer program instructions are executed by the processor.
Illustratively, the computer program instructions constitute a playback component, wherein the playback component, upon completion of installation, initiates a background process and remains resident.
According to still another aspect of the present invention, there is also provided a storage medium. Program instructions are stored on a storage medium. The program instructions are operable, when executed, to perform the audiovisual playback method as described above.
According to yet another aspect of the invention, there is also provided a computer program product comprising program instructions. The program instructions are operable, when executed, to perform the audiovisual playback method as described above.
According to the technical scheme, the audio and video playing method does not limit the format of the code stream data of the audio and video signals at all, supports various code stream data formats and audio and video signals with different resolutions, and can support various mainstream browsers. Meanwhile, the scheme can be realized without the participation of a remote server as the scheme runs at a local client; therefore, the defect that the requirement of deploying a remote server in the prior art is still difficult to meet the requirement of playing multiple paths of videos is effectively overcome, and the playing delay is avoided.
Drawings
The above and other objects, features and advantages of the present invention will become more apparent by describing in more detail embodiments of the present invention with reference to the attached drawings. The accompanying drawings are included to provide a further understanding of the embodiments of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention. In the drawings, like reference numbers generally represent like parts or steps.
Fig. 1 shows a schematic block diagram of an example electronic device for implementing an audio-video playing method and apparatus according to an embodiment of the present invention;
fig. 2 shows a schematic flow diagram of an audiovisual playback method according to an embodiment of the invention;
fig. 3 shows a schematic flow diagram of an audiovisual playback method according to another embodiment of the invention;
fig. 4 shows a schematic timing diagram of an audiovisual playback method according to an embodiment of the invention;
fig. 5 shows a schematic block diagram of an audiovisual playback device in accordance with an embodiment of the invention;
fig. 6 shows a schematic block diagram of an audiovisual playback system in accordance with an embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. It is to be understood that the described embodiments are merely a subset of embodiments of the invention and not all embodiments of the invention, with the understanding that the invention is not limited to the example embodiments described herein. All other embodiments, which can be derived by a person skilled in the art from the embodiments described in the present application without inventive step, shall fall within the scope of protection of the present application.
First, an example electronic device 100 for implementing an audio-video playing method and apparatus according to an embodiment of the present invention is described with reference to fig. 1.
As shown in fig. 1, electronic device 100 includes one or more processors 102, one or more memory devices 104. Optionally, the electronic device 100 may also include an input device 106 and an output device 108, which are interconnected by a bus system 110 and/or other form of connection mechanism (not shown). It should be noted that the components and structure of the electronic device 100 shown in fig. 1 are exemplary only, and not limiting, and that the electronic device may have other components and structures as desired.
The processor 102 may be implemented in hardware using at least one of a microprocessor, a Digital Signal Processor (DSP), a Field Programmable Gate Array (FPGA), and a Programmable Logic Array (PLA). Processor 102 may also be one or a combination of Central Processing Units (CPUs), Graphics Processors (GPUs), Application Specific Integrated Circuits (ASICs), or other forms of processing units having data processing capabilities and/or instruction execution capabilities, and may control other components in electronic device 100 to perform desired functions.
The input device 106 may be a device used by a user to input instructions and may include one or more of a keyboard, mouse, microphone, touch screen, etc.
The output device 108 may output various information (e.g., images and/or sounds) to an external (e.g., user), and may include one or more of a display, a speaker, and the like. Alternatively, the input device 106 and the output device 108 may be integrated together, implemented using the same interactive device (e.g., a touch screen).
Illustratively, an example electronic device for implementing the audio/video playing method and apparatus according to the embodiment of the present invention may be implemented on a device such as a personal computer, a tablet computer, or a smart phone.
In order to at least partially solve the technical problem, an embodiment of the present invention provides an audio/video playing method. Fig. 2 shows a schematic flow diagram of an audiovisual playback method 200 in accordance with an embodiment of the invention. An audio-video playing method according to an embodiment of the present invention will be described below with reference to fig. 2. The method may be implemented by using a playing component, and the following describes the audio/video playing method 200 by taking this as an example. When the user opens the video playing page of the browser, the user may be prompted to download and install the playing component. After the playing component is installed, the background application process can be started on the client without perception, and the resident running is kept, so that the video playing function is realized.
As shown in fig. 2, the method 200 includes the following steps.
Step S210, monitor the local port, and receive a connection request of the browser through the local port.
Illustratively, a local port may refer to a port on a local client, such as a computer, such as an 80 port or a 433 port, etc. The connection request of the browser can be received in real time by monitoring the local port. Specifically, the native application process of the aforementioned play component may enable a two-way communication (Websocket) service when running in the background, so as to implement listening to the native port. It can be appreciated that the Websocket service listens to which local port can be specifically set according to the specific situation. And when a user opens an audio and video playing page associated with the local application process by using the browser, the browser sends a connection request. The connection request may be received by the local application process via the local port.
Step S220, establishing a communication connection with the browser based on the connection request.
It will be appreciated that the local application process, upon receiving the connection request, immediately responds to the request to establish a communication connection with the browser.
The connection between the browser and the local application process can be established through the two steps of step S210 and step S220. The connection relationship between the browser and the local application process can follow a Websocket protocol.
Step S230, receiving a playing request instruction of the audio/video signal sent by the browser through the communication connection established in step S220. The playing request instruction is used for requesting to play a specific audio and video signal. After the browser establishes a communication connection with the local application process, the local application process may receive a play request instruction sent by the browser. Similar to the connection request, the play request command may also be transmitted via the local port.
And step S240, acquiring a streaming address of the audio/video signal according to the playing request instruction.
Illustratively, the play request instruction may include information on a streaming address of the audio-video signal. The streaming address may be an Inter-Process Communication (Inter-Process Communication) or other network address providing a video streaming platform. The Streaming address may be, for example, a Real Time Streaming Protocol (RTSP) address, a Websocket Streaming address, or the like. It will be appreciated that there is one and only one streaming address for each audio-visual signal.
And step S250, establishing network connection with the stream taking address and obtaining code stream data of the audio and video signals from the stream taking address.
The local application process can establish network connection with the stream taking address after acquiring the stream taking address of the audio and video signal. And then the local application process can access the stream taking address and obtain the code stream data of the expected audio and video signals to finish stream taking. The code stream data may refer to data carrying audio and video signals, which may exist in the form of audio and video files. For the code stream data of the video, the H264 coding format and the non-H264 coding format can be divided.
And step S260, transmitting the code stream data to the browser through the communication connection with the browser so as to play the audio and video signals by the browser.
The local application process may send the codestream data to the browser via a communication connection with the browser after obtaining the codestream data. Similarly, the codestream data may also be sent through a local port, such as based on a local loopback address. The browser can play the corresponding audio and video signals after receiving the code stream data.
According to the technical scheme, the audio and video playing method does not limit the format of the code stream data of the audio and video signals at all, supports various code stream data formats and audio and video signals with different resolutions, and can support various mainstream browsers. Meanwhile, the scheme can be realized without the participation of a remote server as the scheme runs at a local client; therefore, the defect that the requirement of deploying a remote server in the prior art is still difficult to meet the requirement of playing multiple paths of videos is effectively overcome, and the playing delay is avoided.
Illustratively, the method 200 may further include: and acquiring a stream taking protocol of the audio and video signals according to the playing request instruction. The streaming protocol may be a standard streaming protocol such as a national standard G28181 protocol and an RTSP protocol. The streaming protocol can also be a third-party streaming protocol customized by a third party. In other words, in this embodiment, the playback component may extend its streaming mode, for example, the streaming mode may be obtained by using a Software Development Kit (SDK) of a downloaded and installed third-party device vendor. Step S250 may include: and based on the stream taking protocol, obtaining the code stream data of the audio and video signals from the stream taking address. It can be understood that the local application process may obtain the code stream data from the stream fetching address according to different stream fetching protocols. Based on the third-party stream taking protocol, the local application process can obtain the code stream data of the third party with the private coding format so as to play audio and video.
The technical scheme can meet the requirements of most users by using the mainstream stream taking protocol. Meanwhile, the situation that the user possibly plays the third-party audio and video signals is also considered, the technical scheme has expandability, supports the stream fetching mode developed by the third party, and can solve the playing problem of the third-party audio and video signals. The use experience of the user is greatly improved.
Illustratively, the method 200 may further include: and acquiring the output mode of the audio/video signal according to the playing request instruction. The output mode of the audio and video signal can represent the output format of the audio and video signal.
Step S255 may also exist after the code stream data of the audio and video signal is acquired from the streaming address and before the code stream data is sent to the browser, i.e., between step S250 and step S260. In step S255, the code stream data is processed according to the acquired output mode to obtain code stream data conforming to the output mode. It can be understood that in this solution, there is no fixed requirement on the format of the bitstream data. The format of the audio/video signal finally played by the browser is only related to the output mode of the audio/video signal in the playing request instruction.
Therefore, the audio and video playing method supports code stream data in various formats, and effectively ensures the generalization of audio and video signals which can be played by the browser. In addition, the audio and video signals are processed at the local client, the audio and video playing pressure is distributed to each client without depending on a remote server, and the end-to-end delay can be reduced to be within 300 ms.
In a specific embodiment, step S255 can be implemented by: in the case where the format of the codestream data is video data in the H264 encoding format, the codestream data is encapsulated into codestream data in the fmp4 format.
After acquiring the code stream data, the local application process firstly judges the format of the code stream data. In this embodiment, the format of the code stream data acquired by the local application process is an H264 encoding format. As will be appreciated by those of ordinary skill in the art, the H264 encoding format is a compressed format for digital video. For the code stream data in this encoding format, the local application process may encapsulate it into code stream data in fmp4 format. The specific method for encapsulating the code stream data in the H264 coding format may be implemented by using the existing or future developed technology, and is not limited in this application.
And transmitting the packaged code stream data from the local application process to the browser through the communication connection so as to play the audio and video signals by the browser. In this embodiment, the playing of the audio/video signal by the browser may be playing the code stream data using a video (video) tag conforming to a media source extension specification (extended MSE). The media source extension specification is a standard for the hypertext markup language (HTML) to support streaming media operations. The video tag is used to define the video. Illustratively, the browser pulls the code stream data through a Range request, and then processes the pulled code stream data through the MediaSource. And finally, the processed code stream data is delivered to a video label. And the video tag identifies the processed code stream data so as to mark the processed code stream data as video. And finally, playing the video signal by the browser.
According to the technical scheme, the video data in the H264 coding format is encapsulated locally without involving a remote server. Thus, no remote server needs to be deployed. In addition, the browser can adopt a GPU hard decoding and hard display mode, so that the utilization rate of a CPU is reduced, and the influence on the system performance is avoided.
In another specific embodiment, step S255 may further include: and decoding the code stream data to obtain brightness and chrominance (YUV) information of the code stream data under the condition that the format of the code stream data is video data in a non-H264 coding format. It is understood that the video data in the non-H264 encoding format refers to video data in encoding formats other than the H264 encoding format, for example, encoding formats such as H265 or mjpeg (motion Joint Photographic Experts group). As described above, after acquiring the code stream data, the local application process may first determine the format of the code stream data. In this embodiment, the format of the code stream data acquired by the local application process is a non-H264 encoding format. The local application process may perform soft decoding or hard decoding on the coded stream data, and the decoding method may be implemented by using any existing or future developed technology, which is not limited herein. After decoding the code stream data, YUV information can be obtained. Where "Y" represents brightness, i.e., a gray scale value. "U" and "V" represent chromaticity, which information may describe the color and saturation of the image, for specifying the color of a pixel in the image.
In this embodiment, the step S260 of sending the code stream data to the browser to play the audio/video signal by the browser includes: and sending YUV information of the code stream data to the browser so as to be subjected to YUV rendering by the browser.
Illustratively, the browser may perform YUV rendering by using YUV information of the codestream data through webgl (web Graphics library). Among them, WebGL is a 3D drawing protocol. YUV rendering may be specifically achieved by the following steps. 1) Initializing WebGL context initContextGL, and acquiring WebGL rendering context getContext (); 2) initializing shader program initProgram; 3) initializing a data cache initBuffers, wherein the data cache initBuffers comprise a vertex coordinate system and a texture coordinate system; 4) creating texture initTexture; 5) initializing YUV texture initTextures; 6) uploading YUV data to texture updateFrameData (width, height, data); 7) display frame data SR _ DisplayFrameData (width, height, data); 8) setting a display region SR _ setdisplayrect (stdisplayrect); 9) the display resource SR _ Destroy is released. According to the steps, a plurality of rendered video frames can be finally obtained, and the browser can play the video frames sequentially to realize the playing of the video signals.
In the technical scheme, YUV information of the video data in the non-H264 format is sent to the browser, and the browser performs YUV rendering. In the technical scheme, YUV rendering is performed by using WebGL, which is equivalent to GPU hard display. Therefore, the resource consumption of the browser can be greatly reduced.
In another specific embodiment, step S255 further includes: for the case where the code stream data is audio data, the code stream data is decoded into Pulse Code Modulation (PCM) code stream data. The format of the codestream data of the Audio data may be an Advanced Audio Coding (AAC) format, a G711 Coding format, or the like. The decoded code stream data is an analog signal with continuous time and continuous values. The PCM can convert the analog signal into a digital signal that is time-discrete and takes on discrete values. And then, the local application process transmits the audio signal to the browser to realize the playing of the audio signal by the browser.
In the technical scheme, the code stream data of the audio data in various coding formats can be converted into the PCM coding format. Therefore, the decoding process can be simplified, the format of the audio signal acquired by the browser is ensured to be uniform, and the workload of the browser is reduced.
Fig. 3 shows a schematic flow chart of an audiovisual playback method according to another embodiment of the present invention. As shown in fig. 3, the browser establishes a communication connection with the local application process. The local application process receives a playing request instruction sent by the browser through the communication connection, and acquires a streaming address and an output mode of the audio and video signals. And then the local application process accesses the stream taking address to obtain the code stream data of the audio and video signal. And the local application process judges whether the code stream data is in an H264 coding format. When the code stream data is in a non-H264 coding format, the code stream data is decoded into a YUV or PCM format and is sent to a browser for playing. And when the code stream data is in the H264 coding format, packaging the code stream data into the fmp4 format and sending the code stream data to the browser for playing. The above processes have been described in detail, and are not described herein for brevity.
Exemplarily, after the step S250, the method 200 further includes: determining whether code stream data is successfully acquired; and for the condition that the code stream data is not successfully acquired, establishing network connection with the stream taking address and acquiring the code stream data of the audio and video signals from the stream taking address again until the code stream data is successfully acquired.
Firstly, the local application process judges whether the code stream data is successfully acquired from the stream-taking address. It will be appreciated that the fetching operation may fail for some reason, such as the local application process failing to successfully establish a network connection with the fetching address, the fetching address not responding, or the network connection being incorrect. When the local application process does not successfully acquire the code stream data from the stream fetching address, the network connection with the stream fetching address is tried to be established again. And code stream data is acquired again after connection. And repeating the steps until the code stream data is successfully acquired. After the code stream data is successfully acquired, the code stream data can be correspondingly processed, and finally the audio and video signals are played.
The operation of circularly trying to fetch the stream can effectively ensure the probability of accurately and timely playing the audio and video signals under the condition of stream fetching failure, and the phenomenon of playing failure is reduced as much as possible.
Illustratively, the audio-video playing method 200 further includes: and receiving a playing control instruction of the audio and video signal sent by the browser through the communication connection. This step may occur in response to a user's operation at any time during the playing of the audio-video signal by the browser. Controlling the browser to play the audio and video signals according to the playing control instruction
The playing control instruction is used for controlling the playing of the audio and video signals. The playback control instruction may be input by a user via an input device 106, such as a mouse, keyboard, etc., into a browser at the front end, which then sends the playback control instruction to the local application process via the communication connection. The playback control instructions may include one or more of the following instructions: pause playing instruction, resume playing instruction, video recording instruction and video electronic amplification instruction. According to the playing control instruction, the local application process can perform corresponding processing on the code stream data so as to control the playing condition of the browser on the audio/video signals. When the local application process receives the pause play instruction, the transmission of the code stream data can be paused. That is, the browser cannot obtain new code stream data, and cannot continue to play the audio/video signal. Correspondingly, the playing resuming instruction can resume transmission of the code stream data in the pause state, and the transmission starting point is consistent with that before the pause, so as to ensure the continuity of playing the audio and video by the browser. The video recording instruction may indicate that the video played by the browser starts to be recorded from the moment the local application process receives the instruction. And when the local application process receives the video recording instruction again, immediately stopping the video recording process. The video amplification instruction can perform equal-scale amplification on a certain specified area in a video picture so as to ensure that an amplified image is not distorted. The video zoom-in command may include coordinates and a zoom-in magnification of the designated area. After receiving the video amplification instruction, the video image can be cut according to the coordinate information of the designated area, and the cut image is amplified according to the amplification factor, so that the video amplification function is realized.
Therefore, the control of the user on the audio and video signal playing condition can be realized based on the playing control instruction. The watching and listening experience of the user is guaranteed. In addition, the user can input a corresponding playing control instruction to the browser according to the requirement. The different play control instructions can greatly meet different user requirements, and the use experience of the user is further guaranteed.
Illustratively, the method 200 further comprises: responding to the operation of breaking the communication connection of the browser, breaking the network connection and stopping obtaining the code stream data of the audio and video signals from the stream taking address; and clearing the cached code stream data.
For example, the disconnection of the communication connection may be realized by the user closing a browser page, stopping the playing of the current audio-video signal, and the like. And when the browser is disconnected from the communication connection with the local application process, the local application process simultaneously disconnects the network connection with the streaming address of the audio/video signal. And after the network connection is disconnected, the local application process stops acquiring the code stream data of the audio and video signals from the stream taking address. It may indicate that the playback has ended for the audiovisual signal. Since the memory is limited, in order to reduce the memory occupation, after each audio/video signal is played, the local application process can clear the previously cached code stream data so as to cache the new code stream data of the audio/video signal.
Therefore, the usability of the local application process is guaranteed, the memory occupation of the local client is reduced, and the working efficiency of the local client is guaranteed.
Fig. 4 shows a schematic timing diagram of an audiovisual playback method according to an embodiment of the present invention. As shown in fig. 4, first, the browser sends a play request instruction to the local application process in response to the operation of the user. The local application process can acquire the streaming address and the output mode of the audio and video signal based on the playing request instruction. And then the local application process is connected with the stream taking address and carries out stream taking on the audio and video signals based on a corresponding stream taking protocol so as to obtain the code stream data of the audio and video signals. In this process, a phenomenon of stream fetching failure may occur, and the local application process may execute the stream fetching operation in a loop until the stream fetching is successful. And after the stream is successfully acquired, the local application process processes the acquired code stream data according to the output mode of the audio/video signal so as to acquire the code stream data conforming to the output mode. And finally, the local application process sends the processed code stream data to the browser, and the browser plays the audio and video signals. In the playing process, the playing condition of the audio and video signals can be controlled in response to the playing control instruction. And stopping playing the audio and video signals when the audio and video playing is finished or the browser is disconnected from the communication connection with the local application process. And the local application program disconnects the network connection with the stream fetching address and stops fetching the stream. And finally, the local application process clears the cached code stream data, and then completes the playing process of the audio and video signals once. The above processes have been described in detail, and are not described herein for brevity.
According to a second aspect of the present invention, an audio/video playing device is provided. Fig. 5 shows a schematic block diagram of an audiovisual playback device 500 according to an embodiment of the present invention.
As shown in fig. 5, the apparatus 500 includes a first receiving module 510, a first communication module 520, a second receiving module 530, a parsing module 540, a second communication module 550, and a transmitting module 560. The modules may respectively execute the steps/functions of the audio/video playing method described above. Only the main functions of the components of the device 500 are described below, and details that have been described above are omitted.
The first receiving module 510 is configured to listen to the local port and receive a connection request of the browser through the local port.
The first communication module 520 is configured to establish a communication connection with the browser based on the connection request.
The second receiving module 530 is configured to receive, via the communication connection, a play request instruction of an audio/video signal sent by the browser
The parsing module 540 is configured to obtain a stream fetching address of the audio/video signal according to the play request instruction.
The second communication module 550 is configured to establish a network connection with the stream fetching address and obtain code stream data of the audio/video signal from the stream fetching address.
The sending module 560 is configured to send the code stream data to the browser via the communication connection, so that the browser plays the audio/video signal.
The first receiving module 510, the first communication module 520, the second receiving module 530, the parsing module 540, the second communication module 550, and the sending module 560 may all be implemented by the processor 102 in the electronic device shown in fig. 1 executing program instructions stored in the storage device 104.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
According to a third aspect of the present invention, an audio-video playing system is provided. Fig. 6 shows a schematic block diagram of an audiovisual playback system 600 in accordance with an embodiment of the present invention. As shown in fig. 6, system 600 includes a processor 610 and a memory 620.
The memory 620 has stored therein computer program instructions for performing the audiovisual playback method described above when executed by the processor 610.
The processor 610 is configured to run the computer program instructions stored in the memory 620 to perform corresponding steps of the audio and video playing method according to the embodiment of the present invention, and is configured to implement the first receiving module 510, the first communication module 520, the second receiving module 530, the parsing module 540, the second communication module 550, and the sending module 560 in the audio and video playing apparatus according to the embodiment of the present invention.
Illustratively, the computer program instructions constitute a playback component. And after the playing component is installed, starting a background process and keeping resident running.
Therefore, real-time monitoring of the local port can be achieved, and real-time playing of audio and video signals is guaranteed.
According to a fourth aspect of the present invention, a storage medium is provided. Program instructions are stored on a storage medium, and when executed, the program instructions are used for executing the audio/video playing method described above and implementing corresponding modules in the audio/video playing apparatus according to the embodiment of the present invention. The storage medium may include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM), a portable compact disc read only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.
According to a fifth aspect of the invention, there is also provided a computer program product comprising program instructions. The program instructions are operable, when executed, to perform the audiovisual playback method as described above.
Although the illustrative embodiments have been described herein with reference to the accompanying drawings, it is to be understood that the foregoing illustrative embodiments are merely exemplary and are not intended to limit the scope of the invention thereto. Various changes and modifications may be effected therein by one of ordinary skill in the pertinent art without departing from the scope or spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as set forth in the appended claims.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described device embodiments are merely illustrative, and for example, the division of the units is only one logical functional division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another device, or some features may be omitted, or not executed.
In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the invention and aiding in the understanding of one or more of the various inventive aspects. However, the method of the present invention should not be construed to reflect the intent: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
It will be understood by those skilled in the art that all of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where such features are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.
The various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or Digital Signal Processor (DSP) may be used in practice to implement some or all of the functions of some of the modules in an audio-video playback device according to embodiments of the present invention. The present invention may also be embodied as apparatus programs (e.g., computer programs and computer program products) for performing a portion or all of the methods described herein. Such programs implementing the present invention may be stored on computer-readable media or may be in the form of one or more signals. Such a signal may be downloaded from an internet website or provided on a carrier signal or in any other form.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by one and the same item of hardware. The usage of the words first, second and third, etcetera do not indicate any ordering. These words may be interpreted as names.
The above description is only for the specific embodiment of the present invention or the description thereof, and the protection scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and the changes or substitutions should be covered within the protection scope of the present invention. The protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (14)
1. An audio and video playing method comprises the following steps:
monitoring a local port, and receiving a connection request of a browser through the local port;
establishing a communication connection with the browser based on the connection request;
receiving a playing request instruction of an audio and video signal sent by the browser through the communication connection;
acquiring a stream taking address of the audio and video signal according to the playing request instruction;
establishing network connection with the stream taking address and acquiring code stream data of the audio and video signals from the stream taking address; and
and sending the code stream data to the browser through the communication connection so as to play the audio and video signals by the browser.
2. The method of claim 1, wherein the method further comprises:
acquiring an output mode of the audio and video signal according to the playing request instruction;
and after the code stream data of the audio and video signals are acquired from the streaming address and before the code stream data are sent to the browser, processing the code stream data according to the output mode to acquire the code stream data conforming to the output mode.
3. The method according to claim 1 or 2, wherein after said obtaining of codestream data of said audio-video signal from said streaming address and before said sending of said codestream data to said browser, the method further comprises:
for the case that the format of the code stream data is video data in an H264 coding format, packaging the code stream data into code stream data in an fmp4 format;
the playing of the audio and video signals by the browser comprises:
and playing the code stream data by using a video tag conforming to the media source expansion specification.
4. The method according to claim 1 or 2, wherein after said obtaining of codestream data of said audio-video signal from said streaming address and before said sending of said codestream data to said browser, the method further comprises:
decoding the code stream data to obtain brightness and chrominance information of the code stream data under the condition that the format of the code stream data is video data in a non-H264 coding format;
the sending the code stream data to the browser so that the browser plays the audio and video signals comprises:
and sending the brightness and the chrominance information of the code stream data to the browser so as to render the brightness and the chrominance by the browser.
5. The method according to claim 1 or 2, wherein after said obtaining of codestream data of said audio-video signal from said streaming address and before said sending of said codestream data to said browser, the method further comprises:
and decoding the code stream data into code stream data modulated by pulse codes under the condition that the code stream data is audio data.
6. The method of any of claims 1 to 5, wherein the method further comprises: acquiring a stream taking protocol of the audio and video signals according to the playing request instruction;
the acquiring of the code stream data of the audio and video signal from the stream taking address comprises: and acquiring the code stream data from the stream taking address based on the stream taking protocol.
7. The method of any one of claims 1 to 6, wherein after the establishing of the network connection with the streaming address and the acquiring of the streaming data of the audio and video signal from the streaming address, the method further comprises:
determining whether the code stream data is successfully acquired;
and for the condition that the code stream data is not successfully acquired, the establishment of the network connection with the stream taking address is executed again, and the code stream data of the audio and video signals are acquired from the stream taking address until the code stream data is successfully acquired.
8. The method of any of claims 1 to 7, wherein the method further comprises:
receiving a playing control instruction of an audio and video signal sent by the browser through the communication connection;
and controlling the browser to play the audio and video signals according to the playing control instruction.
9. The method of claim 8, wherein the playback control instructions include one or more of the following instructions: pause playing instruction, resume playing instruction, video recording instruction and video electronic amplification instruction.
10. The method of any one of claims 1 to 9, wherein the method further comprises:
responding to the operation of the browser for disconnecting the communication connection, disconnecting the network connection and stopping the acquisition of the code stream data of the audio and video signals from the streaming address; and
and clearing the cached code stream data.
11. An audiovisual playback system comprising a processor and a memory, wherein the memory has stored therein computer program instructions for execution by the processor to perform an audiovisual playback method as claimed in any of claims 1 to 10.
12. The system of claim 11, wherein the computer program instructions constitute a playback component, wherein the playback component, upon completion of installation, initiates a background process and remains resident.
13. A storage medium having stored thereon program instructions for performing, when running, the audiovisual playback method of any of claims 1-10.
14. A computer program product comprising program instructions for performing, when running, the audio-visual playback method as claimed in any one of claims 1 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111094908.XA CN114040251A (en) | 2021-09-17 | 2021-09-17 | Audio and video playing method, system, storage medium and computer program product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111094908.XA CN114040251A (en) | 2021-09-17 | 2021-09-17 | Audio and video playing method, system, storage medium and computer program product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114040251A true CN114040251A (en) | 2022-02-11 |
Family
ID=80134516
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111094908.XA Pending CN114040251A (en) | 2021-09-17 | 2021-09-17 | Audio and video playing method, system, storage medium and computer program product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114040251A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114285836A (en) * | 2022-03-03 | 2022-04-05 | 苏州万店掌网络科技有限公司 | Video playing method, device and medium |
| CN115209222A (en) * | 2022-06-15 | 2022-10-18 | 深圳市锐明技术股份有限公司 | Video playing method and device, electronic equipment and readable storage medium |
| CN115904462A (en) * | 2023-02-10 | 2023-04-04 | 中航金网(北京)电子商务有限公司 | Application program upgrading method and device, electronic equipment and readable storage medium |
| CN116055789A (en) * | 2023-03-24 | 2023-05-02 | 杭州星犀科技有限公司 | Live broadcast picture amplifying method, system, equipment and medium based on android system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20180086112A (en) * | 2017-01-20 | 2018-07-30 | 한화에어로스페이스 주식회사 | Apparatus and method for playing back and seeking media in web browser |
| CN111372128A (en) * | 2020-03-11 | 2020-07-03 | 北京旷视科技有限公司 | Video playing method and device, computer equipment and readable storage medium |
| CN112073809A (en) * | 2020-08-09 | 2020-12-11 | 富盛科技股份有限公司 | Method for supporting browser to play video with any coding format |
| CN112312221A (en) * | 2019-07-31 | 2021-02-02 | 广州弘度信息科技有限公司 | Audio and video playing method, storage medium and device |
| CN112422508A (en) * | 2020-10-19 | 2021-02-26 | 武汉烽火众智数字技术有限责任公司 | Method and system for playing video in browser webpage |
-
2021
- 2021-09-17 CN CN202111094908.XA patent/CN114040251A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20180086112A (en) * | 2017-01-20 | 2018-07-30 | 한화에어로스페이스 주식회사 | Apparatus and method for playing back and seeking media in web browser |
| CN112312221A (en) * | 2019-07-31 | 2021-02-02 | 广州弘度信息科技有限公司 | Audio and video playing method, storage medium and device |
| CN111372128A (en) * | 2020-03-11 | 2020-07-03 | 北京旷视科技有限公司 | Video playing method and device, computer equipment and readable storage medium |
| CN112073809A (en) * | 2020-08-09 | 2020-12-11 | 富盛科技股份有限公司 | Method for supporting browser to play video with any coding format |
| CN112422508A (en) * | 2020-10-19 | 2021-02-26 | 武汉烽火众智数字技术有限责任公司 | Method and system for playing video in browser webpage |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114285836A (en) * | 2022-03-03 | 2022-04-05 | 苏州万店掌网络科技有限公司 | Video playing method, device and medium |
| CN115209222A (en) * | 2022-06-15 | 2022-10-18 | 深圳市锐明技术股份有限公司 | Video playing method and device, electronic equipment and readable storage medium |
| CN115209222B (en) * | 2022-06-15 | 2024-02-09 | 深圳市锐明技术股份有限公司 | Video playing method and device, electronic equipment and readable storage medium |
| CN115904462A (en) * | 2023-02-10 | 2023-04-04 | 中航金网(北京)电子商务有限公司 | Application program upgrading method and device, electronic equipment and readable storage medium |
| CN116055789A (en) * | 2023-03-24 | 2023-05-02 | 杭州星犀科技有限公司 | Live broadcast picture amplifying method, system, equipment and medium based on android system |
| CN116055789B (en) * | 2023-03-24 | 2023-08-11 | 杭州星犀科技有限公司 | Live broadcast picture amplifying method, system, equipment and medium based on android system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114040251A (en) | Audio and video playing method, system, storage medium and computer program product | |
| CN109327727B (en) | Live stream processing method in WebRTC and stream pushing client | |
| US20220263885A1 (en) | Adaptive media streaming method and apparatus according to decoding performance | |
| US10567809B2 (en) | Selective media playing method and apparatus according to live streaming and recorded streaming | |
| CN110324671B (en) | Webpage video playing method and device, electronic equipment and storage medium | |
| US10979785B2 (en) | Media playback apparatus and method for synchronously reproducing video and audio on a web browser | |
| US8639086B2 (en) | Rendering of video based on overlaying of bitmapped images | |
| WO2022257699A1 (en) | Image picture display method and apparatus, device, storage medium and program product | |
| CN113542757B (en) | Image transmission method and device for cloud application, server and storage medium | |
| US20160248833A1 (en) | System and method for playing a video on mobile web environments | |
| EP2409223A2 (en) | Hosted application platform with extensible media format | |
| CN110505511B (en) | Method, device and system for playing video in webpage and computing equipment | |
| CN113938470B (en) | Method and device for playing RTSP data source by browser and streaming media server | |
| CN108337246B (en) | Media playback apparatus and media service apparatus preventing playback delay | |
| CN112843676B (en) | Data processing method, device, terminal, server and storage medium | |
| WO2017080175A1 (en) | Multi-camera used video player, playing system and playing method | |
| CN111464828A (en) | Virtual special effect display method, device, terminal and storage medium | |
| CN104010204B (en) | Image information processing method and device | |
| CN114302176A (en) | Video playing method and device | |
| KR102417055B1 (en) | Method and device for post processing of a video stream | |
| CN111083527A (en) | Video playing method and device of application, storage medium and electronic equipment | |
| CN116347158A (en) | Video playing method and device, electronic equipment and computer readable storage medium | |
| US10547878B2 (en) | Hybrid transmission protocol | |
| KR102247892B1 (en) | System for cloud streaming service, method of image cloud streaming service using application code and apparatus for the same | |
| CN114222185A (en) | Video playing method, terminal equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |