CN115988234A - Audio and video processing method and system - Google Patents

Abstract

The invention relates to an audio and video processing method. The method includes: after the browser receives a trigger operation, sends a message to a server, and the server establishes a connection with a mobile terminal; the server receives and processes the video sent by the mobile terminal. stream and audio stream data, and send the video stream data and the audio stream data to the browser; the browser stores the video stream data in the cache, and obtains the web page decoding script and The webpage rendering script analyzes the video stream data and displays the video stream data in real time; the browser plays the audio stream data in real time through an audio decoder. The method can realize high frame rate video transmission of iOS devices, and supports audio synchronization.

Description

Audio and video processing method and system

technical field

The present invention relates to the field of computer technology, in particular to an audio and video processing method and system.

Background technique

The cloud real machine platform can remotely share and use mobile devices. Through centralized management of resources, reasonable scheduling and allocation, resource utilization can be improved, cost and demand can be balanced, and research and development efficiency can be improved. In terms of audio and video processing and transmission, the existing cloud real machine platform does not have perfect support for iOS devices. The screen image acquisition is mainly through installing driver software on the device, such as webdriveragent, and then taking screenshots of the device, and then transmitting them to the client for browsing On the server, and most cloud real machine platforms do not support audio synchronization, which brings great inconvenience to the test task.

Contents of the invention

In order to solve the deficiencies of the prior art, the present invention proposes an audio and video processing method and system, which intends to realize high frame rate video transmission of iOS devices and support audio synchronization.

For realizing the above object, the technical scheme adopted in the present invention comprises:

The first aspect of the present invention discloses an audio and video processing method, including:

After the browser receives the trigger operation, it sends a message to the server, and the server establishes a connection with the mobile terminal;

The server receives and processes the video stream and audio stream data sent by the mobile terminal, and sends the video stream data and the audio stream data to the browser;

The browser stores the video stream data in the cache, obtains the webpage decoding script and the webpage rendering script from the server to analyze the video stream data, and displays the video stream data in real time; The decoder plays the audio stream data in real time.

Further, the mobile terminal is an iOS device.

Further, after receiving the trigger operation, the browser sends a websocket message to the server, and the server establishes a connection by sending a NEED data packet to the mobile terminal.

Further, the processing method of the audio stream data includes the following steps:

The server adopts Waveform Audio File Format to process the audio stream data;

Input the processed data into ffmpeg for parsing;

The audio stream data output by ffmpeg is divided into audio blocks and packaged into blobs and sent to the browser.

Further, the processing method of the video stream data includes the following steps:

The server uses H.264 network abstraction layer unit to process payload data;

Input the processed data into ffmpeg for analysis, and control the video resolution, frame rate and bit rate;

Send the video stream data output by ffmpeg to the browser.

The second aspect of the present invention discloses an audio and video processing system, comprising:

The data connection module, after the browser receives the trigger operation, sends a message to the server, and the server establishes a connection with the mobile terminal;

A data processing module, the server receives and processes the video stream and audio stream data sent by the mobile terminal, and sends the video stream data and the audio stream data to the browser;

A data decoding module, the browser stores the video stream data in the cache, obtains a webpage decoding script and a webpage rendering script from the server to analyze the video stream data, and displays the video stream data in real time; The browser plays the audio stream data in real time through the audio decoder.

The third aspect of the present invention discloses a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned method is realized.

The fourth aspect of the present invention discloses an electronic device, including a processor and a memory;

The memory is used to store operation instructions;

The processor is configured to execute the above method by calling the operation instruction.

The fifth aspect of the present invention discloses a computer program product, including computer programs and/or instructions, characterized in that, when the computer programs and/or instructions are executed by a processor, the steps of the above method are realized.

The beneficial effects of the present invention are:

Using the audio and video processing method and system of the present invention, the method encodes the screen display data of the iOS terminal device and provides the encoded video data to the client using the iOS terminal device remotely, without installing any application or embedded code on the iOS terminal device , while achieving high frame rate, high image quality, low latency, and synchronous audio transmission.

Description of drawings

FIG. 1 is a schematic flowchart of an audio and video processing method provided by an embodiment of the present invention.

FIG. 2 is a schematic structural diagram of an audio and video processing system provided by an embodiment of the present invention.

Detailed ways

In order to enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only It is an embodiment of a part of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

It should be noted that the terms "first" and "second" in the description and claims of the present invention and the above drawings are used to distinguish similar objects, but not necessarily used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements is not necessarily limited to the expressly listed instead, may include other steps or elements not explicitly listed or inherent to the process, method, product or apparatus.

The first aspect of the present invention relates to an audio and video processing method as shown in Figure 1, specifically comprising:

Step S1. After receiving the trigger operation, the browser sends a message to the server, and the server establishes a connection with the mobile terminal.

Wherein, the browser sends a websocket message to the server after receiving the trigger operation of clicking "use mobile phone" by the client, and the server establishes a connection by sending a NEED data packet to the mobile terminal.

The mobile terminal is an iOS device, specifically an iOS mobile phone device.

Before the step S1, the method further includes establishing a connection between the mobile terminal and the server through the USB protocol, enabling the configuration for transmitting screen audio and video data, and completing the interactive operation.

Step S2, the server receives and processes the video stream and audio stream data sent by the mobile terminal, and sends the video stream data and the audio stream data to the browser.

Preferably, the processing method of the audio stream data comprises the following steps:

The server adopts Waveform Audio File Format (wave) to process the audio stream data;

Input the processed data into ffmpeg for parsing;

The audio stream data output by ffmpeg is divided into audio blocks and packaged into blobs and sent to the browser.

Preferably, the processing method of the video stream data comprises the following steps:

The server uses H.264 Network Abstraction Layer Unit (NALU) to process payload data (Payload);

Input the processed data into ffmpeg for analysis, and control the video resolution, frame rate and bit rate;

Send the video stream data output by ffmpeg to the browser.

Step S3, the browser stores the video stream data in the cache, obtains a webpage decoding script and a webpage rendering script from the server to analyze the video stream data, and displays the video stream data in real time; the browsing The device plays the audio stream data in real time through the audio decoder.

Wherein, the browser stores the received H.264 video stream data in the cache after determining that it is a complete NALU.

Using the audio and video processing method and system of the present invention, the method encodes the screen display data of the iOS terminal device and provides the encoded video data to the client using the iOS terminal device remotely, without installing any application or embedded code on the iOS terminal device , while achieving high frame rate, high image quality, low latency, and synchronous audio transmission.

The present invention also relates to an audio and video processing system as shown in Figure 2, the structure comprising:

The data connection module, after the browser receives the trigger operation, sends a message to the server, and the server establishes a connection with the mobile terminal;

A data processing module, the server receives and processes the video stream and audio stream data sent by the mobile terminal, and sends the video stream data and the audio stream data to the browser;

A data decoding module, the browser stores the video stream data in the cache, obtains a webpage decoding script and a webpage rendering script from the server to analyze the video stream data, and displays the video stream data in real time; The browser plays the audio stream data in real time through the audio decoder.

By using the system, the above-mentioned audio and video processing method can be implemented and corresponding technical effects can be achieved.

When using the above method and system to specifically execute audio and video transmission, reference may be made to the following specific embodiments.

1. Initialize the session

(1) Enable hidden device configuration information

(2) Lock and open the transmission endpoint

(3) Waiting to receive PING packets

(4) Respond with PING packet

(5) Wait for the SYNC CWPA packet to receive the device audio timestamp

(6) Create a local timestamp record, put the timestamp into SYNC CWPA and send it

(7) Send ASYN_HPD1

(8) Send ASYN_HPA1

(9) Receive synchronous AFMT and return no error signal

(10) Receive CVRP video timestamp

(11) Reply with local video timestamp

(12) Send a NEED message using the timestamp of step (10)

(13) Receive two ASYN

(14) Receive the CLOK message, create a new timestamp record and reply to the message

(15) Receive the TIME message, use the timestamp created in step (14) and reply to the message

2. Receive data

Video and audio data will be sent by the device, and video NEED packets need to be sent periodically

3. Close the data stream

(1) Send ASYN_HPA0 with the device timestamp in the CWPA SYNC packet to tell the device to stop sending audio data

(2) Send ASYN_HPD0 with empty timestamp to stop video data

(3) Receive stop SYNC packet

(4) Respond to the stop SYNC data packet whose 8 bits are 0

(5) Receive ASYN_RELS of a local video timestamp

(6) Receive an ASYN_RELS with a local timestamp created after the SYNC CLOK message

(7) Release the usb endpoint

(8) Set the device active configuration to usbmux

Embodiments of the present invention also provide a computer-readable storage medium capable of implementing all the steps in the methods in the above-mentioned embodiments, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the above-mentioned All steps of the method in the embodiment.

An embodiment of the present invention also provides an electronic device for performing the above method. As an implementation device of the method, the electronic device is at least equipped with a processor and a memory, and in particular, the memory stores data required for executing the method. and related computer programs, and all the steps of the method are implemented by the processor calling the data in the memory, and the program executes, and corresponding technical effects are obtained.

Preferably, the electronic device may include a bus architecture, and the bus may include any number of interconnected buses and bridges, and the bus may link together various circuits including one or more processors and memories. The bus may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, all of which are well known in the art and therefore will not be further described herein. The bus interface provides the interface between the bus and the receiver and transmitter. The receiver and transmitter can be the same element, a transceiver, providing means for communicating with various other systems over a transmission medium. The processor is responsible for managing the bus and general processing, while memory may be used to store data used by the processor when performing operations.

Additionally, the electronic device may further include components such as a communication module, an input unit, an audio processor, a display, and a power supply. The processor (or called controller, operation control) adopted by it may include a microprocessor or other processor devices and/or logic devices, which receive input and control the operation of various components of the electronic equipment; the memory may be One or more of buffer memory, flash memory, hard drive, removable media, volatile memory, non-volatile memory or other suitable devices, which can store the above-mentioned relevant data information, and can also store execution-related information program, and the processor can execute the program stored in the memory to realize information storage or processing, etc.; the input unit is used to provide input to the processor, such as a button or a touch input device; the power supply is used to provide power to electronic equipment ; The display is used for displaying display objects such as images and text, for example, it may be an LCD display. A communication module is a transmitter/receiver that sends and receives signals via an antenna. A communication module (transmitter/receiver) is coupled to the processor to provide input signals and receive output signals, which may be the same as in conventional mobile communication terminals. Based on different communication technologies, multiple communication modules, such as a cellular network module, a bluetooth module and/or a wireless local area network module, may be provided in the same electronic device. The communication module (transmitter/receiver) is also coupled to a speaker and a microphone via an audio processor to provide audio output via the speaker and receive audio input from the microphone for usual telecommunication functions. Audio processors may include any suitable buffers, decoders, amplifiers, etc. In addition, the audio processor is also coupled to the central processing unit, so that the recording on the machine can be made through the microphone, and the sound stored on the machine can be played through the speaker.

Those skilled in the art should understand that the embodiments of the present invention may be provided as methods, systems, or computer program products. Accordingly, the present invention can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a A system for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising a system of instructions, the The system implements the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams. While preferred embodiments of the present invention have been described, additional changes and modifications can be made to these embodiments by those skilled in the art once the basic inventive concept is appreciated. Therefore, it is intended that the appended claims be construed to cover the preferred embodiment as well as all changes and modifications which fall within the scope of the invention.

The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto, any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention etc. should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the protection scope of the claims.

Claims (9)

1. An audio-video processing method, characterized by comprising:

after receiving the trigger operation, the browser sends a message to a server, and the server is connected with the mobile terminal;

the server receives and processes video stream data and audio stream data sent by the mobile terminal, and sends the video stream data and the audio stream data to the browser;

the browser stores the video stream data into a cache, acquires a webpage decoding script and a webpage drawing script from the server to analyze the video stream data, and displays the video stream data in real time; and the browser plays the audio stream data in real time through an audio decoder.

2. The method of claim 1, wherein the mobile terminal is an iOS device.

3. The method of claim 1, wherein in step S1, after receiving the trigger operation, the browser sends a websocket message to a server, and the server establishes a connection by sending a NEED packet to the mobile terminal.

4. The method according to claim 1, wherein in the step S2, the processing method of the audio stream data comprises the steps of:

the server processes the Audio stream data by adopting a Waveform Audio File Format;

inputting the processed data into ffmpeg for analysis;

and segmenting the audio stream data output by the ffmpeg into audio blocks, packaging the audio blocks into blobs and sending the blobs to the browser.

5. The method according to any one of claims 1 to 4, wherein in the step S2, the processing method of the video stream data comprises the following steps:

the server adopts an H.264 network abstraction layer unit to process the effective load data;

inputting the processed data into ffmpeg for analysis, and controlling the video resolution, the frame rate and the code rate;

and sending the video stream data output by the ffmpeg to the browser.

6. An audio-video processing system, comprising:

the data connection module is used for sending a message to the server after the browser receives the triggering operation, and the server is connected with the mobile terminal;

the server receives and processes video stream data and audio stream data sent by the mobile terminal, and sends the video stream data and the audio stream data to the browser;

the browser stores the video stream data into a cache, acquires a webpage decoding script and a webpage drawing script from the server to analyze the video stream data, and displays the video stream data in real time; and the browser plays the audio stream data in real time through an audio decoder.

7. A computer-readable storage medium, characterized in that a computer program is stored on the storage medium, which computer program, when being executed by a processor, carries out the method of any one of claims 1 to 5.

8. An electronic device comprising a processor and a memory;

the memory is used for storing operation instructions;

the processor is used for executing the method of any one of claims 1 to 5 by calling the operation instruction.

9. A computer program product comprising a computer program and/or instructions, characterized in that the computer program and/or instructions, when executed by a processor, implement the steps of the method of any one of claims 1 to 5.

Images