CN112423076B - Audio screen-throwing synchronous control method, equipment and computer readable storage medium - Google Patents
Audio screen-throwing synchronous control method, equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN112423076B CN112423076B CN202011293135.3A CN202011293135A CN112423076B CN 112423076 B CN112423076 B CN 112423076B CN 202011293135 A CN202011293135 A CN 202011293135A CN 112423076 B CN112423076 B CN 112423076B
- Authority
- CN
- China
- Prior art keywords
- screen
- audio
- throwing
- data
- channel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
- H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/433—Content storage operation, e.g. storage operation in response to a pause request, caching operations
- H04N21/4334—Recording operations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/439—Processing of audio elementary streams
Abstract
The invention discloses an audio screen-throwing synchronous control method, equipment and a computer readable storage medium, wherein the method comprises the following steps: when the cast end starts to transmit the cast data to the cast end through a first channel in the cast connection, starting audio recording of the cast end through the audio recording service; then, acquiring audio data in real time through the audio recording service, and transmitting the audio data to the screen throwing end through a second channel in the screen throwing connection by the thrown end; and finally, synchronously and real-timely analyzing the audio data when the screen throwing end renders the screen throwing data in real time. The humanized audio screen-throwing synchronous control scheme is realized, so that when the screen is thrown to the screen-throwing end for reversely throwing the screen, synchronous reverse screen throwing of pictures and audio can be realized, the reverse screen-throwing function is further perfected, and the screen-throwing use experience of a user is enhanced.
Description
Technical Field
The present invention relates to the field of mobile communications, and in particular, to a method and apparatus for synchronous control of audio projection, and a computer readable storage medium.
Background
In the prior art, with the continuous development of intelligent terminal equipment, the demand of users for screen projection is also higher and higher, and at present, some related software products for reverse screen projection exist on the market, namely, when a mobile phone is projected to a computer, the computer controls the mobile phone. However, in the above scheme, the function of supporting audio synchronous screen projection is not available, that is, the user can only silently synchronize the display content of the mobile phone to the computer terminal, but cannot synchronously project the audio data acquired by the mobile phone terminal to the computer terminal, so it can be seen that the reverse screen projection in the prior art cannot bring more complete and more perfect screen projection use experience to the user in audio synchronization.
Disclosure of Invention
In order to solve the technical defects in the prior art, the invention provides an audio screen-throwing synchronous control method, which comprises the following steps:
when a screen throwing connection is established between a thrown end and a screen throwing end, creating an audio recording service of the thrown end;
when the cast end starts to transmit the cast data to the cast end through a first channel in the cast connection, starting audio recording of the cast end through the audio recording service;
acquiring audio data in real time through the audio recording service, and transmitting the audio data to the screen throwing end through a second channel in the screen throwing connection by the thrown end;
And synchronously and real-timely analyzing the audio data when the screen throwing end renders the screen throwing data in real time.
Optionally, when the cast end establishes the cast connection with the cast end, creating the audio recording service of the cast end includes:
when the screen-throwing end and the screen-throwing end are connected, receiving an audio screen-throwing initial instruction sent by the screen-throwing end to the screen-throwing end;
and creating the audio recording service at the thrown end according to the audio screen throwing initial instruction.
Optionally, when the cast end establishes the cast connection with the cast end, creating an audio recording service of the cast end further includes:
when the screen-throwing end establishes screen-throwing connection with the screen-throwing end, a first channel for transmitting the screen-throwing data is created;
and creating a second channel for transmitting the audio data.
Optionally, when the cast end starts to transmit the cast data to the cast end through a first channel in the cast connection, starting audio recording of the cast end through the audio recording service, including:
monitoring the transmission state of the screen throwing data;
and when the screen-throwing data starts transmission, starting the audio recording service.
Optionally, the acquiring the audio data in real time through the audio recording service, and transmitting the audio data to the screen-throwing end by the thrown end through a second channel in the screen-throwing connection, includes:
acquiring audio sampling data in real time through the audio recording service;
and carrying out compression coding on the audio sampling data according to a preset compression coding format to obtain the audio data.
Optionally, the acquiring the audio data in real time through the audio recording service, and transmitting the audio data to the screen-throwing end by the thrown end through a second channel in the screen-throwing connection, further includes:
presetting a network stream format corresponding to the audio data;
transmitting the audio data in the second channel in the network stream format.
Optionally, when the screen projection end renders the screen projection data in real time, synchronizing and analyzing the audio data in real time includes:
when the screen throwing data are received by the screen throwing device, rendering and displaying a screen throwing picture in real time;
and when the screen throwing end receives the audio data, restoring the audio data into the audio sampling data.
Optionally, when the screen projection end renders the screen projection data in real time, the audio data is synchronized and analyzed in real time, and the method further includes:
Synchronizing a play instruction and a rendering instruction of the audio sampling data and the screen throwing data;
and playing the audio sampling data according to the playing instruction, and simultaneously, rendering the screen throwing data according to the rendering instruction.
The invention also provides an audio screen synchronization control device, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the computer program realizes the steps of the audio screen synchronization control method according to any one of the above steps when being executed by the processor.
The invention also provides a computer readable storage medium, wherein the computer readable storage medium stores an audio screen-on synchronous control program, and the audio screen-on synchronous control program realizes the steps of the audio screen-on synchronous control method according to any one of the above steps when being executed by a processor.
By implementing the audio screen-projection synchronous control method, the equipment and the computer readable storage medium, the audio recording service of the screened terminal is created when the screened terminal establishes screen-projection connection with the screened terminal; then, when the cast end starts to transmit the cast data to the cast end through a first channel in the cast connection, starting audio recording of the cast end through the audio recording service; then, acquiring audio data in real time through the audio recording service, and transmitting the audio data to the screen throwing end through a second channel in the screen throwing connection by the thrown end; and finally, synchronously and real-timely analyzing the audio data when the screen throwing end renders the screen throwing data in real time. The humanized audio screen-throwing synchronous control scheme is realized, so that when the screen is thrown to the screen-throwing end for reversely throwing the screen, synchronous reverse screen throwing of pictures and audio can be realized, the reverse screen-throwing function is further perfected, and the screen-throwing use experience of a user is enhanced.
Drawings
The invention will be further described with reference to the accompanying drawings and examples, in which:
fig. 1 is a schematic diagram of a hardware structure of a mobile terminal according to the present invention;
fig. 2 is a schematic diagram of a communication network system according to an embodiment of the present invention;
FIG. 3 is a flowchart of a first embodiment of an audio projection synchronization control method of the present invention;
FIG. 4 is a flowchart of a second embodiment of the audio projection synchronization control method of the present invention;
FIG. 5 is a flowchart of a third embodiment of an audio projection synchronization control method of the present invention;
FIG. 6 is a flowchart of a fourth embodiment of the audio projection synchronization control method of the present invention;
FIG. 7 is a flowchart of a fifth embodiment of the audio projection synchronization control method of the present invention;
FIG. 8 is a flowchart of a sixth embodiment of an audio projection synchronization control method of the present invention;
FIG. 9 is a flowchart of a seventh embodiment of an audio projection synchronization control method of the present invention;
fig. 10 is a flowchart of an eighth embodiment of the audio projection synchronization control method of the present invention.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
In the following description, suffixes such as "module", "component", or "unit" for representing elements are used only for facilitating the description of the present invention, and have no specific meaning per se. Thus, "module," "component," or "unit" may be used in combination.
The terminal may be implemented in various forms. For example, the terminals described in the present invention may include mobile terminals such as cell phones, tablet computers, notebook computers, palm computers, personal digital assistants (Personal Digital Assistant, PDA), portable media players (Portable Media Player, PMP), navigation devices, wearable devices, smart bracelets, pedometers, and fixed terminals such as digital TVs, desktop computers, and the like.
The following description will be given taking a mobile terminal as an example, and those skilled in the art will understand that the configuration according to the embodiment of the present invention can be applied to a fixed type terminal in addition to elements particularly used for a moving purpose.
Referring to fig. 1, which is a schematic diagram of a hardware structure of a mobile terminal implementing various embodiments of the present invention, the mobile terminal 100 may include: an RF (Radio Frequency) unit 101, a WiFi module 102, an audio output unit 103, an a/V (audio/video) input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, a processor 110, and a power supply 111. Those skilled in the art will appreciate that the mobile terminal structure shown in fig. 1 is not limiting of the mobile terminal and that the mobile terminal may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.
The following describes the components of the mobile terminal in detail with reference to fig. 1:
the radio frequency unit 101 may be used for receiving and transmitting signals during the information receiving or communication process, specifically, after receiving downlink information of the base station, processing the downlink information by the processor 110; and, the uplink data is transmitted to the base station. Typically, the radio frequency unit 101 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, and the like. In addition, the radio frequency unit 101 may also communicate with networks and other devices via wireless communications. The wireless communication may use any communication standard or protocol, including but not limited to GSM (Global System of Mobile communication, global System for Mobile communications), GPRS (General Packet Radio Service ), CDMA2000 (Code Division Multiple Access, CDMA 2000), WCDMA (Wideband Code Division Multiple Access ), TD-SCDMA (Time Division-Synchronous Code Division Multiple Access, time Division synchronous code Division multiple Access), FDD-LTE (Frequency Division Duplexing-Long Term Evolution, frequency Division Duplex Long term evolution), and TDD-LTE (Time Division Duplexing-Long Term Evolution, time Division Duplex Long term evolution), etc.
WiFi belongs to a short-distance wireless transmission technology, and a mobile terminal can help a user to send and receive e-mails, browse web pages, access streaming media and the like through the WiFi module 102, so that wireless broadband Internet access is provided for the user. Although fig. 1 shows a WiFi module 102, it is understood that it does not belong to the necessary constitution of a mobile terminal, and can be omitted entirely as required within a range that does not change the essence of the invention.
The audio output unit 103 may convert audio data received by the radio frequency unit 101 or the WiFi module 102 or stored in the memory 109 into an audio signal and output as sound when the mobile terminal 100 is in a call signal reception mode, a talk mode, a recording mode, a voice recognition mode, a broadcast reception mode, or the like. Also, the audio output unit 103 may also provide audio output (e.g., a call signal reception sound, a message reception sound, etc.) related to a specific function performed by the mobile terminal 100. The audio output unit 103 may include a speaker, a buzzer, and the like.
The a/V input unit 104 is used to receive an audio or video signal. The a/V input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, the graphics processor 1041 processing image data of still pictures or video obtained by an image capturing device (e.g., a camera) in a video capturing mode or an image capturing mode. The processed image frames may be displayed on the display unit 106. The image frames processed by the graphics processor 1041 may be stored in the memory 109 (or other storage medium) or transmitted via the radio frequency unit 101 or the WiFi module 102. The microphone 1042 can receive sound (audio data) via the microphone 1042 in a phone call mode, a recording mode, a voice recognition mode, and the like, and can process such sound into audio data. The processed audio (voice) data may be converted into a format output that can be transmitted to the mobile communication base station via the radio frequency unit 101 in the case of a telephone call mode. The microphone 1042 may implement various types of noise cancellation (or suppression) algorithms to cancel (or suppress) noise or interference generated in the course of receiving and transmitting the audio signal.
The mobile terminal 100 also includes at least one sensor 105, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 1061 according to the brightness of ambient light, and the proximity sensor can turn off the display panel 1061 and/or the backlight when the mobile terminal 100 moves to the ear. As one of the motion sensors, the accelerometer sensor can detect the acceleration in all directions (generally three axes), and can detect the gravity and direction when stationary, and can be used for applications of recognizing the gesture of a mobile phone (such as horizontal and vertical screen switching, related games, magnetometer gesture calibration), vibration recognition related functions (such as pedometer and knocking), and the like; as for other sensors such as fingerprint sensors, pressure sensors, iris sensors, molecular sensors, gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that may also be configured in the mobile phone, the detailed description thereof will be omitted.
The display unit 106 is used to display information input by a user or information provided to the user. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display (Liquid Crystal Display, LCD), an Organic Light-Emitting Diode (OLED), or the like.
The user input unit 107 may be used to receive input numeric or character information and to generate key signal inputs related to user settings and function control of the mobile terminal. In particular, the user input unit 107 may include a touch panel 1071 and other input devices 1072. The touch panel 1071, also referred to as a touch screen, may collect touch operations thereon or thereabout by a user (e.g., operations of the user on the touch panel 1071 or thereabout by using any suitable object or accessory such as a finger, a stylus, etc.) and drive the corresponding connection device according to a predetermined program. The touch panel 1071 may include two parts of a touch detection device and a touch controller. The touch detection device detects the touch azimuth of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends the touch point coordinates to the processor 110, and can receive and execute commands sent from the processor 110. Further, the touch panel 1071 may be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. The user input unit 107 may include other input devices 1072 in addition to the touch panel 1071. In particular, other input devices 1072 may include, but are not limited to, one or more of a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, mouse, joystick, etc., as specifically not limited herein.
Further, the touch panel 1071 may overlay the display panel 1061, and when the touch panel 1071 detects a touch operation thereon or thereabout, the touch panel 1071 is transferred to the processor 110 to determine the type of touch event, and then the processor 110 provides a corresponding visual output on the display panel 1061 according to the type of touch event. Although in fig. 1, the touch panel 1071 and the display panel 1061 are two independent components for implementing the input and output functions of the mobile terminal, in some embodiments, the touch panel 1071 may be integrated with the display panel 1061 to implement the input and output functions of the mobile terminal, which is not limited herein.
The interface unit 108 serves as an interface through which at least one external device can be connected with the mobile terminal 100. For example, the external devices may include a wired or wireless headset port, an external power (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device having an identification module, an audio input/output (I/O) port, a video I/O port, an earphone port, and the like. The interface unit 108 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the mobile terminal 100 or may be used to transmit data between the mobile terminal 100 and an external device.
The processor 110 is a control center of the mobile terminal, connects various parts of the entire mobile terminal using various interfaces and lines, and performs various functions of the mobile terminal and processes data by running or executing software programs and/or modules stored in the memory 109 and calling data stored in the memory 109, thereby performing overall monitoring of the mobile terminal. Processor 110 may include one or more processing units; preferably, the processor 110 may integrate an application processor that primarily handles operating systems, user interfaces, applications, etc., with a modem processor that primarily handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 110.
The mobile terminal 100 may further include a power source 111 (e.g., a battery) for supplying power to the respective components, and preferably, the power source 111 may be logically connected to the processor 110 through a power management system, so as to perform functions of managing charging, discharging, and power consumption management through the power management system.
Although not shown in fig. 1, the mobile terminal 100 may further include a bluetooth module or the like, which is not described herein.
In order to facilitate understanding of the embodiments of the present invention, a communication network system on which the mobile terminal of the present invention is based will be described below.
Referring to fig. 2, fig. 2 is a schematic diagram of a communication network system according to an embodiment of the present invention, where the communication network system is an LTE system of a general mobile communication technology, and the LTE system includes a UE (User Equipment) 201, an e-UTRAN (Evolved UMTS Terrestrial Radio Access Network ) 202, an epc (Evolved Packet Core, evolved packet core) 203, and an IP service 204 of an operator that are sequentially connected in communication.
Specifically, the UE201 may be the terminal 100 described above, and will not be described herein.
The E-UTRAN202 includes eNodeB2021 and other eNodeB2022, etc. The eNodeB2021 may be connected with other eNodeB2022 by a backhaul (e.g., an X2 interface), the eNodeB2021 is connected to the EPC203, and the eNodeB2021 may provide access from the UE201 to the EPC 203.
EPC203 may include MME (Mobility Management Entity ) 2031, hss (Home Subscriber Server, home subscriber server) 2032, other MMEs 2033, SGW (Serving Gate Way) 2034, pgw (PDN Gate Way) 2035 and PCRF (Policy and Charging Rules Function, policy and tariff function entity) 2036, and so on. The MME2031 is a control node that handles signaling between the UE201 and EPC203, providing bearer and connection management. HSS2032 is used to provide registers to manage functions such as home location registers (not shown) and to hold user specific information about service characteristics, data rates, etc. All user data may be sent through SGW2034 and PGW2035 may provide IP address allocation and other functions for UE201, PCRF2036 is a policy and charging control policy decision point for traffic data flows and IP bearer resources, which selects and provides available policy and charging control decisions for a policy and charging enforcement function (not shown).
Although the LTE system is described above as an example, it should be understood by those skilled in the art that the present invention is not limited to LTE systems, but may be applied to other wireless communication systems, such as GSM, CDMA2000, WCDMA, TD-SCDMA, and future new network systems.
Based on the above mobile terminal hardware structure and the communication network system, various embodiments of the method of the present invention are provided.
Example 1
Fig. 3 is a flowchart of a first embodiment of the audio projection synchronization control method of the present invention. An audio screen-throwing synchronous control method, comprising the following steps:
s1, when a screen projection connection is established between a screen projection end and a screen projection end, creating an audio recording service of the screen projection end;
s2, when the cast end starts to transmit the screen casting data to the screen casting end through a first channel in the screen casting connection, starting audio recording of the cast end through the audio recording service;
s3, acquiring audio data in real time through the audio recording service, and transmitting the audio data to the screen throwing end through a second channel in the screen throwing connection by the thrown end;
and S4, synchronously and real-time analyzing the audio data when the screen projection end renders the screen projection data in real time.
In this embodiment, first, when a screen-casting connection is established between a screen-casting end and a screen-casting end, an audio recording service of the screen-casting end is created; then, when the cast end starts to transmit the cast data to the cast end through a first channel in the cast connection, starting audio recording of the cast end through the audio recording service; then, acquiring audio data in real time through the audio recording service, and transmitting the audio data to the screen throwing end through a second channel in the screen throwing connection by the thrown end; and finally, synchronously and real-timely analyzing the audio data when the screen throwing end renders the screen throwing data in real time.
Optionally, in this embodiment, the to-be-cast end uses a mobile phone as an example, and the screen-cast end uses a computer as an example. In this embodiment, considering that the reverse screen-casting method generally starts an executable jar (a program) on the mobile phone by means of a computer adb (a debug mode), so that the executable jar has shell (a control) authority and can perform screen acquisition, and screen-recording is realized, but the executable jar has shell authority but does not have recording authority, and the recording authority must apply for recording by starting an application with context (content), so that the prior art has no scheme of reverse screen-casting of synchronous audio. Therefore, in this embodiment, recording is not possible for the property that jar packets only have shell rights, so this embodiment uses a related application apk (installation application package) manner with context, that is, creates an apk with a Service or an Activity of the above function, in this embodiment, after the creation of the above apk, the apk may be pulled up by an adb shell command, and then starts an AudioRecord Service of recording in the Activity or Service, and performs an audio recording operation.
The method has the beneficial effects that when the screen-throwing connection is established between the thrown end and the screen-throwing end, the audio recording service of the thrown end is established; then, when the cast end starts to transmit the cast data to the cast end through a first channel in the cast connection, starting audio recording of the cast end through the audio recording service; then, acquiring audio data in real time through the audio recording service, and transmitting the audio data to the screen throwing end through a second channel in the screen throwing connection by the thrown end; and finally, synchronously and real-timely analyzing the audio data when the screen throwing end renders the screen throwing data in real time. The humanized audio screen-throwing synchronous control scheme is realized, so that when the screen is thrown to the screen-throwing end for reversely throwing the screen, synchronous reverse screen throwing of pictures and audio can be realized, the reverse screen-throwing function is further perfected, and the screen-throwing use experience of a user is enhanced.
Example two
Fig. 4 is a flowchart of a second embodiment of the audio screen-projection synchronization control method according to the present invention, based on the above embodiment, when a screen-projection connection is established between a screen-projection end and a screen-projection end, the creating an audio recording service of the screen-projection end includes:
S11, when the screen-throwing end and the screen-throwing end are connected, receiving an audio screen-throwing initial instruction sent by the screen-throwing end to the screen-throwing end;
s12, creating the audio recording service at the thrown end according to the audio screen throwing initial instruction.
In this embodiment, first, when the screen-throwing end and the screen-throwing end establish screen-throwing connection, an audio screen-throwing initial instruction sent by the screen-throwing end to the screen-throwing end is received; and then, creating the audio recording service at the thrown end according to the audio projection initial instruction.
Optionally, in this embodiment, when the screen-throwing end establishes a screen-throwing connection with the screen-throwing end, an audio screen-throwing initial instruction sent by the screen-throwing end to the screen-throwing end is received, where the instruction is used to create the audio recording service at the screen-throwing end;
optionally, in this embodiment, the instruction may be further configured to control the cast end to download and install the audio recording service from the specified address;
optionally, in this embodiment, the instruction may further be configured to control the received terminal to receive the audio recording service sent by the screen terminal, and perform installation or enabling of the service.
The method has the advantages that when the screen-throwing end is identified to establish screen-throwing connection with the screen-throwing end, an audio screen-throwing initial instruction sent by the screen-throwing end to the screen-throwing end is received; and then, creating the audio recording service at the thrown end according to the audio projection initial instruction. The method provides an initialization basis for realizing a humanized audio screen-throwing synchronous control scheme, so that when a screen is thrown to a screen-throwing end for reversely throwing, synchronous and reverse screen throwing of pictures and audio can be realized, the reverse screen-throwing function is further perfected, and the screen-throwing use experience of a user is enhanced.
Example III
Fig. 5 is a flowchart of a third embodiment of the audio screen-on synchronization control method according to the present invention, based on the above embodiment, where when the screen-on connection is established between the screen-on end and the screen-on end, the creating an audio recording service of the screen-on end further includes:
s13, when the screen-throwing end and the screen-throwing end are connected, a first channel for transmitting the screen-throwing data is created;
s14, and creating a second channel for transmitting the audio data.
In this embodiment, first, when the cast end establishes a screen casting connection with the screen casting end, a first channel for transmitting the screen casting data is created; then, and creating a second channel for transmitting the audio data.
Optionally, in this embodiment, when the dropped end establishes a drop connection with the drop end, a first channel for transmitting the drop data is created, where the first channel may be a socket channel (socket);
optionally, in this embodiment, when the dropped end establishes a drop connection with the drop end, a second channel for transmitting the audio data is created, and the second channel may be another socket channel different from the socket channel described above;
optionally, in this embodiment, the two socket channels are isolated, so as to avoid affecting one when the other is wrong.
The method has the beneficial effects that when the screen-throwing end establishes screen-throwing connection with the screen-throwing end, a first channel for transmitting the screen-throwing data is created; then, and creating a second channel for transmitting the audio data. The transmission channel setting basis is provided for realizing a humanized audio screen-throwing synchronous control scheme, so that when a screen is thrown to a screen-throwing end for reversely throwing the screen, synchronous and reverse screen throwing of pictures and audio can be realized, the reverse screen throwing function is further perfected, and the screen throwing use experience of a user is enhanced.
Example IV
Fig. 6 is a flowchart of a fourth embodiment of an audio screen-on synchronization control method according to the present invention, based on the above embodiment, when the screen-on end starts to transmit screen-on data to the screen-on end through a first channel in the screen-on connection, the method for starting audio recording on the screen-on end through the audio recording service includes:
s21, monitoring the transmission state of the screen throwing data;
s22, when the screen projection data starts transmission, starting the audio recording service.
In this embodiment, first, a transmission state of the screen projection data is monitored; and then, when the screen-throwing data starts transmission, starting the audio recording service.
Optionally, in this embodiment, when the screen-on data starts transmission, the audio recording Service is started, for example, after the apk is created, the apk may be pulled up by an adb shell command, and then an AudioRecord Service of recording is started in the Activity or Service, and an audio recording operation is performed.
Optionally, in this embodiment, when the screen-drop application is started, an AudioRecord Service of recording is started in the Activity or Service, and an audio recording operation is performed.
The method has the beneficial effects that the transmission state of the screen throwing data is monitored; and then, when the screen-throwing data starts transmission, starting the audio recording service. The starting scheme of the audio recording service is provided for realizing a humanized audio screen-throwing synchronous control scheme, so that when a screen is thrown to a screen-throwing end for reversely throwing the screen, synchronous and reverse screen throwing of pictures and audio can be realized, the reverse screen throwing function is further perfected, and the screen throwing use experience of a user is enhanced.
Example five
Fig. 7 is a flowchart of a fifth embodiment of an audio screen synchronization control method according to the present invention, based on the above embodiment, the method for acquiring audio data in real time through the audio recording service, and transmitting, by the screened terminal, the audio data to the screened terminal through a second channel in the screen connection, includes:
s31, acquiring audio sampling data in real time through the audio recording service;
s32, carrying out compression coding on the audio sampling data according to a preset compression coding format to obtain the audio data.
In this embodiment, first, audio sampling data is obtained in real time through the audio recording service; and then, carrying out compression coding on the audio sampling data according to a preset compression coding format to obtain the audio data.
Alternatively, in this embodiment, considering that only uncompressed sampled data of PCM (pulse code modulation) is recorded in audio, the data amount of such sampled data is huge and is not suitable for network transmission, so in this embodiment, the sampled data is compressed, for example, a compression coding scheme of AAC (audio compression coding) may be adopted, and thus, the data amount of sampled data may be 1/20 of the original data amount, and the load of audio transmission on network bandwidth is greatly reduced.
The beneficial effects of the embodiment are that the audio sampling data are obtained in real time through the audio recording service; and then, carrying out compression coding on the audio sampling data according to a preset compression coding format to obtain the audio data. The compression coding scheme of the audio recording service is provided for realizing a humanized audio screen-throwing synchronous control scheme, so that when a screen is thrown to a screen-throwing end for reversely throwing the screen, synchronous and reverse screen throwing of pictures and audio can be realized, the reverse screen throwing function is further perfected, and the screen throwing use experience of a user is enhanced.
Example six
Fig. 8 is a flowchart of a sixth embodiment of an audio screen synchronization control method according to the present invention, based on the above embodiment, where the audio data is obtained in real time by the audio recording service, and the audio data is transmitted to the screen end by the screen end through a second channel in the screen connection, and further includes:
S33, presetting a network stream format corresponding to the audio data;
s34, transmitting the audio data in the second channel according to the network stream format.
In this embodiment, first, a network stream format corresponding to the audio data is preset; the audio data is then transmitted in the second channel in the network stream format.
Optionally, in this embodiment, as described above, since the network transmission of the audio data is considered, the AAC data is packaged into the network stream format of the ADTS (Audio Data Transport Stream, a transport stream format), so that the current sampling rate of AAC and the related audio parameters can be notified to the screen-throwing end by the throwing end, so that the correct decoding operation can be performed after the network transmission.
The beneficial effects of the embodiment are that the network stream format corresponding to the audio data is preset; the audio data is then transmitted in the second channel in the network stream format. The network transmission format of the audio recording service is provided for realizing a humanized audio screen-throwing synchronous control scheme, so that when a screen is thrown to a screen-throwing end for reversely throwing the screen, synchronous and reverse screen throwing of pictures and audio can be realized, the reverse screen throwing function is further perfected, and the screen throwing use experience of a user is enhanced.
Example seven
Fig. 9 is a flowchart of a seventh embodiment of an audio screen-on synchronization control method according to the present invention, based on the above embodiment, where when the screen-on end renders the screen-on data in real time, the audio data is synchronized and parsed in real time, including:
s41, rendering and displaying a screen projection picture in real time when the screen projection data are received by the screen projection terminal;
s42, when the screen throwing end receives the audio data, the audio data are restored into the audio sampling data.
In this embodiment, first, when the screen-throwing data is received by the screen-throwing device, rendering and displaying a screen-throwing picture in real time; and then, when the screen throwing end receives the audio data, restoring the audio data into the audio sampling data.
Optionally, in this embodiment, as described in the above example, when the screen-throwing end is a computer, the received screen-throwing data is rendered in real time through the display card of the computer, so as to obtain a display screen-throwing picture;
optionally, in this embodiment, when the screen-throwing end is a computer, as described in the above example, the network stream format is parsed by the sound card of the computer to obtain the restored audio sampling data.
The method has the advantages that when the screen throwing data are received by the screen throwing end, the screen throwing picture is rendered and displayed in real time; and then, when the screen throwing end receives the audio data, restoring the audio data into the audio sampling data. The analysis scheme of the screen projection data is provided for realizing a humanized audio screen projection synchronous control scheme, so that when the screen projection is carried out on the screen projection end in a reverse direction, the synchronous reverse screen projection of the picture and the audio can be realized, the reverse screen projection function is further perfected, and the screen projection use experience of a user is enhanced.
Example eight
Fig. 10 is a flowchart of an eighth embodiment of an audio screen-on synchronization control method according to the present invention, based on the above embodiment, where when the screen-on end renders the screen-on data in real time, the audio data is synchronized and parsed in real time, and further includes:
s43, synchronizing a play instruction and a rendering instruction of the audio sampling data and the screen throwing data;
s44, playing the audio sampling data according to the playing instruction, and simultaneously, rendering the screen throwing data according to the rendering instruction.
In this embodiment, first, a play instruction and a rendering instruction of the audio sampling data and the screen projection data are synchronized; and then, playing the audio sampling data according to the playing instruction, and simultaneously, rendering the screen throwing data according to the rendering instruction.
Optionally, in this embodiment, the playing instruction and the rendering instruction of the audio sampling data and the screen throwing data are synchronized, so that a screen throwing picture at the screen throwing end is kept synchronous with the screen throwing audio;
optionally, in this embodiment, when the dropped terminal is composed of a plurality of sound sources, audio data corresponding to different sound sources are transmitted through separate channels respectively;
optionally, in this embodiment, for audio data corresponding to different audio sources, parsing is performed at the screen-throwing end to obtain corresponding audio sampling data, and one or more items of audio sampling data are selected to be played synchronously with the screen-throwing picture.
The method has the advantages that the playing instruction and the rendering instruction of the audio sampling data and the screen throwing data are synchronized; and then, playing the audio sampling data according to the playing instruction, and simultaneously, rendering the screen throwing data according to the rendering instruction. The synchronous control scheme for the screen projection is provided for realizing a humanized audio screen projection synchronous control scheme, so that when the screen projection is carried out on the screen projection end in a reverse direction, the synchronous reverse screen projection of the picture and the audio can be realized, the reverse screen projection function is further perfected, and the screen projection use experience of a user is enhanced.
Example nine
Based on the above embodiment, the present invention further provides an audio screen synchronization control device, where the device includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, where the computer program when executed by the processor implements the steps of the audio screen synchronization control method according to any one of the above claims.
It should be noted that the above device embodiments and method embodiments belong to the same concept, the specific implementation process of the device embodiments is detailed in the method embodiments, and technical features in the method embodiments are correspondingly applicable to the device embodiments, which are not repeated herein.
Examples ten
Based on the above embodiment, the present invention further provides a computer readable storage medium, where an audio screen-on synchronization control program is stored, where the audio screen-on synchronization control program, when executed by a processor, implements the steps of the audio screen-on synchronization control method according to any one of the above embodiments.
It should be noted that the medium embodiment and the method embodiment belong to the same concept, the specific implementation process of the medium embodiment and the method embodiment are detailed, and technical features in the method embodiment are correspondingly applicable in the medium embodiment, which is not repeated herein.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising instructions for causing a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The embodiments of the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those having ordinary skill in the art without departing from the spirit of the present invention and the scope of the claims, which are to be protected by the present invention.
Claims (6)
1. An audio screen-throwing synchronous control method is characterized by comprising the following steps:
when a screen throwing connection is established between a thrown end and a screen throwing end, creating an audio recording service of the thrown end;
when the cast end starts to transmit the cast data to the cast end through a first channel in the cast connection, starting audio recording of the cast end through the audio recording service;
acquiring audio data in real time through the audio recording service, and transmitting the audio data to the screen throwing end through a second channel in the screen throwing connection by the thrown end;
when the screen throwing end renders the screen throwing data in real time, synchronously and real-time analyzing the audio data;
when the screen-throwing connection is established between the thrown end and the screen-throwing end, the audio recording service of the thrown end is established, and the method comprises the following steps:
when the screen-throwing end and the screen-throwing end are connected, receiving an audio screen-throwing initial instruction sent by the screen-throwing end to the screen-throwing end;
creating the audio recording service at the thrown end according to the audio screen throwing initial instruction;
wherein, the liquid crystal display device comprises a liquid crystal display device,
the initial instruction is used for creating the audio recording service at the thrown end;
or the initial instruction is used for controlling the thrown end to download and install the audio recording service from the appointed address;
Or the initial instruction is used for controlling the received terminal to receive the audio recording service sent by the screen terminal and installing or starting the audio recording service;
acquiring audio sampling data in real time through the audio recording service;
when the screen throwing data are received by the screen throwing device, rendering and displaying a screen throwing picture in real time;
and when the screen throwing end renders the screen throwing data in real time, synchronizing and analyzing the audio data in real time, wherein the method comprises the following steps of:
synchronizing a play instruction and a rendering instruction of the audio sampling data and the screen throwing data;
playing the audio sampling data according to the playing instruction, and simultaneously, rendering the screen throwing data according to the rendering instruction;
when a plurality of sound sources are arranged at the thrown end, respectively transmitting the audio data corresponding to different sound sources through isolated channels;
respectively analyzing the audio data corresponding to different sound sources at the screen throwing end to obtain corresponding audio sampling data, and selecting one or more audio sampling data to be synchronously played with the screen throwing picture;
when the screen-throwing connection is established between the thrown end and the screen-throwing end, the audio recording service of the thrown end is created, and the method further comprises the following steps:
When the screen-throwing end establishes screen-throwing connection with the screen-throwing end, creating the first channel for transmitting the screen-throwing data;
and creating the second channel for transmitting the audio data;
the first channel is a socket channel, the second channel is another socket channel different from the socket channel, and the two socket channels are isolated, so that the influence on the other socket channel caused by one error is avoided;
when the cast end starts to transmit the cast data to the cast end through a first channel in the cast connection, starting audio recording of the cast end through the audio recording service, including:
monitoring the transmission state of the screen throwing data;
when the screen-throwing data starts transmission, starting the audio recording service;
wherein, the liquid crystal display device comprises a liquid crystal display device,
starting an executable jar program on the thrown end through the screen throwing end in an adb debugging mode, and acquiring and recording a screen through shell control authority by the executable jar program;
creating an apk installation application program package through a related application with context, and setting a Service or an Activity through the apk installation application program package;
And pulling up the apk installation application program package through an adb shell control command, starting an AudioRecord recording Service for recording in the Activity or Service, and performing audio recording operation through the AudioRecord recording Service.
2. The method for synchronously controlling the screen projection of audio according to claim 1, wherein the step of acquiring the audio data in real time by the audio recording service and transmitting the audio data from the screen-projected end to the screen-projected end through the second channel in the screen-projected connection comprises:
and carrying out compression coding on the audio sampling data according to a preset compression coding format to obtain the audio data.
3. The method for synchronously controlling the screen projection of audio according to claim 2, wherein the audio data is obtained in real time by the audio recording service, and the audio data is transmitted to the screen projection end by the projected end through a second channel in the screen projection connection, further comprising:
presetting a network stream format corresponding to the audio data;
transmitting the audio data in the second channel in the network stream format.
4. The method for synchronously controlling the screen projection of the audio according to claim 3, wherein the step of synchronously and real-time analyzing the audio data when the screen projection end renders the screen projection data in real time comprises the steps of:
And when the screen throwing end receives the audio data, restoring the audio data into the audio sampling data.
5. An audio projection synchronization control device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, which when executed by the processor, implements the steps of the audio projection synchronization control method according to any one of claims 1 to 4.
6. A computer readable storage medium, wherein an audio screen-on synchronization control program is stored on the computer readable storage medium, and the audio screen-on synchronization control program, when executed by a processor, implements the steps of the audio screen-on synchronization control method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011293135.3A CN112423076B (en) | 2020-11-18 | 2020-11-18 | Audio screen-throwing synchronous control method, equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011293135.3A CN112423076B (en) | 2020-11-18 | 2020-11-18 | Audio screen-throwing synchronous control method, equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112423076A CN112423076A (en) | 2021-02-26 |
| CN112423076B true CN112423076B (en) | 2023-05-05 |
Family
ID=74773024
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011293135.3A Active CN112423076B (en) | 2020-11-18 | 2020-11-18 | Audio screen-throwing synchronous control method, equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112423076B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112905289A (en) * | 2021-03-10 | 2021-06-04 | Oppo广东移动通信有限公司 | Application picture display method, device, terminal, screen projection system and medium |
| CN115442643A (en) * | 2021-06-03 | 2022-12-06 | Oppo广东移动通信有限公司 | Screen projection method and device, wireless terminal, screen projection equipment and storage medium |
| CN113746911A (en) * | 2021-08-26 | 2021-12-03 | 科大讯飞股份有限公司 | Audio processing method and related device, electronic equipment and storage medium |
| CN115016749A (en) * | 2021-09-08 | 2022-09-06 | 荣耀终端有限公司 | Screen projection content control method, device, system and storage medium |
| CN115065852B (en) * | 2022-08-22 | 2022-11-01 | 深圳创维－Rgb电子有限公司 | Sound and picture synchronization method and device, electronic equipment and readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2011150456A (en) * | 2010-01-20 | 2011-08-04 | Seiko Epson Corp | Content display device and display control method of the same |
| CN106612445A (en) * | 2015-10-21 | 2017-05-03 | 中国移动通信集团公司 | Wireless one-screen audio channel control method, transmitting device and receiving device |
| CN108512810A (en) * | 2017-02-24 | 2018-09-07 | 北京九五智驾信息技术股份有限公司 | The system that mobile terminal is communicated with car-mounted terminal |
| CN109275130A (en) * | 2018-09-13 | 2019-01-25 | 锐捷网络股份有限公司 | A kind of throwing screen method, apparatus and storage medium |
| CN111381954A (en) * | 2018-12-27 | 2020-07-07 | 深圳市优必选科技有限公司 | Audio data recording method, system and terminal equipment |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6405255B1 (en) * | 1996-07-01 | 2002-06-11 | Sun Microsystems, Inc. | Mixing and splitting multiple independent audio data streams in kernel space |
-
2020
- 2020-11-18 CN CN202011293135.3A patent/CN112423076B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2011150456A (en) * | 2010-01-20 | 2011-08-04 | Seiko Epson Corp | Content display device and display control method of the same |
| CN106612445A (en) * | 2015-10-21 | 2017-05-03 | 中国移动通信集团公司 | Wireless one-screen audio channel control method, transmitting device and receiving device |
| CN108512810A (en) * | 2017-02-24 | 2018-09-07 | 北京九五智驾信息技术股份有限公司 | The system that mobile terminal is communicated with car-mounted terminal |
| CN109275130A (en) * | 2018-09-13 | 2019-01-25 | 锐捷网络股份有限公司 | A kind of throwing screen method, apparatus and storage medium |
| CN111381954A (en) * | 2018-12-27 | 2020-07-07 | 深圳市优必选科技有限公司 | Audio data recording method, system and terminal equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112423076A (en) | 2021-02-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112423076B (en) | Audio screen-throwing synchronous control method, equipment and computer readable storage medium | |
| CN108572764B (en) | Character input control method and device and computer readable storage medium | |
| CN110175062B (en) | Screen refreshing frame rate method of mobile terminal, mobile terminal and storage medium | |
| CN111708503B (en) | Screen projection control method, device and computer readable storage medium | |
| CN112799577B (en) | Method, terminal and storage medium for projecting small window | |
| CN112822538A (en) | Screen projection display method, screen projection device, terminal and storage medium | |
| CN112689191B (en) | Screen projection control method, terminal and computer readable storage medium | |
| CN109358940A (en) | Desktop synchronization, mobile terminal and computer readable storage medium | |
| CN110052024B (en) | Game vibration adjusting method, adjusting device, mobile terminal and storage medium | |
| CN113596231B (en) | Screen-throwing display control method, device and computer readable storage medium | |
| CN112492340B (en) | Live broadcast audio acquisition method, mobile terminal and computer readable storage medium | |
| CN112437472B (en) | Network switching method, equipment and computer readable storage medium | |
| CN113176869A (en) | Screen-projecting audio time delay control method and device and computer readable storage medium | |
| CN108900503A (en) | Data communications method, communication processing equipment, terminal and readable storage medium storing program for executing | |
| CN109684020B (en) | Theme switching method, device and computer readable storage medium | |
| CN109561221B (en) | Call control method, device and computer readable storage medium | |
| CN112887776B (en) | Method, equipment and computer readable storage medium for reducing audio delay | |
| CN113179307B (en) | Audio delay control method, equipment and computer readable storage medium | |
| CN111970668B (en) | Bluetooth audio control method, device and computer readable storage medium | |
| CN112492450B (en) | Sound parameter regulation and control method, device and computer readable storage medium | |
| CN114307136A (en) | Game screen-throwing power consumption control method and device and computer readable storage medium | |
| CN110287381B (en) | Page control node searching method, terminal and computer readable storage medium | |
| CN114126032A (en) | Information synchronization method, intelligent terminal and storage medium | |
| CN109710576B (en) | File display method, terminal and storage medium | |
| CN109144747B (en) | Data processing method, terminal and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20230411 Address after: Room 2207, Dongjun Huarong Plaza, No. 196, Section 1, Wanjiali Middle Road, Hehuayuan Street, Furong District, Changsha City, Hunan Province, 410000 Applicant after: Hunan Jiajia Intelligent Technology Co.,Ltd. Address before: 518000 6-8 / F, 10-11 / F, 6 / F, 6-10 / F, C zone, Han's innovation building, No. 9018, Beihuan Avenue, high tech Zone, Nanshan District, Shenzhen City, Guangdong Province Applicant before: NUBIA TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |