CN111432161B - Audio state visual playing and feedback method, device and terminal - Google Patents
Audio state visual playing and feedback method, device and terminal Download PDFInfo
- Publication number
- CN111432161B CN111432161B CN202010356399.2A CN202010356399A CN111432161B CN 111432161 B CN111432161 B CN 111432161B CN 202010356399 A CN202010356399 A CN 202010356399A CN 111432161 B CN111432161 B CN 111432161B
- Authority
- CN
- China
- Prior art keywords
- audio
- bitmap
- curve
- local
- far
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/15—Conference systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/431—Generation of visual interfaces for content selection or interaction; Content or additional data rendering
- H04N21/4312—Generation of visual interfaces for content selection or interaction; Content or additional data rendering involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/439—Processing of audio elementary streams
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/8106—Monomedia components thereof involving special audio data, e.g. different tracks for different languages
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
An audio state visual playing and feedback method comprises the following steps: generating a resampling sequence sample of the locally acquired audio and a resampling sequence sample of the locally played audio; drawing a local acquisition audio curve bitmap and a local play audio curve bitmap, and overlapping the local acquisition audio curve bitmap and the local play audio curve bitmap into a local video frame for displaying; packing and sending the resample sequence samples of the locally acquired audio and the resample sequence samples of the locally played audio; unpacking and splitting the received far-end feedback data packet to obtain a resampling sequence sample of the collected audio of the remote terminal and a resampling sequence sample of the played audio of the remote terminal; and drawing the far-end collected audio curve bitmap and the far-end played audio curve bitmap, and overlapping the far-end collected audio curve bitmap and the far-end played audio curve bitmap into a far-end video frame for displaying. The invention also provides an audio state visual playing and feedback device, which visually and intuitively feeds back the audio states of the local terminal and the remote terminal to a user to help the user to quickly know the current state.
Description
Technical Field
The invention relates to the technical field of multimedia conference systems, in particular to a method for playing and feeding back audio states of all terminals in a multimedia conference system.
Background
In an existing multimedia conference system or multimedia communication system, a system terminal processes audio, and includes, at a transmitting side: audio acquisition, audio coding, audio frequency are sent, reach the receiving side through network transmission, include at the receiving side: audio receiving, audio decoding and audio playing. And the participants judge whether the audio processing of the current terminals is normal or not through the voice interaction.
Because the system terminal lacks a means for intuitively reflecting the processing state of each link of the audio, when the audio interaction is not smooth, if a person of the system cannot respond to the voice of the opposite party, the person usually needs to inquire whether the opposite party can hear the voice of the opposite party or exchange with the help of third-party software for judging a general problem, the state of each key link of the audio processing of the local terminal and the remote terminal cannot be intuitively fed back to the user, and the efficiency is very low.
Disclosure of Invention
In order to solve the defects in the prior art, the invention aims to provide a method, a device and a terminal for audio state visual playing and feedback, which can help a user to quickly know the current state and narrow the problem positioning range by visually feeding back the states of all key links of audio processing of a local terminal and a remote terminal to the user.
In order to achieve the above object, the audio status visual playing and feedback method provided by the present invention comprises the following steps:
generating a resampling sequence sample of the locally acquired audio and a resampling sequence sample of the locally played audio;
drawing a local acquisition audio curve bitmap and a local play audio curve bitmap, and overlapping the local acquisition audio curve bitmap and the local play audio curve bitmap into a local video frame for displaying;
packing and sending the resample sequence samples of the locally acquired audio and the resample sequence samples of the locally played audio;
unpacking and splitting the received far-end feedback data packet to obtain a resampling sequence sample of the collected audio of the remote terminal and a resampling sequence sample of the played audio of the remote terminal;
and drawing the far-end collected audio curve bitmap and the far-end played audio curve bitmap, and overlapping the far-end collected audio curve bitmap and the far-end played audio curve bitmap into a far-end video frame for displaying.
Further, the step of generating resample sequence samples of the locally acquired audio and the resample sequence samples of the locally played audio further comprises:
performing secondary sampling on the local audio acquisition sample, acquiring resampling data of the local audio acquisition sample, and generating a resampling sequence sample of the local audio acquisition sample;
and carrying out secondary sampling on the playing samples of the local audio to obtain the resampling data of the playing samples of the local audio, and generating the resampling sequence samples of the local playing audio.
Further, the step of drawing a locally collected audio curve bitmap and a locally played audio curve bitmap, and superimposing the drawn locally collected audio curve bitmap and the locally played audio curve bitmap into a local video frame for display further includes:
drawing a local acquisition audio curve bitmap according to the resampling sequence sample of the local acquisition audio;
drawing a curve bitmap of the local playing audio according to the resample sequence sample of the local playing audio;
and overlaying the local acquisition audio curve bitmap and the local playing audio curve bitmap to a local video frame for display.
Further, the method comprises the steps of,
and respectively indicating the corresponding local acquisition audio curve bitmap and local play audio curve bitmap by drawing microphone and loudspeaker icons, and overlapping the local acquisition audio curve bitmap and the local play audio curve bitmap into a local video frame.
Further, the step of drawing the far-end collected audio curve bitmap and the far-end played audio curve bitmap, and superimposing the drawn far-end collected audio curve bitmap and the far-end played audio curve bitmap into a far-end video frame for display further includes:
drawing a far-end collected audio curve bitmap according to the obtained resampling sequence sample of the collected audio of the remote terminal;
drawing a far-end playing audio curve bitmap according to the obtained resampling sequence sample of the playing audio of the remote terminal;
and superposing the far-end collected audio curve bitmap and the far-end played audio curve bitmap to a received far-end video frame for displaying.
Further, the method comprises, in combination with,
and respectively indicating the corresponding far-end collected audio curve bitmap and the far-end played audio curve bitmap by drawing microphone and loudspeaker icons, and overlapping the far-end collected audio curve bitmap and the far-end played audio curve bitmap into a far-end video frame.
In order to achieve the above object, the present invention further provides an audio status visual playing and feedback device, which comprises a secondary sampling module, a curve drawing module, a video superimposing module, a packing and summarizing module, a feedback information transceiving module, and an unpacking and splitting module, wherein,
the secondary sampling module is used for respectively carrying out secondary sampling on the local audio acquisition sample and the local audio playing sample to generate a resampling sequence sample of the local acquisition audio and a resampling sequence sample of the local playing audio;
the curve drawing module is used for drawing a collected audio curve bitmap and a played audio curve bitmap according to the resampled sequence samples of the collected audio and the resampled sequence samples of the played audio;
the video overlapping module overlaps the collected audio curve bitmap and the played audio curve bitmap into a video frame;
the packaging and summarizing module is used for packaging the resampling sequence samples of the locally acquired audio and the resampling sequence samples of the locally played audio generated by the secondary sampling module to generate a feedback data packet;
the feedback information transceiving module is used for transmitting the feedback data packet generated by the packaging and summarizing module to the remote terminal through the network and forwarding the received feedback data packet of the remote terminal to the unpacking and splitting module;
and the unpacking and splitting module is used for unpacking and splitting the feedback data packet of the remote terminal and sending the obtained resampling sequence sample of the far-end collected audio and the resampling sequence sample of the far-end played audio to the curve drawing module.
Further, the curve drawing module may draw the curve,
respectively drawing a local acquisition audio curve bitmap and a local play audio curve bitmap according to the local acquisition audio resampling sequence sample and the local play audio resampling sequence sample generated by the secondary sampling module;
and respectively drawing a far-end collected audio curve bitmap and a far-end played audio curve bitmap according to the resampling sequence sample of the far-end collected audio and the resampling sequence sample of the far-end played audio sent by the unpacking and splitting module.
In order to achieve the above object, the present invention further provides a multimedia conference system terminal, which includes the audio state visual playing and feedback device described above, and implements the steps of the audio state visual playing and feedback method described above.
To achieve the above object, the present invention further provides a computer-readable storage medium having a computer program stored thereon, where the computer program can be executed by a processor to perform the steps of audio state visual playing and feedback.
The audio state visual playing and feedback method, device and terminal can be used as an optional function in a multimedia conference system, can be started when voice interaction is abnormal, and help a user to quickly and intuitively know the current audio state and preliminarily judge the fault range by visually feeding back the states of each key link of audio processing of the local terminal and the opposite terminal. The audio processing links of the main feedback comprise: the method comprises the following steps of acquiring the audio of the terminal of the local side, playing the audio of the terminal of the local side, acquiring the audio of the terminal of the opposite side and playing the audio of the terminal of the opposite side.
The states of all key links of the audio processing of the local terminal and the remote terminal are visually and intuitively fed back to the user, so that the user can be helped to quickly know the current state, and the problem positioning range is narrowed.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention and not to limit the invention. In the drawings:
fig. 1 is a flow chart of an audio state visualization playing and feedback method according to the present invention;
fig. 2 is a schematic structural diagram of an audio status visual playing and feedback device according to the present invention; .
Detailed Description
The preferred embodiments of the present invention will be described in conjunction with the accompanying drawings, and it will be understood that they are described herein for the purpose of illustration and explanation and not limitation.
Example 1
Fig. 1 is a flowchart of an audio state visual playing and feedback method according to the present invention, and the audio state visual playing and feedback method of the present invention will be described in detail with reference to fig. 1.
First, in step 101, a local audio acquisition sample is sub-sampled to obtain re-sampled data of the local audio acquisition sample, and a re-sampled sequence sample of the local audio acquisition sample is formed.
In the embodiment of the invention, the sampling interval for performing secondary sampling on the audio acquisition samples acquired at the local terminal is the length of an audio coding frame, which is usually 20ms, namely, one sample is extracted every 20ms to form a resampling sequence sample.
In step 102, the playing samples of the local audio are sub-sampled to obtain the re-sampled data of the local audio playing samples, and a re-sampled sequence sample of the local audio is formed.
In the embodiment of the present invention, the sampling interval for performing subsampling on the play sample of the local audio is the length of an audio coding frame, which is usually 20ms, that is, one sample is extracted every 20ms to form a resample sequence sample.
In the embodiment of the invention, the audio samples output to the audio playing device at the local terminal are subjected to secondary sampling, the resampling data of the playing audio is obtained, and the resampling sequence samples of the local playing audio are formed.
In step 103, according to the resample sequence samples of the local acquisition audio and the resample sequence samples of the local playing audio, respectively drawing a local acquisition audio curve bitmap and a local playing audio curve bitmap, and superimposing the local acquisition audio curve bitmap and the local playing audio curve bitmap into a local video frame.
Preferably, in the embodiment of the present invention, when drawing the local capture audio curve bitmap and the local play audio curve bitmap, the microphone and speaker icons may be drawn to respectively indicate the corresponding capture audio curve bitmap and the corresponding play audio curve bitmap, and the capture audio curve bitmap and the play audio curve bitmap are superimposed on the local video frame.
In step 104, after rendering the local video overlaid with the locally-acquired audio curve bitmap and the locally-played audio curve bitmap, the audio acquisition and playing states of the local terminal are displayed in the local video window.
In the embodiment of the invention, if a sample (audio is not collected currently or the playable counterpart audio is not obtained) is not collected or played in a certain drawing process, the mute processing is performed, that is, a 0 horizontal straight line is drawn.
In step 105, the formed resample sequence samples of the locally collected audio and the resample sequence samples of the locally played audio are packed to form a feedback data packet and sent to the remote terminal through the network.
In the embodiment of the invention, when a feedback data packet required to be transmitted is received, the resampling sequence samples of the locally acquired audio and the resampling sequence samples of the locally played audio are regularly packaged into the feedback data packet according to a certain packaging format, and the feedback data packet is fed back to the remote terminal through a network.
In step 106, the feedback data packet sent by the remote terminal is received and unpacked and split, and the resampling sequence sample of the collected audio of the remote terminal and the resampling sequence sample of the played audio of the remote terminal are obtained.
In the embodiment of the invention, any terminal of the multimedia conference system can inform the appointed remote terminal to feed back the corresponding audio state to the terminal through the multimedia conference system signaling; after receiving a feedback data packet sent by a designated remote terminal, the feedback data packet is analyzed and extracted according to a packaging format, and a resampling sequence sample of the designated remote terminal for currently acquiring the audio and a resampling sequence sample of the playing audio are respectively obtained.
In step 107, the collected audio curve bitmap and the played audio curve bitmap of the designated remote terminal are respectively drawn according to the resampled sequence sample of the currently collected audio and the resampled sequence sample of the played audio of the designated remote terminal and are superposed into the received far-end video frame.
Preferably, when drawing the collected audio curve bitmap and the played audio curve bitmap of the designated remote terminal, the audio curve bitmap may be formed by drawing a microphone and speaker icon indicating a corresponding sample curve of the collected audio and a corresponding sample curve of the played audio, respectively, and superimposed on the received video frame of the designated remote terminal.
In step 108, rendering the far-end video overlaid with the collected audio curve bitmap and the played audio curve bitmap of the remote terminal, and displaying the audio collection and playing state of the specified remote terminal on a far-end video window.
Example 2
In the embodiment of the present invention, an audio status visualization playing and feedback device is further provided, which includes a secondary sampling module 201, a curve drawing module 202, a video superimposing module 203, a packaging and summarizing module 204, a feedback information transceiving module 205, and an unpacking and splitting module 206, wherein,
the secondary sampling module 201 is used for carrying out secondary sampling on the collected samples of the local audio to obtain the resampling data of the collected samples of the local audio and form resampling sequence samples of the local collected audio; performing secondary sampling on a playing sample of the local audio to obtain resampling data of the playing sample of the local audio to form a resampling sequence sample of the local audio; one path of the formed resample sequence samples of the locally acquired audio and the resample sequence samples of the locally played audio is sent to the curve drawing module 202, and the other path of the formed resample sequence samples of the locally acquired audio and the resample sequence samples of the locally played audio is sent to the packing and summarizing module 204.
In the embodiment of the present invention, the sampling interval of the sub-sampling module 201 is the length of the audio coding frame, which is usually 20ms, that is, one sample is extracted every 20ms to form a new resample sequence sample.
The curve drawing module 202 is used for drawing a curve bitmap of the locally acquired audio and a curve bitmap of the locally played audio respectively according to the formed resampled sequence sample of the locally acquired audio and the resampled sequence sample of the locally played audio; it receives the resample sequence sample of the collected audio and the resample sequence sample of the played audio from the designated remote terminal sent by the unpacking and splitting module 206, and draws the collected audio curve bitmap and the played audio curve bitmap of the designated remote terminal respectively.
Preferably, in the embodiment of the present invention, when drawing the collected audio curve bitmap and the played audio curve bitmap of the specified remote terminal, the corresponding collected audio sample curve bitmap and the corresponding played audio sample curve bitmap may be respectively indicated by drawing the microphone and speaker icons.
The video overlapping module 203 overlaps the curve bitmap of the locally acquired audio and the curve bitmap of the locally played audio from the curve drawing module 202 to a local video frame (performs pixel-level overlapping), renders the overlapped local video frame, and sends the rendered local video frame to a local video window for display; the curve bitmap of the far-end collected audio and the curve bitmap of the far-end played audio from the curve drawing module 202 are superimposed to the received far-end video frame (pixel-level superimposition), and the superimposed far-end video frame is rendered and then sent to a far-end video window for display.
And a packing and summarizing module 204, which receives the resample sequence samples of the locally collected audio and the resample sequence samples of the locally played audio from the secondary sampling module 201, packs and encapsulates the resample sequence samples into a feedback data packet according to a certain packing format, and sends the feedback data packet to the feedback information transceiver module 205.
A feedback information transceiving module 205, which transmits the feedback data packet from the packetizing and summarizing module 204 to the remote terminal through the network; the feedback data packet from the remote terminal is received through the network and sent to the unpacking and splitting module 206.
And an unpacking and splitting module 206, which unpacks and splits the feedback data packet from the remote terminal of the feedback information transceiving module 205, obtains the resample sequence sample of the collected audio and the resample sequence sample of the played audio of the remote terminal, and sends the resample sequence samples to the curve drawing module 202.
Example 3
In an embodiment of the present invention, a terminal is further provided, which includes the above-mentioned audio state visual playing and feedback device, and implements the above-mentioned audio state visual playing and feedback method.
Example 4
In an embodiment of the present invention, there is also provided a computer readable storage medium having stored thereon a computer program which when run performs the steps of the audio state visualization playing and feedback method as described above.
Example 5
The invention can help the user to quickly know the current state and narrow the problem positioning range, such as:
scene 1: if the local video window microphone curve is in a mute state (the waveform is a 0-level curve), the local video window microphone curve may be caused by hardware failure of local microphone equipment or software setting (such as too small acquisition volume setting) and the like, so that the local audio is not acquired;
scene 2: the local video window microphone curve displays sound, and the opposite side video window loudspeaker curve is in a mute state, which may be the reason of network packet loss, system server failure and the like, and the opposite side cannot normally acquire the audio data of the local side;
scene 3: the local video window microphone curve and the counterpart video window loudspeaker curve are both displayed as sound, but the counterpart video window microphone curve is displayed in a mute state, which may be that the hardware failure of the counterpart loudspeaker device causes that the sound of the counterpart is not heard, or the hardware failure or software setting of the counterpart microphone device causes that the audio of the counterpart is not collected, and may also cause that the counterpart temporarily does not respond to our inquiry;
scene 4: a local video window microphone curve, an opposite video window loudspeaker curve and a microphone curve are displayed to be voiced, but the local video window loudspeaker curve is displayed to be in a mute state, which may be the reason of network packet loss, system server failure and the like, and the local terminal cannot normally acquire the audio data of the opposite side;
scene 5: the local video window microphone curve and the loudspeaker curve, and the counterpart video window microphone curve and the loudspeaker curve are all displayed as sound, and the counterpart sound may not be played correctly due to hardware failure or software setting problems of the loudspeaker used by the local terminal.
Those of ordinary skill in the art will understand that: although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that changes may be made in the embodiments and/or equivalents thereof without departing from the spirit and scope of the invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (8)
1. A visual playing and feedback method of audio state is applied to a multimedia conference system terminal and comprises the following steps:
generating a resampling sequence sample of the locally acquired audio and a resampling sequence sample of the locally played audio;
drawing a local acquisition audio curve bitmap and a local play audio curve bitmap, and overlapping the local acquisition audio curve bitmap and the local play audio curve bitmap into a local video frame for displaying;
the step of drawing the local collection audio curve bitmap and the local playing audio curve bitmap and overlapping the drawn local collection audio curve bitmap and the local playing audio curve bitmap into a local video frame for displaying further comprises the following steps:
drawing a local acquisition audio curve bitmap according to the resampling sequence sample of the local acquisition audio;
drawing a curve bitmap of the local playing audio according to the resample sequence sample of the local playing audio;
superposing the local acquisition audio curve bitmap and the local playing audio curve bitmap to a local video frame for displaying;
packing and sending the resample sequence samples of the locally acquired audio and the resample sequence samples of the locally played audio;
unpacking and splitting the received far-end feedback data packet to obtain a resampling sequence sample of the collected audio of the remote terminal and a resampling sequence sample of the played audio of the remote terminal;
drawing a far-end collected audio curve bitmap and a far-end played audio curve bitmap, and overlapping the far-end collected audio curve bitmap and the far-end played audio curve bitmap into a far-end video frame for displaying;
and when the voice interaction is abnormal, judging the fault state and the fault range by combining the local acquisition audio curve bitmap, the local playing audio curve bitmap, the remote acquisition audio curve bitmap and the local visual curve bitmap of the remote playing audio curve bitmap.
2. The audio state visual playback and feedback method according to claim 1, wherein said step of generating resampled sequence samples of locally captured audio and resampled sequence samples of locally played audio further comprises:
performing secondary sampling on the local audio acquisition sample, acquiring resampling data of the local audio acquisition sample, and generating a resampling sequence sample of the local audio acquisition sample;
and carrying out secondary sampling on the playing samples of the local audio to obtain the resampling data of the playing samples of the local audio, and generating the resampling sequence samples of the local playing audio.
3. The audio state visual playback and feedback method according to claim 1, wherein said step of overlaying said locally captured audio curve bitmap and said locally played audio curve bitmap into a local video frame for display further comprises,
and respectively indicating the corresponding local acquisition audio curve bitmap and local play audio curve bitmap by drawing microphone and loudspeaker icons, and overlapping the local acquisition audio curve bitmap and the local play audio curve bitmap into a local video frame.
4. The audio status visual playing and feedback method according to claim 1, wherein said step of rendering the far-end captured audio curve bitmap and the far-end played audio curve bitmap and superimposing them on the far-end video frame for display further comprises:
drawing a far-end collected audio curve bitmap according to the obtained resampling sequence sample of the collected audio of the remote terminal;
drawing a far-end playing audio curve bitmap according to the obtained resampling sequence sample of the playing audio of the remote terminal;
and superposing the far-end collected audio curve bitmap and the far-end played audio curve bitmap to a received far-end video frame for displaying.
5. The audio state visual playback and feedback method according to claim 4, further comprising,
and respectively indicating the corresponding far-end collected audio curve bitmap and the far-end played audio curve bitmap by drawing microphone and loudspeaker icons, and overlapping the far-end collected audio curve bitmap and the far-end played audio curve bitmap into a far-end video frame.
6. An audio state visual playing and feedback device is applied to a multimedia conference system and comprises a secondary sampling module, a curve drawing module, a video superposition module, a packaging and summarizing module, a feedback information receiving and sending module and an unpacking and splitting module,
the secondary sampling module is used for respectively carrying out secondary sampling on the local audio acquisition sample and the local audio playing sample to generate a resampling sequence sample of the local acquisition audio and a resampling sequence sample of the local playing audio;
the curve drawing module is used for drawing a local collected audio curve bitmap and a local played audio curve bitmap respectively according to the resampled sequence samples of the local collected audio and the resampled sequence samples of the local played audio;
the video overlapping module overlaps the local acquisition audio curve bitmap and the local playing audio curve bitmap to a local video frame for displaying;
the packaging and summarizing module is used for packaging the resampling sequence samples of the locally acquired audio and the resampling sequence samples of the locally played audio generated by the secondary sampling module to generate a feedback data packet;
the feedback information transceiving module is used for sending the feedback data packet generated by the packaging and summarizing module to a remote terminal through a network and forwarding the received feedback data packet of the remote terminal to the unpacking and splitting module;
the unpacking and splitting module is used for unpacking and splitting the feedback data packet of the remote terminal and sending the obtained resampling sequence sample of the far-end collected audio and the resampling sequence sample of the far-end played audio to the curve drawing module;
the curve drawing module is also used for respectively drawing a far-end collected audio curve bitmap and a far-end played audio curve bitmap according to the resampling sequence sample of the far-end collected audio and the resampling sequence sample of the far-end played audio sent by the unpacking and splitting module;
and when the voice interaction is abnormal, judging the fault state and the fault range by combining the local acquisition audio curve bitmap, the local playing audio curve bitmap, the remote acquisition audio curve bitmap and the local visual curve bitmap of the remote playing audio curve bitmap.
7. A multimedia conference system terminal, characterized in that it comprises the audio state visualization playing and feedback device of claim 6 and executes the audio state visualization playing and feedback method of any one of claims 1-5.
8. A computer-readable storage medium, on which a computer program is stored, the computer program being executable by a processor to perform the audio state visualization playing and feedback method of any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010356399.2A CN111432161B (en) | 2020-04-29 | 2020-04-29 | Audio state visual playing and feedback method, device and terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010356399.2A CN111432161B (en) | 2020-04-29 | 2020-04-29 | Audio state visual playing and feedback method, device and terminal |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111432161A CN111432161A (en) | 2020-07-17 |
| CN111432161B true CN111432161B (en) | 2022-02-01 |
Family
ID=71558555
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010356399.2A Active CN111432161B (en) | 2020-04-29 | 2020-04-29 | Audio state visual playing and feedback method, device and terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111432161B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104052423A (en) * | 2013-03-15 | 2014-09-17 | 骷髅头有限公司 | Customizing Audio Reproduction Devices |
| CN105120070A (en) * | 2015-07-16 | 2015-12-02 | 努比亚技术有限公司 | Methods and terminal for playing music during call |
| CN106128470A (en) * | 2016-06-29 | 2016-11-16 | 努比亚技术有限公司 | The device and method of display audible spectrum |
| CN109785860A (en) * | 2019-01-08 | 2019-05-21 | 北京中创信测科技股份有限公司 | A kind of method that four domain six-dimension intelligent of 2G3G4G voice quality failure is delimited |
| CN110223709A (en) * | 2019-05-31 | 2019-09-10 | 维沃移动通信有限公司 | A kind of recording frequency spectrum display methods and terminal device |
| CN110602529A (en) * | 2019-09-12 | 2019-12-20 | 广州虎牙科技有限公司 | Live broadcast monitoring method and device, electronic equipment and machine-readable storage medium |
| CN110619894A (en) * | 2019-09-30 | 2019-12-27 | 北京淇瑀信息科技有限公司 | Emotion recognition method, device and system based on voice oscillogram |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8300841B2 (en) * | 2005-06-03 | 2012-10-30 | Apple Inc. | Techniques for presenting sound effects on a portable media player |
-
2020
- 2020-04-29 CN CN202010356399.2A patent/CN111432161B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104052423A (en) * | 2013-03-15 | 2014-09-17 | 骷髅头有限公司 | Customizing Audio Reproduction Devices |
| CN105120070A (en) * | 2015-07-16 | 2015-12-02 | 努比亚技术有限公司 | Methods and terminal for playing music during call |
| CN106128470A (en) * | 2016-06-29 | 2016-11-16 | 努比亚技术有限公司 | The device and method of display audible spectrum |
| CN109785860A (en) * | 2019-01-08 | 2019-05-21 | 北京中创信测科技股份有限公司 | A kind of method that four domain six-dimension intelligent of 2G3G4G voice quality failure is delimited |
| CN110223709A (en) * | 2019-05-31 | 2019-09-10 | 维沃移动通信有限公司 | A kind of recording frequency spectrum display methods and terminal device |
| CN110602529A (en) * | 2019-09-12 | 2019-12-20 | 广州虎牙科技有限公司 | Live broadcast monitoring method and device, electronic equipment and machine-readable storage medium |
| CN110619894A (en) * | 2019-09-30 | 2019-12-27 | 北京淇瑀信息科技有限公司 | Emotion recognition method, device and system based on voice oscillogram |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111432161A (en) | 2020-07-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9622052B1 (en) | Methods for using a caption device with a mobile device | |
| EP2901669B1 (en) | Near-end indication that the end of speech is received by the far end in an audio or video conference | |
| US11482240B2 (en) | Presentation of communications | |
| US20170127249A1 (en) | Method and On-Vehicle Terminal for Communication with Vehicles in Vehicle Fleet | |
| US20050221771A1 (en) | Receiving and sending method of mobile TV phone and mobile TV phone terminal | |
| EP1135921B1 (en) | System and method for ip-based communication having speech generated text | |
| WO2002003693B1 (en) | Advanced set top terminal having a video call feature | |
| JP2019050554A (en) | Method and apparatus for providing voice service | |
| CN111614929A (en) | Acoustic positioning video conference method, system and storage medium based on 5G transmission | |
| CN111432161B (en) | Audio state visual playing and feedback method, device and terminal | |
| WO2017157168A1 (en) | Method, terminal, system and computer storage medium for video calling | |
| CN109802968B (en) | Conference speaking system | |
| CN115514989B (en) | Data transmission method, system and storage medium | |
| JP5508605B1 (en) | COMMUNICATION CONTROL SYSTEM, COMMUNICATION CONTROL METHOD, COMMUNICATION CONTROL PROGRAM, TERMINAL, AND TERMINAL PROGRAM | |
| US20150237304A1 (en) | Method and device for processing an audio input state | |
| JP4385501B2 (en) | Customer terminal message output system | |
| US11431767B2 (en) | Changing a communication session | |
| CN116668626B (en) | Multipath video pre-monitoring method and system | |
| JP2008028884A (en) | Call center system for tv telephone communication | |
| CN109302542A (en) | Long-distance monitoring method, device, equipment and storage medium based on GSM | |
| JP2015095736A (en) | Communication system for on-vehicle device, on-vehicle device, call failure presentation method and call failure presentation program | |
| WO2022252928A1 (en) | Screen projection method and apparatus, wireless terminal, screen projection device and storage medium | |
| CN107819846A (en) | A kind of long-range connection method and device | |
| WO2001080573A2 (en) | Apparatus and method for remote troubleshooting | |
| TWM636507U (en) | A radio system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |