CN109635133B

CN109635133B - Visual audio playing method and device, electronic equipment and storage medium

Info

Publication number: CN109635133B
Application number: CN201811547567.5A
Authority: CN
Inventors: 蒋义勇
Original assignee: Tencent Music Entertainment Technology Shenzhen Co Ltd
Current assignee: Tencent Music Entertainment Technology Shenzhen Co Ltd
Priority date: 2018-12-18
Filing date: 2018-12-18
Publication date: 2022-12-09
Anticipated expiration: 2038-12-18
Also published as: CN109635133A

Abstract

The invention discloses a visual audio playing method and device, electronic equipment and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: acquiring a first time stamp and a second time stamp; determining a third timestamp according to the first timestamp and the second timestamp; according to the third time stamp, acquiring the target audio characteristics from the stored audio characteristics and the time stamp corresponding to the audio characteristics; generating and rendering an image indicating a target audio feature; and when the audio is played to the time indicated by the third timestamp, displaying the rendered image. According to the method and the device, the current playing time and the total time required by the current image generation and image rendering are comprehensively considered to determine the third time stamp, and further determine the target audio characteristics to perform the image generation and rendering, so that the rendered image is displayed when the audio is played to the time indicated by the third time stamp, and therefore the audio played at the same time is consistent with the displayed image, and the effect of synchronizing sound and pictures is achieved.

Description

Visual audio playing method and device, electronic equipment and storage medium

Technical Field

The present invention relates to the field of computer technologies, and in particular, to a method and an apparatus for playing a visual audio, an electronic device, and a storage medium.

Background

With the gradual development of computer technology, people increasingly use electronic devices such as mobile phones and tablet computers in work and life. When a user uses the electronic device to play audio, in order to meet the visual requirement of the user, the image matched with the audio can be displayed while the audio is played, so that the effect of visually playing the audio is realized.

In the related art, as shown in fig. 1, the acquired audio data is usually decoded first to obtain decoded audio data. And then sending the decoded audio data to a sound card buffer area, and acquiring the decoded audio data from the sound card buffer area by the sound card for playing. However, if the speed of storing the audio data into the sound card buffer is higher than the playing speed, the decoded audio data currently stored into the sound card buffer may have a playing delay, so that an error may exist between the currently displayed image and the currently played audio data, and a problem may occur that the currently played audio is not matched with the currently displayed image.

Disclosure of Invention

The embodiment of the invention provides a visual audio playing method and device, electronic equipment and a storage medium, which can solve the problems in the related art. The technical scheme is as follows:

in a first aspect, a visual audio playing method is provided, where the method includes:

acquiring a first time stamp and a second time stamp, wherein the first time stamp is used for indicating the current playing time, and the second time stamp is used for indicating the total time length of image generation and rendering at the current playing time;

determining a third timestamp according to the first timestamp and the second timestamp, wherein the third timestamp is used for indicating the time when the total time length is elapsed from the current playing time;

acquiring target audio characteristics from the stored audio characteristics and timestamps corresponding to the audio characteristics according to the third timestamps;

generating and rendering an image indicating the target audio feature;

and displaying the rendered image when the audio is played to the time indicated by the third timestamp.

Optionally, before obtaining the first timestamp and the second timestamp, the method further includes:

acquiring an audio data packet;

decoding the audio data packet to obtain decoded audio data;

and determining and storing the audio characteristics of the decoded audio data and the time stamp corresponding to the audio characteristics of the decoded audio data.

Optionally, the determining the audio feature of the decoded audio data and the timestamp corresponding to the audio feature of the decoded audio data includes:

performing Fast Fourier Transform (FFT) on the decoded audio data to obtain audio characteristics of the decoded audio data;

and determining the product of the number of channels for playing the audio, the sampling rate of the audio and the bit depth of the audio, and taking the quotient of the total data quantity of the audio data obtained by decoding at the current moment and before the current moment and the product as the time stamp corresponding to the decoded audio features.

Optionally, after the decoding the audio data packet to obtain decoded audio data, the method further includes:

storing the decoded audio data to a sound card buffer area;

and acquiring the decoded audio data from the sound card buffer area through a sound card to play.

Optionally, the obtaining, according to the third timestamp, the target audio feature from the stored audio feature and a timestamp corresponding to the audio feature includes:

searching audio features with corresponding timestamps equal to the third timestamp from the stored audio features and timestamps corresponding to the audio features;

if the target audio characteristic is found, determining the found audio characteristic as the target audio characteristic;

if the audio feature is not found, acquiring the audio feature corresponding to the timestamp with the minimum difference value between the third timestamps from the stored audio features and timestamps corresponding to the audio features; or acquiring an audio feature corresponding to a timestamp which is smaller than the third timestamp and has the smallest difference with the third timestamp; or acquiring an audio feature corresponding to a timestamp which is greater than the third timestamp and has the smallest difference with the third timestamp; and determining the obtained audio features as the target audio features.

Optionally, the obtaining the first timestamp and the second timestamp includes:

determining a product of the number of channels playing the audio, the sampling rate of the audio and the bit depth of the audio, and taking a quotient value between the product and the total data amount of the audio data played at the current time and before the current time as the first timestamp;

acquiring the average generation duration of a preset number of frame images generated before the image used for indicating the target audio characteristic is generated and the average rendering duration of the preset number of frame images;

and determining the sum of the average generation time length and the average rendering time length as the total time length for generating and rendering the image at the current playing time, and taking the time stamp corresponding to the total time length as the second time stamp.

In a second aspect, a visual audio playing device is provided, the device including:

the device comprises a first acquisition module and a second acquisition module, wherein the first acquisition module is used for acquiring a first time stamp and a second time stamp, the first time stamp is used for indicating the current playing time, and the second time stamp is used for indicating the total time length for generating and rendering images at the current playing time;

a first determining module, configured to determine a third timestamp according to the first timestamp and the second timestamp, where the third timestamp is used to indicate a time when the total duration elapses from the current playing time;

the second obtaining module is used for obtaining the target audio characteristics from the stored audio characteristics and the timestamps corresponding to the audio characteristics according to the third timestamps;

the generating module is used for generating and rendering an image used for indicating the target audio characteristics;

and the display module is used for displaying the rendered image when the audio is played to the moment indicated by the third timestamp.

Optionally, the apparatus further comprises:

the third acquisition module is used for acquiring the audio data packet;

the decoding module is used for decoding the audio data packet to obtain decoded audio data;

and the second determining module is used for determining and storing the audio characteristics of the decoded audio data and the time stamp corresponding to the audio characteristics of the decoded audio data.

Optionally, the first determining module includes:

the first determining submodule is used for carrying out FFT (fast Fourier transform) on the decoded audio data to obtain the audio characteristics of the decoded audio data;

and the second determining submodule is used for determining the product of the number of channels for playing the audio, the sampling rate of the audio and the bit depth of the audio, and taking the quotient of the total data volume of the audio data obtained by decoding at the current moment and before the current moment and the product as the timestamp corresponding to the decoded audio characteristic.

Optionally, the apparatus further comprises:

the storage module is used for storing the decoded audio data into a sound card buffer area;

and the playing module is used for acquiring the decoded audio data from the sound card buffer area through a sound card and playing the audio data.

Optionally, the second obtaining module includes:

the searching submodule is used for searching the audio features of which the corresponding time stamps are equal to the third time stamp from the stored audio features and the time stamps corresponding to the audio features;

a third determining submodule, configured to determine, if the target audio feature is found, the found audio feature as the target audio feature;

the third determining submodule is further configured to, if the audio feature is not found, obtain, from the stored audio feature and the timestamp corresponding to the audio feature, the audio feature corresponding to the timestamp with the smallest difference between the third timestamps; or acquiring an audio feature corresponding to a timestamp which is smaller than the third timestamp and has the smallest difference with the third timestamp; or acquiring an audio feature corresponding to a timestamp which is larger than the third timestamp and has the smallest difference value with the third timestamp; and determining the obtained audio features as the target audio features.

Optionally, the first obtaining module includes:

a fourth determining sub-module, configured to determine a product of the number of channels playing the audio, the sampling rate of the audio, and the bit depth of the audio, and use a quotient between a total data amount of audio data played at and before the current time and the product as the first timestamp;

the acquisition submodule is used for acquiring the average generation duration of a preset number of frame images generated before the images used for indicating the target audio characteristics are generated and the average rendering duration of the preset number of frame images;

and the fifth determining submodule is used for determining the sum of the average generation time length and the average rendering time length as the total time length of image generation and rendering at the current playing time, and taking a time stamp corresponding to the total time length as the second time stamp.

In a third aspect, an electronic device is provided, which includes:

a processor and a memory;

wherein the memory has stored therein at least one instruction, which is loaded and executed by the processor to implement the visual audio playing method according to the first aspect.

In a fourth aspect, a computer-readable storage medium is provided, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement the visual audio playing method according to the first aspect.

According to the embodiment of the invention, the third timestamp is determined by acquiring the first timestamp and the second timestamp, comprehensively considering the current playing time and the total time required for image generation and image rendering at the current playing time. And then according to the third timestamp, acquiring the target audio characteristics from the stored audio characteristics and the timestamp corresponding to the audio characteristics, generating an image for indicating the target audio characteristics and rendering the image, so that the finally obtained image corresponds to the audio data played by the third timestamp, and displaying the rendered image when the audio is played to the moment indicated by the third timestamp, thereby ensuring that the audio data played at the same moment is consistent with the displayed image, achieving the effect of sound-picture synchronization, and solving the problem that the currently played audio is not matched with the currently displayed image in the related technology.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a flow chart of a visual audio playing in the prior art provided by an embodiment of the present invention;

fig. 2 is a flowchart of a visual audio playing method according to an embodiment of the present invention;

fig. 3 is a flowchart of another visual audio playing method provided by the embodiment of the present invention;

FIG. 4 is a schematic diagram of a display image according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of another display image provided by an embodiment of the invention;

FIG. 6 is a schematic diagram of yet another display image provided by an embodiment of the invention;

fig. 7 is a flowchart of another visual audio playing provided by the embodiment of the present invention;

fig. 8 is a schematic structural diagram of a visual audio playing apparatus according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Fig. 2 is a flowchart of a visual audio playing method according to an embodiment of the present invention, and referring to fig. 2, the method includes the following steps:

step 201: the method comprises the steps of obtaining a first time stamp and a second time stamp, wherein the first time stamp is used for indicating the current playing time, and the second time stamp is used for indicating the total time length of image generation and rendering at the current playing time.

Step 202: and determining a third time stamp according to the first time stamp and the second time stamp, wherein the third time stamp is used for indicating the time when the total time length is elapsed from the current playing time.

Step 203: and acquiring the target audio feature from the stored audio feature and the timestamp corresponding to the audio feature according to the third timestamp.

Step 204: and generating and rendering an image indicating the target audio characteristic.

Step 205: and when the audio is played to the time indicated by the third timestamp, displaying the rendered image.

In summary, in the embodiments of the present invention, a first timestamp and a second timestamp are obtained, where the first timestamp is used to indicate a current playing time, and the second timestamp is used to indicate a total duration of image generation and rendering at the current playing time; and comprehensively considering the current playing time and the total time length required for image generation and image rendering at the current playing time, and determining a third timestamp. And then according to the third timestamp, acquiring the target audio characteristics from the stored audio characteristics and the timestamp corresponding to the audio characteristics, generating an image for indicating the target audio characteristics and rendering the image, so that the finally obtained image corresponds to the audio data played by the third timestamp, and displaying the rendered image when the audio is played to the moment indicated by the third timestamp, thereby ensuring that the audio data played at the same moment is consistent with the displayed image, achieving the effect of sound-picture synchronization, and solving the problem that the currently played audio is not matched with the currently displayed image in the related technology.

Optionally, before acquiring the first timestamp and the second timestamp, the method further includes:

acquiring an audio data packet;

decoding the audio data packet to obtain decoded audio data;

Optionally, determining the audio feature of the decoded audio data and the timestamp corresponding to the audio feature of the decoded audio data includes:

performing FFT on the decoded audio data to obtain audio characteristics of the decoded audio data;

determining the product of the number of channels playing the audio, the sampling rate of the audio and the bit depth of the audio, and taking the quotient of the total data volume of the audio data decoded at the current moment and before the current moment and the product as the time stamp corresponding to the decoded audio features.

Optionally, after decoding the audio data packet to obtain decoded audio data, the method further includes:

storing the decoded audio data into a sound card buffer area;

and acquiring the decoded audio data from the sound card buffer area through the sound card for playing.

Optionally, according to the third timestamp, obtaining the target audio feature from the stored audio feature and the timestamp corresponding to the audio feature includes:

if the audio features are found, determining the found audio features as target audio features;

if the audio feature is not found, acquiring the audio feature corresponding to the timestamp with the minimum difference value between the third timestamps from the stored audio features and the timestamps corresponding to the audio features; or acquiring the audio features which are smaller than the third time stamps and correspond to the time stamps with the minimum difference value; or acquiring the audio features which are larger than the third time stamps and correspond to the time stamps with the minimum difference value; and determining the acquired audio features as target audio features.

Optionally, obtaining the first timestamp and the second timestamp comprises:

determining the product of the number of channels playing the audio, the sampling rate of the audio and the bit depth of the audio, and taking the quotient of the total data volume of the audio data played at the current time and before the current time and the product as a first time stamp;

acquiring the average generation duration of a preset number of frame images generated before an image used for indicating the target audio characteristic is generated and the average rendering duration of the preset number of frame images;

and determining the sum of the average generation time length and the average rendering time length as the total time length for generating and rendering the image at the current playing time, and taking a time stamp corresponding to the total time length as a second time stamp.

All the above optional technical solutions can be combined arbitrarily to form an optional embodiment of the present invention, which is not described in detail herein.

Fig. 3 is a flowchart of a visual audio playing method according to an embodiment of the present invention, which is applied to an electronic device. The embodiment of the present invention will now be described in detail with reference to the embodiment shown in fig. 1, and with reference to fig. 3, the method comprises the following steps:

step 301: and acquiring an audio data packet, and decoding the audio data packet to obtain decoded audio data.

The embodiment of the present invention may be applied to a scenario where an electronic device plays any audio, and each audio is composed of multiple audio data packets, and when the audio data packets are obtained in step 301, the audio data packets are usually obtained one by one, and accordingly, the audio data packets are decoded, that is, each time an audio data packet is obtained, the audio data packet is decoded to obtain decoded audio data.

It should be noted that the compression of the audio is generally implemented by encoding the recorded analog audio signal into a digital audio signal, so as to increase the transmission rate during transmission and reduce the occupied storage resources during storage. Then, in the embodiment of the present invention, the obtained data packet is a digital audio signal, and when playing audio, the digital audio signal needs to be converted into an analog audio signal, and then played through the sound card, so that when obtaining the audio data packet, the audio data packet needs to be decoded to obtain decoded audio data, so as to perform subsequent playing. The decoded audio data is generally PCM (Pulse Code Modulation) data.

After decoding the audio data packet to obtain decoded audio data, storing the decoded audio data into a sound card buffer area; and then the electronic equipment can acquire the decoded audio data from the sound card buffer area through the sound card to play.

It should be noted that the sound card buffer is a buffer inside the sound card and is used for storing decoded audio data required by the sound card to play audio. And the sound card acquires the decoded audio data from the sound card buffer zone according to a certain play speed to play.

Step 302: and determining and storing the audio characteristics of the decoded audio data and the time stamp corresponding to the audio characteristics of the decoded audio data.

The audio features at least include any one of frequency spectrum and loudness, but may also include others, such as beats. The time stamp corresponding to the audio feature of the decoded audio data is: and displaying an image indicating the audio characteristics of the decoded audio data.

When determining the audio feature of the decoded audio data, the FFT may be performed on the decoded audio data to obtain the audio feature of the decoded audio data. When the time stamp corresponding to the audio feature of the decoded audio data is determined, the product of the number of channels playing the audio, the sampling rate of the audio and the bit depth of the audio can be determined, and the quotient of the product and the total data amount of the audio data decoded at the current time and before the current time is used as the time stamp corresponding to the decoded audio feature.

It should be noted that the number of channels playing the audio refers to the number of sound devices in the sound card when the audio is played, the sampling rate of the audio refers to the number of times of sampling the audio in unit time, the bit depth of the audio refers to the number of bits of each sampling value, and the bit depth generally includes 8bits and 16bits. The quotient between the total data volume of the audio data decoded at and before the current time and the product is: the total data amount of the audio data decoded at and before the current time is divided by the product.

Because the decoded audio data is a continuous time signal, the decoded audio data can be sampled to obtain discrete sampling values of the continuous time signal, and then the discrete sampling values are subjected to FFT (fast Fourier transform) to convert the discrete sampling values of the continuous time signal from a time domain to a frequency domain to obtain the audio characteristics of the continuous time signal. When the audio characteristic is a frequency spectrum, the frequency spectrum of the continuous-time signal can be obtained through the above process. When the audio feature comprises loudness, it is also necessary to calculate the loudness of the continuous-time signal from the frequency of the continuous-time signal. Of course, when the audio features further include other features such as a beat, the spectrum of the continuous-time signal can be obtained by the above method, and the beat of the decoded audio data is extracted from the spectrum.

After obtaining the audio feature of the decoded audio data and the timestamp corresponding to the audio feature of the decoded audio data, the audio feature of the decoded audio data and the timestamp corresponding to the audio feature of the decoded audio data may be stored. For example, the audio features of the decoded audio data and the timestamps corresponding to the audio features of the decoded audio data may be stored directly in a ring buffer of the electronic device. Of course, the corresponding relationship between the audio features and the time stamps may be established first, and the time stamps corresponding to the audio features of the decoded audio data may be stored in the ring buffer in the form of the corresponding relationship.

Step 303: and acquiring a first time stamp and a second time stamp, wherein the first time stamp is used for indicating the current playing time, and the second time stamp is used for indicating the total time length of image generation and rendering at the current playing time.

The first time stamp is the current playing time of the audio, and the second time stamp is the time that is reached after the total time of image generation and rendering from the current playing time of the audio.

When the first time stamp is obtained, the product of the number of channels playing the audio, the sampling rate of the audio, and the bit depth of the audio may be determined, and a quotient between the current time and the product of the total data amount of the audio data played before the current time and the product may be used as the first time stamp. When the second timestamp is acquired, an average generation time length of a preset number of frame images generated before the image indicating the target audio feature is generated and an average rendering time length of the preset number of frame images can be acquired; and determining the sum of the average generation time length and the average rendering time length as the total time length for generating and rendering the image at the current playing time, and taking a time stamp corresponding to the total time length as a second time stamp.

It should be noted that, a quotient between the total data amount of the audio data played at the current time and before the current time and the product is: the total data amount of the audio data played at and before the current time is divided by the product.

In the embodiment of the present invention, the second timestamp is determined by generating the average generation duration and the average rendering duration of the preset number of frame images generated before the image indicating the target audio feature is generated, that is, the average generation duration of the preset number of frame images generated before is used as the generation duration for generating the image indicating the target audio feature, and the average rendering duration of the preset number of frame images generated before is used as the rendering duration for rendering the image indicating the target audio feature, so that the duration required for generating and rendering the image indicating the target audio feature can be predicted more accurately, and the determined second timestamp is more accurate, thereby facilitating accurate subsequent determination of the target audio feature.

In addition, when the number of images generated before the image indicating the target audio feature is generated is less than the preset number, an average generation time length and an average rendering time length of all images generated before the image indicating the target audio feature is generated may be acquired, and a sum between the average generation time length and the average rendering time length of all images generated before may be determined as the second time stamp. Wherein when the image indicating the target audio feature is the first frame image in which the audio is played, i.e., there is no image generated before the image indicating the target audio feature is generated, then it may be determined that the second timestamp is zero.

In another possible implementation, the second timestamp is directly set to zero when fewer than a preset number of images are generated before the image indicative of the target audio feature is generated.

Step 304: and determining a third time stamp according to the first time stamp and the second time stamp, wherein the third time stamp is used for indicating the time when the total time length is elapsed from the current playing time.

When the third timestamp is determined according to the first timestamp and the second timestamp, namely, a sum value between the first timestamp and the second timestamp is determined, and the sum value is determined as the third timestamp.

In the embodiment of the invention, a certain time length is required for generating and rendering each frame of image, and the data amount required to be processed by the electronic equipment at different times is different, that is, the processing capacity of the electronic equipment at different times may be different, so that the time lengths required for generating and rendering images at different times are different. And the speed of storing the audio data into the sound card buffer is higher than the playing speed, so that the situation that the playing of the decoded audio data currently stored into the sound card buffer is delayed may occur. Therefore, the third timestamp is determined according to the current playing time and the total time length of image generation and rendering from the current playing time, and the third timestamp can be determined more accurately so as to perform subsequent processing, and the effect that the audio data played at the same time is consistent with the displayed image is achieved.

Step 305: and acquiring the target audio characteristic from the stored audio characteristic and the timestamp corresponding to the audio characteristic according to the third timestamp.

When the target audio feature is obtained from the stored audio feature and the time stamp corresponding to the audio feature according to the third time stamp, the time stamp matching the third time stamp may be determined from the time stamp corresponding to the stored audio feature, then the audio feature corresponding to the time stamp may be obtained from the stored audio feature, and the obtained audio feature may be determined as the target audio feature.

In a possible implementation manner, when the target audio feature is obtained from the stored audio feature and the timestamp corresponding to the audio feature according to the third timestamp, the audio feature whose corresponding timestamp is equal to the third timestamp may be searched from the stored audio feature and the timestamp corresponding to the audio feature; and if so, determining the found audio features as target audio features. If the audio feature is not found, acquiring the audio feature corresponding to the timestamp with the minimum difference value between the third timestamps from the stored audio features and the timestamps corresponding to the audio features; or acquiring an audio feature corresponding to a timestamp which is smaller than the third timestamp and has the smallest difference with the third timestamp; or acquiring an audio feature corresponding to a timestamp which is larger than the third timestamp and has the smallest difference with the third timestamp; and determining the acquired audio features as target audio features.

The time stamp corresponding to the audio feature is a time when an image indicating the audio feature is generated and rendered, and the rendered image is displayed. By determining the audio features corresponding to the timestamps equal to the third timestamp as the target audio features, the image display time and the audio playing time can be matched, so that the images to be displayed at any time can be displayed at the time when the audio is played, and the played audio data and the displayed images are kept consistent.

In addition, in playing the time stamp capable of corresponding the stored audio feature, there may be no time stamp equal to the third time stamp. Then the audio feature whose corresponding timestamp is closest to the third timestamp may be determined to be the target audio feature, or the audio feature corresponding to a timestamp that is less than and closest to the third timestamp may be determined to be the target audio feature; or the audio characteristic corresponding to the timestamp which is greater than the third timestamp and closest to the third timestamp is the target audio characteristic. Thus, when the timestamp equal to the third timestamp does not exist, the target audio characteristic is determined through the three modes, so that the difference value between the timestamp corresponding to the determined target audio characteristic and the third timestamp is small, and the finally played audio data is guaranteed to be matched with the displayed image.

Step 306: and generating and rendering an image indicating the target audio characteristic.

The style of the image for indicating the target audio feature may be preset, for example, the image may be a rectangle, a circle, a wave, a flame, a constellation, or other patterns, when the image for indicating the target audio feature is generated and rendered, a corresponding image may be generated according to a change of the target audio feature, and the generated image is rendered according to the set rendering parameters, so as to display images of different styles, increase playability, and improve user viscosity.

For example, assuming that the target audio feature is a frequency spectrum, as shown in fig. 4, if the pattern of the set image is a rectangle, values in the frequency spectrum may be represented by the height of the rectangle; if the pattern of the arranged image is wave-shaped, as shown in fig. 5, values in the frequency spectrum can be represented as undulations in the longitudinal direction of the wave; as shown in fig. 6, if the pattern of the set image is a flame shape, a value in the spectrum may be represented by a height of the flame, thereby generating and rendering a spectrogram indicating the spectrum. Other shapes, such as diamond, water drop, etc., may of course be provided. For example, if the audio feature is loudness and the pattern of the image that can be set is a circle, the change in loudness may be represented by the radius of the circle, thereby generating and rendering an image indicating loudness. Of course, other patterns, such as rectangles, diamonds, constellation patterns, etc., may be provided.

In addition, in generating an image, a generation time period for generating the image may be obtained by counting from a time when image generation is started until the image indicating the target audio feature is completely generated and stopping. Accordingly, when rendering an image, the time from the moment when the image rendering is started may be counted until the time when the image rendering indicating the target audio feature is completed is stopped, resulting in the rendering time period for rendering the image.

Step 307: and when the audio is played to the time indicated by the third timestamp, displaying the rendered image.

Since the image is generated and rendered according to the target audio feature determined by the third timestamp, when the image is an image matching the time indicated by the third timestamp, the image is displayed when the audio is played to the time indicated by the third timestamp, that is, when the playing time is the time indicated by the third timestamp, the image matching the time is displayed, so that the effect of sound-picture synchronization can be achieved.

In a possible implementation manner, as shown in fig. 7, the visual audio playing method provided in the embodiment of the present invention may be implemented by a decoding module, an audio feature determining module, a feature buffer, a UI (User Interface), an image generating module, an image rendering module, a sound card buffer, and a sound card of an electronic device.

The electronic equipment acquires the audio data packet, decodes the audio data packet through the decoding module, and then sends the decoded audio data to the audio characteristic calculation module and the sound card buffer area through the decoding module. And the sound card acquires the decoded audio data from the sound card buffer area to play. When receiving the decoded audio data, the audio characteristic determination module determines the audio characteristics of the decoded audio data and the time stamps corresponding to the audio characteristics of the decoded audio data, and stores the time stamps in the characteristic buffer area. The UI obtains the current time and the total data volume of audio data played before the current time from the sound card so as to determine a first time stamp, obtain a second time stamp, determine a third time stamp according to the first time stamp and the second time stamp, and obtain target audio characteristics from audio characteristics stored in the characteristic buffer zone and time stamps corresponding to the audio characteristics according to the third time stamp. The UI then sends the target audio feature to the image generation module. The image generating module generates an image used for indicating the target audio characteristics and sends the image to the image rendering module, and the image rendering module renders the image so that the electronic equipment can display the rendered image through the display screen at the moment indicated by the third time stamp of the audio playing value.

Fig. 8 is a schematic structural diagram of a visual audio playing apparatus according to an embodiment of the present invention. Referring to fig. 8, the apparatus includes: a first acquisition module 801, a first determination module 802, a second acquisition module 803, a generation module 804, and a display module 805.

A first obtaining module 801, configured to obtain a first timestamp and a second timestamp, where the first timestamp is used to indicate a current playing time, and the second timestamp is used to indicate a total duration of image generation and rendering at the current playing time;

a first determining module 802, configured to determine a third timestamp according to the first timestamp and the second timestamp, where the third timestamp is used to indicate a time when the total duration elapses from the current playing time;

a second obtaining module 803, configured to obtain, according to the third timestamp, the target audio feature from the stored audio feature and the timestamp corresponding to the audio feature;

a generating module 804, configured to generate and render an image indicating a target audio feature;

and a display module 805, configured to display the rendered image when the audio is played to a time indicated by the third timestamp.

Optionally, the apparatus further comprises:

the third acquisition module is used for acquiring the audio data packet;

Optionally, the first determining module 802 includes:

and the second determining submodule is used for determining the product of the number of channels for playing the audio, the sampling rate of the audio and the bit depth of the audio, and taking the quotient of the total data volume of the audio data obtained by decoding at the current moment and before the current moment and the product as the time stamp corresponding to the decoded audio features.

Optionally, the apparatus further comprises:

and the playing module is used for acquiring the decoded audio data from the sound card buffer area through the sound card and playing the audio data.

Optionally, the second obtaining module 803 includes:

the searching submodule is used for searching the audio features with the corresponding timestamps equal to the third timestamp from the stored audio features and the timestamps corresponding to the audio features;

the third determining submodule is used for determining the searched audio features as target audio features if the audio features are searched;

the third determining submodule is further used for acquiring the audio feature corresponding to the timestamp with the minimum difference value with the third timestamp from the stored audio feature and the timestamp corresponding to the audio feature if the audio feature is not found; or acquiring the audio features which are smaller than the third time stamps and correspond to the time stamps with the minimum difference value; or acquiring an audio feature corresponding to a timestamp which is larger than the third timestamp and has the smallest difference with the third timestamp; and determining the acquired audio features as target audio features.

Optionally, the first obtaining module 801 includes:

a fourth determining sub-module, configured to determine a product of the number of channels playing the audio, the sampling rate of the audio, and the bit depth of the audio, and use a quotient between a total data amount of audio data played at and before the current time and the product as a first timestamp;

an obtaining submodule, configured to obtain an average generation duration of a preset number of frame images generated before an image indicating the target audio feature is generated, and an average rendering duration of the preset number of frame images;

and the fifth determining submodule is used for determining the sum of the average generation time length and the average rendering time length as the total time length of image generation and rendering at the current playing time, and taking a time stamp corresponding to the total time length as a second time stamp.

It should be noted that: in the visual audio playing device provided in the above embodiment, when the visual audio is played, only the division of the above functional modules is used for illustration, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules, so as to complete all or part of the above described functions. In addition, the visual audio playing device and the visual audio playing method provided by the above embodiments belong to the same concept, and specific implementation processes thereof are described in the method embodiments and are not described herein again.

Fig. 9 shows a block diagram of an electronic device according to an exemplary embodiment of the present invention. The electronic device 900 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III, motion Picture Experts compression standard Audio Layer 3), an MP4 player (Moving Picture Experts Group Audio Layer IV, motion Picture Experts compression standard Audio Layer 4), a notebook computer, or a desktop computer. Electronic device 900 may also be referred to by other names as user equipment, portable electronic device, laptop electronic device, desktop electronic device, and so on.

In general, electronic device 900 includes: a processor 901 and a memory 902.

Processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and so on. The processor 901 may be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). Processor 901 may also include a main processor and a coprocessor, where the main processor is a processor for Processing data in a wake state, and is also called a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the display screen. In some embodiments, the processor 901 may further include an AI (Artificial Intelligence) processor for processing computing operations related to machine learning.

Memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory, as well as non-volatile memory, such as one or more magnetic disk storage devices, flash memory storage devices. In some embodiments, a non-transitory computer readable storage medium in memory 902 is used to store at least one instruction for execution by processor 901 to implement a visual audio playback method provided by method embodiments of the present invention.

In some embodiments, the electronic device 900 may further optionally include: a peripheral interface 903 and at least one peripheral. The processor 901, memory 902, and peripheral interface 903 may be connected by buses or signal lines. Various peripheral devices may be connected to the peripheral interface 903 via a bus, signal line, or circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 904, a touch display screen 905, a camera 906, an audio circuit 907, a positioning component 908, and a power supply 909.

The peripheral interface 903 may be used to connect at least one peripheral related to I/O (Input/Output) to the processor 901 and the memory 902. In some embodiments, the processor 901, memory 902, and peripheral interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 901, the memory 902 and the peripheral interface 903 may be implemented on a separate chip or circuit board, which is not limited by this embodiment.

The Radio Frequency circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuitry 904 communicates with communication networks and other communication devices via electromagnetic signals. The radio frequency circuit 904 converts an electrical signal into an electromagnetic signal to transmit, or converts a received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 904 comprises: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and so forth. The radio frequency circuitry 904 may communicate with other electronic devices via at least one wireless communication protocol. The wireless communication protocols include, but are not limited to: metropolitan area networks, various generation mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and/or WiFi (Wireless Fidelity) networks. In some embodiments, the radio frequency circuit 904 may further include NFC (Near Field Communication) related circuits, which are not limited in this respect.

The display screen 905 is used to display a UI (User Interface). The UI may include graphics, text, icons, video, and any combination thereof. When the display screen 905 is a touch display screen, the display screen 905 also has the ability to capture touch signals on or over the surface of the display screen 905. The touch signal may be input to the processor 901 as a control signal for processing. At this point, the display 905 may also be used to provide virtual buttons and/or a virtual keyboard, also referred to as soft buttons and/or a soft keyboard. In some embodiments, the display 905 may be one, providing a front panel of the electronic device 900; in other embodiments, the number of the display screens 905 may be at least two, and each of the display screens is disposed on a different surface of the electronic device 900 or has a foldable design; in still other embodiments, the display 905 may be a flexible display disposed on a curved surface or on a folded surface of the electronic device 900. Even more, the display screen 905 may be arranged in a non-rectangular irregular figure, i.e. a shaped screen. The Display panel 905 may be made of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other materials.

The camera assembly 906 is used to capture images or video. Optionally, camera assembly 906 includes a front camera and a rear camera. Generally, a front camera is disposed on a front panel of an electronic apparatus, and a rear camera is disposed on a rear surface of the electronic apparatus. In some embodiments, the number of the rear cameras is at least two, and each rear camera is any one of a main camera, a depth-of-field camera, a wide-angle camera and a telephoto camera, so that the main camera and the depth-of-field camera are fused to realize a background blurring function, the main camera and the wide-angle camera are fused to realize panoramic shooting and a VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, camera assembly 906 may also include a flash. The flash lamp can be a monochrome temperature flash lamp or a bicolor temperature flash lamp. The double-color-temperature flash lamp is a combination of a warm-light flash lamp and a cold-light flash lamp, and can be used for light compensation at different color temperatures.

Audio circuit 907 may include a microphone and a speaker. The microphone is used for collecting sound waves of a user and the environment, converting the sound waves into electric signals, and inputting the electric signals to the processor 901 for processing, or inputting the electric signals to the radio frequency circuit 904 for realizing voice communication. For stereo capture or noise reduction purposes, the microphones may be multiple and located at different locations of the electronic device 900. The microphone may also be an array microphone or an omni-directional pick-up microphone. The speaker is used to convert electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The loudspeaker can be a traditional film loudspeaker and can also be a piezoelectric ceramic loudspeaker. When the speaker is a piezoelectric ceramic speaker, the speaker can be used for purposes such as converting an electric signal into a sound wave audible to a human being, or converting an electric signal into a sound wave inaudible to a human being to measure a distance. In some embodiments, audio circuit 907 may also include a headphone jack.

The positioning component 908 is used to locate a current geographic Location of the electronic device 900 to implement a navigation or LBS (Location Based Service). The Positioning component 908 may be a Positioning component based on the GPS (Global Positioning System) in the united states, the beidou System in china, the graves System in russia, or the galileo System in the european union.

The power supply 909 is used to supply power to the various components in the electronic device 900. The power source 909 may be alternating current, direct current, disposable or rechargeable. When the power source 909 includes a rechargeable battery, the rechargeable battery may support wired charging or wireless charging. The rechargeable battery may also be used to support fast charge technology.

In some embodiments, the electronic device 900 also includes one or more sensors 910. The one or more sensors 910 include, but are not limited to: an acceleration sensor 911, a gyro sensor 912, a pressure sensor 913, a fingerprint sensor 914, an optical sensor 915, and a proximity sensor 916.

The acceleration sensor 911 may detect the magnitude of acceleration in three coordinate axes of a coordinate system established with the electronic device 900. For example, the acceleration sensor 911 may be used to detect the components of the gravitational acceleration in three coordinate axes. The processor 901 can control the touch display 905 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 911. The acceleration sensor 911 may also be used for acquisition of motion data of a game or a user.

The gyro sensor 912 may detect a body direction and a rotation angle of the electronic device 900, and the gyro sensor 912 and the acceleration sensor 911 cooperate to acquire a 3D motion of the user on the electronic device 900. The processor 901 can implement the following functions according to the data collected by the gyro sensor 912: motion sensing (such as changing the UI according to a user's tilting operation), image stabilization at the time of photographing, game control, and inertial navigation.

The pressure sensor 913 may be disposed on a side bezel of the electronic device 900 and/or underneath the touch display screen 905. When the pressure sensor 913 is disposed on the side frame of the electronic device 900, the user's holding signal of the electronic device 900 may be detected, and the processor 901 performs left-right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 913. When the pressure sensor 913 is disposed at a lower layer of the touch display 905, the processor 901 controls the operability control on the UI interface according to the pressure operation of the user on the touch display 905. The operability control comprises at least one of a button control, a scroll bar control, an icon control and a menu control.

The fingerprint sensor 914 is used for collecting a fingerprint of the user, and the processor 901 identifies the user according to the fingerprint collected by the fingerprint sensor 914, or the fingerprint sensor 914 identifies the user according to the collected fingerprint. Upon identifying that the user's identity is a trusted identity, processor 901 authorizes the user to perform relevant sensitive operations including unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings, etc. The fingerprint sensor 914 may be disposed on the front, back, or side of the electronic device 900. When a physical button or vendor Logo is provided on the electronic device 900, the fingerprint sensor 914 may be integrated with the physical button or vendor Logo.

The optical sensor 915 is used to collect ambient light intensity. In one embodiment, the processor 901 may control the display brightness of the touch display 905 based on the ambient light intensity collected by the optical sensor 915. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 905 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 905 is turned down. In another embodiment, the processor 901 can also dynamically adjust the shooting parameters of the camera assembly 906 according to the ambient light intensity collected by the optical sensor 915.

The proximity sensor 916, also referred to as a distance sensor, is typically disposed on a front panel of the electronic device 900. The proximity sensor 916 is used to capture the distance between the user and the front of the electronic device 900. In one embodiment, when the proximity sensor 916 detects that the distance between the user and the front face of the electronic device 900 gradually decreases, the processor 901 controls the touch display 905 to switch from the bright screen state to the dark screen state; when the proximity sensor 916 detects that the distance between the user and the front surface of the electronic device 900 becomes gradually larger, the processor 901 controls the touch display 905 to switch from the breath screen state to the bright screen state.

Those skilled in the art will appreciate that the configuration shown in fig. 9 is not limiting to the electronic device 900 and may include more or fewer components than shown, or some components may be combined, or a different arrangement of components may be used.

A non-transitory computer readable storage medium, wherein instructions of the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the visual audio playing method of fig. 2 or fig. 3 described above.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and should not be taken as limiting the scope of the present invention, which is intended to cover any modifications, equivalents, improvements, etc. within the spirit and scope of the present invention.

Claims

1. A visual audio playing method, the method comprising:

acquiring a first time stamp and a second time stamp, wherein the first time stamp is used for indicating the current playing time, and the second time stamp is used for indicating the total time length for generating and rendering the image at the current playing time;

generating and rendering an image indicating the target audio feature;

2. The method of claim 1, wherein prior to obtaining the first timestamp and the second timestamp, further comprising:

acquiring an audio data packet;

decoding the audio data packet to obtain decoded audio data;

3. The method of claim 2, wherein the determining the audio characteristics of the decoded audio data and the timestamps to which the audio characteristics of the decoded audio data correspond comprises:

4. The method of claim 2, wherein after decoding the audio data packet to obtain decoded audio data, further comprising:

storing the decoded audio data to a sound card buffer area;

5. The method of claim 1, wherein obtaining the target audio feature from the stored audio feature and the timestamp corresponding to the audio feature according to the third timestamp comprises:

if the audio feature is not found, acquiring the audio feature corresponding to the timestamp with the minimum difference value from the stored audio features and timestamps corresponding to the audio features; or acquiring an audio feature corresponding to a timestamp which is smaller than the third timestamp and has the smallest difference with the third timestamp; or acquiring an audio feature corresponding to a timestamp which is larger than the third timestamp and has the smallest difference value with the third timestamp; and determining the obtained audio features as the target audio features.

6. The method of claim 1, wherein the obtaining the first timestamp and the second timestamp comprises:

determining a product among the number of channels playing the audio, the sampling rate of the audio and the bit depth of the audio, and taking a quotient between a total data amount of audio data played at the current time and before the current time and the product as the first time stamp;

acquiring an average generation duration of a preset number of frame images generated before an image used for indicating the target audio feature is generated and an average rendering duration of the preset number of frame images;

and determining the sum of the average generation time length and the average rendering time length as the total time length for generating and rendering the image at the current playing time, and taking a time stamp corresponding to the total time length as the second time stamp.

7. A visual audio playback apparatus, the apparatus comprising:

and the display module is used for displaying the rendered image when the audio is played to the time indicated by the third timestamp.

8. The apparatus of claim 7, wherein the apparatus further comprises:

the third acquisition module is used for acquiring the audio data packet;

9. The apparatus of claim 8, wherein the first determining module comprises:

the first determining submodule is used for carrying out Fast Fourier Transform (FFT) on the decoded audio data to obtain the audio characteristics of the decoded audio data;

10. The apparatus of claim 8, wherein the apparatus further comprises:

and the playing module is used for acquiring the decoded audio data from the sound card buffer area through a sound card and playing the decoded audio data.

11. The apparatus of claim 7, wherein the second obtaining module comprises:

the searching submodule is used for searching the audio features with the corresponding timestamps equal to the third timestamps from the stored audio features and the timestamps corresponding to the audio features;

a third determining sub-module, configured to determine, if found, the found audio feature as the target audio feature;

the third determining sub-module is further configured to, if the audio feature is not found, obtain, from the stored audio feature and the timestamp corresponding to the audio feature, an audio feature corresponding to a timestamp with a minimum difference value between the audio feature and the third timestamp; or acquiring an audio feature corresponding to a timestamp which is smaller than the third timestamp and has the smallest difference with the third timestamp; or acquiring an audio feature corresponding to a timestamp which is larger than the third timestamp and has the smallest difference value with the third timestamp; and determining the obtained audio features as the target audio features.

12. The apparatus of claim 7, wherein the first obtaining module comprises:

and the fifth determining submodule is used for determining the sum of the average generation time length and the average rendering time length as the total time length for generating and rendering the image at the current playing time, and taking a time stamp corresponding to the total time length as the second time stamp.

13. An electronic device, characterized in that the electronic device comprises:

a processor and a memory;

wherein the memory has stored therein at least one instruction that is loaded and executed by the processor to implement the visual audio playing method of any one of claims 1 to 6.

14. A computer-readable storage medium having stored therein at least one instruction, which is loaded and executed by a processor, to implement the visual audio playback method as claimed in any one of claims 1 to 6.