WO2022110259A1

WO2022110259A1 - Vibration generation method, vibration control method, and related device thereof

Info

Publication number: WO2022110259A1
Application number: PCT/CN2020/133228
Authority: WO
Inventors: 郑亚军
Original assignee: 瑞声声学科技(深圳)有限公司; 瑞声光电科技(常州)有限公司
Priority date: 2020-11-24
Filing date: 2020-12-02
Publication date: 2022-06-02
Also published as: CN112466267A; CN112466267B

Abstract

A vibration generation method and apparatus, a vibration control method and system, an electronic device, and a storage medium. The vibration generation method comprises: step S22, identifying an audio beat in an audio signal; step S24, obtaining a plurality of note energies of the audio signal according to the audio beat; step S26, respectively performing normalization preprocessing on the plurality of note energies to obtain a plurality of normalized energies; and step S28, generating, according to the plurality of normalized energies, a haptic vibration file corresponding to the audio signal. By adoption of the method, the haptic vibration effect can be flexibly matched with the beat of the audio signal, and the matching accuracy is high.

Description

Vibration generation method, vibration control method and related equipment

technical field

The present invention relates to the technical field of vibration drive, and in particular, to a vibration generation method and device thereof, a vibration control method and system thereof, an electronic device and a storage medium.

Background technique

With the improvement of living standards, people have higher and higher requirements for the listening experience of music. People not only want to have a good hearing experience, but also want to have a tactile feeling.

In the related art, a vibration motor is provided in the music playing device, and when the music is playing, the motor is controlled to vibrate so that the music playing device produces a haptic vibration effect.

However, in the current vibration control method, due to the mismatch between the tactile vibration effect generated by the motor and the rhythm of the music, when playing music, the auditory feeling and the tactile feeling are not coordinated, resulting in poor user listening experience.

beneficial effect

Based on this, in view of the above technical problems, it is necessary to provide a vibration generation method and its device, a vibration control method and its system, an electronic device and a storage device that can flexibly match the tactile vibration effect with the beat of the audio signal and have high matching accuracy. medium.

The present invention provides a vibration generation method, the method includes:

Identify audio beats in audio signals;

Acquiring a plurality of note energies of the audio signal according to the audio beat;

Perform normalization preprocessing on a plurality of the note energies respectively to obtain a plurality of normalized energies; wherein, the note energies and the normalized energies are set in one-to-one correspondence;

A haptic vibration file corresponding to the audio signal is generated from a plurality of the normalized energies.

Preferably, the acquiring a plurality of note energies of the audio signal according to the audio beat includes:

Acquire multiple notes in the audio beat according to a preset time interval; wherein, the time interval between two adjacent said notes is the preset time interval;

Obtain the note energy for each said note.

Preferably, the step of obtaining the note energy of each of the notes includes:

In the same audio beat, according to the amplitude of the audio signal corresponding to each of the notes, the note energy of each of the notes is obtained by calculation.

Preferably, performing normalization preprocessing on a plurality of the note energies respectively, and obtaining a plurality of normalized energies includes:

In the same audio beat, the normalized energy corresponding to each of the notes is obtained according to the maximum value of the energy of the multiple notes and the note energy of each of the notes.

Preferably, the step of generating a haptic vibration file corresponding to the audio signal according to the normalized energy includes:

Marking the note corresponding to the normalized energy higher than the preset energy threshold as a tactile vibration point;

According to each of the haptic vibration points, a haptic vibration file corresponding to the audio signal is generated.

The present invention provides a vibration control method, the vibration control method includes the steps of the vibration generation method of the present invention, and further includes:

Extract the haptic vibration points in the haptic vibration file;

generating a vibration driving voltage corresponding to the tactile vibration point according to each of the tactile vibration points;

According to the vibration driving voltage, the motor is driven to vibrate to generate a haptic vibration effect.

The present invention provides a vibration generating device, comprising:

an audio processing module for identifying audio beats in the audio signal; and,

The calculation processing module is used for obtaining multiple note energies of the audio signal according to the audio beat; for performing normalization preprocessing on the multiple note energies respectively, and obtaining multiple normalized energies, wherein the The note energy is set in a one-to-one correspondence with the normalized energy; it is used for generating a haptic vibration file corresponding to the audio signal according to a plurality of the normalized energy.

The present invention provides a vibration control system, which includes:

The vibration generating device of the present invention is used for inputting a tactile vibration file;

A tactile vibration identification module for extracting the tactile vibration points in the tactile vibration file, and also for generating a vibration driving voltage corresponding to the tactile vibration points according to each of the tactile vibration points; and,

A vibration unit, configured to vibrate according to the vibration driving voltage to generate a haptic vibration effect.

The present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the vibration generating method of the present invention and/or the steps of the present invention are implemented. The steps of the vibration control method described above.

The present invention provides a computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the steps of the vibration generating method of the present invention and/or the steps of the vibration generation method of the present invention are implemented. The steps of the vibration control method.

The above-mentioned vibration generation method and device thereof, calculate and process the note energy of each note by a normalization optimization algorithm to obtain the normalized energy of each note, and generate a tactile vibration file corresponding to the audio signal according to the normalized energy , the tactile vibration effect and the beat of the audio signal are flexibly matched. At the same time, since the normalized energy of each note is higher than the note energy of the note, the problem of insufficient local audio intensity is effectively avoided, and the haptic vibration effect and audio signal are improved. The accuracy of matching the beats of each other, and the coordination between auditory perception and tactile perception is good, which provides conditions for improving the user's listening experience; in addition, the above vibration generation method is applied to the vibration control method and its system, When the audio signal is played, the above-mentioned vibration control method and system thereof can generate a tactile vibration effect according to the tactile vibration file, and the tactile vibration effect is highly matched with the beat of the audio signal, which improves the coordination between the user's auditory experience and tactile experience, Effectively improve the user's listening experience.

Description of drawings

Fig. 1 is the schematic flow chart of the vibration generation method of the present invention;

Fig. 2 is the schematic flow chart of step S24 of Fig. 1;

3 is a schematic flowchart of step S28 in FIG. 1;

4 is a schematic flowchart of the vibration control method of the present invention;

Fig. 5 is the structural block diagram of the vibration generating device of the present invention;

FIG. 6 is a structural block diagram of the vibration control system of the present invention.

Embodiments of the present invention

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

Referring to Figure 1, the present invention provides a vibration generation method, comprising the following steps:

Step 22, identifying the audio beat in the audio signal.

Step 24: Acquire multiple note energies of the audio signal according to the audio beat.

Step 26: Perform normalization preprocessing on a plurality of the note energies respectively to obtain a plurality of normalized energies; wherein the note energies and the normalized energies are set in a one-to-one correspondence.

Step 28: Generate a haptic vibration file corresponding to the audio signal according to the plurality of normalized energies.

The above vibration generation method, in the vibration generation method, calculates and processes the note energy of each note through a normalization optimization algorithm to obtain the normalized energy of each note, and generates a corresponding audio signal according to the normalized energy. The haptic vibration file realizes the flexible matching of the haptic vibration effect and the beat of the audio signal. At the same time, since the normalized energy of each note is higher than the note energy of the note, the problem of insufficient local audio intensity is effectively avoided, and the haptic vibration effect is improved. The accuracy of matching with the beat of the audio signal, and the coordination between the auditory feeling and the tactile feeling are good, which is beneficial to improve the user's listening experience.

In order to further understand the above method, please refer to Figures 1-3 at the same time, and each step of the above method will be described in detail below:

Step S22, identifying the audio beat in the audio signal.

Specifically, before the step S22, a signal preprocessing process can be added according to actual needs, and the signal preprocessing process is: importing an audio file, acquiring the audio signal of the audio file, extracting the audio feature information of the audio signal, according to the The audio feature information identifies beat moments of a plurality of beat points, and obtains an audio beat sequence Bp from the audio signal according to the beat moments of the plurality of beat points, and the audio beat sequence Bp includes a plurality of audio beats.

Step S24: Acquire multiple note energies of the audio signal according to the audio beat.

Specifically, the step S24 includes:

Step S241: Acquire multiple notes in the audio beat according to a preset time interval; wherein, the time interval between two adjacent notes is the preset time interval, denoted as T ₀ .

Specifically, in the step S241, the preset time interval T ₀ can be specifically determined according to the actual application situation, and the notes in the same audio beat are represented as T(n), where n is the same audio frequency The number of notes in a beat, for example, in this embodiment, the preset time interval is specifically set to be a quarter of the duration of a beat, that is, it is equivalent to that there are four notes with the same time interval in each beat. Here, n represents the 1st, 2nd, 3rd, and 4th note numbers in the same audio beat, and the first to fourth notes in the same audio beat are represented as T(1), T(2), T( 3), T(4).

Step S242, acquiring the note energy of each of the notes.

Wherein, the note energy in the same audio beat is represented as P ₀ (n), and in this embodiment, the note energy of the first to fourth notes in the same audio beat is represented as P ₀ (1 ), P ₀ (2), P ₀ (3), P ₀ (4).

Specifically, in the step S242, within the same audio beat, the note energy of each of the notes is calculated and obtained according to the amplitude of the audio signal corresponding to each of the notes. Meanwhile, before step S242, the sampling rate of the audio signal needs to be detected in advance, which is denoted as f _s .

The specific calculation formula for the note energy P ₀ (n) is expressed as:

P ₀ (n) = sqrt(X ₀ (t _n ) ² + X ₀ (t _n+1 ) ² +…+X ₀ (t _n+k ) ² )/(k+1),

K=T ₀ ×f _s ,

In the formula, X ₀ (t _n ) is the amplitude of the audio signal corresponding to the time t _n of the nth note, and X ₀ (t _n+k ) is the audio frequency corresponding to the time t _n +k of the nth note. The amplitude of the signal, where time t _n is the start time of the nth note, time tn _+k is the end time of the nth note, k is the number of sampling points of the nth note, and T ₀ is the adjacent two The time interval of notes, f _s is the sampling rate.

The second note T(2) of the third audio beat occurs at 3 seconds and lasts for 0.2 seconds (that is, the time interval T ₀ is 0.2 seconds), and the signal sampling rate fs is 48000. in:

The number of sampling points of the second note T(2) of the third audio beat k=0.2×48000=9600,

Calculate the corresponding time of each sampling point in turn, the time of the first sampling point (ie the starting time) t _n =3×48000=144000, the time of the second sampling point t _n+1 =3×48000+1= 144001,..., the time of the 9600th sampling point (ie the termination time) t _n+k =3×48000+9600=144000+9600;

Calculate the amplitude corresponding to each sampling point according to the formula X ₀ ( _t _n )= sin(2π·100·t _n /f _s ). Specifically, the amplitude X ₀ ( 144000)= sin(2π·100·144000 /9600), the amplitude X ₀ (144001)= sin(2π·100·144001 /9600), , the second sampling point time t _n+1 The amplitude X ₀ (144000+9600)= sin[2π·100·(144000+9600)/9600] corresponding to the time t _n+k of 9600 sampling points.

According to the above values, calculate the normalized energy corresponding to the second note T(2) of the third audio beat P ₀ (2)= sqrt(X ₀ (144000) ² + X ₀ (144001) ² + …+ X ₀ (144000+9600) ² )/(9600+1).

It should be noted that, in other embodiments, the step S242 may also select a specific calculation method of the note energy according to the actual situation.

Step S26, performing normalization preprocessing on a plurality of the note energies respectively to obtain a plurality of normalized energies; wherein the note energies and the normalized energies are set in a one-to-one correspondence.

Wherein, the normalized energy in the same audio beat is represented as P(n), and in this embodiment, the normalized energy of the first to fourth notes in the same audio beat is represented as P in turn (1), P(2), P(3), P(4).

Specifically, in the step S26, within the same audio beat, the normalized energy corresponding to each of the notes is obtained according to the maximum value of the energy of a plurality of the notes and the note energy of each of the notes.

In the same beat, the calculation formula of the normalized energy P(n) corresponding to any note is expressed as:

P(n) = P ₀ (n)/max(P ₀ (n)),

In the formula, P(n) represents the normalized energy corresponding to any note in the beat, max(P ₀ (n)) represents the maximum note energy in the note energy sequence P ₀ (n) in the same audio beat, and In this embodiment, max(P ₀ (n)) is the maximum value among the note energies P ₀ (1), P ₀ (2), P ₀ (3), and P ₀ (4).

Step S28, generating a haptic vibration file corresponding to the audio signal according to a plurality of the normalized energies.

Specifically, the step S28 includes:

Step S281, marking the note corresponding to the normalized energy higher than the preset energy threshold as a tactile vibration point.

Among them, the preset energy threshold value is expressed as Ph, and the value of the preset energy threshold value Ph can be specifically set according to the actual application. On the contrary, the larger the value of the preset energy threshold Ph is, the less haptic vibration effects the audio signal can match, and in this embodiment, the preset energy threshold Ph is set to 0.5·max (P( n)), max(P(n)) represents the maximum normalized energy in the normalized energy sequence P(n) within the same audio beat, that is, the preset energy threshold Ph is set to be within the same audio beat One-half of the largest normalized energy in the normalized energy sequence P(n).

Specifically, it is determined whether the four normalized energies (P(1), P(2), P(3), and P(4)) in each audio beat are higher than the energy threshold Ph (ie, 0.5·max( P(n))), if yes, then the four normalized energies (P(1), P(2), P(3), P(4)) corresponding to the normalized energy higher than the energy threshold Ph The notes are marked as tactile vibration points.

It should be noted that by setting the preset energy threshold Ph to 0.5·max(P(n)), the distribution of the haptic vibration points on the audio signal can be made more uniform, and the haptic vibration can be evenly generated when the audio signal is played It avoids that the vibration frequency is too high and the vibration pain is caused to the user, and the vibration frequency is too low and the tactile vibration effect is not obvious, which effectively improves the user's tactile experience and makes the experience better.

Step S282, generating a haptic vibration file corresponding to the audio signal according to each of the haptic vibration points.

Referring to FIG. 4 , the present invention provides a vibration control method applying the above vibration generation method, and the vibration control method includes the following steps:

Step 302, extracting the haptic vibration points in the haptic vibration file.

Step 304: Generate a vibration driving voltage corresponding to each of the tactile vibration points according to each of the tactile vibration points.

Step 306 , according to the vibration driving voltage, drive the motor to vibrate to generate a haptic vibration effect.

Above-mentioned vibration control method, when playing audio signal, above-mentioned vibration control method can produce tactile vibration effect according to tactile vibration file, this tactile vibration effect and the audio rhythm matching degree of audio signal are high, improve the user's auditory feeling and tactile feeling. It can effectively improve the user's listening experience.

It should be understood that although the steps in the flowcharts of FIGS. 1-4 are shown in sequence according to the arrows, these steps are not necessarily executed in the sequence shown by the arrows. Unless explicitly stated herein, the execution of these steps is not strictly limited to the order, and these steps may be performed in other orders. Moreover, at least a part of the steps in FIGS. 1-4 may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but may be executed at different times. The execution of these steps or stages The order is also not necessarily sequential, but may be performed alternately or alternately with other steps or at least a portion of the steps or phases within the other steps.

Referring to FIG. 5, the present invention provides a vibration generating device 100, which includes an audio processing module 101 and a calculation processing module 102, wherein:

The audio processing module 101 is used to identify the audio beat in the audio signal;

The calculation processing module 102 is configured to obtain multiple note energies of the audio signal according to the audio beat; and is configured to perform normalization preprocessing on the multiple note energies to obtain multiple normalized energies, Wherein, the note energy and the normalized energy are set in a one-to-one correspondence; it is used for generating a haptic vibration file corresponding to the audio signal according to a plurality of the normalized energy.

The audio processing module 101 is further configured to acquire a plurality of notes in the audio beat according to a preset time interval, wherein the time interval between two adjacent said notes is the preset time interval; The notes corresponding to the normalized energy above the preset energy threshold are marked as haptic vibration points.

The calculation processing module 102 is further configured to obtain a plurality of note energies of the audio signal according to the audio beat, specifically, to obtain the note energy of each of the notes; Perform normalization preprocessing respectively to obtain a plurality of normalized energies, wherein the note energy and the normalized energy are set in a one-to-one correspondence; it is used to generate a The tactile vibration file corresponding to the audio signal, specifically, used to identify the note corresponding to the normalized energy higher than the preset energy threshold is marked as a tactile vibration point, and is used to generate the corresponding tactile vibration point according to each of the tactile vibration points. The haptic vibration file corresponding to the audio signal.

Referring to FIG. 6, the present invention provides a vibration control system 200, which includes the above-mentioned vibration generating device 100, a tactile vibration recognition module 201, and a vibration unit 202, wherein:

The vibration generating apparatus 100 is used for inputting a haptic vibration file.

The tactile vibration identification module 201 is configured to extract the tactile vibration points in the tactile vibration file, and is also configured to generate a vibration driving voltage corresponding to the tactile vibration point according to each of the tactile vibration points.

The vibration unit 202 is configured to vibrate according to the vibration drive voltage to generate a tactile vibration effect; specifically, the structural form of the vibration unit 202 can be specifically selected according to the actual use situation, for example, in this embodiment, The vibration unit 202 is a vibration motor.

For the specific definition of the vibration generating device, please refer to the above definition of the vibration generating method, and for the specific definition of the vibration control system, please refer to the above definition of the vibration control method, which will not be repeated here. Each module in the above vibration generating device or vibration control system may be implemented in whole or in part by software, hardware and combinations thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

The present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the vibration generating method of the present invention and/or the steps of the present invention are implemented. The steps of the vibration control method described above. It should be noted that, the above-mentioned vibration generating device and vibration control system can be applied to the electronic device.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage In the medium, when the computer program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database or other media used in the various embodiments provided by the present invention may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (Random Access Memory) Access Memory, RAM) or external cache memory. By way of illustration and not limitation, RAM may take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (Dynamic Random Access Memory). Access Memory, DRAM), etc.

The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features It is considered to be the range described in this specification.

The above-mentioned embodiments only represent several embodiments of the present invention, and the descriptions thereof are more specific and detailed, but should not be construed as a limitation on the scope of the invention patent. It should be pointed out that for those skilled in the art, without departing from the concept of the present invention, several modifications and improvements can be made, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the appended claims.

Claims

A vibration generation method, characterized in that the method comprises:

Identify audio beats in audio signals;

Acquiring a plurality of note energies of the audio signal according to the audio beat;

Perform normalization preprocessing on a plurality of the note energies respectively to obtain a plurality of normalized energies; wherein, the note energies and the normalized energies are set in one-to-one correspondence;

A haptic vibration file corresponding to the audio signal is generated from a plurality of the normalized energies.
The vibration generation method according to claim 1, wherein the acquiring a plurality of note energies of the audio signal according to the audio beat comprises:

Acquire multiple notes in the audio beat according to a preset time interval; wherein, the time interval between two adjacent said notes is the preset time interval;

Obtain the note energy for each said note.
The vibration generation method according to claim 2, wherein the step of obtaining the note energy of each of the notes comprises:

In the same audio beat, according to the amplitude of the audio signal corresponding to each of the notes, the note energy of each of the notes is obtained by calculation.
The vibration generation method according to claim 2, wherein the normalization preprocessing is performed on a plurality of the note energies respectively, and the acquisition of the plurality of normalized energies comprises:

In the same audio beat, the normalized energy corresponding to each of the notes is obtained according to the maximum value of the energy of the multiple notes and the note energy of each of the notes.
The vibration generation method according to claim 4, wherein the step of generating a haptic vibration file corresponding to the audio signal according to the normalized energy comprises:

Marking the note corresponding to the normalized energy higher than the preset energy threshold as a tactile vibration point;

According to each of the haptic vibration points, a haptic vibration file corresponding to the audio signal is generated.
A vibration control method, characterized in that the vibration control method includes the steps of the vibration generation method described in any one of the above claims 1-5, and the vibration control method further includes:

Extract the haptic vibration points in the haptic vibration file;

generating a vibration driving voltage corresponding to the tactile vibration point according to each of the tactile vibration points;

According to the vibration driving voltage, the motor is driven to vibrate to generate a haptic vibration effect.
A vibration generating device, characterized in that the vibration generating device comprises:

an audio processing module for identifying audio beats in the audio signal; and,

The calculation processing module is used for obtaining multiple note energies of the audio signal according to the audio beat; for performing normalization preprocessing on the multiple note energies respectively, and obtaining multiple normalized energies, wherein the The note energy is set in a one-to-one correspondence with the normalized energy; it is used for generating a haptic vibration file corresponding to the audio signal according to a plurality of the normalized energy.
A vibration control system, characterized in that the vibration control system comprises:

The vibration generating device of claim 7, for inputting a tactile vibration file;

A tactile vibration identification module for extracting the tactile vibration points in the tactile vibration file, and also for generating a vibration driving voltage corresponding to the tactile vibration points according to each of the tactile vibration points; and,

A vibration unit, configured to vibrate according to the vibration driving voltage to generate a haptic vibration effect.
An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the vibration generation method according to any one of claims 1 to 5 when the processor executes the computer program and/or the steps of the vibration control method of claim 6.
A computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the steps and/or the rights of the method for generating vibrations described in any one of claims 1 to 5 are realized when the computer program is executed. The steps of the vibration control method of claim 6.