CN116185165B

CN116185165B - Haptic sensation generation method, system, device and computer storage medium

Info

Publication number: CN116185165B
Application number: CN202210692429.6A
Authority: CN
Inventors: 柳慧芬; 陈松; 何亮
Original assignee: Wuhan Silicon Integrated Co Ltd
Current assignee: Wuhan Silicon Integrated Co Ltd
Priority date: 2022-06-17
Filing date: 2022-06-17
Publication date: 2024-04-02
Anticipated expiration: 2042-06-17
Also published as: CN116185165A

Abstract

The embodiment of the application discloses a method, a system, equipment and a computer storage medium for generating touch sense, wherein the method comprises the following steps: acquiring audio data; segmenting the audio data to determine at least one audio data segment; classifying the at least one audio data segment, and determining vibration type labels corresponding to the at least one audio data segment respectively; performing haptic effect analysis according to the at least one audio data segment and the vibration category label corresponding to the at least one audio data segment, and determining the haptic effect data corresponding to the at least one audio data segment; driving waveform data is generated from haptic effect data corresponding to each of the at least one audio data segment, and haptic output is generated in response to the driving waveform data. Thus, the method can realize the conversion of audio data to haptic effects and enhance the haptic perception effect.

Description

Haptic sensation generation method, system, device and computer storage medium

Technical Field

The present disclosure relates to the field of haptic generation, and in particular, to a haptic generation method, system, device, and computer storage medium.

Background

Haptic is a haptic pressure feedback technique that utilizes the feel of a user's touch by applying haptic feedback effects, such as force, vibration, and motion, to the user. Among them, touch-enabled devices have become increasingly popular. For example, mobile and other devices may be configured with a touch-sensitive display such that a user can provide input by touching portions of the display. As another example, a tactile enabled surface separate from the display may be used for input, such as a touch pad, mouse, or other device.

In the related art, some touch-enabled devices utilize haptic effects, e.g., haptic effects configured to simulate texture or friction on a touch surface. In some devices, these haptic effects may be related to audio or other effects output by the device. However, existing solutions do not take into account the comprehensiveness of the audio data when it is converted to haptic sensation, and are limited to the preliminary processing of the audio signal, resulting in poor haptic effects being output.

Disclosure of Invention

The application provides a haptic generation method, a haptic generation system, haptic generation equipment and a computer storage medium, which can realize conversion of audio data to haptic effects and enhance haptic perception effects.

The technical scheme of the application is realized as follows:

In a first aspect, embodiments of the present application provide a haptic generation method, the method including:

acquiring audio data;

performing segmentation processing on the audio data to determine at least one audio data segment;

classifying the at least one audio data segment, and determining vibration class labels corresponding to the at least one audio data segment respectively;

performing haptic effect analysis according to vibration category labels corresponding to the at least one audio data segment respectively, and determining haptic effect data corresponding to the at least one audio data segment respectively;

driving waveform data is generated from haptic effect data corresponding to each of the at least one audio data segment, and haptic output is generated in response to the driving waveform data.

In some embodiments, the segmenting the audio data to determine at least one audio data segment includes:

preprocessing the audio data to obtain envelope information;

and judging the envelope information and a preset threshold, and if the envelope value in the envelope information is lower than the preset threshold, determining that the audio data position corresponding to the envelope value is in a non-connection state so as to obtain the at least one audio data segment.

In some embodiments, the preprocessing the audio data to obtain envelope information includes:

band-pass filtering is carried out on the audio data to obtain filtered audio data;

rectifying and normalizing the filtered audio data to obtain the envelope information; wherein the envelope information comprises time domain envelope information or energy envelope information.

In some embodiments, the classifying the at least one audio data segment to determine vibration class labels corresponding to the at least one audio data segment, includes:

extracting time domain features of the audio data segment to obtain time domain feature information;

extracting frequency domain characteristics of the audio data segment to obtain frequency domain characteristic information;

and classifying according to the time domain characteristic information and/or the frequency domain characteristic information to determine a vibration type label corresponding to the audio data segment.

In some embodiments, the time domain feature information comprises envelope fluctuation feature information and audio feature duration information, the frequency domain feature information comprising short-time spectral density distribution information; the method further comprises the steps of:

determining a first characteristic result in the time domain characteristic information according to the envelope fluctuation characteristic information; wherein the first characteristic result comprises a single undulation or a plurality of undulations;

Determining a second feature result in the time domain feature information according to the audio feature duration information; wherein the second characteristic result includes a long vibration or a short vibration;

determining a third characteristic result in the frequency domain characteristic information according to the short-time frequency spectrum density distribution information; wherein the third feature result comprises a continuum, line spectrum, or mixed spectrum;

correspondingly, the classifying processing according to the time domain feature information and/or the frequency domain feature information, and determining the vibration class label corresponding to the audio data segment, includes:

and determining a vibration type label corresponding to the audio data segment based on at least one of the first feature result, the second feature result and the third feature result.

In some embodiments, the determining the haptic effect data corresponding to each of the at least one audio data segment according to the haptic effect analysis performed by the vibration class label corresponding to each of the at least one audio data segment includes:

performing vibration analysis according to the vibration type label corresponding to the audio data segment to obtain a vibration result corresponding to the audio data segment;

and determining the haptic effect data corresponding to the audio data segment according to the obtained vibration result.

performing data analysis according to the audio data segment to obtain amplitude envelope information corresponding to the audio data segment;

and determining the haptic effect data corresponding to the audio data segment according to the obtained vibration result and the amplitude envelope information.

In some embodiments, the generating driving waveform data from haptic effect data corresponding to each of the at least one audio data segment, the generating haptic output in response to the driving waveform data, comprises:

matching the haptic effect data corresponding to each of the at least one audio data segment in a preset vibration library to determine driving waveform data;

and receiving a trigger instruction, and driving a brake according to the trigger instruction and the driving waveform data so as to generate a touch output.

In some embodiments, the generating driving waveform data from haptic effect data corresponding to each of the at least one audio data segment, in response to the driving waveform data, to generate the haptic output, further comprises:

Obtaining model parameters of the brake;

calculating haptic effect data corresponding to each of the at least one audio data segment by the model parameters to generate the driving waveform data;

In a second aspect, embodiments of the present application provide a haptic generation system, comprising:

the audio data analysis unit is used for acquiring audio data, carrying out segmentation processing on the audio data and determining at least one audio data segment;

the classifying unit is used for classifying the at least one audio data segment and determining vibration type labels corresponding to the at least one audio data segment;

a haptic effect analysis unit for receiving the at least one audio data segment and the vibration category label corresponding to each of the at least one audio data segment and performing haptic effect analysis to determine haptic effect data corresponding to each of the at least one audio data segment;

and a haptic generation unit for generating driving waveform data according to haptic effect data corresponding to each of the at least one audio data segment, and generating a haptic output in response to the driving waveform data.

In some embodiments, the audio data analysis unit is configured to pre-process the audio data to obtain envelope information; and judging the envelope information and a preset threshold, and if the envelope value in the envelope information is lower than the preset threshold, determining that the audio data position corresponding to the envelope value is in a non-connection state so as to obtain the at least one audio data segment.

In some embodiments, the audio data analysis unit is further configured to perform band-pass filtering processing on the audio data after the audio data is acquired, to obtain filtered audio data; rectifying and normalizing the filtered audio data to obtain the envelope information; wherein the envelope information comprises time domain envelope information or energy envelope information.

In some embodiments, the classifying unit is configured to perform time domain feature extraction on the audio data segment to obtain time domain feature information; extracting frequency domain characteristics of the audio data segment to obtain frequency domain characteristic information; and classifying according to the time domain characteristic information and/or the frequency domain characteristic information to determine a vibration type label corresponding to the audio data segment.

In some embodiments, the time domain feature information comprises envelope fluctuation feature information and audio feature duration information, the frequency domain feature information comprising short-time spectral density distribution information;

the classifying unit is configured to determine a first feature result in the time domain feature information according to the envelope fluctuation feature information, determine a second feature result in the time domain feature information according to the audio feature duration information, determine a third feature result in the frequency domain feature information according to the short-time spectrum density distribution information, and classify according to the first feature result and the second feature result in the time domain feature information and/or the third feature result in the frequency domain feature information to obtain a vibration class label corresponding to the audio data segment.

In some embodiments, the first feature result comprises: single relief or multiple relief;

the second feature result includes: long vibration or short vibration;

the third feature result includes: a continuous spectrum, a line spectrum, or a mixed spectrum.

In some embodiments, the haptic effect analysis unit is configured to perform vibration analysis according to a vibration type tag corresponding to the audio data segment, to obtain a vibration result corresponding to the audio data segment; and performing data analysis according to the audio data segment to obtain amplitude envelope information corresponding to the audio data segment; and determining haptic effect data corresponding to the audio data segment according to the amplitude envelope information and the vibration result.

In some embodiments, the haptic generation unit includes a haptic data generation unit, a haptic driving unit, and a brake; wherein,

the haptic data generation unit is used for matching the haptic effect data corresponding to each of the at least one audio data segment in a preset vibration library to generate driving waveform data;

the touch driving unit is used for receiving a trigger instruction and driving the brake according to the trigger instruction and the driving waveform data so as to generate touch output.

In some embodiments, the haptic data generating unit is further configured to obtain model parameters of the brake, and calculate haptic effect data corresponding to each of the at least one audio data segment by using the model parameters, so as to generate the driving waveform data.

In some embodiments, the haptic generation system further comprises a data store; wherein,

the data storage is used for storing audio data to be processed;

the audio data analysis unit is connected with the data memory and used for acquiring the audio data from the data memory.

In a third aspect, an embodiment of the present application provides an electronic device, including:

A memory for storing a computer program capable of running on the processor;

a processor for executing the haptic generation method of any one of the first aspects when running the computer program.

In a fourth aspect, embodiments of the present application provide a computer storage medium storing a computer program which, when executed by at least one processor, implements the haptic generation method of any one of the first aspects.

The embodiment of the application provides a haptic sensation generation method, a haptic sensation generation system, haptic sensation generation equipment and a computer storage medium, wherein audio data are acquired; segmenting the audio data to determine at least one audio data segment; classifying the at least one audio data segment, and determining vibration type labels corresponding to the at least one audio data segment respectively; performing haptic effect analysis according to the at least one audio data segment and the vibration category label corresponding to the at least one audio data segment, and determining the haptic effect data corresponding to the at least one audio data segment; driving waveform data is generated from haptic effect data corresponding to each of the at least one audio data segment, and haptic output is generated in response to the driving waveform data. Thus, after the audio data is segmented, the corresponding haptic effect data can be determined by classifying the obtained audio data segments and performing haptic effect analysis processing, and then the corresponding haptic output is generated according to the generated driving waveform data; thereby enabling the conversion of audio data into haptic effects and enhancing haptic effects.

Drawings

FIG. 1 is a schematic flow chart of a haptic generation method according to an embodiment of the present application;

FIG. 2 is a schematic flow chart of a segmentation process in a haptic generation method according to an embodiment of the present application;

FIG. 3 is a schematic flow chart of a classification process in a haptic sensation generation method according to an embodiment of the present disclosure;

FIG. 4 is a schematic flow chart of haptic generation in a haptic generation method according to an embodiment of the present application;

FIG. 5 is a schematic diagram of the composition and structure of a haptic generation system according to an embodiment of the present application;

FIG. 6 is a schematic diagram of the structure of a haptic generation unit in a haptic generation system according to an embodiment of the present application;

FIG. 7 is a schematic diagram of the composition of another haptic generation system provided in an embodiment of the present application;

FIG. 8 is a schematic diagram of a detailed structure of a haptic generation system according to an embodiment of the present application;

FIG. 9 is a detailed flow chart of a haptic generation method according to an embodiment of the present application;

FIG. 10 is a schematic flow chart of a classification process in a haptic sensation generation method according to an embodiment of the present application;

FIG. 11 is a schematic flow chart of determining haptic effect data in a haptic generation method according to an embodiment of the present application;

Fig. 12 is a schematic diagram of a specific hardware structure of an electronic device according to an embodiment of the present application;

fig. 13 is a schematic diagram of a composition structure of an electronic device according to an embodiment of the present application.

Detailed Description

For a more complete understanding of the features and technical content of the embodiments of the present application, reference should be made to the following detailed description of the embodiments of the present application, taken in conjunction with the accompanying drawings, which are for purposes of illustration only and not intended to limit the embodiments of the present application.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the present application only and is not intended to be limiting of the present application.

In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is to be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with one another without conflict. It should also be noted that the term "first/second/third" in reference to the embodiments of the present application is used merely to distinguish similar objects and does not represent a specific ordering for the objects, it being understood that the "first/second/third" may be interchanged with a specific order or sequence, if allowed, to enable the embodiments of the present application described herein to be implemented in an order other than that illustrated or described herein.

It will be appreciated that haptic sensation is an important way for humans to perceive external information, a haptic pressure feedback technique that utilizes the sense of touch of a user by applying haptic feedback effects, such as force, vibration, and motion, to the user. The use of the vibrant technique in a touch enabled device can integrate spatial feel, sound and touch, providing a more complete sensory experience. For example, the inclusion of audio portions or audio data in a movie or video game can produce haptic-like audio effects.

In the related art, although there have been technical schemes for converting audio data into haptic sensation. In these existing solutions, however, the sampled audio file can be converted manually on the one hand. The designer listens to, analyzes and determines the sampled audio file, and selects sampled audio features, producing haptic effects from the selected features; however, the manual design has limited intervention resources and low efficiency; on the other hand, in the technology of vibration along with sound, particularly for the vibration effect of music audio, no clear direction and method for analyzing the vibration effect exist at present; for the method of audio conversion haptic, limited to the preliminary processing of the audio signal, there is no accurate guidance of the haptic effect.

Based on this, embodiments of the present application provide a haptic generation method, which may include: acquiring audio data; segmenting the audio data to determine at least one audio data segment; classifying the at least one audio data segment, and determining vibration type labels corresponding to the at least one audio data segment respectively; performing haptic effect analysis according to the at least one audio data segment and the vibration category label corresponding to the at least one audio data segment, and determining the haptic effect data corresponding to the at least one audio data segment; driving waveform data is generated from haptic effect data corresponding to each of the at least one audio data segment, and haptic output is generated in response to the driving waveform data. Thus, after the audio data is segmented, the corresponding haptic effect data can be determined by classifying the obtained audio data segments and performing haptic effect analysis processing, and then the corresponding haptic output is generated according to the generated driving waveform data; thereby enabling the conversion of audio data into haptic effects and enhancing haptic effects.

In another embodiment of the present application, reference is made to fig. 1, which is a schematic flow chart illustrating a haptic generation method according to an embodiment of the present application. As shown in fig. 1, the method may include:

S101: audio data is acquired.

It should be noted that the haptic generation method provided in the embodiments of the present application may be applied to an electronic device with haptic generation requirements. Here, the electronic device may be, for example, a computer, a smart phone, a tablet computer, a notebook computer, a palm computer, a personal digital assistant (Personal Digital Assistant, PDA), a portable media player (Portable Media Player, PMP), a navigation device, a wearable device, or the like, which is not particularly limited in the embodiments of the present application.

It should also be noted that audio data is a form of sound signal, different audio data having different inherent characteristics. Wherein the intrinsic features can be divided into three levels, namely a physical sample level of the lowest level, an acoustic feature level of the middle level and a semantic level of the highest level. The physical sample level contains characteristics such as sampling frequency, time scale, sample, format, code and the like; the acoustic feature level includes features including a perceptual feature and an acoustic feature, wherein the perceptual feature includes tones, pitches, melodies, rhythms, and the like, and the acoustic feature includes energy, zero crossing rates, linear predictive coding (Linear Predictive Coding, LPC) coefficients, and a structured representation of audio, and the like; the semantic level comprises a music narrative, an audio object description, a voice recognition text and the like, and the embodiment of the application can process the characteristics of the acoustic feature level of the audio data to be processed to obtain the haptic effect data corresponding to the audio data.

It should be further noted that, in the embodiment of the present application, the audio data may be acquired by an audio acquisition device. The audio acquisition device can be integrated in the electronic equipment or independent of the electronic equipment, so that after the audio acquisition device acquires the audio data, the audio data is required to be sent to the electronic equipment for processing of converting the audio data into the haptic effect data.

S102: and carrying out segmentation processing on the audio data to determine at least one audio data segment.

It should be noted that, the audio data is segmented by the envelope information, and if the envelope value in the envelope information is lower than the preset threshold value, it may be confirmed that the audio data is not connected here, so as to obtain a plurality of audio data segments. In other words, several audio data segments are obtained here in a disconnected state. In this way, in the subsequent processing, the segmented audio data segments can be sequentially input into the classification unit for classification processing; or when the classifying units are multiple, the segmented audio data segments can be classified in parallel, so that the processing difficulty of the audio data is reduced, and meanwhile, the processing efficiency of the audio data can be improved.

In the embodiment of the present application, in the process of performing the segmentation processing on the audio data, the segmentation processing may be performed according to a fixed length, or may be performed according to an indefinite length. Briefly, in an embodiment of the present application, at least one audio data segment after the segmentation process may have the same or different lengths, which is not specifically limited herein.

In some embodiments, for step S102, referring to fig. 2, a schematic flow diagram of segmentation processing in another haptic generation method provided in an embodiment of the present application is shown, as shown in fig. 2, where the step may include:

s201: preprocessing the audio data to obtain envelope information;

s202: and judging the envelope information and a preset threshold, and if the envelope value in the envelope information is lower than the preset threshold, determining that the audio data position corresponding to the envelope value is in a non-connection state so as to obtain the at least one audio data segment.

In a specific embodiment, the preprocessing the audio data to obtain envelope information may include:

It should be noted that, in the embodiment of the present application, when filtering processing is performed on audio data, since any wave can be decomposed into several kinds of superposition of sine wave and cosine wave, the filtering process, that is, the weighting process, is performed, and the filtering effect is to give different weights to different signal components, so as to reduce noise of the audio data. Thus, in a specific implementation manner, the filtering processing of the audio data in the embodiment of the present application may be a band-pass filtering processing implemented by a low-pass filter, so as to filter out a signal outside the range of the fundamental frequency of the rhythm, so as to obtain low-frequency band data or medium-frequency band data.

It should be further noted that in the embodiment of the present application, the audio data is rectified and normalized, and may be converted into envelope information for representation, so that the audio data is subjected to threshold judgment subsequently, and further continuity of the audio data is determined, and segmentation of the audio data is implemented at discontinuous positions of the audio data. The rectification mode can take absolute value or square value, and interpolate connecting maximum value points to output time domain envelope information or energy envelope information. Thus, in a specific implementation manner, taking the time domain envelope information as an example, the threshold value may be determined according to the preset threshold value and the time domain envelope information, and if the envelope value in the time domain envelope information is lower than the preset threshold value, it may be determined that the audio data is not connected here, and a plurality of audio data segments may be obtained.

S103: and classifying the at least one audio data segment, and determining vibration type labels corresponding to the at least one audio data segment.

In the embodiment of the present application, in the process of classifying at least one audio data segment, classification processing may be sequentially performed on each audio data segment, so that the resource pressure in each classification processing process may be reduced; the classification processing can be performed on a plurality of audio data segments at the same time, so that the classification processing speed of the audio data segments can be increased.

In some embodiments, for step S103, referring to fig. 3, a schematic flow chart of a classification process in still another haptic generation method provided in an embodiment of the present application is shown. As shown in fig. 3, this step may include:

s301: extracting time domain features of the audio data segment to obtain time domain feature information;

s302: extracting frequency domain characteristics of the audio data segment to obtain frequency domain characteristic information;

s303: and classifying according to the time domain characteristic information and/or the frequency domain characteristic information to determine a vibration type label corresponding to the audio data segment.

Here, the time domain feature information includes envelope fluctuation feature information and audio feature duration information, and the frequency domain feature information includes short-time spectral density distribution information. Accordingly, in some embodiments, the determining the vibration class label corresponding to the audio data segment according to the classification processing performed by the time domain feature information and/or the frequency domain feature information may include:

accordingly, the classifying processing according to the time domain feature information and/or the frequency domain feature information, and determining the vibration class label corresponding to the audio data segment may include:

In the embodiment of the present application, when determining the vibration tag, any one of the feature results may be selected for classification, for example, classification may be performed according to the first feature result, classification may be performed according to the second feature result, or classification may be performed according to the third feature result; any two feature results may be selected for classification, for example, classification according to a first feature result and a second feature, classification according to a first feature result and a third feature, classification according to a second feature result and a third feature, classification according to a first feature result, a second feature result and a third feature result, and finally determining a vibration class label corresponding to the audio data segment.

It should be further noted that, in the embodiment of the present application, the judging manner of the single undulation and the multiple undulations may be that the case that the envelope undulation characteristic is higher than the preset threshold is marked as 1, the case that the envelope undulation characteristic is lower than the preset threshold is marked as 0, the undulation occurs once when the segment of '01' or '10' occurs, and the undulation occurs multiple times when the segment of '01' or '10' occurs alternately multiple times. The method for judging the long vibration and the short vibration may be that the long vibration is judged when the duration of the audio feature is greater than a preset threshold value, and the short vibration is judged when the duration of the audio feature is less than or equal to the preset threshold value. The continuous spectrum, the line spectrum or the mixed spectrum may be judged in such a way that the spectrum is continuous on the frequency axis, whereas the spectrum is continuous on the time axis. Other cases are mixed spectra.

For example, for one audio data segment, the corresponding vibration category label may include 'hit', 'bump', 'run', 'pull string', and so on. Wherein, if the three obtained characteristic results are single fluctuation, short vibration and continuous spectrum in turn, the three characteristic results can be classified as 'hit', 'collision'; if the three obtained characteristic results are in turn a plurality of times of fluctuation, long vibration and line spectrum, the three characteristic results can be classified as 'string pulling'; if the three obtained characteristic results are in turn a plurality of fluctuations, long oscillations or short oscillations, a mixed spectrum, it can be classified as 'operation'; but is not limited in any way herein.

For one piece of audio data, the corresponding characteristic results may include, for example, single relief, multiple relief, long vibration, short vibration, continuous spectrum, line spectrum, or mixed spectrum, etc. Wherein in some embodiments, the feature results may be combined directly into class labels, such as directly taking a single undulation as one class label; but is not limited in any way herein.

S104: and carrying out haptic effect analysis according to the vibration type labels corresponding to the at least one audio data segment respectively, and determining the haptic effect data corresponding to the at least one audio data segment respectively.

It should be noted that, in the embodiment of the present application, one audio data segment corresponds to a plurality of vibration type tags, and the vibration modes corresponding to the plurality of vibration type tags and the amplitude envelope information corresponding to the audio data segment determine the haptic effect data corresponding to the audio data segment.

In some embodiments, the determining the haptic effect data corresponding to each of the at least one audio data segment according to the haptic effect analysis performed by the vibration class label corresponding to each of the at least one audio data segment may further include:

It should be noted that, in the embodiment of the present application, each audio data segment may correspond to one or more vibration type labels, and each vibration type label corresponds to a corresponding vibration type. All vibration types corresponding to the same audio data segment form a vibration result corresponding to the audio data segment.

It should be further noted that, in this embodiment of the present application, the vibration type of the audio data segment may be determined to be long vibration, short vibration or combined vibration according to the vibration type tag, and the amplitude, frequency, and duration of the haptic effect may be determined according to the amplitude, frequency, and duration of the audio data, where the two may be combined to obtain haptic effect data, where the haptic effect data may be matched with each other not only in the sound effect type and the vibration type, but also in the sound effect parameter and the vibration parameter, so as to form a rich and vivid haptic output.

S105: driving waveform data is generated from haptic effect data corresponding to each of the at least one audio data segment, and haptic output is generated in response to the driving waveform data.

In some embodiments, for step S105, referring to fig. 4, a schematic flow chart of haptic generation in a haptic generation method provided in an embodiment of the present application is shown. As shown in fig. 4, this step may include:

s401: matching the haptic effect data corresponding to each of the at least one audio data segment in a preset vibration library to determine driving waveform data;

s402: and receiving a trigger instruction, and driving the brake according to the trigger instruction and the driving waveform data so as to generate a touch output.

In the embodiment of the present application, the required driving waveform data is obtained by matching the obtained haptic effect data in the preset vibration library, and then the brake is driven according to the driving waveform data and the trigger command, so as to output the haptic sensation corresponding to the audio data segment. Wherein haptic effect data corresponding to one audio data segment may be matched to a plurality of different vibration types and corresponding vibration parameters, and driving waveform data is formed from the plurality of vibration types and corresponding vibration parameters. The vibration type may be long vibration, short vibration, combined vibration, or the like, and the vibration parameter may include amplitude, frequency, or the like of the vibration, but is not particularly limited herein.

obtaining model parameters of the brake;

In this embodiment of the present application, the model parameters of the actuator and the haptic effect data may be combined, and the desired haptic effect may be simulated by vibration of the actuator to obtain the desired driving waveform data. In addition, in some embodiments, after the haptic effect data is calculated, the obtained driving waveform data may also be stored in a preset vibration library, so that the required driving waveform data can be obtained through matching of the preset vibration library.

The embodiment of the application provides a touch generating method, which is used for acquiring audio data; segmenting the audio data to determine at least one audio data segment; classifying the at least one audio data segment, and determining vibration type labels corresponding to the at least one audio data segment respectively; performing haptic effect analysis according to the at least one audio data segment and the vibration category label corresponding to the at least one audio data segment, and determining the haptic effect data corresponding to the at least one audio data segment; driving waveform data is generated from haptic effect data corresponding to each of the at least one audio data segment, and haptic output is generated in response to the driving waveform data. Thus, after the audio data is segmented, the corresponding haptic effect data can be determined by classifying the obtained audio data segments and performing haptic effect analysis processing, and then the corresponding haptic output is generated according to the generated driving waveform data; thereby enabling the conversion of audio data into haptic effects and enhancing haptic effects.

In some embodiments of the present application, reference is made to fig. 5, which illustrates a schematic diagram of the composition of a haptic generation system provided by embodiments of the present application. As shown in fig. 5, the system may include:

an audio data analysis unit 501, configured to obtain audio data, perform segmentation processing on the audio data, and determine at least one audio data segment;

a classification unit 502, configured to perform classification processing on the at least one audio data segment, and determine vibration category labels corresponding to the at least one audio data segment respectively;

a haptic effect analysis unit 503, configured to receive the at least one audio data segment and the vibration class label corresponding to each of the at least one audio data segment and perform haptic effect analysis, and determine haptic effect data corresponding to each of the at least one audio data segment;

a haptic generation unit 504 for generating driving waveform data from haptic effect data corresponding to each of the at least one audio data segment, responsive to the driving waveform data to generate a haptic output.

It should be noted that the haptic generation system provided in the embodiments of the present application may be integrated into an electronic device having haptic generation requirements. Here, the electronic device may be, for example, a computer, a smart phone, a tablet computer, a notebook computer, a palm computer, a personal digital assistant (Personal Digital Assistant, PDA), a portable media player (Portable Media Player, PMP), a navigation device, a wearable device, or the like, which is not particularly limited in the embodiments of the present application.

It should be noted that, the audio data analysis unit 501 segments the acquired audio data on a time axis to obtain at least one audio data segment; each audio data segment is then classified using classification unit 502, where each audio data segment may correspond to one or more vibration class labels. The haptic effect analysis unit 503 may output corresponding haptic effect data according to the classification result and the audio data segment; and the haptic generation unit 504 may generate a corresponding driving waveform according to the haptic effect data to form a haptic output.

In some embodiments, the audio data analysis unit 501 is configured to pre-process the audio data to obtain envelope information; and judging the envelope information and a preset threshold, and if the envelope value in the envelope information is lower than the preset threshold, determining that the audio data position corresponding to the envelope value is in a non-connection state so as to obtain the at least one audio data segment.

It should be noted that, the audio data is segmented by the envelope information, and if the envelope value in the envelope information is lower than the preset threshold value, it may be confirmed that the audio data is not connected here, so as to obtain a plurality of audio data segments. In other words, several audio data segments are obtained here in a disconnected state. Thus, in the subsequent processing, the segmented audio data segments may be sequentially input into the classification unit 502 for classification processing; or when the classifying units 502 are multiple, the segmented audio data segments can be classified in parallel, so that the processing difficulty of the audio data is reduced, and meanwhile, the processing efficiency of the audio data can be improved.

In some embodiments, the audio data analysis unit 501 is further configured to perform band-pass filtering processing on the audio data after the audio data is acquired, to obtain filtered audio data; and rectifying and normalizing the filtered audio data to obtain the envelope information.

In an embodiment of the present application, the envelope information may include time domain envelope information or energy envelope information.

It should be noted that, in the embodiment of the present application, when filtering processing is performed on audio data, since any wave can be decomposed into several kinds of superposition of sine wave and cosine wave, the filtering process is a weighting process, and the filtering effect is to give different weights to different signal components, so as to reduce noise of the audio data.

It should also be noted that when higher frequency components of the signal are allowed to pass through a filter, such a filter is called a high pass filter. When lower frequency components of the signal are allowed to pass through a filter, such a filter is called a low pass filter. When only components in a certain frequency range of the signal are allowed to pass through the filter, such a filter is called a band-pass filter. When components in a certain frequency range in the signal are not allowed to pass through the filter, such a filter is called a band-reject filter. Thus, in a specific implementation manner, the filtering processing of the audio data according to the embodiments of the present application may be a band-pass filtering processing implemented by a low-pass filter, so as to filter out high-frequency signals outside the range of the fundamental frequency of the rhythm.

In some embodiments, the classifying unit 502 is configured to perform time domain feature extraction on the audio data segment to obtain time domain feature information; extracting frequency domain characteristics of the audio data segment to obtain frequency domain characteristic information; and classifying according to the time domain characteristic information and/or the frequency domain characteristic information to determine a vibration type label corresponding to the audio data segment.

In the embodiment of the present application, in the process of classifying at least one audio data segment, classification processing may be sequentially performed on each audio data segment, so that the resource pressure in each classification processing process may be reduced; the classification processing can be performed on a plurality of audio data segments at the same time, so that the classification processing speed of the audio data segments can be increased. Here, the embodiment of the present application performs the classification processing on each audio data segment in turn.

It should be further noted that, in this embodiment of the present application, a plurality of time domain feature information and a plurality of frequency domain feature information may exist in the same audio data segment at the same time, on this basis, different time domain feature information and frequency domain feature information respectively correspond to different vibration class labels, that is, the same audio data segment may correspond to a plurality of different vibration class labels, and at the same time, for one audio data segment, only time-frequency feature information may be extracted for classification processing, only frequency domain feature information may be extracted for classification processing, or both time domain feature information and frequency domain feature information may be extracted for classification processing, and in particular, adjustment of a classification method may be performed according to different requirements.

In some embodiments, the time domain feature information may include envelope fluctuation feature information and audio feature duration information, and the frequency domain feature information includes short-time spectral density distribution information. Accordingly, the classifying unit 502 is configured to determine a first feature result in the time domain feature information according to the envelope fluctuation feature information, determine a second feature result in the time domain feature information according to the audio feature duration information, determine a third feature result in the frequency domain feature information according to the short-time spectrum density distribution information, and determine a vibration class label corresponding to the audio data segment based on at least one of the first feature result, the second feature result, and the third feature result.

In the embodiment of the present application, for the same audio data segment, the first feature result, the second feature result, and the third feature result may be determined at the same time. Wherein the first characteristic result is used to determine a heave frequency characteristic of the vibration, the first characteristic result may comprise, for example: single relief or multiple relief; the second characteristic result is used to determine the duration of the vibration, which may include, for example: long vibration or short vibration; the third characteristic result is used to determine the continuity of the vibration, and may include, for example: a continuous spectrum, a line spectrum, or a mixed spectrum.

Here, the single undulation and the multiple undulations may be determined by marking a case in which the envelope undulation characteristic is higher than a preset threshold value as 1, a case in which the envelope undulation characteristic is lower than the preset threshold value as 0, undulation once when a segment of '01' or '10' occurs, and undulation multiple times when a segment of '01' or '10' alternately occurs multiple times. The method for judging the long vibration and the short vibration may be that the long vibration is judged when the duration of the audio feature is greater than a preset threshold value, and the short vibration is judged when the duration of the audio feature is less than or equal to the preset threshold value. The continuous spectrum, the line spectrum or the mixed spectrum may be judged in such a way that the spectrum is continuous on the frequency axis, whereas the spectrum is continuous on the time axis. Other cases are mixed spectra.

In some embodiments, the haptic effect analysis unit 503 is configured to perform vibration analysis according to the vibration type label corresponding to the audio data segment to obtain a vibration result corresponding to the audio data segment; and performing data analysis according to the audio data segment to obtain amplitude envelope information corresponding to the audio data segment; and determining haptic effect data corresponding to the audio data segment according to the amplitude envelope information and the vibration result.

It should be noted that, in this embodiment of the present application, the vibration type of the audio data segment may be determined to be long vibration, short vibration or combined vibration according to the vibration type tag, and the amplitude, frequency, and duration of the haptic effect may be determined according to the amplitude, frequency, and duration of the audio data, where the two may be combined to obtain haptic effect data, where the haptic effect data may be matched with each other not only in the sound effect type and the vibration type, but also in the sound effect parameter and the vibration parameter, so as to form a rich and vivid haptic output.

In some embodiments, referring to FIG. 6, a schematic diagram of the composition of a haptic generation unit in a haptic generation system is shown. As shown in fig. 6, the haptic generation unit 504 includes a haptic data generation unit 601, a haptic driving unit 602, and a stopper 603; wherein,

The haptic data generating unit 601 is configured to match haptic effect data corresponding to each of the at least one audio data segment in a preset vibration library, and generate driving waveform data;

the haptic driving unit 602 is configured to receive a trigger instruction, and drive the actuator 603 according to the trigger instruction and the driving waveform data, so as to generate a haptic output.

In the embodiment of the present application, according to the obtained haptic effect data, the required driving waveform data is obtained by matching in the preset vibration library, and according to the driving waveform data and the trigger command, the actuator 603 is driven, and the haptic sensation corresponding to the audio data segment is output. Wherein haptic effect data corresponding to one audio data segment may be matched to a plurality of different vibration types and corresponding vibration parameters, and driving waveform data is formed from the plurality of vibration types and corresponding vibration parameters. The vibration type may be long vibration, short vibration, combined vibration, or the like, and the vibration parameter may include amplitude, frequency, or the like of the vibration, but is not particularly limited herein.

In some embodiments, the haptic data generating unit 504 is further configured to obtain model parameters of the actuator, and calculate haptic effect data corresponding to each of the at least one audio data segment by using the model parameters, so as to generate the driving waveform data.

In the embodiment of the present application, the model parameters of the actuator and the haptic effect data are combined, and the desired haptic effect is simulated by the vibration of the actuator to obtain the desired driving waveform data. In addition, it should be noted that after the haptic effect data is calculated, the obtained driving waveform data may also be stored in a preset vibration library, so that the required driving waveform data may be obtained through matching of the preset vibration library.

In some embodiments, reference is made to fig. 7, which illustrates a schematic of the composition of another haptic generation system provided by embodiments of the present application. As shown in fig. 7, the haptic generation system may also include a data store 505; wherein,

the data memory 505 is configured to store audio data to be processed;

the audio data analysis unit 501 is connected to the data memory 505, and is configured to obtain the audio data from the data memory 505.

It should be noted that, in the embodiment of the present application, the data storage 505 is one of memories, and the audio data stored in the data storage 505 may be in a lossless format (for example, WAV, FLAC, APE, ALAC, wavPack) or a lossy format (for example, MP3, AAC, ogg Vorbis, opus) or the like. In addition, the audio data analysis unit 501 needs to perform preprocessing on the audio data when acquiring the audio data, where the preprocessing may include format conversion, time length interception, and the like, in order to acquire audio data of a certain length.

Embodiments of the present application disclose a haptic generation system, comprising: the audio data analysis unit is used for acquiring audio data, carrying out segmentation processing on the audio data and determining at least one audio data segment; the classifying unit is used for classifying the at least one audio data segment and determining vibration class labels corresponding to the at least one audio data segment respectively; the haptic effect analysis unit is used for receiving and analyzing at least one audio data segment and a vibration type label corresponding to the at least one audio data segment respectively, and determining haptic effect data corresponding to the at least one audio data segment respectively; and a haptic generation unit for generating driving waveform data according to haptic effect data corresponding to each of the at least one audio data segment, and generating a haptic output in response to the driving waveform data. In this way, the audio data is analyzed and classified by the audio data analysis unit and the classification unit, the corresponding haptic effect data is determined by the haptic effect analysis unit, and finally the haptic sensation corresponding to the audio data is output by the haptic sensation generation unit; thereby enabling the conversion of audio data into haptic effects and enhancing haptic effects.

In yet another embodiment of the present application, reference is made to fig. 8, which is a schematic diagram illustrating a detailed composition of a haptic generation system according to an embodiment of the present application, based on the haptic generation system described in the previous embodiment. As shown in fig. 8, the system may include: a data memory 801, an audio data analysis unit 802, a classification unit 803, a haptic effect analysis unit 804, a haptic data generation unit 805, a haptic drive unit 806, a haptic control unit 807, and a brake 808.

In embodiments of the present application, a system and method for converting audio data into haptic effects is provided herein. The data storage 801 acquires audio data with a certain length, and the audio data analysis unit 802 performs start-stop point detection on the data to form a plurality of independent audio data segments; then, sequentially inputting the audio data segments into a classification unit 803 for classification, wherein the classification unit 803 classifies the audio data segments according to the envelope, the frequency spectrum and the data length characteristics of the audio data segments, and outputs corresponding vibration type labels; the audio data segment and the vibration type tag are input to the haptic effect analysis unit 804, and the haptic effect analysis unit 804 outputs the corresponding haptic effect data. Finally, the haptic data generation unit 805 will combine the brake model parameters and the haptic effect data to generate driving waveform data; the haptic driving unit 806 drives the actuator 808 according to the trigger instruction and the driving waveform data of the haptic control unit 807 to form a haptic sensation.

On the basis of the haptic generation system provided in the foregoing embodiment, referring to fig. 9, a detailed flowchart of a haptic generation method provided in the embodiment of the present application is shown, as shown in fig. 9, where the method includes:

s901: audio data of a certain length is acquired.

Specifically, a certain length of audio data is acquired from the data memory 801.

S902: and detecting starting and ending points and dividing the starting and ending points into a plurality of audio data segments.

Specifically, the audio data analysis unit 802 is used to perform start-stop point detection on the audio data, so as to form a plurality of independent audio data segments. In some embodiments, a method of audio data segmentation may include:

the audio data is subjected to low-pass filtering to filter high frequencies outside the fundamental frequency range of the rhythm;

rectifying (the rectifying method can be absolute value or square) the filtered data, normalizing, interpolating and connecting maximum points, and outputting envelope information, such as a time domain envelope or an energy envelope;

and judging a delay threshold according to the preset threshold and the envelope information, and if the envelope value in the envelope information is lower than the preset threshold, considering that the audio data is not connected at the position. Finally, a plurality of audio data segments are obtained.

S903: each audio data segment enters a classification unit for classification.

Specifically, the audio data segments are sequentially input to the classification unit 803 for classification, and the classification unit 803 classifies the audio data segments according to the envelope, spectrum and data length characteristics of the audio data segments and outputs corresponding vibration type labels.

In some embodiments, referring to fig. 10, a schematic flow chart of a classification process in a haptic generation method according to an embodiment of the present application is shown, as shown in fig. 10, the method may include: (feature extraction is not sequential):

step one: extracting audio data time domain characteristics-envelope fluctuation characteristics: single relief and multiple relief. The fluctuation judging method can be as follows: and taking the average value of energy (or the result of other rectifying modes) as EM1 and the median value of energy EM2 to obtain a threshold value K1XE1+K2 XEM 2, marking that the energy (or the result of other rectifying modes) exceeds the threshold value as 1, marking that the energy (or the result of other rectifying modes) is lower than the threshold value as 0, judging that the duration of 1 or 0 of the marking is lower than a preset time threshold value as 1, and inverting. The resulting continuous 0 segment and continuous 1 segment undulate once when the '01' or '10' segment appears and multiple times when the '01' or '10' segment alternates multiple times.

Step two: the audio data time domain feature is extracted, namely the audio feature duration, specifically the continuous audio time when the energy (or the result of other rectification modes) exceeds the threshold value of K3+E1+K4. And the vibration is marked as a long vibration when the preset time threshold value 2 is exceeded, otherwise, the vibration is marked as a short vibration.

Step three: extracting the frequency domain characteristics of the audio data, namely the short-time frequency spectrum density distribution, specifically, carrying out short-time Fourier transform on the audio data frame by frame, wherein the frequency spectrum is continuous spectrum when being continuous on a frequency axis, and is line spectrum when being continuous on a time axis. Other cases are mixed spectra.

Step four: and according to the obtained characteristic results, carrying out single fluctuation and multiple fluctuation, long vibration, short vibration, continuous spectrum and line spectrum, and carrying out touch classification.

In particular, for music, it may be expressed as 'hit', 'plucked string', 'pulled string', 'played', and the like. In particular, for daily events, it is specifically classified into 'friction', 'collision', 'operation', and the like. For example, single heave, short vibration, continuous spectrum can be classified as 'strike', 'bump'. Multiple relief, long vibration, line spectrum, which can be classified as 'pull string'. Multiple fluctuations, long or short oscillations, mixed spectra, can be classified as 'operational'.

It should be noted that the features illustrated in the present invention may be changed by some simple calculations to form other methods, not specifically how to calculate, and the core features may represent the time-frequency characteristics of the audio. The classification can be expanded and deleted according to the characteristic result, and is not particularly the mapping from the characteristic to the classification name, and the core is the time-frequency characteristic summary of objective touch, so that the reverse association from the audio to the touch is formed.

S904: haptic effects are analyzed in conjunction with the classification results and the audio data.

Specifically, the data segment and the vibration type tag are input to the haptic effect analysis unit 804, and the haptic effect analysis unit 804 outputs corresponding haptic effect data.

In some embodiments, referring to fig. 11, which is a schematic flow chart illustrating determining haptic effect data in a haptic generation method provided in an embodiment of the present application, as shown in fig. 11, a haptic effect analysis unit may include: and according to the classification result of the classification unit, the corresponding objective touch sense is known to be long vibration, short vibration and combined vibration. Mapping is performed through a vibration effect preset by touch sense. Or further for a particular independent vibration, the vibration amplitude envelope is formed from the envelope of the audio data, the particular method may be linear or non-linear mapping. The driving frequency used by the haptic sensation is determined from the fundamental frequency of the audio, and may be a linear or a nonlinear mapping in particular.

S905: driving waveform information is generated from the haptic effect data.

Specifically, the haptic data generation unit 805 will generate driving waveform data in combination with the model parameters of the actuator 808 and the haptic effect data.

In some embodiments, the haptic effect generation unit may include: and matching the haptic effect data with a pre-stored vibration effect library to obtain driving waveform data required by vibration, or calculating the driving waveform data required by the brake motion model according to the vibration effect.

S906: the actuator drives the brake to form a touch sense under the triggering of the controller.

Specifically, the haptic driver drives the actuator according to the trigger instruction of the haptic control unit and the driving waveform data and information to form the haptic sensation.

The haptic sensation generation system and method provided by the embodiment of the application have the following key flow: the embodiment of the application provides a haptic sensation generation method, which is based on the detailed description of the specific implementation of the foregoing embodiment, and it can be seen that according to the technical scheme of the foregoing embodiment, the audio data analysis unit segments the acquired audio data on the time axis, the classification unit classifies each segment of audio data, and the haptic effect analysis unit outputs the haptic effect configuration according to the classification result and the audio data. The haptic effect generation unit generates a corresponding driving waveform according to the effect configuration. The haptic control unit enables the haptic driving unit according to the vibration information, thereby driving the actuator to form a haptic sensation. In this way, conversion of audio data to haptic effects can be achieved, enhancing haptic effects.

In a further embodiment of the present application, a computer storage medium is provided, which stores a haptic generation program which, when executed by at least one processor, implements the steps of the method of any of the preceding embodiments.

Based on the above-mentioned computer storage medium, referring to fig. 12, a specific hardware structure diagram of an electronic device provided in an embodiment of the present application is shown. As shown in fig. 12, the electronic device 120 may include: a communication interface 1201, a memory 1202 and a processor 1203; the various components are coupled together by a bus system 1204. It is appreciated that the bus system 1204 is used to facilitate connected communications between these components. The bus system 1204 includes a power bus, a control bus, and a status signal bus in addition to the data bus. But for clarity of illustration, the various buses are labeled as bus system 1204 in fig. 12. The communication interface 1201 is configured to receive and send signals during the process of receiving and sending information with other external network elements;

a memory 1202 for storing a computer program capable of running on the processor 1203;

a processor 1203 configured to, when executing the computer program, perform:

acquiring audio data;

segmenting the audio data to determine at least one audio data segment;

classifying the at least one audio data segment, and determining vibration type labels corresponding to the at least one audio data segment respectively;

Performing haptic effect analysis according to the at least one audio data segment and the vibration category label corresponding to the at least one audio data segment, and determining the haptic effect data corresponding to the at least one audio data segment;

It is to be appreciated that the memory 1202 in embodiments of the present application may be either volatile memory or nonvolatile memory, or may include both volatile and nonvolatile memory. The nonvolatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable EPROM (EEPROM), or a flash Memory. The volatile memory may be random access memory (Random Access Memory, RAM) which acts as an external cache. By way of example, and not limitation, many forms of RAM are available, such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (Double Data Rate SDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), and Direct memory bus RAM (DRRAM). The memory 1202 of the systems and methods described herein is intended to comprise, without being limited to, these and any other suitable types of memory.

While the processor 1203 may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the method described above may be performed by integrated logic circuitry in hardware or instructions in software in the processor 1203. The processor 1203 described above may be a general purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present application may be embodied directly in hardware, in a decoded processor, or in a combination of hardware and software modules in a decoded processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in the memory 1202, and the processor 1203 reads the information in the memory 1202 and performs the steps of the above method in combination with its hardware.

It is to be understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For a hardware implementation, the processing units may be implemented within one or more application specific integrated circuits (Application Specific Integrated Circuits, ASIC), digital signal processors (Digital Signal Processing, DSP), digital signal processing devices (DSP devices, DSPD), programmable logic devices (Programmable Logic Device, PLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units configured to perform the functions described herein, or a combination thereof.

For a software implementation, the techniques described herein may be implemented with modules (e.g., procedures, functions, and so on) that perform the functions described herein. The software codes may be stored in a memory and executed by a processor. The memory may be implemented within the processor or external to the processor.

Optionally, as another embodiment, the processor 1203 is further configured to perform the steps of the method of any of the previous embodiments when running the computer program.

In some embodiments, referring to fig. 13, a schematic diagram of a composition structure of an electronic device 120 according to an embodiment of the present application is shown. As shown in FIG. 13, the electronic device 120 includes at least the haptic generation system 50 of any of the foregoing embodiments.

In the embodiment of the present application, for the electronic device 120, the audio data may be analyzed and classified by the audio data analysis unit and the classification unit, then the haptic effect analysis unit determines the corresponding haptic effect data, and finally the haptic generation unit outputs the haptic sensation corresponding to the audio data; thereby enabling the conversion of audio data into haptic effects and enhancing haptic effects.

It should be noted that, in this application, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

The foregoing embodiment numbers of the present application are merely for describing, and do not represent advantages or disadvantages of the embodiments.

The methods disclosed in the several method embodiments provided in the present application may be arbitrarily combined without collision to obtain a new method embodiment.

The features disclosed in the several product embodiments provided in the present application may be combined arbitrarily without conflict to obtain new product embodiments.

The features disclosed in the several method or apparatus embodiments provided in the present application may be arbitrarily combined without conflict to obtain new method embodiments or apparatus embodiments.

The foregoing is merely specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think about changes or substitutions within the technical scope of the present application, and the changes and substitutions are intended to be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A haptic generation method, the method comprising:

acquiring audio data;

Classifying the at least one audio data segment, and determining vibration class labels corresponding to the at least one audio data segment respectively, wherein the vibration class labels comprise events corresponding to the at least one audio data segment respectively;

2. The method of claim 1, wherein the segmenting the audio data to determine at least one audio data segment comprises:

preprocessing the audio data to obtain envelope information;

3. The method of claim 2, wherein preprocessing the audio data to obtain envelope information comprises:

4. The method of claim 1, wherein classifying the at least one audio data segment to determine vibration class labels for each of the at least one audio data segment comprises:

5. The method of claim 4, wherein the time domain characteristic information comprises envelope fluctuation characteristic information and audio characteristic duration information, and the frequency domain characteristic information comprises short-time spectral density distribution information; the method further comprises the steps of:

6. The method of claim 1, wherein the determining haptic effect data for each of the at least one segment of audio data based on haptic effect analysis from the vibration class label for each of the at least one segment of audio data comprises:

7. The method of claim 1, wherein the determining haptic effect data for each of the at least one segment of audio data based on haptic effect analysis from the vibration class label for each of the at least one segment of audio data comprises:

8. The method of claim 1, wherein generating drive waveform data from haptic effect data corresponding to each of the at least one audio data segment, responsive to the drive waveform data, to generate a haptic output comprises:

9. The method of claim 8, wherein the generating driving waveform data from haptic effect data corresponding to each of the at least one audio data segment, responsive to the driving waveform data, to generate a haptic output, further comprises:

obtaining model parameters of the brake;

10. A haptic generation system, the haptic generation system comprising:

the classification unit is used for classifying the at least one audio data segment, and determining vibration class labels corresponding to the at least one audio data segment respectively, wherein the vibration class labels comprise events corresponding to the at least one audio data segment respectively;

11. The haptic generation system of claim 10, wherein,

the audio data analysis unit is used for preprocessing the audio data to obtain envelope information; and judging the envelope information and a preset threshold, and if the envelope value in the envelope information is lower than the preset threshold, determining that the audio data position corresponding to the envelope value is in a non-connection state so as to obtain the at least one audio data segment.

12. The haptic generation system of claim 11, wherein,

the audio data analysis unit is further used for carrying out band-pass filtering processing on the audio data after the audio data are acquired, so as to obtain filtered audio data; rectifying and normalizing the filtered audio data to obtain the envelope information; wherein the envelope information comprises time domain envelope information or energy envelope information.

13. The haptic generation system of claim 10, wherein,

the classifying unit is used for extracting time domain features of the audio data segment to obtain time domain feature information; extracting frequency domain characteristics of the audio data segment to obtain frequency domain characteristic information; and classifying according to the time domain characteristic information and/or the frequency domain characteristic information to determine a vibration type label corresponding to the audio data segment.

14. A haptic generation system as recited in claim 13 wherein said time domain feature information includes envelope fluctuation feature information and audio feature duration information, said frequency domain feature information includes short-time spectral density distribution information;

15. The haptic generation system of claim 14, wherein,

the first feature result includes: single relief or multiple relief;

the second feature result includes: long vibration or short vibration;

16. The haptic generation system of claim 10, wherein,

the haptic effect analysis unit is used for carrying out vibration analysis according to the vibration type label corresponding to the audio data segment to obtain a vibration result corresponding to the audio data segment; performing data analysis according to the audio data segment to obtain amplitude envelope information corresponding to the audio data segment; and determining haptic effect data corresponding to the audio data segment according to the amplitude envelope information and the vibration result.

17. The haptic generation system of claim 10, the haptic generation unit comprising a haptic data generation unit, a haptic drive unit, and a brake; wherein,

18. The haptic generation system of claim 17,

the haptic data generating unit is further configured to obtain model parameters of the brake, calculate haptic effect data corresponding to each of the at least one audio data segment by using the model parameters, and generate the driving waveform data.

19. An electronic device, the electronic device comprising:

a memory for storing a computer program capable of running on the processor;

a processor for performing the method of any of claims 1 to 9 when the computer program is run.

20. A computer storage medium storing a computer program which, when executed by at least one processor, implements the method of any one of claims 1 to 9.