KR20150028737A - Haptic conversion system using frequency shifting - Google Patents

Abstract

Provided is a system that converts an input into one or more haptic effects using frequency shifting. The system receives an input signal. The system further performs a fast Fourier transform of the input signal. The system further shifts one or more frequencies of the transformed input signal to one or more frequencies within a shift-to frequency range. The system further performs an inverse fast Fourier transform of the frequency-shifted signal, where the inversely transformed signal forms a haptic signal. The system further generates the one or more haptic effects based on the haptic signal.

Description

TECHNICAL FIELD [0001] The present invention relates to a haptic conversion system using a frequency transition,

One embodiment relates generally to a device, and more particularly, to a device that generates haptic effects.

Haptic is a tactile and force feedback technique that utilizes a user's sense of touch by applying haptic feedback effects (i.e., "haptic effects") such as force, vibration and motion. Devices such as mobile devices, touch screen devices, and personal computers can be configured to generate haptic effects. Generally, a call to embedded hardware (such as an actuator) that may cause haptic effects may be programmed into the device's operating system (OS). These calls specify which haptic effect to play. For example, when a user interacts with a device using, for example, a button, a touch screen, a lever, a joystick, a wheel, or some other control, the device's OS sends a playback command Can be transmitted. Embedded hardware then generates the appropriate haptic effect.

The devices may be configured to adjust the output of the haptic effects with the output of other components such as audio, such that the haptic effects are contained within the other component. For example, an audio effects developer can develop audio effects that can be output by the device, such as machine gun fire, explosion, or car crashes. In addition, other types of content, such as video effects, may be developed and later output by the device. The haptic effect developer may then make a haptic effect on the device, and the device may be configured to output the haptic effect with other components. However, this process generally requires the personal judgment of a haptic effect developer to produce a haptic effect that accurately compensates for audio effects or other types of content. A poorly made haptic effect that does not complement audio effects or other types of content can produce an overall harmonic effect if the haptic effect does not "fit " well with audio effects or other content. This type of user experience is generally not desired.

One embodiment is a system that transforms an input to one or more haptic effects using a frequency transition. The system receives the input signal. The system also performs fast Fourier transform of the input signal. The system also transitions one or more frequencies of the transformed input signal to one or more frequencies within a shift-to frequency range. The system also performs an inverse fast Fourier transform of the frequency shifted signal, where the inverse transformed signal forms a haptic signal. The system also generates one or more haptic effects based on the haptic signal.

Further embodiments, details, advantages and modifications will become apparent from the following detailed description of the preferred embodiments, which is to be read in connection with the accompanying drawings.
1 is a block diagram of a system according to one embodiment of the present invention;
2 is a diagram illustrating a frequency transition of a plurality of frequencies of an input signal, according to an embodiment of the present invention, wherein the frequency transition is used to convert the input signal into a haptic signal.
Figure 3 is a flow diagram of the functionality of a haptic transformation module, in accordance with an embodiment of the present invention.
4 is a flow diagram of a first exemplary method for converting an input signal to a haptic signal, in accordance with an embodiment of the present invention.
5 is a flow diagram of a second exemplary method for converting an input signal to a haptic signal, in accordance with an embodiment of the present invention.
6 is a flow diagram of a third exemplary method for converting an input signal to a haptic signal, in accordance with an embodiment of the present invention.
7 is a flow diagram of a fourth exemplary method for converting an input signal to a haptic signal, in accordance with an embodiment of the present invention.
8 is a flow diagram of a fifth exemplary method for converting an input signal to a haptic signal, in accordance with an embodiment of the present invention.

One embodiment is a system that can automatically derive a haptic signal associated with an input signal. An example of an input signal may be an audio signal, where the audio signal may be a component of a multimedia file, such as an audio file or a video file. The derived haptic signal includes frequency content transited to a range compatible with a haptic output device such as an actuator. Thus, the system can transmit the derived haptic signal to the haptic output device and generate one or more haptic effects based on the derived haptic signal. Alternatively, the system may store the haptic signal to be reproduced later in a synchronous manner with other signals. According to this embodiment, the system may utilize a fast Fourier transform of the input signal to derive an associated haptic signal. The system can analyze the frequency content of the input signal and can detect frequencies with the highest amplitudes (using different approaches and in different ways), and detect these frequencies (by different techniques) It can be shifted to the frequency range around the resonance frequency of the output device. Different strategies may be used to classify and transition the frequency content of the input signal, as will be described in more detail below.

Figure 1 shows a block diagram of a system 10 in accordance with one embodiment of the present invention. In one embodiment, the system 10 is part of a mobile device, and the system 10 provides haptic conversion functionality to the mobile device. In another embodiment, the system 10 is part of a wearable device, and the system 10 provides a haptic translation function to the wearable device. Examples of wearable devices include wristbands, headbands, glasses, rings, ankle bands, arrays integrated into a garment, or any other type of device that a wearer can wear or carry on the body. Some wearable devices can be "haptically enabled ", which means that the devices include a mechanism for generating haptic effects. In another embodiment, the system 10 is separate from a device (e.g., a mobile device or wearable device) and remotely provides a haptic translation function to the device. Although depicted as a single system, the functionality of the system 10 may be implemented as a distributed system. The system 10 includes a bus 12 or other communication mechanism for communicating information and a processor 22 for processing information coupled to the bus 12. The processor 22 may be any type of general purpose or special purpose processor. The system 10 further includes a memory 14 for storing information and instructions to be executed by the processor 22. The memory 14 may be comprised of any combination of random access memory (RAM), read only memory (ROM), static storage such as magnetic or optical disk, or any other type of computer readable media.

Computer readable media can be any available media that can be accessed by processor 22 and can include both volatile and nonvolatile media, removable and non-removable media, communication media, and storage media. Communication media may include computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any other form of information delivery medium known in the art . ≪ / RTI > The storage medium may be RAM, flash memory, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, a hard disk, a removable disk, a compact disk read- , Or any other form of storage medium known in the art.

In one embodiment, memory 14 stores software modules that provide functionality when executed by processor 22. These modules, in one embodiment, include an operating system 15 that provides operating system functionality to the system 10 as well as to the remainder of the mobile device. These modules further include a haptic conversion module 16 that uses a frequency transition to convert the input to one or more haptic effects, as described in more detail below. In certain embodiments, the haptic conversion module 16 may comprise a plurality of modules, where each module provides a specific individual function of converting the input to one or more haptic effects using a frequency transition . The system 10 will typically include one or more additional application modules 18 to include additional functionality, such as Integrator ^TM software from Immersion Corporation.

The system 10 may be used to provide mobile wireless network communications, such as infrared, wireless, Wi-Fi, or cellular network communications, in embodiments that transmit and / or receive data to and / And further includes a communication device 20 such as an interface card. In other embodiments, the communication device 20 provides a wired network connection, such as an Ethernet connection or a modem.

The processor 22 is also coupled to a display 24, such as a liquid crystal display (LCD), that displays, via the bus 12, a graphical representation or user interface to the user. The display 24 may be a touch sensitive input device, such as a touch screen, configured to transmit signals to and receive signals from the processor 22, and may be a multi-touch touch screen.

In one embodiment, the system 10 further includes an actuator 26. The processor 22 may send a haptic signal associated with the generated haptic effect to the actuator 26 and the actuator 26 may in turn transmit the haptic signal to the actuator 26 such as vibrotactile haptic effects, electrostatic friction haptic effects, And outputs haptic effects. The actuator 26 includes an actuator driving circuit. Actuator 26 may be, for example, an electric motor, an electromagnetic actuator, a voice coil, a shape memory alloy, an electro-active polymer, a solenoid, an eccentric rotating mass motor (ERM), a linear resonant actuator (LRA) A piezoelectric actuator, a high-bandwidth actuator, an electro-active polymer (EAP) actuator, an electrostatic friction display, or an ultrasonic vibration generator. In alternate embodiments, the system 10 may include one or more additional actuators (not illustrated in FIG. 1) in addition to the actuator 26. The actuator 26 is an example of a haptic output device wherein the haptic output device is configured to output haptic effects such as vibrotactile haptic effects, electrostatic friction haptic effects, or deformed haptic effects in response to a drive signal . In alternate embodiments, the actuator 26 may be replaced with any other type of haptic output device. In addition, in other alternative embodiments, the system 10 may not include an actuator 26, and a device separate from the system 10 may include an actuator or other haptic output device that generates haptic effects , The system 10 transmits the generated haptic signals to the device via the communication device 20.

In one embodiment, the system 10 further includes a speaker 28. The processor 22 may transmit the audio signal to the speaker 28, which in turn outputs the audio effects. The speaker 28 may be any type of speaker such as, for example, a dynamic speaker, an electrodynamic speaker, a piezoelectric speaker, a magnetostrictive speaker, an electrostatic speaker, a ribbon and a flat magnetic speaker, a bending wave speaker, a flat speaker, a heil air motion transducer ), A plasma arc speaker, and a digital speaker. In alternative embodiments, the system 10, in addition to the speaker 28, may include one or more additional speakers (not illustrated in FIG. 1). In addition, in other alternative embodiments, the system 10 may not include a speaker 28, and a separate device from the system 10 may include a speaker for outputting audio effects, Transmits audio signals to the device via the communication device (20).

In one embodiment, the system 10 further comprises a sensor 30. The sensor 30 may be any type of sensor capable of sensing and controlling a sound, a movement, an acceleration, a bio signal, a distance, a flow, a force / pressure / strain / bend, a humidity, a linear position, an orientation / tilt, Such as, but not limited to, light, vibration, or visible light intensity. The sensor 30 may also be configured to convert the detected energy or other physical characteristics into an electrical signal or any signal representing the virtual sensor information. The sensor 30 may be an accelerometer, an electrocardiograph, an electroencephalograph, a near field, an electric ophthalmometer, an electric palate, an electric skin reaction sensor, a capacitive sensor, a Hall effect sensor, an infrared sensor, an ultrasonic sensor, (Or bend sensor), force sensitive resistor, load cell, LuSense CPS ² 155, miniature pressure transducer, piezoelectric sensor, strain gage, hygrometer, linear position touch sensor, linear potentiometer (or slider), linear variable differential transformer, compass, A temperature sensor (such as a thermometer, a thermocouple, a resistance temperature detector, a thermistor, or a temperature conversion integrated circuit), a temperature sensor, A microphone, a photometer, an altimeter, a bio-monitor, a camera, or a light-dependent resistor. In alternative embodiments, the system 10, in addition to the sensor 30, may include one or more additional sensors (not illustrated in FIG. 1). In some of these embodiments, the sensor 30 and one or more additional sensors may be part of an array of sensors, or a collection of some other type of sensors. In addition, in other alternative embodiments, the system 10 may not include the sensor 30, and a device separate from the system 10 may detect one type of energy, or other physical characteristic, Or other physical characteristics into electrical or other types of signals representing virtual sensor information. The device can then transmit the converted signal to the system 10 via the communication device 20. [

FIG. 2 illustrates a frequency transition of a plurality of frequencies of an input signal, according to an embodiment of the present invention, wherein the frequency transition is used to convert the input signal into a haptic signal. FIG. 2 shows a graph 210 that is a representation of an audio signal, which is an input signal of one type. In alternative embodiments, the audio signal may be replaced by another type of input signal, such as a video signal, an acceleration signal, an orientation signal, an ambient light signal, or other type of signal that may include data captured by the sensor .

According to this embodiment, a fast Fourier transform may be performed on an audio signal, where the transformed audio signal is represented by a graph 220. A fast Fourier transform is a mapping of signals or other functions defined in one area such as space or time to other areas such as wavelength or frequency.

The one or more frequencies of the transformed audio signal may then be classified and transited according to one of the plurality of frequency transition methods, wherein the frequency shifted audio signal is represented by graph 230. More specifically, the frequencies of the transformed signal are classified by a characteristic (such as amplitude), and one, some, or all of the classified frequencies of the transformed signal are selected, Frequency to a frequency within the "after" frequency range.

Exemplary techniques for classifying and transiting frequencies are further described below as part of exemplary methods for converting an input signal to a haptic signal, with reference to Figures 4, 5, 6, 7, and 8. However, those skilled in the art will appreciate that the exemplary frequency shifting techniques described below are exemplary frequency shifting techniques in accordance with the illustrative embodiments, and that, in alternative embodiments, It will be appreciated that frequency shift techniques can be categorized and transited according to the techniques. In addition, the exemplary techniques described below may be used to generate a set of proprietary haptic effects, where one set of haptic effects will produce a different "feel" based on the selected frequency transition technique. In addition, the frequency transition technique used can be predefined by the system, or customized by the system administrator, the haptic effect developer, the user, or some other individual.

According to this embodiment, an inverse fast Fourier transform may be performed on the frequency shifted audio signal, where the inverse transformed audio signal is represented by graph 240. The haptic signal can then be formed based on the inversely transformed audio signal. The haptic signal can then be transmitted to a haptic output device, such as an actuator, where the haptic signal is used by the haptic output device to output one or more haptic effects. Alternatively, the haptic signal may be stored in a storage medium such as RAM, flash memory, ROM, EPROM, EEPROM, register, hard disk, removable disk, CD-ROM, or any other form of storage medium. The haptic signal can then be retrieved from the storage medium and transmitted to the haptic output device, so that the haptic output device can output one or more haptic effects based on the haptic signal.

FIG. 3 illustrates a flow diagram of the functionality of a haptic transformation module (such as the haptic transformation module 16 of FIG. 1), in accordance with one embodiment of the present invention. In one embodiment, the functionality of FIG. 3 as well as FIG. 4, FIG. 5, FIG. 6, FIG. 7, and FIG. 8 may each be stored in memory or other computer readable or type media and executed by a processor It is implemented by software. In other embodiments, each function may be implemented in hardware (e.g., through the use of an application specific integrated circuit (ASIC), programmable gate array (PGA), field programmable gate array (FPGA) And the like.

Flow begins and proceeds to 300. At 300, an input signal is received. In certain embodiments, the input signal may be an audio signal. In other embodiments, the input signal may be a video signal. In other embodiments, the input signal may be an acceleration signal. The flow then continues to 310.

At 310, a fast Fourier transform is performed on the input signal. The flow then continues to 320.

At 320, the amplitudes of the plurality of frequencies of the transformed input signal are normalized. In certain embodiments, as a result of the fast Fourier transform of the input signal, two vectors are obtained: one vector containing frequencies, and another vector containing sizes corresponding to those frequencies. A maximum value of all sizes may be found and a maximum value may be used to normalize all sizes so that a magnitude vector having a value less than 1 is obtained. In other words, a vector containing magnitudes can be normalized to one. In certain embodiments, 320 may be omitted. The flow then continues to 330.

At 330, the frequencies of the transformed input signal are categorized. The frequencies may be classified based on the characteristics of the frequencies. In certain embodiments, the frequency of the transformed input signal is determined by an exemplary method of converting an input signal to a haptic signal, as described in more detail below with respect to Figures 4, 5, 6, 7, Lt; RTI ID = 0.0 > a < / RTI > In certain embodiments, 330 may be omitted. The flow then continues to 340.

At 340, one or more frequencies are selected from the frequencies of the transformed input signal. One, some or all of the frequencies of the transformed input signal may be selected. In certain embodiments, the frequency of the transformed input signal is determined by an exemplary method of converting an input signal to a haptic signal, as described in more detail below with respect to Figures 4, 5, 6, 7, May be selected according to a selection technique that is part of one of the methods. In certain embodiments, 340 may be omitted. The flow then continues to 350.

At 350, selected frequencies of the transformed input signal are clustered. By clustering, the selected frequencies can be reduced to a lesser number of frequencies. In certain embodiments, the selected frequencies of the transformed input signal are converted to a haptic signal by converting an input signal to a haptic signal, as described in more detail below with respect to Figures 4, 5, 6, 7, Clustering techniques that are part of one of the methods can be clustered. In addition, in certain embodiments, 350 may be omitted. The flow then continues to 360.

At 360, the clustered frequencies of the transformed input signal transitions to within the "after" frequency range. By transition, each clustered frequency is shifted from its original frequency (i.e., the "shift-from" frequency) to the new frequency (ie, the "after" frequency). In certain embodiments, the clustered frequencies of the transformed input signal may be transformed into an example of converting an input signal to a haptic signal, as described in more detail below with respect to Figures 4, 5, 6, 7, And can be transited according to a frequency transition technique which is part of one of the methods. Further, in a specific embodiment, a haptic signal is formed based on the obtained frequency shifted signal. The flow then proceeds to 370.

At 370, an inverse fast Fourier transform is performed on the frequency shifted signal. In certain embodiments, a haptic signal is formed based on the obtained inverse transformed signal. The flow then proceeds to 380.

At 380, the inversely transformed signal is enveloped by the original input signal. This may be a one-to-one point (sample) multiplication between the original input signal and the inverse transformed signal. Another way to do envelope processing is to extract the envelope of the original input signal and multiply the inversely transformed signal by this envelope. Alternatively, the inversely transformed signal may be used as it is without being enveloped, or in other words, multiplied by an envelope having a constant magnitude of one. In certain embodiments, a haptic signal is formed based on the obtained enveloped signal. In alternate embodiments, 380 may be omitted. The flow then continues to 390.

At 390, one or more haptic effects are generated based on the haptic signal. In some embodiments, a haptic signal may be transmitted to a haptic output device, such as actuator 26 of FIG. 1, to generate one or more haptic effects. Then, the flow ends.

4 is a flow diagram of a first exemplary method for converting an input signal to a haptic signal, in accordance with an embodiment of the present invention. Flow begins and proceeds to 410. At 410, a fast Fourier transform is performed on an input signal such as an audio signal, a video signal, an acceleration signal, or any other signal that may contain data captured by the sensor. The flow then continues to 420.

At 420, the frequencies of the transformed input signal are sorted by their amplitude. The flow then proceeds to 430.

At 430, a predetermined number of frequencies having the largest amplitudes are selected from the frequencies of the transformed input signal. The predetermined number of frequencies may be equal to the width of the predetermined frequency range. The predetermined frequency range may be in the "after" frequency range. The flow then proceeds to 440.

At 440, the selected frequencies are shifted to the frequency range after the threshold. In certain embodiments, the frequency range after the cloth may include the resonant frequency of the haptic output device. In some of the embodiments, the haptic output device may be an actuator. In addition, in certain embodiments, selected frequencies may be moved by assigning an amplitude (derived from a fast Fourier transform value) of each selected frequency to a frequency within the frequency range after the frequency. The flow then proceeds to 450.

At 450, all fast Fourier transform values of frequencies not within the frequency range after the thousandth are nullified. The flow then proceeds to 460.

At 460, an inverse fast Fourier transform of the frequency shifted signal is performed. The flow then proceeds to 470.

At 470, the inverse transformed signal is enveloped by the original input signal to form a haptic signal. This may be a one-to-one point (sample) multiplication between the original input signal and the inverse transformed signal. Another way to do envelope processing is to extract the envelope of the original input signal and multiply the inversely transformed signal by this envelope. Alternatively, the inversely transformed signal may be used as it is without being enveloped, or in other words, multiplied by an envelope having a constant magnitude of one. Then, the flow ends.

5 illustrates a flow diagram of a second exemplary method for converting an input signal to a haptic signal, in accordance with an embodiment of the present invention. Flow begins and proceeds to 510. At 510, a fast Fourier transform is performed on an input signal such as an audio signal, a video signal, an acceleration signal, or any other signal that may contain data captured by the sensor. The flow then continues to 520.

At 520, the amplitudes of the frequencies of the transformed input signal are normalized. In certain embodiments, as a result of the fast Fourier transform of the input signal, two vectors are obtained: one vector containing frequencies, and another vector containing sizes corresponding to those frequencies. A maximum value of all sizes may be found and a maximum value may be used to normalize all sizes so that a magnitude vector having a value less than 1 is obtained. In other words, a vector containing magnitudes can be normalized to one. The flow then continues to 530.

At 530, the frequencies of the transformed input signal are sorted by their normalized amplitude. The flow then continues to 540.

At 540, one or more frequencies having amplitudes greater than a predetermined threshold (e.g., 0.1) are selected from the frequencies of the converted input signal. The selected frequencies may be identified as frequencies "fs ", and other frequencies may be ignored. The flow then proceeds to 550.

At 550, the remaining frequencies fs are clustered as follows. the frequency fs [i + 1] having the second largest amplitude, starting from the frequency fs [i] having the largest amplitude, for each frequency fs [i] all frequencies within [i] -x Hz to fs [i] + x Hz] (e.g., x = 1) are considered. When a frequency having an amplitude larger than the frequency fs [i] is within this range, the frequency fs [i] is removed. By applying this clustering technique, only frequencies apart by more than x Hz are present. These remaining frequencies may be identified as frequencies "fc ". The flow then continues to 560.

At 560, the remaining frequencies fc are grouped into one or more blocks B (fc), where B (fc [i]) = [fc [i] y Hz to fc [i] + y Hz], where y is a smaller number than the width of the predetermined frequency range. The predetermined frequency range may be in the "after" frequency range. The frequency content of each block B (fc [i]) then transitions to a frequency in the frequency range (e. G., 200 to 300 Hz for an actuator having a resonant frequency of 250 Hz) The blocks B (fc) may be separated by a predetermined distance d (e. G., D = 1 Hz) within the frequency range after the fringe, so that the content shifted from the different frequencies will feel different. For example, a first block corresponding to a frequency having the largest amplitude may be shifted to center at 250 Hz, and a second block corresponding to a frequency having the second largest amplitude may be shifted to 248 Hz, The third block corresponding to a frequency having the third largest amplitude may be shifted to center at 252 Hz and the lower and upper limits of the frequency range (e.g., 200 Hz to 300 Hz) , The same is true. In certain embodiments, one or more frequencies of the frequency content of each block may be transitioned by assigning an amplitude (derived from a fast Fourier transform value) of each frequency to a frequency within the frequency range after the frequency. The flow then continues to 570.

At 570, all Fast Fourier Transform values of frequencies not within the frequency range after the threshold are zero. The flow then continues to 580.

At 580, an inverse fast Fourier transform of the frequency shifted signal is performed. The flow then continues to 590.

At 590, to form a haptic signal, the inversely transformed signal is enveloped by the original input signal. This may be a one-to-one point (sample) multiplication between the original input signal and the inverse transformed signal. Another way to do envelope processing is to extract the envelope of the original input signal and multiply the inversely transformed signal by this envelope. Alternatively, the inversely transformed signal may be used as it is without being enveloped, or in other words, multiplied by an envelope having a constant magnitude of one. The enveloped signal can have an envelope close to the envelope of the original input signal and can be used as a haptic signal. Then, the flow ends.

Figure 6 illustrates a flow diagram of a third exemplary method for converting an input signal to a haptic signal, in accordance with an embodiment of the present invention. Flow begins and proceeds to 610. At 610, a fast Fourier transform is performed on an input signal such as an audio signal, a video signal, an acceleration signal, or any other signal that may contain data captured by the sensor. The flow then continues to 620.

At 620, the amplitudes of the frequencies of the transformed input signal are normalized. In certain embodiments, as a result of the fast Fourier transform of the input signal, two vectors are obtained: one vector containing frequencies, and another vector containing sizes corresponding to those frequencies. A maximum value of all sizes may be found and a maximum value may be used to normalize all sizes so that a magnitude vector having a value less than 1 is obtained. In other words, a vector containing magnitudes can be normalized to one. The flow then continues at 630.

At 630, the frequencies of the transformed input signal are sorted by their normalized amplitude. The flow then continues to 640.

At 640, one or more frequencies having amplitudes greater than a predetermined threshold (e.g., 0.1) are selected from the frequencies of the transformed input signal. The selected frequencies may be identified as frequencies "fs ", and other frequencies may be ignored. The flow then proceeds to 650.

At 650, the remaining frequencies fs are clustered as follows. the frequency fs [i + 1] having the second largest amplitude, starting from the frequency fs [i] having the largest amplitude, for each frequency fs [i] all frequencies at [i] -x Hz to fs [i] + x Hz] (e.g., x = 1) are considered. When a frequency having an amplitude larger than the frequency fs [i] is within this range, the frequency fs [i] is removed. By applying this clustering, there are only frequencies apart by more than x Hz. These remaining frequencies may be identified as frequencies "fc ". The flow then continues to 660.

At 660, the remaining frequencies fc are grouped into one or more blocks B (fc), where B (fc [i]) = [fc [i] -y Hz to fc [i] + y Hz], where y is a smaller number than the width of the predetermined frequency range. The predetermined frequency range may be in the "after" frequency range. The frequency content of each block B (fc [i]) then transitions to a frequency in the frequency range (e. G., 200 to 300 Hz for an actuator having a resonant frequency of 250 Hz) The selection of the locations of the frequencies in the frequency range after the cloth can be random. In certain embodiments, one or more frequencies of the frequency content of each block may be transitioned by assigning an amplitude (derived from a fast Fourier transform value) of each frequency to a frequency within the frequency range after the frequency. The flow then proceeds to 670.

At 670, all fast Fourier transform values of frequencies not within the frequency range after the thousandth are zero. The flow then proceeds to 680.

At 680, an inverse fast Fourier transform of the frequency shifted signal is performed. The flow then proceeds to 690.

At 690, to form a haptic signal, the inverse transformed signal is enveloped by the original input signal. This may be a one-to-one point (sample) multiplication between the original input signal and the inverse transformed signal. Another way to do envelope processing is to extract the envelope of the original input signal and multiply the inversely transformed signal by this envelope. Alternatively, the inversely transformed signal may be used as it is without being enveloped, or in other words, multiplied by an envelope having a constant magnitude of one. The enveloped signal can have an envelope close to the envelope of the original input signal and can be used as a haptic signal. Then, the flow ends.

Figure 7 illustrates a flow diagram of a fourth exemplary method for converting an input signal to a haptic signal, in accordance with an embodiment of the present invention. Flow begins and proceeds to 700. At 700, a fast Fourier transform is performed on an input signal, such as an audio signal, a video signal, an acceleration signal, or any other signal that may contain data captured by the sensor. The flow then proceeds to 710.

At 710, the amplitudes of the frequencies of the transformed input signal are normalized. In certain embodiments, as a result of the fast Fourier transform of the input signal, two vectors are obtained: one vector containing frequencies, and another vector containing sizes corresponding to those frequencies. A maximum value of all sizes may be found and a maximum value may be used to normalize all sizes so that a magnitude vector having a value less than 1 is obtained. In other words, a vector containing magnitudes can be normalized to one. The flow then proceeds to 720.

At 720, the frequencies of the transformed input signal are sorted by their normalized amplitude. The flow then proceeds to 730.

At 730, one or more frequencies having amplitudes that are greater than a predetermined threshold (e.g., 0.1) are selected from the frequencies of the transformed input signal. The selected frequencies may be identified as frequencies "fs ", and other frequencies may be ignored. The flow then proceeds to 740.

At 740, the remaining frequencies fs are clustered as follows. the frequency fs [i + 1] having the second largest amplitude, starting from the frequency fs [i] having the largest amplitude, for each frequency fs [i] all frequencies at [i] -x Hz to fs [i] + x Hz] (e.g., x = 1) are considered. When a frequency having an amplitude larger than the frequency fs [i] is within this range, the frequency fs [i] is removed. By applying this clustering, there are only frequencies apart by more than x Hz. These remaining frequencies may be identified as frequencies "fc ". The flow then proceeds to 750.

At 750, the remaining frequencies fc are clustered as follows. the surface under the amplitude spectrum curve for the range [fc [i] -y Hz to fc [i] + y Hz] is calculated for each frequency "fc [i]" at fc. If the surface is larger than the predetermined threshold, the frequency fc [i] is maintained. Otherwise, the frequency fc [i] is removed. The remaining frequencies may be identified as frequencies "fd ". The flow then proceeds to 760.

At 760, the remaining frequencies fd are grouped into one or more blocks B (fd), where B (fd [i]) = [fd [i] -z Hz to fd [i] + z Hz], where z is a smaller number than the width of the predetermined frequency range. The predetermined frequency range may be in the "after" frequency range. The frequency content of each block B (fd [i]) then transitions to a frequency within the frequency range (e. G., 200 to 300 Hz for an actuator with a 250 Hz resonant frequency) The selection of the locations of the frequencies in the frequency range after the cloth can be random. In certain embodiments, one or more frequencies of the frequency content of each block may be transitioned by assigning an amplitude (derived from a fast Fourier transform value) of each frequency to a frequency within the frequency range after the frequency. The flow then proceeds to 770.

At 770, all Fast Fourier Transform values of frequencies not within the frequency range after the threshold are zero. The flow then proceeds to 780.

At 780, an inverse fast Fourier transform of the frequency shifted signal is performed. The flow then proceeds to 790.

At 790, to form a haptic signal, the inverse transformed signal is enveloped by the original input signal. This may be a one-to-one point (sample) multiplication between the original input signal and the inverse transformed signal. Another way to do envelope processing is to extract the envelope of the original input signal and multiply the inversely transformed signal by this envelope. Alternatively, the inversely transformed signal may be used as it is without being enveloped, or in other words, multiplied by an envelope having a constant magnitude of one. The enveloped signal can have an envelope close to the envelope of the original input signal and can be used as a haptic signal. Then, the flow ends.

Figure 8 shows a flow diagram of a fifth exemplary method for converting an input signal to a haptic signal, in accordance with an embodiment of the present invention. At 810, a fast Fourier transform is performed on an input signal, such as an audio signal, a video signal, an acceleration signal, or any other signal that may contain data captured by the sensor. The flow then continues to 820.

At 820, the amplitude spectrum of the transformed input signal is estimated. The flow then continues to 830.

At 830, the frequencies of the transformed input signal are grouped into adjacent blocks b, where each block contains x consecutive frequencies. The flow then proceeds to 840.

At 840, the surface under the amplitude spectrum (identified as surface S (b)) is estimated for each block b, and each surface S (b) is normalized to 1 using the maximum of all surfaces S do. The flow then proceeds to 850.

At 850, each block b with a surface S (b) below a predetermined threshold is removed. The flow then proceeds to 860.

At 860, the frequency content of one or more blocks with the largest surfaces is shifted to a random location within the "after the thousand" frequency range and centered thereon. The flow then proceeds to 870.

At 870, all Fast Fourier Transform values of frequencies not within the frequency range after the threshold are zero. The flow then proceeds to 880.

At 880, an inverse fast Fourier transform of the frequency shifted signal is performed. The flow then continues to 890.

At 890, to form a haptic signal, the inversely transformed signal is enveloped by the original input signal. This may be a one-to-one point (sample) multiplication between the original input signal and the inverse transformed signal. Another way to do envelope processing is to extract the envelope of the original input signal and multiply the inversely transformed signal by this envelope. Alternatively, the inversely transformed signal may be used as it is without being enveloped, or in other words, multiplied by an envelope having a constant magnitude of one. The enveloped signal can have an envelope close to the envelope of the original input signal and can be used as a haptic signal. Then, the flow ends.

As such, in one embodiment, the system may use a frequency transition to generate a haptic signal from an input signal, such as an audio signal, a video signal, or an acceleration signal. The system can thus convert input data, such as audio data, video data, or acceleration data, into haptic effects, without requiring any filtering of the input signal. Thus, all the contents of the input signal can be used to generate the haptic signal.

The features, structures, or characteristics of the invention described throughout this specification may be combined in any suitable manner in one or more embodiments. For example, the use of "one embodiment", "certain embodiments", "a specific embodiment", "specific embodiments", or other similar expressions throughout this specification is to be interpreted relative to the embodiment Structure or characteristic described may be included in at least one embodiment of the present invention. Thus, the appearances of the phrase " one embodiment, "" certain embodiments," " And the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

It will be appreciated by those skilled in the art that the present invention as discussed above may be practiced with the steps in a different order and / or with the elements differing from what is disclosed. Thus, while the present invention has been described on the basis of these preferred embodiments, it will be apparent to those skilled in the art that various modifications, changes and alternative constructions will become apparent to those skilled in the art without departing from the spirit and scope of the invention. Therefore, in order to determine the scope of the invention, reference should be made to the appended claims.

Claims (15)

A computer implemented method of transforming an input into one or more haptic effects using a frequency transition,
Receiving an input signal;
Performing a fast Fourier transform of the input signal;
Transitioning one or more frequencies of the transformed input signal to one or more frequencies within a shift-to frequency range;
Performing an inverse fast Fourier transform of the frequency shifted signal, wherein the inversely transformed signal forms a haptic signal; And
And generating the one or more haptic effects based on the haptic signal. 2. The method of claim 1, further comprising: classifying the frequencies of the transformed input signal; And
And selecting one or more frequencies from the frequencies of the transformed input signal. 3. The method of claim 2, further comprising clustering the one or more selected frequencies of the transformed input signal. 4. The method of claim 3, further comprising normalizing the amplitude of the frequencies of the transformed input signal. 3. The method of claim 2, wherein classifying the frequencies of the transformed input signal further comprises classifying the frequencies by amplitude;
Wherein selecting the one or more frequencies of the transformed input signal further comprises selecting one or more frequencies having the largest amplitudes;
Wherein transitioning the one or more selected frequencies of the transformed input signal further comprises moving the one or more selected frequencies to a frequency range after the cloth, wherein the frequency range after the cloth includes the resonant frequency of the haptic output device Gt; 6. The method of claim 5, wherein moving the one or more selected frequencies of the estimated input signal comprises:
Assigning an amplitude derived from a fast Fourier transform value of each selected frequency to a frequency in the frequency range after the frequency; And
And nulling all Fast Fourier Transform values of frequencies not within the frequency range after the threshold. 5. The method of claim 4, wherein classifying the frequencies of the transformed input signal further comprises classifying the frequencies by their normalized amplitudes;
Wherein selecting the one or more frequencies of the transformed input signal further comprises selecting one or more frequencies having amplitudes greater than a predetermined threshold;
Clustering the one or more selected frequencies of the transformed input signal may include, for each selected frequency, removing the selected frequency if there is a frequency within a first predetermined range having an amplitude greater than the amplitude of the selected frequency Further comprising:
Wherein transitioning the one or more selected frequencies of the transformed input signal comprises grouping the one or more clustered frequencies in one or more blocks, the width of each block being greater than a width of the second Shifting the frequency content of each block into the frequency range after the threshold, and allowing the shifted frequency content of each block to be focused on a frequency within the frequency range after the threshold ≪ / RTI > 8. The method of claim 7, wherein transiting the frequency content of each block to a frequency within the frequency range
Assigning the fast Fourier transform values of frequencies of each group to frequencies within the frequency range after the frequency; And
Further comprising zeroing all fast Fourier transform values of frequencies not within the frequency range after the threshold. 8. The method of claim 7, wherein each block is disposed at a random location within the frequency range after the cloth;
Wherein clustering the one or more selected frequencies of the transformed input signal comprises calculating for each clustered frequency a surface under the amplitude spectrum curve for a third predetermined range, And removing the clustered frequencies. 3. The method of claim 2, further comprising: estimating an amplitude spectrum of the transformed input signal;
Wherein classifying the frequencies of the transformed input signal further comprises grouping the frequencies into a plurality of adjacent blocks, each block comprising one or more adjacent frequencies;
Wherein selecting the one or more frequencies of the transformed input signal comprises estimating a surface under an amplitude spectrum for each block, normalizing the surface of each block to a maximum value, Further comprising removing each block having a surface;
Wherein transitioning the one or more selected frequencies of the transformed input signal comprises: transitioning the frequency content of one or more blocks having the largest surfaces within the frequency range after the transition, Further comprising causing content to be focused on a frequency within the frequency range after the threshold. A system for converting an input to one or more haptic effects using a frequency transition,
A memory configured to store a haptic conversion module; And
And a processor configured to execute the haptic transformation module stored on the memory;
The haptic transformation module is configured to receive an input signal;
Wherein the haptic transformation module is further configured to perform a fast Fourier transform of the input signal;
The haptic transformation module is further configured to transition one or more frequencies of the transformed input signal to one or more frequencies within a frequency range subsequent to the transition;
Wherein the haptic conversion module is further configured to perform an inverse fast Fourier transform of the frequency shifted signal, the inverse transformed signal forming a haptic signal;
Wherein the haptic conversion module is further configured to generate the one or more haptic effects based on the haptic signal. 12. The haptic transducer of claim 11, wherein the haptic conversion module is further configured to classify the frequencies of the transformed input signal;
Wherein the haptic conversion module is further configured to select one or more frequencies from the frequencies of the transformed input signal. 13. The system of claim 12, wherein the haptic conversion module is further configured to cluster the one or more selected frequencies of the transformed input signal. 13. The haptic transducer of claim 12, wherein the haptic conversion module is further configured to classify the frequencies by amplitude;
The haptic transformation module is also configured to select one or more frequencies having the largest amplitudes;
Wherein the haptic transformation module is further configured to move the one or more selected frequencies to a frequency range after the cloth, wherein the frequency range includes a resonant frequency of the haptic output device. 11. A computer-readable medium having stored thereon instructions which, when executed by a processor, cause the processor to implement the method of any one of claims 1 to 10.

Images