WO2021141212A1

WO2021141212A1 - System and method for converting sound into tactile effect

Info

Publication number: WO2021141212A1
Application number: PCT/KR2020/013836
Authority: WO
Inventors: 김수영; 추승원; 김정훈
Original assignee: 김수영
Priority date: 2020-01-06
Filing date: 2020-10-12
Publication date: 2021-07-15
Also published as: KR102286822B1; KR20210088264A

Abstract

The present invention relates to a system for converting a sound into a tactile effect, which converts a natural sound into a tactile signal and provides the tactile signal in real time, whereby the natural sound frequency of a wiredly or wirelessly provided image or sound is converted into a vibration signal and provided as a tactile feedback to a user in real time, and thus the user can be provided with a realistic tactile sensation corresponding to the image and sound.

Description

Systems and methods for converting sound into tactile effects

The present invention relates to a system and method for providing real-time conversion of natural sound into tactile signals.

BACKGROUND With the development of IT technology, users who use computer devices, game machines, and various portable devices require more realistic interaction with these devices. Accordingly, interest in a method of providing a hardware environment capable of more realistically transmitting tactile information in addition to the existing visual and auditory information transmission is increasing.

Recently, in addition to the information delivery method that relied on only video and sound, tactile effects (skin sensation) are additionally provided so that users can feel a more realistic sense of reality.

Korean Patent Application Laid-Open No. 10-2009-0112767, 'Haptic Feedback System with Stored Effects,' creates a virtual vibration pattern as a pulse wave in advance and stores it in the memory. It is a method to transmit the vibration effect by calling the appropriate vibration pattern.

Since this vibration effect simply provides a pre-made vibration pattern, it is not possible to provide a tactile sense that matches the screen and sound in real time. It is practically difficult to produce a vibration pattern corresponding to the image.

Also, since the tactile sense transmitted to the user through the vibration pattern is a pre-made vibration pattern, it is impossible to receive realistic tactile sense corresponding to real-time images and sounds.

Accordingly, it is an object of the present invention to provide a system and method for providing sound provided by a device to a user as real-time tactile feedback.

The present invention according to the above object is a system for converting a sound into a tactile effect, and a driving device 100 that converts an input sound into a tactile signal and provides it to the user is provided, and the driving device 100 is generally The MCU 10 for controlling the operation of the driving device 100, the power input unit 12 to supply power through the external input power and the internal power supply, and the power supply by the external input power of the power input unit 12 If this is not the case, the internal battery 14 for supplying power to the driving device 100, and a wired/wireless communication module 18 for inputting and outputting control signals and user input data from the host to the driving device 100 through wired/wireless communication And, a sound signal input unit 20 for receiving natural sound for converting into a vibration signal, a sound processor 22 for amplifying the input natural sound analog signal and converting it into a signal most suitable for vibration, and the sound processor ( 22) and a driving circuit 24 for controlling the converted vibration signal, and an electroactive transducer 26 driven through the driving circuit 24 to generate a tactile effect.

In addition, the present invention provides a method for converting a sound into a tactile effect, comprising the steps of: receiving an analog signal, which is natural sound, to a driving device; amplifying the inputted analog signal; An electroactive transducer adjusted to operate in a resonance band amplified for each frequency, comprising the steps of adjusting to be maintained, removing noise from the sound signal, and amplifying the sound signal from which the noise has been removed by amplifying each frequency It is characterized in that it consists of a step of generating a real-time tactile signal by transmitting a frequency to the

In addition, the present invention provides a method for converting a sound into a tactile effect, comprising the steps of: receiving an analog signal, which is natural sound, to a driving device; amplifying the inputted analog signal; Adjusting to be maintained, removing noise from the acoustic signal, amplifying the noise-removed acoustic signal by amplifying for each frequency band, and a plurality of adjusted to operate in the amplified resonance band for each frequency band It is characterized in that it consists of a step of generating a real-time tactile signal by transmitting a frequency to an electroactive transducer of

The present invention has the effect of providing a realistic tactile sense suitable for images and sounds by converting the provided natural sound of an image or sound into a vibration signal in real time and providing it to the user as a tactile feedback.

1 is a block diagram showing the overall configuration of a system for converting a sound into a tactile effect according to an embodiment of the present invention;

2 to 3 are detailed configuration diagrams of a system for converting a sound into a tactile effect according to an embodiment of the present invention;

4 is a view for explaining a method of amplifying for each frequency according to an embodiment of the present invention;

5 to 6 are flowcharts illustrating a method of converting a sound into a tactile effect according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram showing the overall configuration of a system for converting a sound into a tactile effect according to an embodiment of the present invention.

The system for converting a sound into a tactile effect of the present invention includes a driving device 100 that converts an input sound into a tactile signal and provides it to the user, and the driving device 100 generally operates the driving device 100 . MCU 10 for controlling the, power input unit 12 for supplying power through external input power and internal power, and internal power when power is not supplied by external input power of the power input unit 12 The battery 14 supplied to the driving device 100 and the power input of the driving device 100 are controlled, and data are transmitted and received through wired/wireless communication between the driving device 100 in which the user input device is installed and the host. The input control unit 16 to control, the wired/wireless communication module 18 for inputting and outputting the control signal and user input data from the host to the driving device 100 through wired/wireless communication, and a natural sound for converting into a vibration signal is input A receiving sound signal input unit 20, a sound processor 22 that amplifies the input natural sound analog signal and converts it into a signal most suitable for vibration, and a driving circuit that controls the vibration signal converted through the sound processor 22 a furnace 24, an electroactive transducer 26 driven through the driving circuit 24 to generate a tactile effect, a switch unit 28 receiving a user's input signal to the driving device 100, and driving; It is installed in the device 100 and consists of a sensor 30 that detects the state of the driving device 100, and a camera 31 that captures an image and collects information included in the captured image together with a user identification function.

The MCU 10 controls the power of the driving device 100 while controlling the operation of the sound processor 22 that converts the input natural sound into a vibration signal, and the electroactive converter 26 that generates a tactile effect through vibration. Control.

The power input unit 12 is for supplying power to the driving device 100 , and supplies power to the driving device 100 through an external input power source and an internal battery.

When input power is supplied from the outside, the power supply through the internal battery is cut off. On the contrary, when external power is not input, power is controlled to be input to the driving device 100 by the internal battery.

The internal battery 14 may be a disposable battery, and as another embodiment, when there is no external power input as a rechargeable power source, the charged power is input to the driving device 100 and the rechargeable power source is charged. As a method, it may be in the form of wired charging using USB, micro USB, etc. and wireless charging using a coil.

In addition, it may be filled by contact in the form of a plate spring using metal-to-metal contact.

The input control unit 16 controls the input power by the power input unit 12 and the battery 14 and transmits user input information by the switch unit 28 including a small LCD, a touch pad, and a camera to the host computer. Control to exchange data through wired/wireless communication.

The driving device 100 of the present invention may include various handheld devices including smart phones, game device controllers, AR/VR devices, etc., and these portable devices are connected to a host computer by wire or wireless to provide data will send and receive

In this case, the user input information may be transmitted to the host computer or signal data including the sound of the host computer may be transmitted to the driving device 100 .

The wired/wireless communication module 18 manages data transmission/reception using the driving device 100, and receives acoustic data input from the host computer in connection with the input control unit 16 or user input information through wired/wireless communication with the host computer. B) It transmits and receives status information of the driving device 100 and manages to control the status of the driving device 100 through control information received from the host computer.

The sound signal input unit 20 receives an analog sound converted into a vibration signal, and receives an analog signal to be converted into a vibration signal through the sound processor 22 .

The driving device 100 of the present invention provides a tactile signal by connecting the frequency of the input natural sound as it is to vibration, so that real-time tactile information can be transmitted.

At this time, when the input sound signal is a digital signal, it is converted into an analog signal using a DAC (Digital to Analog Converter) to generate a vibration signal.

The sound processor 22 converts the natural sound input through the sound signal input unit 20 into a vibration signal for providing tactile feedback, and converts the input analog signal into a predetermined ratio by the amplification unit 21 . After amplification, the frequency signal is adjusted to maintain a constant desired gain through automatic gain control, and noise is removed from the amplified sound signal by a differential amplifier 34 (Differential Apmlifier).

The sound signal amplified by the amplification unit 21 is adjusted to keep the frequency signal constant by the automatic gain control unit 32, and the automatic gain control unit 32 amplifies the signal having a large amplitude and becomes saturated when the signal is It is controlled to minimize noise caused by distortion. Among analog sound signals, a sound signal with a small amplitude is amplified and a sound signal with a large amplitude is amplified only enough to not be saturated, so that the sound frequency at a constant level is maintained.

The differential amplifier 34 is an amplifier capable of obtaining an output proportional to the potential difference between the two input signals, and since both signal lines are affected by the surrounding noise, there is no significant change in the characteristics of operation due to the difference between the two signal lines. This takes advantage of the fact that ambient noise can be minimized or attenuated by operating as if there is no actual noise effect.

The sound signal converted by the sound processor 22 is amplified once again by the frequency-specific amplifier 23 as shown in FIG. 2 .

At this time, the method of amplifying the sound signal uses a compression method of compressing and transforming the amplified signal.

The compression method used in the present invention is an automatic signal processing method, and when the input or output of each frequency is too strong or weakly input or output within the operating frequency range, it is strengthened or weakened at a certain compression ratio.

Referring to FIG. 4 , the amplification is strengthened as it goes down from 200 Hz, which is the midpoint of the 20 to 450 Hz, which is the midpoint of the 20 to 450 Hz, the sensible frequency band that human skin is most sensitive to, and the 200 to 300 Hz band increases gradually as it goes up. Up to 300~450Hz, as the vibration speed increases, the number of vibrations increases, but the sensitivity of the skin to vibration decreases. Therefore, to increase or maintain the skin's sensitivity to vibration, amplification should be performed within the amplification rate range of the 200~300Hz band. will be regulated

This is a case in which a vibration signal of a frequency band of 20 to 450 Hz is transmitted by one electroactive converter 26, and when a plurality of electroactive converters 26 are installed in the driving device 100, the following method is used. Control.

3 illustrates a configuration in which a plurality of electroactive transducers 26 are installed and bands are set differently according to acoustic characteristics through the amplification unit 29 for each frequency band to provide a tactile signal.

In order to realize the vibration wavelength closest to sound, it is good to decompose the frequency band of sound into multiple zones and convert several bands into vibration wavelengths and output them, because all sounds are not a single frequency, but multiple frequency bands. to be.

As an embodiment of the present invention, there is provided a signal amplification method for each band that reduces the error in target gain for each frequency by dividing the frequency of the sound signal into a low-pitched range, a mid-tone range, and a high-pitched range.

The sensible first band low frequency frequency set in the present invention is 20 to 450 Hz, the tangible second band mid-range frequency is 800 Hz to 1.3 kHz, and the sensible third band high frequency frequency may be 2.8 to 3.3 kHz, In the case of installing more electroactive transducers 26 in the driving device 100 as an embodiment of the present invention, the frequency for each sound band set above can be set in a more subdivided manner.

Among the audible frequencies of 20 to 20,000 Hz, the band where the sensation is most transmitted to the skin is 20 to 450 Hz, which is the first band of sensation, and the resonance frequency bandwidth provided through the electroactive transducer 26 is set to 20. It is manufactured at ~450 Hz.

Next, the configuration for setting the resonant frequency bandwidth of the electroactive converter 26 will be described below.

The sensible second band, 800 Hz ~ 1.3 kHz frequency band, is a frequency band that occurs the most along with the tangible first band among general sounds that are not special sounds, and an electroactive transducer 26 resonating according to this band is manufactured.

The sensory third band, 2.8 ~ 3.3 kHz frequency band, is a frequency band that occurs frequently after the first and second bands, and the electroactive transducer 26 resonating according to this band is manufactured.

The configuration for manufacturing the electroactive converter 26 resonating in the frequency band of the first, second, and third bands of the tangible will be described as follows.

As an embodiment, the resonance band is adjusted by using a moving mass installed in the electroactive transducer 26, and if the weight of the installed moving mass is increased, the resonance band is lowered, and if the weight is reduced, the resonance band is increased. Adjust.

In addition, by increasing the spring constant (spring constant), it is also possible to use the movement of the resonance band to a higher frequency band.

As another embodiment, when the material of the electroactive transducer 26 is used as a piezoelectric element along with the increase and decrease of the moving mass and the adjustment of the spring constant, the piezoelectric constant and the elastic constant by selectively using the material And by adjusting the frequency constant (Frequency constant), etc. to manufacture the electroactive converter 26 suitable for each band.

When a plurality of electroactive transducers 26 set to operate in different resonant frequency bands are installed in the driving device, a much richer sense of vibration than simply installing a plurality of the same electroactive transducers 26 resonating only at a frequency of one band. This can be done to increase realism.

For example, if electroactive transducers 26 operating in different resonant frequency bands are respectively installed on both sides of the driving device, the left and right vibration sensations are generated differently, so that more realistic vibrations corresponding to the screen and image are provided to the user it can be done

Referring back to Figure 1, the noise is removed by the sound processor 22 and the amplified sound signal is provided to the electroactive converter 26 through the driving circuit 24, and the electroactive converter 26 is used by a person. It generates a vibration signal to be transmitted to the tactile sense that can be felt.

As an embodiment of the present invention, the electroactive transducer 26 may be an actuator, and in addition to this, a linear resonant actuator (LRA), a stepper motor, a solenoid motor, etc. may be used.

In the present invention, since the input natural sound is transmitted to the user as a tactile signal as it is, an actuator having a wide resonance band and a fast response time should be used.

The response time for the human tactile nerve to feel the sense of simultaneity should be within 10 ms, so it is better to selectively use an actuator suitable for it.

As a preferred embodiment of the present invention, the electroactive transducer 26 may be an EAP actuator using an electroactive polymer (EAP) or a piezoelectric actuator using a piezoelectric element.

As described above, the electroactive transducer 26 must deliver all of the senses in the range of 20 to 450 Hz, which is the frequency at which the human body feels the most tactile sense, and the sound processor 22 processes the natural sound to generate the driving circuit 24 . If the electroactive transducer 26 fails to generate a tactile signal suitable for the received frequency even if it is transmitted through it, a realistic tactile sense cannot be transmitted to the user.

To this end, in the present invention, a piezo actuator having a response speed of 5 ms or less and a wide resonant frequency band is installed in the driving device 100 to generate an immediate and realistic tactile signal.

One or more electroactive transducers 26 may be installed in the driving device 100 , and it is also possible to change the installation location according to the user's preference.

The switch unit 28 is installed in the driving device 100 to input user input information, and the input information can be transmitted to a host computer connected to the driving device 100, and through this, A signal to control operation may be input.

The sensor 30 may be installed in the driving device 100, for example, an acceleration sensor, a gyro sensor, a proximity sensor, an illuminance sensor, a hall sensor, a motion sensor, a temperature and humidity sensor, a heart rate sensor, a fingerprint recognition sensor, an ultrasonic sensor, etc. The installed sensor allows detection information of the installed sensor to be input to the driving device 100 as needed, and the input detection information may also be transmitted to a host computer.

The camera 31 may take an image and use it as a user identification means, and may function to collect information included in the image taken with it.

For example, by scanning by the camera 31, such as a QR code, barcode, etc., it is possible to shoot an image in which data is provided, and the provided data can be shown to the user through the host computer.

In addition, it is possible to analyze the captured image to analyze the information of the object included in the image, and to synthesize the AR image with the captured image and provide it through the host computer.

Hereinafter, a method of converting an input natural sound into a tactile effect according to the present invention will be described with reference to FIGS. 5 to 6 .

The driving device of the present invention receives an analog signal that is natural sound through wired/wireless communication, and in this case, when the input sound signal is a digital signal, it is converted back into an analog signal by the DAC.

The input analog signal is amplified at a constant rate by the amplifier, and then the signal is adjusted to maintain a constant desired gain through automatic gain control.

Automatic gain control amplifies a signal with a large amplitude and adjusts it to minimize noise due to distortion when it is saturated. Among analog sound signals, a sound signal with a small amplitude is amplified and a sound signal with a large amplitude is not saturated. By amplifying only the degree, a constant level of sound frequency is maintained.

The sound signal whose gain is adjusted by automatic gain control passes through the differential amplifier 34 to remove noise.

The noise-removed sound signal is amplified once again for each frequency, and when the input or output of each frequency is too high or low within the operating frequency range, it is strengthened or weakened according to a certain compression ratio, so that the human skin feels the most sensitive. It is adjusted by the perceived frequency band.

As an embodiment, the vibration slows down toward the bottom of 200 Hz, which is the midpoint of the 20-450 Hz band, which people feel most sensitive to, so that the skin feels the vibration weakly, so the amplification is strengthened, and the 200-300 Hz band gradually increases the amplification. Up to 300~450Hz, as the vibration speed increases, the number of vibrations increases, but the sensitivity of the skin to vibration decreases. Therefore, to increase or maintain the skin's sensitivity to vibration, amplification should be performed within the amplification rate range of the 200~300Hz band. use the control method.

The sound signal amplified for each frequency is transmitted to the electroactive transducer 26 to provide a tactile signal suitable for the amplified frequency to the user, and at this time, the electroactive transducer 26 can operate according to the amplified resonance band for each frequency. is in an adjusted state.

An actuator can be used as an example of the electroactive transducer 26, and the weight and spring constant of the moving body installed in the actuator can be adjusted by adjusting the vibration to match the resonant band for each amplified frequency, and together with the material of the actuator When using as a piezoelectric element, the resonance band is adjusted to operate according to the transmitted sound signal by mixing and adjusting the piezoelectric constant, elastic constant, and frequency constant of the material.

Although specific embodiments have been described in the above description of the present invention, various modifications may be made without departing from the scope of the present invention. Therefore, the scope of the present invention should not be defined by the described embodiments, but should be defined by the claims and equivalents to the claims.

The driving device 100 of the present invention is a handheld device, and includes a smartphone, a tablet PC, a console game controller, a phone mounted gamepad, a mouse, a phone case tactile generating device, a remote control, a PC input device, a joystick, a VR/AR device, It can be installed in game gear, laptop computer, media player and its auxiliary devices equipped with various switch buttons, desktop computer, touch-sensitive TV, embedded touch-sensitive display panel (ATM, kiosk, touch-type panel), human robot It can be installed and used where a tactile signal is needed, such as various robots including

In addition, it can be installed in auxiliary devices installed in transportation devices including vehicles, ships, airplanes, and trains to transmit tactile signals, and to medical devices that calm the brain with fine vibrations using music to relieve pain, etc. can be installed.

Claims

A system for converting sound into a tactile effect, the system comprising:

A driving device 100 that converts the input sound into a tactile signal and provides it to the user is provided;

The driving device 100 includes an MCU 10 for controlling the operation of the driving device 100 as a whole,

a power input unit 12 for supplying power through an external input power source and an internal power source;

an internal battery 14 for supplying internal power to the driving device 100 when power is not supplied by the external input power of the power input unit 12;

A wired/wireless communication module 18 for inputting and outputting control signals and user input data from the host to and from the driving device 100 through wired/wireless communication;

An acoustic signal input unit 20 for receiving natural sound for converting into a vibration signal;

A sound processor 22 that amplifies the input natural sound analog signal and converts it into a signal most suitable for vibration;

a driving circuit 24 for controlling the vibration signal converted through the sound processor 22;

A system for converting sound into a tactile effect, characterized in that it comprises an electroactive transducer (26) driven through the driving circuit (24) to generate a tactile effect.
According to claim 1,

Converting sound into a tactile effect, characterized in that an input control unit 16 that controls the power input of the driving device 100 and transmits and receives data through wired/wireless communication between the driving device 100 and the host is configured system that does.
According to claim 1,

A system for converting sound into a tactile effect, characterized in that the switch unit (28) for inputting a user's input signal to the driving device (100) is configured.
According to claim 1,

A system for converting sound into a tactile effect, characterized in that a sensor (30) installed in the driving device (100) to detect the state of the driving device (100) is configured.
According to claim 1,

A system for converting sound into a tactile effect, characterized in that a camera (31) for capturing an image and collecting information included in the captured image together with a user identification function is configured.
According to claim 1,

A system for converting sound into a tactile effect, characterized in that the amplifier 21 is configured to amplify the input analog signal at a predetermined ratio.
According to claim 1,

The sound processor 22 is configured to amplify a sound signal with a small amplitude among analog sound signals and amplify a sound signal with a large amplitude to the extent that it is not saturated, so that the gain of the sound frequency is maintained. A system that transforms the characteristic sound into a tactile effect.
According to claim 1,

The system for converting sound into a tactile effect, characterized in that the sound processor (22) includes a differential amplifier (34) that removes noise by converting the amplified sound signal into a differential signal.
According to claim 1,

A system for converting sound into a tactile effect, characterized in that the frequency-by-frequency amplifier (23) for amplifying the sound signal converted through the sound processor (22) for each frequency is configured.
According to claim 1,

A plurality of electroactive transducers 26 are installed to provide tactile signals by setting different bands according to acoustic characteristics through the amplification unit 29 for each frequency band,

The perceived first band, the low frequency frequency, is 20~450Hz, the perceived second band, the midrange frequency, is 800Hz~1.3kHz, and the third band, the high frequency frequency, is 2.8~3.3kHz. A system that converts the sound you hear into a tactile effect.
A method for converting sound into a tactile effect, the method comprising:

receiving an analog signal, which is a natural sound, to a driving device;

amplifying the input analog signal;

adjusting the amplified sound signal to maintain a constant desired gain;

removing noise from the sound signal;

Amplifying the sound signal from which the noise has been removed by amplifying each frequency;

A method for converting sound into a tactile effect, comprising the step of generating a real-time tactile signal by transmitting a frequency to an electroactive transducer (26) adjusted to operate in a resonance band amplified for each frequency.
A method for converting sound into a tactile effect, the method comprising:

receiving an analog signal, which is a natural sound, to a driving device;

amplifying the input analog signal;

adjusting the amplified sound signal to maintain a constant desired gain;

removing noise from the sound signal;

Amplifying the noise-removed sound signal by amplifying each frequency band;

A method of converting sound into a tactile effect, comprising the step of generating a real-time tactile signal by transmitting a frequency to a plurality of electroactive transducers (26) adjusted to operate in the resonance band amplified for each frequency band.
12. The method of claim 11,

In the step of amplifying by frequency,

It is a step in which the amplification is strengthened as it goes down from 200Hz, which is the midpoint of the 20-450Hz, the midpoint of the frequency band that human skin feels most sensitive to, and the amplification is gradually increased as it goes up in the 200-300Hz band. A method of converting sound into a tactile effect, characterized in that
13. The method of claim 12,

In the step of amplifying each frequency band,

The first band, which is the low frequency band, is 20 to 450 Hz, the perceived second band, the middle frequency, is 800 Hz to 1.3 kHz, and the third band, the high frequency, is 2.8 to 3.3 kHz. A method of converting sound into a tactile effect, characterized in that