KR101496546B1 - Musical instrument training system and method using haptic stimuli - Google Patents

Abstract

An instrument training system utilizing a haptic stimulus, a haptic stimulation feedback device, and a haptic stimulation feedback method are disclosed. The musical instrument education system includes a haptic stimulation feedback device that detects a performance error based on a sound wave generated by a musical instrument and generates and transmits haptic data according to a performance error; a haptic stimulation feedback device that generates a haptic stimulation based on the received haptic data; . The haptic stimulation feedback apparatus includes a sound wave receiving unit for sensing a sound wave generated from a musical instrument, a sound wave computing unit for calculating a performance error between a frequency of a preset reference rate and a frequency of the sensed sound wave, a type of a haptic stimulus Or a haptic stimulus determination unit for determining the strength of the haptic data, and a haptic data transmission unit for transmitting the haptic data.

Description

TECHNICAL FIELD The present invention relates to a musical instrument training system and method using haptic stimulation,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to musical instrument education using haptic stimulation, and more particularly, to a technique for providing immediate haptic stimulation feedback when a musical instrument is not playing a correct pitch.

Haptic feedback, which can be felt when a person touches an object with his or her fingers, includes touch feedback (Tactile Feedback), which is felt when an object touches the surface of the skin, and kinesthetic force feedback, which is felt when movement of joints and muscles is disturbed. .

These haptic feedbacks are the most common and used first in the medical field. Due to the nature of the medical field, especially the surgical surgical (invasive surgery) field, it is difficult for the trainee to master and master the procedure through trial and error, and the haptic feedback, which is a relatively economical means, is utilized .

That is, a three-dimensional image that can directly perform a lesion on a virtual patient while viewing a three-dimensional anatomical structure on a computer screen is displayed on a computer screen in real time. By using a mechanical receptor such as a small pin that moves by compressed air or electricity, the tactile sensation actually touching the skin tissues is transmitted to realize an environment similar to the actual situation.

These haptic stimuli are perceived differently depending on the speed and force of rubbing the object, and the physical properties of the object. Therefore, in order for a person to recognize a different type of haptic stimulus, the physical properties of the object must be changed. As a technique related to this, a plurality of microprojections are formed on the contact surface of an object to be brought into contact with the hand, and the length of the microprojections is changed to change the interval between the microprojections, and the displacement or frequency of the microprojection vibration is adjusted, And the like are provided.

On the other hand, in the field of music education, learning through lecturers or educators is the main learning method, but because it requires considerable high cost, the trainees can not always come up with lecturers or educators. Therefore, in the process of mastering the instrument without their help, it takes a lot of time to learn many learning elements one by one.

Especially, it is very difficult for beginners to judge whether they are doing correctly or not, such as pitch or technique, so the effect of education is very limited. In order to overcome this problem, pitches provide a visual information of the pitch error through products such as a chromatic tuner, which helps the instructor to play the correct note.

That is, a vibration electrical signal is received from the tuned musical instrument, and a digital signal is derived and converted based on the frequency of the received signal. The digital signal is compared with a digital signal representing the frequency of the tuning reference rate, Through the display device.

However, the provision of such visual information is limited only to the degree to which the pianist tunes the musical instrument or corrects his / her erroneous point after the performance, or may not be able to see the musical score during the performance, have. This may mean that the trainee needs to be given an additional amount of time and effort by the specialist.

As a solution to this problem, a system has been developed that provides feedback to the trainee using a different sensory channel without using visual feedback. Or, by limiting the types of instruments, a method to reduce the relative factors to be considered has been devised. In fact, systems such as MusiQ of Adventus of Canada have already been commercialized in the case of keyboard musical instruments such as piano. A feedback system has been developed for avoiding posture disturbance of a stringed instrument when using haptic feedback instead of visual feedback.

It is necessary to study the feedback for the correction of the pitch that occupies a large part of the musical instrument performance training, especially the way to prevent the attention by the visual feedback.

It is an object of the present invention to provide a musical instrument education system which can give immediate and diverse feedbacks to an instructor based on sound waves generated from musical instruments.

It is another object of the present invention to provide a haptic stimulation feedback device capable of providing haptic stimulation of various types and strengths in accordance with the degree of error of a sound wave generated from a musical instrument.

It is still another object of the present invention to provide a haptic stimulation feedback method performed in a musical instrument education system.

According to one aspect of the present invention, there is provided a musical instrument education system including a haptic stimulation feedback device for detecting a performance error based on a sound wave generated by a musical instrument and generating and transmitting haptic data according to a performance error, And a haptic stimulus generator for generating a haptic stimulus based on the haptic data received from the feedback device.

Here, the musical instrument may include a string or wind instrument.

The haptic stimulation feedback device includes a sound wave receiving unit for sensing a sound wave generated from a musical instrument, a sound wave computing unit for analyzing a frequency of the sound wave sensed by the sound wave receiving unit and calculating a performance error between a preset reference frequency and a tuning reference rate, A haptic stimulus determination unit for determining the type or intensity of the haptic stimulus based on the haptic stimulus, and a haptic data transmission unit for transmitting the haptic data according to the type or intensity of the determined haptic stimulus.

Further, the sound wave computing unit may be applied with a filter based on the frequency spectrum and the noise level of the musical instrument.

Further, the sound wave computing unit may find a pitch of the sensed sound wave using Fast Fourier Transform (Fast Fourier Transform) and calculate a performance error based on the preset score data.

Furthermore, the haptic stimulation determination unit can determine the type or intensity of the haptic stimulus depending on the degree of performance error, the number of times, and the type of the musical instrument.

Here, the haptic stimulus generator may generate a haptic stimulus composed of at least one physical force among vibration, twist, and force.

According to another aspect of the present invention, there is provided a haptic stimulation feedback apparatus including: a sound wave receiving unit for sensing a sound wave generated from a musical instrument; and a haptic stimulation feedback unit for analyzing a frequency of a sound wave sensed by the sound wave receiving unit, A haptic stimulus determination unit for determining a type or strength of the haptic stimulus according to the performance error, and a haptic data transmission unit for transmitting the haptic data according to the type or intensity of the determined haptic stimulus .

Here, the sound wave computing unit may be applied with a filter based on the frequency spectrum and the noise level of the musical instrument.

Here, the sound wave computing unit may find a pitch of a sound wave detected by using a Fast Fourier Transform (FFT), and may calculate a performance error based on preset score data.

Here, the haptic stimulus determination unit may determine the type or strength of the haptic stimulus depending on the degree of performance error, the number of times, and the type of the musical instrument.

According to another aspect of the present invention, there is provided a haptic stimulation feedback method comprising the steps of sensing a sound wave generated by a musical instrument, comparing a sensed sound wave with a predetermined reference frequency, Determining a type or strength of the haptic data and generating haptic data according to the generated haptic data; and transmitting the haptic data to the device generating the haptic stimulus to generate the haptic stimulus.

Here, the step of comparing the sensed sound wave with the frequency of the predetermined reference ratio to determine the type or intensity of the haptic stimulus and generating the haptic data according to the frequency spectrum and the noise level of the musical instrument, .

Further, it is possible to find the pitch of the sound wave detected by using the Fast Fourier Transform (Fast Fourier Transform), and to calculate the performance error based on the preset score data.

Furthermore, the type or intensity of the haptic stimulus can be determined according to the degree and frequency of the performance error, and the type of musical instrument.

According to the musical instrument education system of the present invention as described above, it is possible to apply to various musical instruments such as strings and wind musical instruments by overcoming the disadvantages of existing musical instrument education systems which are limited to keyboard musical instruments and provide only visual feedback, And an appropriate means for preventing posture disturbance can be provided.

In addition, using the haptic stimulation feedback apparatus according to the present invention as described above, it can be utilized as a useful path for transmitting additional information together with visual feedback.

Further, by using the haptic stimulation feedback method according to the present invention as described above, it is possible to intuitively perceive a mistake by providing instantaneous haptic feedback at the time of erroneous playing, and to estimate the time and cost required for correct performance by correcting It has the advantage of saving.

As a result, the present invention has an advantage in that it provides a higher education efficiency to the trainee in the training of the musical instrument, and can acquire a skill for playing the musical instrument with ease.

1 is a conceptual view for explaining an operation of a musical instrument education system according to an embodiment of the present invention.
FIG. 2 is a diagram for explaining comparison of frequency information of a played note and a tuning reference rate according to an embodiment of the present invention. Referring to FIG.
FIG. 3 is a diagram for explaining comparison between played notes and music score data according to an embodiment of the present invention.
4 is a block diagram illustrating a haptic stimulation feedback apparatus according to an embodiment of the present invention.
5 is a block diagram for explaining a sound wave receiving unit and a sound wave operating unit of a haptic stimulation feedback apparatus according to an embodiment of the present invention.
FIG. 6 is a diagram illustrating an example of a frequency spectrum of an instrument according to an embodiment of the present invention. Referring to FIG.
7 is a flowchart illustrating a haptic stimulation feedback method according to an embodiment of the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. Like reference numerals are used for like elements in describing each drawing.

The terms first, second, A, B, etc. may be used to describe various elements, but the elements should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Do not.

First, the terms used in the present application will be briefly described as follows.

A filter refers to an acoustic filter. May be for obtaining a sound wave of a specific wavelength region according to a spectrum of an input sound wave or may be for attenuating the intensity of an input sound wave.

Since it can cut unnecessary part according to the above-mentioned frequency, it is possible to use a low-cut filter (low cut) for cutting out a low frequency band, a high cut filter for cutting out a high frequency band, a band pass filter Pass filter). It is also possible to use a band reject filter that selects and removes only a specific frequency, or a combination thereof.

Haptic technology is a technology that allows a user to manipulate input devices of various game machines such as a joystick, a mouse, a keyboard, a touch screen, or the like and manipulate input devices of a computer, so that the user can feel vibration, movement feeling, Haptic stimulation refers to the vibration, movement feeling, and force transmitted to the user using this haptic technique.

Pitch refers to the pitch of the note. Physically, it refers to the difference in frequency of sound, and it is felt low when the frequency is high and when the frequency is low.

In 1834, the Stuttgart conference decided to tune the frequency of the warmth of A4 (440 Hz) at 440 Hz, and adopted this pitch at the London meeting in 1939. The pitch displayed by the note name is the relative pitch based on a certain note, which is calculated based on the warm sound (A4) sound of 440 Hz. That is, the sound pitch of the warmth (A4) is the pitch of the sound wave having the frequency of 440 Hz. In the Western standard scale, the frequency of this warmth (A4) sound is multiplied by 1/12 (about 1.059) The tuning reference rate is determined.

Frequency Spectrum is a method of describing wave components such as light waves, radio waves, and sound waves. Sound waves resulting from the combination of different components of frequency are called compound sounds. For example, the sound of a musical instrument that sounds regular is superimposed on components of different frequencies or strengths.

This is expressed as a function of frequency or a result of Fourier transform (Fourier transform), and is represented by the height (vertical axis) of the line representing the amplitude of the harmonic sound on the horizontal axis representing the frequency.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram for explaining the operation of the musical instrument education system 10 according to an embodiment of the present invention. The components of the musical instrument education system 10 will be described with reference to FIG.

The musical instrument education system 10 includes a haptic stimulation feedback apparatus 200 for detecting a performance error based on a sound wave generated by the musical instrument 100 and generating and transmitting haptic data according to a performance error, And a haptic stimulus generator 300 for generating a haptic stimulus based on the haptic data received from the haptic stimulus generator 300.

The musical instrument 100 may include a string or wind instrument.

The musical instrument 100 may include a musical instrument generating a sound wave, and may include a violin, a string instrument such as a cello, a trumpet, a wind instrument such as a flute, and the like.

The haptic stimulation feedback apparatus 200 includes a sound wave receiving unit 210 that senses a sound wave generated from the musical instrument 100 and a sound source unit 210 that analyzes the frequency of the sound wave sensed by the sound wave receiving unit 210, A haptic stimulus determination unit 230 for determining the type or strength of the haptic stimulus on the basis of the performance error, a haptic stimulus determination unit 230 for determining the type or intensity of the haptic stimulus based on the performance error, And a transmitter 240.

The sound wave computing unit 220 can be applied to the filter 221 based on the frequency spectrum and the noise level of the musical instrument 100. Since the musical instrument 100 generates sound waves having various frequencies, it is possible to selectively filter only a desired frequency band through an acoustic filter.

Also, the sound wave computing unit 220 can find the pitch of the sound wave detected by using the Fast Fourier Transform (Fast Fourier Transform), and can calculate the performance error based on the preset score data or the tuning reference rate.

The sound wave detected by the sound wave receiving unit 210 can be detected using the fast Fourier transform to obtain the accurate sound of the played sound. Through the fast Fourier transform, the played musical instrument sounds can be analyzed at the ratio of the fundamental frequency f and its frequency components.

In addition, the sound wave computing unit 220 may use frequency information of a tuning reference rate stored in advance in order to compare frequencies of the sensed sound waves with a predetermined reference frequency. The frequency information of the tuning reference rate can be extracted based on the score data or the tuning reference rate stored in advance using the storage device. The score data stored by the user's input has information on the note name of the music being played, and the frequency of the reference pitch rate, which is the correct pitch of each note name, can be extracted from the score data.

The haptic stimulus determination unit 230 may determine the type or intensity of the haptic stimulus depending on the degree and frequency of the performance error, and the type of the musical instrument 100.

The degree of the performance error can be classified into a completely incorrect, a little wrong, a normal performance, etc., and it is also possible to provide accumulated haptic stimulation according to the occurrence frequency of the performance error. Also, the type or intensity of the haptic stimulus may be determined in consideration of the characteristics of the musical instrument 100.

The haptic stimulus generator 300 may generate a haptic stimulus based on the haptic data received from the haptic stimulus feedback apparatus 200, and the haptic stimulus may be composed of at least one physical force among vibration, twist, and force .

In receiving the haptic data from the haptic stimulation feedback device 200 and reproducing the haptic stimulus, the transmission and reception of the haptic data may be implemented using a general network. For example, wireless communication such as Wi-Fi, Bluetooth, and Zigbee can be used.

A force feedback device, a twist device, or a vibration feedback device is required for providing a haptic stimulus. Between the haptic stimulation transmission device and the haptic stimulation feedback device 200, a wireless communication or audio output terminal A wired connection is required.

FIG. 2 is a view for explaining a comparison of frequency information of a played note and a tuning reference rate according to an embodiment of the present invention. FIG. 3 is a graph illustrating comparison between played notes and score data according to an embodiment of the present invention. And FIG. A method of comparing the frequency of the played note and the tuning reference rate will be described with reference to FIG. 2 to FIG.

The haptic stimulation feedback device 200 can compute the performance error by comparing the frequency of the tune reference rate with the sound wave generated from the musical instrument 100. The haptic stimulation feedback device 200 may have the correct pitch of the note name by the standard tuning method using a storage device in advance. For example, the correct pitch of the warmth (A4) sound is 440Hz, and the pitch of the other note names can be determined based on this, which may be stored in advance.

Referring to FIG. 3, the frequency of the coarse reference rate can be extracted based on the score data previously stored in the storage device of the haptic stimulation feedback device 200. The score data has information on the note name of the music to be played, and the frequency of the reference pitch rate, which is the correct pitch of each note name, can be extracted from the score data.

FIG. 4 is a block diagram illustrating a haptic stimulation feedback apparatus 200 according to an embodiment of the present invention. FIG. 5 is a block diagram of a sound wave receiving unit 210 of a haptic stimulation feedback apparatus 200 according to an embodiment of the present invention. And a sound wave computing unit 220 according to an embodiment of the present invention. 6 is a diagram illustrating an example of a frequency spectrum of an instrument according to an embodiment of the present invention. The haptic stimulation feedback device 200 will now be described with reference to FIGS. 4 through 6. FIG.

The haptic stimulation feedback apparatus 200 includes a sound wave receiving unit 210 that senses a sound wave generated from the musical instrument 100 and a sound source unit 210 that analyzes the frequency of the sound wave sensed by the sound wave receiving unit 210, A haptic stimulus determination unit 230 for determining the type or intensity of the haptic stimulus according to the performance error, a haptic data transmission unit 230 for transmitting the haptic data according to the type or intensity of the determined haptic stimulus, 240).

5, the sound wave receiver 210 can read a sound played by a user using a tablet PC or a PC microphone (MIC) to sense a sound wave generated from the musical instrument 100. FIG.

The sound wave computing unit 220 can be applied to the filter 221 based on the frequency spectrum and the noise level of the musical instrument 100.

Referring to FIG. 6, the musical instrument has a frequency region band that can emit a distinctive tone and sound for each type of musical instrument. In other words, there is a region where no sound is emitted by any musical instrument more than a certain frequency. However, it is not that the musical instruments occupy only one frequency, and the range of the musical instruments is considerably wide in percussion, wind and strings, but there may be a high frequency band and a low frequency band.

For example, there may be a violin from the low-level contrabass to mid-range cello, viola and high-frequency. Therefore, the frequency spectrum of each musical instrument may be different and varied in its distribution.

5, based on the frequency spectra and the noise level of various musical instruments as described above, it is possible to obtain a sound wave of a specific wavelength region according to a spectrum of an input sound wave using an acoustic filter, It is possible. It is possible to cut high or low frequency band (High Cut, Low Cut) according to need, pass only specific band, or remove Band Pass Filter, Band Reject Filter.

Therefore, the acoustic filter can be used to filter out noise in other sound name regions based on the tone to be measured.

In addition, the sound wave computing unit 220 can generate a haptic stimulus that is sensitive or insensitive to a performance error between the played note and the frequency of the coarse reference rate by applying a weight to a specific frequency band.

For example, when a player plays a warm sound (A4), a weight is applied to a frequency band around the frequency of 440 Hz, which is a reference ratio by the standard tuning method, so that a haptic stimulus .

The sound wave computing unit 220 finds the pitch of a sound wave sensed by the sound wave receiver 210 using a Fast Fourier Transform and generates a pitch error signal based on the frequency of the preset score data or the tuning reference rate, Can be calculated.

For this purpose, the sound wave computing unit 220 may include one or more processors 222. The sound wave computing unit 220 can perform the fast Fourier transform to analyze the sound of the played instrument as a ratio of the fundamental frequency f and its frequency components.

In addition, the sound wave computing unit 220 may use frequency information of a tuning reference rate stored in advance in order to compare the frequency of the sound wave sensed by the sound wave receiver 210 with a predetermined reference frequency. For example, The correct pitch is 440Hz, and the pitch of the other pitch names can be set based on this.

The frequency information of the tuning reference rate may be extracted based on the score data stored in advance in the storage device of the haptic stimulation feedback device 200. [ The score data stored by the user's input has information on the note name of the music being played, and the frequency of the reference pitch rate, which is the correct pitch of each note name, can be extracted from the score data.

The musical score data may be a musical score or an external musical score file already stored in the system by way of user input or the like through MIDI, XML or other data format, You can compare the pitch of a sound.

As a result, the performance error calculated by the sound wave computing unit 220 is the difference between the most ideal frequency on the standard tuning law and the tone of the sound wave generated from the musical instrument 100 found through the fast Fourier transform.

The haptic stimulus determination unit 230 may determine the type or intensity of the haptic stimulus depending on the degree and frequency of the performance error calculated by the sound wave computing unit 220 and the type of the musical instrument 100.

If the performance error occurs over a certain level, haptic stimulation can be generated. If the degree of the performance error is so severe that a sound corresponding to the boundary between the two notes or a completely different note is played, it can be noticed that the haptic stimulus is completely wrong. In contrast, when the note to be played is within the normal region, the haptic stimulus may be regarded as being played normally, and the haptic stimulus may not be generated.

The boundary region and the normal region which are bases for the generation of the haptic stimulus can be set by the user. For example, in the case of warmer (A4) sound, 440 Hz may be used as a reference, and 434 Hz to 446 Hz may be set as a normal sound, that is, a normal sound. Also, 428 Hz to 434 Hz or 446 Hz to 453 Hz may be set as a slightly different region, and 428 Hz or less or 453 Hz or more may be set as a boundary region.

Haptic stimuli of different kinds and strength can be generated depending on whether the notes to be played are in the boundary region, the slightly different region, or the normal region.

It is conceivable to provide different feedbacks, such as completely wrong, somewhat wrong, normal playing depending on whether the notes played in the above manner are in the boundary region, the slightly different region, and the normal region, and the type and intensity of the provided haptic feedback It will be possible to set more than two or three levels.

The intensity and type of the haptic stimulus may vary depending on the purpose of the instrument training and the type of the instrument. For example, in order to help the beginner's correct pitch, the strength of the haptic feedback may be mapped to the pitch error of the pitch, The effect can be expected by generating a strong stimulus even in the case of a slight error in performance.

In the case of extracting the frequency of the tuning reference rate from the score data, it is possible to give a strong stimulus when the note in the score is completely erroneously played, and to differently map the type and strength of the haptic stimulus depending on how many errors are caught . For example, you can choose to apply a vibration of 100 Hz for a wrong note, 60 Hz for a wrong note, and 20 Hz for a wrong note. It is possible.

7 is a flowchart illustrating a haptic stimulation feedback method according to an embodiment of the present invention. The haptic stimulation feedback method will be described with reference to FIG.

The haptic stimulation feedback method includes a step S710 of detecting a sound wave generated from the musical instrument 100 using Fast Fourier Transform (S710), a step S710 of detecting a frequency of a tuning reference rate preset from the sensed sound wave, (Step S720) of determining the type or strength of the haptic stimulus by calculating the performance error, calculating haptic stimulus (step S720), transmitting the haptic data to the device generating the haptic stimulus to generate haptic stimulation (step S730) As shown in FIG.

The step S720 of determining the type or intensity of the haptic stimulus by comparing the sensed sound wave with the frequency of the predetermined tuning reference rate and generating the haptic data according to the determined frequency and frequency is based on the frequency spectrum and the noise level of the musical instrument 100, (221) may be applied.

Since the musical instrument 100 generates sound waves having various frequencies, it is possible to selectively filter only a desired frequency band through an acoustic filter. Therefore, the acoustic filter can be used to filter out noise in other sound name regions based on the tone to be measured.

In addition, in the haptic stimulation feedback method, haptic stimulation is generated by sensitively or insensitively responding to the performance error between the played note and the tuning reference rate by adjusting the width of the frequency band set in the normal range narrowly or broadly And applying a weight according to the error to the haptic stimulus can cause stronger or weaker stimulation.

In comparing the frequency of the sensed sound wave with the preset reference tuning reference rate, a pitch of the sensed sound is detected using Fast Fourier Transform, and the played instrument sound has a fundamental frequency f and its frequency component Can be analyzed in the contained ratio. At this time, the performance error can be calculated based on the pitch of the sensed sound wave and the preset score data.

The process of comparing the frequency of the sensed sound wave with the frequency of the predetermined tuning reference rate may use the frequency information of the tuning reference rate stored in advance in the storage device of the haptic stimulation feedback device 200. [

The frequency information of the tuning reference rate may be extracted based on the score data stored in advance in the database of the haptic stimulation feedback apparatus 200. [ The score data stored by the user's input has information on the note name of the music being played, and the frequency of the reference pitch rate, which is the correct pitch of each note name, can be extracted from the score data.

As a result, it is possible to calculate the difference between the most ideal frequency on the extracted standard tuning rule and the exact pitch of the sensed sound wave generated from the musical instrument 100 found through the fast Fourier transform.

The step S720 of determining the type or strength of the haptic stimulus by comparing the sensed sound waves with the frequency of the predetermined tuning reference rate and generating the haptic data according to the determined type may include the degree and frequency of the performance error, The type or strength of the haptic stimulus can be determined.

The intensity and type of the haptic stimulus may vary depending on the purpose of the instrument training and the type of the instrument. For example, in order to help the beginner's correct pitch, the intensity of the haptic feedback can be mapped to the pitch error of the pitch, and in the case of the training of the expert, the effect can be expected by generating a strong stimulus .

In the case of extracting the frequency of the tuning reference rate from the score data, it is possible to give a strong stimulus when the note in the score is completely erroneously played, and to differently map the type and strength of the haptic stimulus depending on how many errors are caught . For example, you can choose to apply a vibration of 100 Hz for a wrong note, 60 Hz for a wrong note, and 20 Hz for a wrong note. It is possible.

The step of generating the haptic stimulus by transmitting the haptic data to the device generating the haptic stimulus may generate the haptic stimulus composed of at least one physical force of vibration, twist, and force.

The frequency of the sensed sound wave is compared with a frequency of a predetermined tuning reference rate to determine the type or intensity of the haptic stimulus and the generated haptic data is transmitted to the device generating the haptic stimulus. Can be implemented. For example, wireless communication such as Wi-Fi, Bluetooth, and Zigbee can be used.

This technique needs to provide real-time analysis of the music played by the instructor in real time, so it can be used as a mobile device or a computer application with built-in microphone function. In order to provide the haptic stimulus, a force feedback device, a twist device, or a vibration feedback device is required, and a wired connection is required between the haptic transmission device and the application, such as a wireless communication or audio output terminal capable of transmitting and receiving data.

10: instrument training system 100: instrument
200: haptic stimulation feedback device 210: sound wave receiver
211: microphone 220: sound wave computing unit
221: filter 222: processor
230: Haptic stimulus determination unit 240: Haptic data transmission unit
300: Haptic stimulation generator

Claims (16)

A haptic stimulation feedback device that detects a performance error based on a sound wave generated by a musical instrument when a user plays a musical instrument and generates and transmits haptic data according to the performance error; And
A haptic stimulus generator for generating a haptic stimulus to the user based on the haptic data received from the haptic stimulation feedback device;
, ≪ / RTI &
The haptic stimulation feedback device includes:
A sound wave receiver for sensing a sound wave of the musical instrument performance generated by the musical instrument;
A sound wave computing unit for analyzing a frequency of a sound wave sensed by the sound wave receiver and calculating a performance error with a predetermined reference rate;
A haptic stimulus determining unit that determines the type or strength of the haptic stimulus according to the performance error of whether the sound wave is in a normal region, a region where difference occurs, or a boundary region;
And a haptic data transmission unit for transmitting haptic data according to the type or strength of the determined haptic stimulus to the haptic stimulation generator. The method according to claim 1,
The musical instrument
A string or a wind instrument. delete The method according to claim 1,
The sound wave computing unit
Characterized in that a filter is applied based on a frequency spectrum and a noise level of the musical instrument The method according to claim 1,
The sound wave computing unit
Wherein a pitch of the sensed sound wave is detected using a Fast Fourier Transform (Fast Fourier Transform), and the performance error is calculated based on preset score data. The method according to claim 1,
The haptic stimulation determination unit may determine,
Wherein the type or strength of the haptic stimulus is determined according to whether the sound wave is in the normal region, the difference region, or the boundary region, in addition to the number of times of the performance error and the type of the musical instrument. The method according to claim 1,
The haptic stimulation generator
Wherein the haptic stimulus comprises at least one physical force selected from the group consisting of a vibration, a twist, and a force. A haptic stimulation feedback device included in an instrument training system,
A sound wave receiver for sensing a sound wave of the musical instrument performance generated by the user when the user plays the musical instrument;
A sound wave operating unit for analyzing a frequency of the sound wave sensed by the sound wave receiving unit and calculating a performance error with a predetermined tuning reference rate;
A haptic stimulus determining unit that determines the type or strength of the haptic stimulus according to the performance error of whether the sound wave is in a normal region, a region where difference occurs, or a boundary region; And
And a haptic data transmitter for transmitting haptic data according to the determined type or intensity of the haptic stimulus to the haptic stimulation generator of the musical instrument education system to provide haptic stimulation feedback to the user. The method of claim 8,
The sound wave computing unit
Wherein a filter can be applied based on a frequency spectrum and a noise level of the musical instrument. The method of claim 8,
The sound wave computing unit
Wherein a pitch of the sensed sound wave is detected using a Fast Fourier Transform (FFT), and the performance error is calculated based on preset score data. The method of claim 8,
The haptic stimulation determination unit may determine,
Wherein the type or strength of the haptic stimulus is determined according to whether the sound wave is in the normal region, the difference region, or the boundary region, in addition to the number of times of the performance error and the type of the musical instrument. Detecting a sound wave for the musical instrument performance generated by the user in playing the musical instrument in the haptic stimulation feedback apparatus of the musical instrument education system;
Wherein the haptic stimulation feedback device calculates a performance error by comparing the sensed sound wave with a preset reference frequency of a coarse reference rate and determines a haptic response based on the performance error of whether the sound wave is in a normal region, Determining a type or strength of the stimulus and generating haptic data according to the determined type; And
And transmitting the haptic data to a haptic stimulation generator of the musical instrument education system for generating a haptic stimulus to the user to generate the haptic stimulus based on the haptic data. The method of claim 12,
Wherein the step of generating the haptic data comprises:
Wherein a filter is applied based on a frequency spectrum and a noise level of the musical instrument. The method of claim 12,
Wherein the step of generating the haptic data comprises:
A pitch of the sensed sound wave is detected using a Fast Fourier Transform (FFT), and the performance error is calculated based on preset score data. The method of claim 12,
Wherein the step of generating the haptic data comprises:
Wherein the type or intensity of the haptic stimulus is determined according to whether the sound wave is in the normal region, the difference region, or the boundary region, in addition to the number of the play errors and the type of the musical instrument. The method of claim 12,
Wherein generating the haptic data based haptic stimulus comprises:
Wherein the haptic stimulation generator generates the haptic stimulus composed of at least one physical force of vibration, twisting, and force.

Images