KR100312750B1 - Virtual musical performance apparatus and method thereof using sensor - Google Patents

Abstract

The present invention relates to a virtual performance device and a method using a sensor for sensing the user's motion to generate a corresponding MIDI sound.

The virtual performance device using the sensor according to the present invention, the selection unit for selecting a song to play from the stored song; A tap sensor for detecting the strength and speed of the tapping by the user; A music player which generates accompaniment according to the selected song's melody and controls and outputs tempo and sound intensity according to a control signal; A graphic display unit which visualizes and displays current music information of the played song as a graphic; And a central control unit for inputting the selection signal to read data of the corresponding song in the memory and outputting the data to the music player, and introducing the sensor signal to control the music player and the graphic display unit.

According to the present invention, accompaniment is automatically output according to the melody by tapping the tab according to the melody of the played song according to the displayed graphic screen, so that the user can easily play the song, and the speed and volume of the played song, Height can be played with more interest by visualizing each object as a three-dimensional visualization.

Description

Virtual musical performance apparatus and method using sensor

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a virtual playing device, and more particularly, to a virtual playing device using a sensor for sensing a user's motion and generating a corresponding MIDI sound, and a method thereof.

In general, in order to play music, there is a method of playing using a real physical instrument directly, and there is a method of transforming and playing music originally recorded using a mixer. There are also a variety of ways to play using MIDI or musical instrument digital interface instruments.

In the case of using a real instrument, the music is played by the interaction between the instrument and the player when a certain force is applied to the instrument with the correct movement according to the score that the player wants to play. In the case of using a mixer or the like, certain parts of the recorded music or sound are changed in frequency or amplitude to create new music sound with modulated sound. Here, the electronic or mid-instrument has an interface similar to that of a conventional musical instrument, for example, a piano or a violin-like interface, while its operation is performed electronically.

In order to play music as desired through such an electronic piano or similar electronic musical instrument, the actual musical performance capable of playing the piano or the same musical instrument as the basic configuration well or conducting with an orchestra And skill is required. Therefore, it is very difficult to play musical instruments in general except for those who have learned to play musical instruments professionally or who have learned with special effort.

In addition, there is a method of playing a MIDI or electronic music related software from a computer or synthesizer and selecting from among pre-stored songs and editing and playing it as desired even if there is no direct performance. However, current MIDI-related software does not provide the ability to easily and easily modify the song in real time while playing. In addition, the current MIDI-related software has a lack of display support except for the acoustic aspect of music. Existing entertainment devices that follow the music in real time include beat mann and dance dance revolution, but these entertainment devices are designed to match the music. Doing so requires a relatively passive user component.

Accordingly, as an active subject who transforms music, the speed of each song, the height and melody of each note, the intensity of each note, and the progress of each note are changed and played according to the user's motion, so that the user cannot handle the instrument skillfully. The ability to match emotions with a few beats required a variety of performance devices that would allow you to feel not only hearing but also visually playing a fairly complex piece of music.

The present invention was created in response to the above-described demands, and the user is played by reflecting the speed of the selected song according to the action, the height of each note, the melody sound intensity, and the like, and displaying a 3D graphic image to actually play the music. The object of the present invention is to provide a virtual performance device and a method using a sensor that can visually feel as well as hearing.

1 is a block diagram of a virtual performance device using a sensor according to the present invention.

2 is a flowchart illustrating a virtual playing process in the apparatus shown in FIG. 1.

3 is a sheet music for explaining that accompaniment is output from a music player according to the operation of the tap sensor of FIG.

4 is a screen of an exemplary embodiment in which music information such as tempo and volume of a song played in the display unit of FIG. 1 is displayed in correspondence with the universe of a virtual space.

<Code Description of Main Parts of Drawing>

10 ... selection, 12 ... tap sensor

14 ... head track sensor, 16 ... data glove

18 ... memory, 20 ... central control

22 ... music player, 24 ... speaker

26 ... graphic control unit, 28 ... display unit.

In order to achieve the above object, a predetermined song according to the present invention outputs a playing sound for the song according to a sensor or a keyboard for detecting a user's tap operation, a tablet, a tablet screen, and changes to a visual image. In the virtual playing method to play,

Detecting a tap operation by a user in response to a melody of the song and a motion through a data glove;

Generating accompaniment about the melody of the selected song when the tap motion is detected, and controlling the tempo and volume of the played song according to the data glove detection signal; And

And a process of visualizing and displaying objects representing the height, tempo, tap strength, and volume of the melody of the selected song to help the user play as a change in size and speed on the screen. do.

The display is characterized in that the three-dimensional display.

According to the sensor that detects the user's tap motion, the virtual performance device that changes and plays the song performance and visual image,

A memory for storing a performance song of melody data MEDI;

A selection unit for selecting a song to play from the stored songs;

A tap sensor for detecting the strength and speed of the tapping by the user;

A music player which generates accompaniment according to the selected song's melody and controls and outputs tempo and sound intensity according to a control signal;

A graphic display unit which visualizes and displays current music information of the played song as a graphic; And

And a central control unit for inputting the selection signal to read data of the corresponding song in the memory and outputting the data to the music player, and introducing the sensor signal to control the music player and the graphic display unit.

The apparatus may further include a head track sensor for displaying three-dimensionally according to the movement of the user's head.

The graphic display unit may display a head mounted display (HMD), a projector, a monitor, or a liquid crystal display (LCD) screen.

And a data glove for outputting a sensor signal to the central control unit in order to adjust the volume and tempo of the music played according to the user's hand motion.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

1 is a block diagram of a virtual performance device using a sensor according to the present invention.

The apparatus shown in FIG. 1 includes a memory 18 for storing melody data of a performance song; A selection unit 10 for selecting a song that the user wants to play in the memory 18, a tap sensor 12 which senses the strength and speed of the user's tactile touch with a finger according to the melody, and a tempo of the output song Data glove (16) for controlling the volume, the head track sensor 14 for detecting the user's head movement, the head position information by the head track sensor 14 to change the user's perspective 3 An object indicating the tempo and volume of the song played according to the intensity of tapping detected by the display unit 28, the head track sensor 14, the data glove 16, and the tap sensor 12, which are displayed in the dimensional graphics. Graphical control unit 26 for visualizing graphically and three-dimensional display such as head mounted display (HMD), and music player 22 for outputting the MIDI sound of the selected song according to the tap signal detected by the tap sensor 12 and A central control unit 20 which inputs a selection signal and reads data of a corresponding song into the memory and outputs the data to a music player, and inflows the sensor signal to control the music player 22 and the graphic controller 26; Equipped.

2 is a flowchart illustrating a virtual playing process in the apparatus shown in FIG. 1.

Referring to the flowchart of FIG. 2, the operation is generally divided into steps 30 through which the user selects a song to be played through the selection unit 10 among songs in which the melody data is stored in the memory 18, and detects a sensor. In step 32, the user senses tapping with one hand according to the melody of the selected music, position of the user's head, and hand movement of the data glove 16 mounted on the other hand of the user; The tap detected in step 34 and step 32 of adjusting the tempo and volume of the song played according to the hand gesture detection signal by the data glove 16 while simultaneously generating an accompaniment for the corresponding melody of the selected song according to the Display objects that change correspondingly according to the fast speed on the screen, and display them three-dimensionally from the user's point of view according to the head track sensor 14. It is divided into 36 steps:

In detail, the central controller 20 reads the corresponding song from the memory 18 according to the selection signal of the selector 10 and outputs the corresponding song to the music player 22. When the user taps the tap sensor 12 according to the user's emotion at a predetermined strength and speed according to the melody of the song being played, the central controller 20 may play the music player 22 and the graphic controller according to the sensor signal. 26). The music player 22 outputs accompaniment according to the melody of the selected song according to the tap signal sensed by the tap sensor 12, and controls the strength of the melody sound of the performance song according to the intensity detected by the tap sensor 12.

3 is a score for explaining that the accompaniment is automatically output from the music player 22 according to each tap when tapping the tap sensor 12 in accordance with the main melody track of the song to be played. Here, reference numerals 37 and 39 denote respective accompaniment tracks output according to the first tap and the third tap that the user taps, and reference numeral 38 denotes a case in which there is no accompaniment in the second tap. By tapping, you play the corresponding melody and play the generated accompaniment track automatically. When the melody is played, the melody is played by tapping the tap sensor 12 with a finger every time the melody is to be played, and the accompaniment is automatically played accordingly. After the accompaniment is finished, the performance is stopped while waiting for a finger input to play the next melody, and when the melody performance is slower than the original speed, the accompaniment also becomes a standby state according to the melody. Also, if the user attempts to play a melody note while the accompaniment is played because the tap operation for the melody is too fast, it is not played but is played on time. At this time, to speed up or slow down the tempo of the music being played is controlled by raising or lowering the data glove 16. The central controller 20 controls the music player 22 according to the signal detected by the data glove 16.

Meanwhile, the graphic controller 26 visually shapes the pitch and speed of the melody to control the speed of movement of each object in the graphic, and controls the size of the object according to the strength of the beat so that the user is equipped with the display unit 28. Display in three dimensions. Therefore, the user can see a three-dimensional screen reflecting the time and intensity of the user's finger tapping according to the melody and taste.

When the head track sensor 14 is mounted on the user's head and transmits the user's head movement information to the central controller 20, the central controller 20 controls the graphic controller 26 so that the user can three-dimensionally change the head movement. Allows music information to be displayed to be displayed.

In addition, the central control unit 20 controls the volume of the playing music output from the music player 22 by introducing a detection signal of the degree that the user wearing the data glove 16 clenched his fist and straightened.

4 is a screen of an exemplary embodiment in which information such as tempo and volume of a song played in the display unit of FIG. 1 is displayed in correspondence with the universe of a virtual space.

In FIG. 4, small asteroids around the earth 42 are located in the background of the cosmic celestial body to show information of music to be played, such as tempo and volume of the music. The size of the earth 42 is represented by the volume of the music currently being played. As the volume increases, the size of the earth 42 increases, and when the size decreases, the size of the earth 42 is displayed smaller. Planet 40 at the top of the screen represents the height of the currently playing note, planet 43 represents the notes already played, planet 44 represents the current melody note and the melody note represents the next melody note. The connected planets up to (46) are sequentially changed colors until the next note is played, so that the user can easily know when the next note is to be played.

In the apparatus shown in FIG. 1, each sensor is connected to the central control unit 20 using the RS-232C protocol, and main components may be implemented using a computer and a sound expansion card. It can be replaced by a keyboard, tablet, or touch screen. In addition, by using a communication method between modules using a protocol (API) for music information exchange, it can be executed in connection with an existing music playing system. In addition, from the present invention, if the display is an LCD (liquid crystal display) instead of the HMD and the volume adjustment button and the tap sensor 12 are mounted on the main body of the player, various modified embodiments such as a simple portable playing device are possible.

It takes a lot of time and effort to learn to play a traditional instrument, according to the present invention, the accompaniment is automatically output according to the melody by tapping the tab in accordance with the melody of the performance in accordance with the graphic screen displayed to easily play the song The speed, volume, and melody height of the song being played can be displayed with three-dimensional visualization of each object, making it possible to play with more interest. It provides the ability to play easily by infants and those who lack training on musical instruments.

Claims (7)

In the virtual playing method of playing a predetermined song by playing a sensor or a keyboard that detects the user's tap operation, the tablet (tablet), the touch screen, etc., the sound of the song and changing to a visual image, Detecting a tap operation by a user in response to a melody of the song and a motion through a data glove; Generating and playing an accompaniment for the melody of the selected song when the tap motion is detected; And In order to help the user play, the information about the height, tempo, tap strength, and volume of the melody of the selected song is visualized and displayed on the screen as a change in size and speed of the object corresponding to each note. Virtual playing method using a sensor, characterized in that it comprises a process for. The method of claim 1, further comprising controlling a tempo and a volume of a performance song according to the data glove detection signal. The method of claim 1, The displaying process is a virtual playing device using a sensor characterized in that the three-dimensional display. A memory for storing a performance song of melody data; A selection unit for selecting a song to play from the stored songs; A tap sensor for detecting the strength and speed of the tapping by the user; A music player which generates accompaniment according to the selected song's melody and controls and outputs tempo and sound intensity according to a control signal; A graphic display unit which visualizes and displays current music information of the played song as a graphic; And A sensor having a central control unit for inputting the selection signal to read data of the corresponding song in the memory and outputting the data to the music player, and introducing the sensor signal to control the music player and the graphic display unit; Virtual playing device used. The virtual performance device using the sensor according to claim 3, further comprising a head track sensor for displaying three-dimensionally according to the movement of the user's head. The virtual performance device using a sensor according to claim 3, wherein the graphic display unit displays a head mounted display (HMD). 4. The virtual playing apparatus according to claim 3, further comprising a data glove for outputting a sensor signal to the central control unit in order to adjust the volume and tempo of the playing song according to a user's hand motion.