KR102128315B1 - An apparatus for arranging MIDI music based on virtual instrument visualization, method thereof and computer recordable medium storing program to perform the method - Google Patents

Abstract

The present invention relates to a virtual music visualization-based MIDI music arrangement device, a method therefor, and a computer-readable recording medium in which a program for performing the method is recorded. The present invention is a communication unit for receiving a gesture image of a user's gesture through a camera device, a projection unit for projecting a virtual image visualizing a musical instrument image corresponding to a plurality of tracks of MIDI music, and a three-dimensional image of the gesture image. After converting the coordinates into 3D coordinates of the virtual image, a gesture recognition unit that recognizes a user's gesture according to the converted 3D coordinates, and the MIDI music according to the recognized user's gesture while playing the MIDI music An apparatus for arranging MIDI music, comprising a virtual arranging section for arranging a plurality of tracks, and a computer-readable recording medium in which a method therefor and a program for performing the method are recorded.

Description

An apparatus for arranging MIDI music based on virtual instrument visualization, method thereof and computer recordable medium storing program to perform the method}

The present invention relates to a MIDI music arrangement technology, and more specifically, a device for arranging MIDI music through gesture recognition for a visualized virtual instrument by visualizing a virtual instrument, a method therefor, and a program for performing the method are recorded Computer readable recording media.

The existing MIDI music editor basically provides a GUI (Graphic User Interface) using a keyboard and mouse. However, it is not intuitive for a user such as a MIDI music composer or arranger to change the volume of individual notes (note, one note). The user adjusts the volume for each note on the editor GUI and repeats the process of listening to the edited song again. This process is time consuming.

(Patent 1) Korean Patent Publication No. 2008-0074977 published on August 13, 2008 (name: music editing device and music editing method)

An object of the present invention is to recognize a user's gesture such as conducting an orchestra while a song is being played so that when the user wants to arrange MIDI music, the work time is reduced and intuitively arranged, an interface that can arrange music in real time In providing.

In addition, another object of the present invention is to provide an interface capable of simultaneously arranging some or all of a plurality of tracks through visualization of a virtual instrument.

Apparatus for arranging MIDI music according to a preferred embodiment of the present invention for achieving the above object is a communication unit for receiving a gesture image of a user's gesture through the camera device, and a plurality of tracks of the MIDI music Gesture recognition for projecting a virtual image visualizing a corresponding instrument image, and converting the three-dimensional coordinates of the gesture image into three-dimensional coordinates of the virtual image, and then recognizing the user's gesture according to the converted three-dimensional coordinates And a virtual arranging unit for arranging a plurality of tracks of the MIDI music according to the recognized user's gesture while playing the MIDI music.

The virtual image is a virtual stage, a plurality of track images corresponding to each of the plurality of tracks of the MIDI music, and an entire track image corresponding to all of the plurality of tracks of the MIDI music, It includes a button image disposed on the border of the virtual stage, the track image is an instrument image representing the instrument playing the track, a volume image representing the size of the volume playing the track and whether the track is selected Contains optional images to represent.

When the gesture recognition unit recognizes the user's gesture of selecting or deselecting at least one track of the plurality of track images and the entire track image, the virtual arrangement unit tracks at least one track according to the recognized user's gesture. It is characterized by selecting or deselecting.

When the gesture recognition unit recognizes the number of bit points that are inflection points in which the direction of the trajectory of the command gesture is changed, the virtual arrangement device changes the time signature of the playing MIDI music according to the number of bit points.

When the gesture recognition unit calculates the speed of the commanding gesture through the time required to complete one commanding gesture, the virtual arrangement device changes the tempo of the playing MIDI music according to the calculated speed and plays it. It is characterized by.

을 통해 산출하면, 상기 가상편곡부는 산출된 속도의 차이에 따라 상기 재생 중인 미디 음악의 연주법을 변경하여 재생하는 것을 특징으로 한다. When the total length of the trajectory that the gesture recognition unit forms one command gesture is N, the point on the trajectory is x, and the gesture speed at point x is s(x), the speed at the bit point of the trajectory of the command gesture is The difference in speed D of the section other than the bit point

When calculated through, the virtual arrangement unit is characterized in that it plays by changing the playing method of the MIDI music being played according to the difference in the calculated speed.

When the gesture recognition unit calculates the size of the commanding gesture through the distance between two or more bitpoints among the plurality of bitpoints recognized in one commanding gesture, the virtual arrangement unit changes the volume of the MIDI music being played according to the calculated size. It is characterized by playing.

The apparatus for arranging MIDI music according to an embodiment of the present invention further includes a storage unit for storing a plurality of musical instrument images, and the virtual arrangement unit extracts musical instrument information for each of a plurality of tracks from the MIDI music, and stores the storage information. It is determined whether there is a musical instrument image corresponding to musical instrument information of each track of the MIDI music among a plurality of musical instrument images, and if an musical instrument image corresponding to musical instrument information of the track exists, selects the corresponding musical instrument image If the instrument image corresponding to the instrument information of the track does not exist, the audio signal reproduced by playing the corresponding track and the audio signal of the virtual instrument corresponding to the plurality of instrument images stored in the storage unit are compared to obtain an audio signal. It is characterized in that the instrument image of the virtual instrument having the highest similarity is selected, and the plurality of tracks are visualized using the selected instrument image to generate the virtual image.

The method for arranging MIDI music according to a preferred embodiment of the present invention for achieving the above object generates a virtual image visualizing an instrument image corresponding to a plurality of tracks of MIDI music, and generates the generated virtual image. Projecting, playing the MIDI music, receiving a gesture image of a user's gesture through a camera device, and converting the three-dimensional coordinates of the gesture image into three-dimensional coordinates of the virtual image Thereafter, the user's gesture is recognized according to the converted 3D coordinates, and the MIDI music is arranged according to the recognized user's gesture.

The step of arranging the MIDI music is characterized by changing the beat of the playing MIDI music according to the number of the bitpoints when the number of bitpoints, which are inflection points in which the direction of the trajectory of the command gesture is changed, is changed.

The step of arranging the MIDI music is characterized in that when the speed of the command gesture is calculated through the time required for one command gesture to be completed, the tempo of the playing MIDI music is changed and played according to the calculated speed. Should be

을 통해 산출하면, 산출된 속도의 차이에 따라 상기 재생 중인 미디 음악의 연주법을 변경하여 재생하는 것을 특징으로 한다. In the arranging the MIDI music, when the total length of the trajectory forming one command gesture is N, the point on the trajectory is x, and the speed of the gesture at point x is s(x), the bit point in the trajectory of the command gesture D is the difference between the velocity at and the velocity of the section other than the bit point.

When calculated through, it is characterized in that the playing method of the MIDI music being played is changed and played according to the difference in the calculated speed.

In the arranging of the MIDI music, if the size of the command gesture is calculated through a distance between two or more bit points among a plurality of bit points recognized in one command gesture, the volume of the MIDI music being played is changed according to the calculated size. It is characterized by playing.

The step of projecting the virtual image includes extracting instrument information for each of a plurality of tracks from the MIDI music, and whether there is an instrument image corresponding to the instrument information of each track of the MIDI music among a plurality of previously stored instrument images. Determining whether there is an instrument image corresponding to the instrument information of the track, and selecting the corresponding instrument image, and as a result of the determination, an instrument image corresponding to the instrument information of the track exists If not, comparing the audio signal reproduced by playing the corresponding track and the audio signal of the virtual instrument corresponding to the plurality of instrument images stored in the storage unit, and selecting the instrument image of the virtual instrument having the highest similarity of the audio signal; And generating the virtual image by visualizing a plurality of tracks using the selected instrument image.

A computer-readable recording medium in which a program for performing a method for arranging MIDI music according to a preferred embodiment of the present invention for achieving the above-described object visualizes an instrument image corresponding to a plurality of tracks of MIDI music Generating a virtual image, projecting the generated virtual image, playing the MIDI music, receiving a gesture image of a user's gesture through a camera device, and three-dimensional of the gesture image After converting the coordinates to the three-dimensional coordinates of the virtual image, and recognizing the user's gesture according to the converted three-dimensional coordinates, and arranging the MIDI music comprising the arrangement of the MIDI music according to the recognized gesture of the user A program that performs a method for doing so is recorded.

According to the present invention, while the MIDI music is being played, a gesture of the user, such as an orchestra conductor, can be recognized, and the song can be edited, so that the user can arrange the MIDI music in real time and intuitively. Moreover, the present invention provides a plurality of virtual instruments virtually so that a user can arrange multi-tracks simultaneously.

1 is a view for explaining a virtual music visualization based MIDI music arrangement system according to an embodiment of the present invention.
2 is a block diagram illustrating a configuration of a MIDI music arrangement system based on virtual instrument visualization according to an embodiment of the present invention.
3 is a flowchart illustrating a method of visualizing a virtual instrument of a plurality of tracks constituting MIDI music according to an embodiment of the present invention.
4 is a screen example for explaining a virtual instrument visualization method of a plurality of tracks constituting MIDI music according to an embodiment of the present invention.
5 is a flowchart illustrating a method for arranging MIDI music based on virtual instrument visualization according to an embodiment of the present invention.
6 to 9 are screen examples for explaining a method of selecting or deselecting a track according to a track selection or deselection gesture.
10 to 18 are screen examples for explaining a method of arranging a selected track according to the present command gesture.
19 and 20 are screen examples for explaining a method of performing a function assigned to a button according to a button selection gesture.

Prior to the detailed description of the present invention, terms or words used in the present specification and claims described below should not be interpreted as being limited to a conventional or lexical meaning, and the inventor may use his own invention in the best way. In order to explain, it should be interpreted as meanings and concepts consistent with the technical spirit of the present invention based on the principle that it can be properly defined as a concept of terms. Therefore, the embodiments shown in the embodiments and the drawings described in this specification are only the most preferred embodiments of the present invention, and do not represent all of the technical spirit of the present invention, and various equivalents can be substituted at the time of application. It should be understood that there may be water and variations.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. At this time, it should be noted that the same components are denoted by the same reference numerals in the accompanying drawings. In addition, detailed descriptions of well-known functions and configurations that may obscure the subject matter of the present invention will be omitted. For the same reason, in the accompanying drawings, some components are exaggerated, omitted, or schematically illustrated, and the size of each component does not entirely reflect the actual size.

First, a description will be given of a MIDI music arrangement system based on virtual instrument visualization according to an embodiment of the present invention. 1 is a view for explaining a virtual music visualization based MIDI music arrangement system according to an embodiment of the present invention. 2 is a block diagram illustrating a configuration of a MIDI music arrangement system based on virtual instrument visualization according to an embodiment of the present invention.

Referring to FIG. 1, the MIDI music arrangement system 10 includes a virtual arrangement device 100 and a camera device 200. The virtual arrangement device 100 is formed based on a head mounted display (HMD), and the camera device 200 is formed by adding a communication function based on a camera. The camera device 200 photographs the user's gesture and provides the captured user's gesture image to the virtual arrangement device 100. The virtual arrangement device 100 may recognize the user's gesture through the user's gesture image provided from the camera device 200 while playing the MIDI music, and arrange the MIDI music according to the recognized gesture.

Then, the virtual arrangement device 100 will be described in more detail. 2, the virtual arrangement device 100 includes a projection unit 110, an audio unit 120, a sensor unit 130, a communication unit 140, a storage unit 150, and a control unit 160, as shown in FIG. .

The projection unit 110 is for projecting a virtual image constituting a virtual space for arranging MIDI music to a user. Such a virtual image may provide a virtual music playing space in which various musical instruments are played. The projection unit 110 is not shown, but includes a projector and an optical system. The projector radiates an image output from the control unit 160 through a micro display panel. Then, the optical system includes a reflector, a prism, and a lens, and reflects the image emitted from the projector through the reflector or refracts through the prism to project it to the user's eye (pupil) through the lens. Thus, the enlarged virtual image is visible in front of the user's eyes. In particular, the virtual image projected by the projection unit 110 has a predetermined 3D coordinate based on a preset reference point. The user's gesture is recognized according to the three-dimensional coordinates. The 3D coordinates of the virtual image will be described in more detail below.

The audio unit 120 includes a microphone (MIC) and a speaker (SPK). The audio unit 120 receives a user's audio signal through a microphone (MIC), and provides the input audio signal to the control unit 160. In addition, the audio unit 120 outputs an audio signal provided from the control unit 160 through a speaker SPK. This audio signal can be MIDI music.

The sensor unit 130 is for detecting the movement of the virtual arrangement device 100. Since the virtual arrangement device 100 is worn on the user's head, when the movement of the virtual arrangement device 100 is sensed through the sensor unit 130, the user's gaze direction can be known. The sensor unit 130 detects the motion of the virtual arrangement device 100 and provides the detected motion to the control unit 160 through coordinates. The coordinates provided by the sensor unit 130 to the control unit 160 include x, y, z, yaw, and pitch (of a 3D Cartesian coordinate system derived according to the movement of the virtual arrangement device 100 from a preset reference point) pitch), and rolls. The sensor unit 130 may be implemented through one or more sensors, which may be an accelerometer, a gyroscope, a magnetometer, or the like.

The communication unit 140 may directly communicate with the camera device 200 through an ISM band (Industrial Scientific Medical band). For example, the communication unit 140 may be a Bluetooth module, an infrared communication module, or the like. When the communication unit 140 receives a gesture image of a user photographed by the camera apparatus 200 from the camera apparatus 200, the communication unit 140 provides the received gesture image of the user to the controller 160. The communication unit 140 may include an RF (Radio Frequency) transmitter (Tx) for up-converting and amplifying the frequency of the transmitted signal, and an RF receiver (Rx) for amplifying the received signal with low noise and down-converting the frequency. In addition, the communication unit 140 may include a modem for modulating the transmitted signal and demodulating the received signal.

The storage unit 150 serves to store programs and data necessary for the operation of the virtual arrangement device 100, and may be divided into a program area and a data area. The program area may store a program for controlling the overall operation of the virtual arrangement device 100, an operating system (OS) for booting the virtual arrangement device 100, an application program for arranging MIDI music, and the like. The data area is an area in which user data generated according to the use of the virtual arrangement device 100 is stored. That is, the data area stores MIDI music, arranged MIDI music, musical instrument images, and the like. Various types of data stored in the storage unit 150 may be deleted, changed, or added according to user manipulation. In addition, the storage unit 150 may be implemented through a hard disk drive (HDD), a solidestat disk drive (SDD), read access memory (RAM), read only memory (ROM), FLASH, EEPROM, or the like.

The controller 160 may control the overall operation of the virtual arrangement device 100 and the signal flow between the internal blocks of the virtual arrangement device 100, and may perform a data processing function for processing data. The control unit 160 loads and executes an application stored in the storage unit 150, and if necessary, allocates a buffer as a temporary storage space. In particular, the control unit 160 includes, for example, a central processing unit (CPU), an application processor (AP), a micro-controller, a graphics processing unit (GPU), and a digital signal. It may be implemented through a processor (DSP: Digital Signal Processor).

The control unit 160 includes a gesture recognition unit 161 and a virtual arrangement unit 163. The gesture recognition unit 161 is for recognizing the user's gesture from the user's gesture image received from the camera device 200. After recognizing the gesture, the gesture recognition unit 161 outputs the recognition result to the virtual arrangement unit 163. The virtual arrangement unit 163 may generate a virtual image constituting a virtual space where a user can arrange MIDI music. The virtual arrangement unit 163 provides the generated virtual image to the projection unit 110. In addition, the virtual arrangement unit 163 may receive MIDI music stored in the storage unit 150 and reproduce it. The virtual arrangement unit 163 provides the reproduced music to the audio unit 120 and outputs it through the speaker SPK of the audio unit 120. Also, the virtual arrangement unit 163 may arrange MIDI music according to the gesture recognized by the gesture recognition unit 161. The virtual arrangement unit 163 may store the arranged music in the storage unit 150 or provide it to the audio unit 120 to output it through the speaker SPK of the audio unit 120. The detailed operation of the control unit 160 including the gesture recognition unit 161 and the virtual arrangement unit 163 will be described in more detail below.

Next, the camera device 200 will be described in more detail. 2, the camera device 200 includes a communication module 210 and a camera module 220.

The communication module 210 is for communicating with the virtual arrangement device 100 through the ISM band. For example, the communication module 210 may be a Bluetooth module, an infrared communication module, or the like. The communication module 210 transmits the image captured by the camera module 220 to the virtual arrangement device 100. The communication module 210 may include an RF (Radio Frequency) transmitter (Tx) for up-converting and amplifying the frequency of the transmitted signal and an RF receiver (Rx) for amplifying the received signal with low noise and down-converting the frequency. . In addition, the communication module 210 may include a modem that modulates the transmitted signal and demodulates the received signal.

The camera module 220 is for photographing a user's gesture. In particular, the camera module 220 is arranged so that when the user wears the virtual arrangement device 100, the front of the user can be photographed. Although not shown, the camera module 220 includes an infrared light emitting module (IR light), a depth image sensor, and a color image sensor. The color image sensor receives light reflected from a subject and converts it into an electrical signal, and may be implemented based on a CCD (Charged Coupled Device), a CMOS (Complementary Metal-Oxide Semiconductor), or the like. The infrared light emitting module emits infrared light in units of pixels. The depth image sensor receives the infrared light emitted by the infrared light emitting module in units of pixels, and measures the distance (or sometimes expressed in depth) that infrared light is reflected back to the subject. Accordingly, the control unit 160 may derive three-dimensional coordinates for each pixel of the subject of the image photographed through the camera device 200 according to the embodiment of the present invention. A representative example of the infrared light emitting module (IR light), the depth image sensor, and the color image sensor of the camera device 200 may be a Kinect sensor. The camera device 200 may further include an analog-to-digital converter, and convert the electrical signal output from the image sensor into a digital sequence and output it to the controller 160.

The camera module 220 is for photographing a user's gesture. In particular, the camera module 220 is arranged so that when the user wears the virtual arrangement device 100, the user's gesture including the movement of the user's hand and the movement of the stick held by the user's hand is captured. The camera module 220 includes an image sensor, and the image sensor receives light reflected from a subject and converts it into an electric signal, and may be implemented based on a CCD (Charged Coupled Device), CMOS (Complementary Metal-Oxide Semiconductor), or the like. Can. The camera module 220 may further include an analog-to-digital converter, and convert the electrical signal output from the image sensor into a digital sequence to output pixel values (eg, RGB values) of each pixel. Can. In particular, the camera module 220 includes a 3D sensor. The 3D sensor is a sensor for obtaining 3D coordinates of each pixel of an image in a non-contact manner. The camera module 220 may detect the coordinate values (eg, x, y, and z values) of three-dimensional coordinates for each pixel of the image captured through the 3D sensor while simultaneously photographing the object. At this time, the coordinate values of the 3D coordinates are coordinate values when the focus of the camera unit 110 is 0 (0, 0, 0). The 3D sensor can use various types of sensors using laser, infrared light, and visible light. These 3D sensors include laser (3D) scanners using any of TOP (Time of Flight), phase shift (Phase-shift) and Online Waveform Analysis, laser 3D scanners using optical triangulation, optics using white light or modulated light. Method 3D scanner, Handheld Real Time PHOTO, Optical 3D scanner, Pattern Projection or Line Scanning optical method, Laser method full body scanner, Photogrammetry photo method scanner, Kinect Fusion And a real-time scanner used. As described above, the camera module 220 according to an embodiment of the present invention generates a gesture image by photographing a user's gesture. The gesture image includes color values (eg, RGB values) and three-dimensional coordinate values (eg, x, y, and z values) for each pixel.

Then, the virtual instrument visualization method of the MIDI music arrangement system 10 described above will be described. 3 is a flowchart illustrating a method of visualizing a virtual instrument of a plurality of tracks constituting MIDI music according to an embodiment of the present invention. 4 is a screen example for explaining a virtual instrument visualization method of a plurality of tracks constituting MIDI music according to an embodiment of the present invention.

Referring to FIG. 3, the virtual arrangement unit 163 loads MIDI music having a plurality of tracks from the storage unit 150 in step S110. The MIDI music is a digital file, and the virtual arrangement unit 163 extracts musical instrument information for each of a plurality of tracks from the MIDI music in step S120. The instrument information indicates the instrument playing the track. For example, the instrument information of the first to fourth tracks (Track 1, 2, 3, 4) may be a violin, a small drum, a large drum, an electric guitar, and the like.

The virtual arrangement unit 163 searches for the storage unit 150 in step S130, and the instrument information (violin, small book) of each track (Track 1, 2, 3, 4) among the plurality of musical instrument images stored in the storage unit 150. , Large drum, electric guitar) determines whether or not an instrument image exists.

As a result of the determination in step S130, if there is an instrument image corresponding to the instrument information of the track, the virtual arrangement unit 163 selects the instrument image corresponding to the track in step S140.

On the other hand, as a result of the determination in step S130, if there is no musical instrument image corresponding to the instrument information of the track, the virtual arrangement unit 163 plays the corresponding track in step S150, and the stored audio signal and the stored in the storage unit 150 The instrument image of the instrument having the highest similarity to the audio signal is selected by comparing with the audio signal of the virtual instrument corresponding to the plurality of instrument images.

For example, in the case of a track designated to make a sound of "Y guitar's electric guitar," the search result of the storage unit 150 shows that the image of the supported virtual instrument is {"stringed instrument", "guitar guitar", "piano", "drum" } If there are only four, select the image with the most similar "guitar". At this time, in order to increase the accuracy of the acoustic identification, the audio signals ({"stringed instrument" sounds) played by each of the virtual instruments capable of supporting the audio signals reproduced by the track (the result of playing with the "Y guitar electric guitar" sound) As a result, the virtual instrument with the most similar audio signal is finally selected by comparing with the result of playing with "guitar guitar", the result of playing with "piano", and the result of playing with "drum"}.

At this time, the comparison of the most similar audio signal is performed using pattern recognition (sound identification) technology. In addition, the meaning of'play' does not need to output the audio signal'play' of the track through the speaker SPK of the audio unit 120, and is used only for comparison.

As described above, when the instrument image for each track is selected, the virtual arrangement unit 163 generates a virtual image by visualizing a plurality of tracks using the instrument image selected in step S160, and the generated virtual image is projected ( 110).

An example of such a screen is shown in FIG. 4. As shown, the virtual image includes a virtual stage ST. In addition, the virtual image further includes a plurality of track images (Track 1, 2, 3, 4) corresponding to each of the plurality of tracks and an entire track image (ALL TRACK) corresponding to the entire plurality of tracks. The plurality of track images (Track 1, 2, 3, 4) and the entire track image (ALL TRACK) are disposed in the virtual stage ST.

4, track images of the first to fourth tracks (Track 1, 2, 3, 4) are shown. For example, any one track image (eg, Track 4) includes an instrument image 21 representing an instrument playing the track, a volume image 23 representing the volume of the volume playing the track, and whether the track is selected. It includes a selection image 25 representing. The ALL track image (ALL TRACK) includes text 27 indicating that it is an entire track image, for example, “ALL TRACK” and a selection image 29 indicating whether all tracks are selected. Also, the virtual image further includes one or more buttons BT. The button BT is preferably disposed on the border of the virtual stage ST so as not to cover the virtual stage ST.

Then, the MIDI music arrangement method based on the virtual instrument visualization of the aforementioned MIDI music arrangement system 10 will be described. 5 is a flowchart illustrating a method for arranging MIDI music based on virtual instrument visualization according to an embodiment of the present invention. 6 to 9 are screen examples for explaining a method of selecting or deselecting a track according to a track selection or deselection gesture. 10 to 18 are screen examples for explaining a method of arranging a selected track according to the present command gesture. 19 and 20 are screen examples for explaining a method of performing a function assigned to a button according to a button selection gesture.

Referring to FIG. 5, in a state in which a plurality of tracks are visualized and a virtual image is projected, the virtual arrangement unit 163 of the control unit 160 plays MIDI music including the plurality of tracks previously loaded in step S210. During playback of MIDI music, the camera module 220 of the camera device 200 continuously photographs the user's gesture, and the communication module 210 transmits the captured user's gesture image to the virtual arrangement device 100.

Then, the gesture recognition unit 161 of the control unit 160 of the virtual arrangement device 100 continuously receives the gesture image through the communication unit 140. Accordingly, the gesture recognition unit 161 may recognize the user's gesture corresponding to the virtual image from the gesture image received in step S220. That is, the gesture image taken by the camera module 220 has three-dimensional coordinates based on the focus of the camera module 220, and the gesture recognition unit 161 refers to these three-dimensional coordinates based on a predetermined reference point of the virtual image. It can be converted into 3D coordinates, and the gesture of the user corresponding to the virtual image can be recognized according to the converted 3D coordinates. For example, the gesture recognition unit 161 may recognize a gesture pointing to a specific track or a specific button in the virtual image, or pointing out the virtual stage ST of the virtual image.

Subsequently, the virtual arrangement unit 163 determines whether the gesture recognized by the gesture recognition unit 161 is a track selection gesture for selecting or deselecting the track in step S230. As a result of the determination in step S230, if the gesture recognized by the gesture recognition unit 161 is a track selection gesture, the virtual arrangement unit 163 selects or deselects the track according to the corresponding gesture in step S240. A method of selecting or deselecting a track according to a track selection gesture is illustrated in FIGS. 6 to 9.

Referring to FIG. 6, as in gesture S1, when the gesture recognition unit 161 detects a track selection gesture of a user pointing a certain track image (eg, Track 2) over a predetermined time through a stick, the virtual arrangement unit ( 163) selects a corresponding track (eg, Track 2). As another example, as in gesture S2, when the gesture recognition unit 161 detects a track selection gesture of a user pointing a certain track image (eg, Track 4) over a predetermined time through a finger, the virtual arrangement unit 163 may Select the appropriate track (eg Track 4). On the other hand, when the gestures S1 and S2 occur simultaneously, two tracks (eg, Track 2, 4) may be selected at the same time. At this time, the virtual arrangement unit 163 may change the color, brightness, saturation, or brightness of the selected image 25 among the track images in order to visually indicate that it is the selected track.

Referring to FIG. 7, as in gesture S3, when the gesture recognition unit 161 detects a user's track selection gesture maintaining a fisted state, the virtual arrangement unit 163 may select a previously selected track (eg, Track 2, If the selected state of 4) is maintained, and the gesture recognition unit 161 detects the track selection gesture of the user pointing the track image (eg, Track 1) for which the gesture recognition unit 161 is not selected for a predetermined time or longer, the virtual arrangement unit ( 163) selects a corresponding track (eg, Track 1) in addition to the previously selected track (eg, Track 2, 4).

Referring to FIG. 8, as in gesture S5, when the gesture recognition unit 161 detects a user's track selection gesture maintaining a fisted state, the virtual arrangement unit 163 may select a previously selected track (eg, Track 1, 2, 4), and the gesture recognition unit 161 determines a track image (eg, Track 1) of any one of the previously selected tracks (eg, Track 1, 2, 4), such as gesture S6. When the user's track selection gesture pointing for a time or more is sensed, the virtual arrangement unit 163 deselects the corresponding track (eg, Track 1) among the previously selected tracks (eg, Track 1, 2, 4).

Referring to FIG. 9, as in gesture S7, the gesture recognition unit 161 points to the entire track image (ALL TRACK) for a predetermined time or more, and then continuously points to two or more track images (eg, Track 1, 2, 4). When the track selection gesture, which is a drag operation, is sensed, the virtual arrangement unit 163 may select tracks (eg, Tracks 1, 2, 4) corresponding to two or more track images that are consecutively pointed.

As described above, according to the present invention, some or all of a plurality of tracks may be selected or deselected in various ways according to a user's track selection gesture.

On the other hand, as a result of the determination in step S230, if the gesture recognized by the gesture recognition unit 161 is not a track selection gesture, the virtual arrangement unit 163 determines whether the gesture recognized by the gesture recognition unit 161 in step S250 is a command gesture. To discriminate. As a result of the determination in step S250, if the gesture recognized by the gesture recognition unit 161 is a command gesture, the virtual arrangement unit 163 measures the beat, tempo, and method of playing the selected track of the MIDI music being played according to the corresponding gesture in step S260. Arrange and play at least one of the volumes. A method of arranging selected tracks of MIDI music being played according to the command gesture is illustrated in FIGS. 10 to 18.

First, referring to FIG. 10, when the gesture recognition unit 161 recognizes that the trajectory according to the movement of the stick according to the command gesture forms a predetermined shape, the virtual arrangement unit 163 is a MIDI being played according to the formed shape. Arrange and play the beat of the selected track of music. That is, the virtual arrangement unit 163 reproduces the beat of the selected track of the MIDI music being played according to the shape of the command gesture. In FIG. 10, ①, ②, ③, and ④ among the trajectories of the command gesture represent bit points, and these bit points are inflection points in which the direction of the trajectory of the command gesture is changed. The gesture recognition unit 161 may recognize the shape of the command gesture through the number of inflection points, that is, the number of bit points.

For example, as in the gesture E1, the gesture recognition unit 161 recognizes that the number of bit points is two if the inflection points formed according to the trajectory of the command gesture using the stick are two such as ① and ②. Accordingly, the virtual arrangement unit 163 reproduces the beat of one word of the selected track of the MIDI music being played by changing the time signature to 2 beats. As another example, as in gesture E2, if the gesture recognition unit 161 has three inflection points formed according to a trajectory in a command gesture using a stick, it is recognized that the number of bit points is three. Accordingly, the virtual arrangement unit 163 reproduces the beat of one word of the selected track of the MIDI music being played by changing the time signature to 3 beats. As another example, as in gesture E3, if the gesture recognition unit 161 has four inflection points formed according to the trajectory of a command gesture using a stick, such as ①, ②, ③ and ④, it recognizes that the number of bit points is three. do. Accordingly, the virtual arrangement unit 163 reproduces the beat of one word of the selected track by changing it to 4 beats.

Next, referring to FIG. 11, the gesture recognition unit 161 calculates the speed of the command gesture through the time it takes for one command gesture to be completed, and the virtual arrangement unit 163 calculates the calculated speed. Then, the tempo of the selected track of the MIDI music being played is changed and played. That is, when the gesture recognition unit 161 recognizes the speed of the command gesture, the virtual arrangement unit 163 changes and plays the tempo of the selected track of the MIDI music being played according to the speed.

For example, the gesture E4 represents a case in which the time required to complete one command gesture (4 beat command) is 1 second, and the gesture E5 is the time required for the same command gesture (4 beat command) equal to E4 is 0.5 seconds. Indicates. In this way, the gesture recognition unit 161 calculates the time required for one command gesture, that is, the speed of the command gesture, and the virtual arrangement unit 163 selects the selected track of the MIDI music being played according to the calculated speed. Play by changing the tempo. Accordingly, the virtual arrangement unit 163 can quickly change and play the tempo of the corresponding track in the case of the gesture E5 compared to the gesture E4.

Next, referring to FIG. 12, the gesture recognition unit 161 calculates a speed difference in a section other than the bit point and the bit point of the command gesture, and the virtual arrangement unit 163 according to the calculated difference in speed. Plays by changing the playing method of the selected track of the MIDI music being played. In FIG. 12, ①, ②, ③, and ④ among the trajectories of the command gesture represent bit points, and these bit points are sections in which the direction of the trajectory of the command gesture is changed, that is, inflection points. The gesture recognition unit 161 has the speed at the bit points ①, ②, ③, and ④ of the command gesture and a section other than the bit point, that is, the speed between sections ① and ②, between ② and ③, and between sections ③ and ④. Upon recognizing the difference, the virtual arrangement unit 163 changes and plays the selected track of the MIDI music being played according to the difference in speed.

When the total length of a locus forming a single command gesture is N, a certain point on the trajectory is x, and the gesture speed at this point is s(x), the gesture recognition unit 161 bit points and bit points of the command gesture The speed difference (D) in the other sections is calculated through Equation 1 below to provide a virtual arrangement unit 163 to determine a performance method.

That is, according to Equation 1, the speed difference (D) between the bit point of the command gesture and the section other than the bit point can be expressed as the sum of the absolute values of acceleration at each point x of the trajectory of the command gesture. The virtual arrangement unit 163 changes to a performance method in which one sound is shortened as the calculated speed difference D is relatively large, and one sound is long as the calculated speed difference D is relatively small.

For example, (E6), (E7), and (E8) in FIG. 12 are graphs showing changes in the speed of each section while one command gesture is completed. In the case of (E8), the speed difference (D) in the section other than the bit point of the command gesture and the bit point is relatively largest, (E6) is relatively small, (E7) is less than (E8), (E6). Accordingly, when there is a commanding gesture such as (E8), the virtual arrangement unit 163 changes to a playing method in which one note is played relatively shorter than (E7). In addition, when there is a commanding gesture such as (E6), the virtual arrangement unit 163 changes to a playing method in which one note is played longer than (E7). And if there is a command gesture such as (E7), the virtual arrangement unit 163 changes to a playing method that plays one note relatively longer than (E8) and plays one note relatively shorter than (E6).

These methods include legato (so that one note is played long, so that there is no break between notes), non legato (slightly shorter than the original note length), portato (3/4 length of the original note) Sound shorter), staccato (sound shorter than half the length of the original note).

Next, referring to FIG. 12, the gesture recognition unit 161 calculates the size of the command gesture, and the virtual arrangement unit 163 changes the volume of the selected track according to the calculated size and plays it. At this time, the gesture recognition unit 161 may calculate the relative size of the command gesture through the distance between two or more bit points among the plurality of bit points recognized in one command gesture. In addition, according to the size of the command gesture calculated by the gesture recognition unit 161, the virtual arrangement unit 163 increases the volume and plays the volume up, and the smaller the command gesture size, the lower the volume.

For example, in the case of the gesture E9, the gesture recognition unit 161 may calculate the distance d1 between the bit points ② and ④ according to the first command gesture (4 beat command). Also, in the case of gesture E10, which is the same command gesture (4 beat command) as E9, the gesture recognition unit 161 may calculate the distance d2 from the bit points ② and ④ according to one command gesture (4 beat command). When d1<d2, the virtual arrangement unit 163 changes the volume to a lower level than gesture E10 in case of gesture E9 and plays the volume higher than gesture E9 in case of gesture E10 according to the size of the command gesture (d1<d2). To play.

On the other hand, until now, although the command gesture using the stick has been described, as shown in the gestures E11 and E12 shown in FIG. 14, the command gesture can be performed with either one of the right hand and the left hand without a stick. That is, according to an embodiment of the present invention, the time signature can be adjusted according to the shape of the command gesture using the hand, that is, the number of bit points. You can adjust the tempo according to the speed of your hand gesture. You can adjust the playing method according to the speed difference at the beat point of the hand gesture. And the volume can be adjusted according to the size of the hand gesture using the hand.

Next, referring to FIG. 15, when the gesture recognition unit 161 detects a user's gesture to maintain a fisted state, as in the gesture E13, the virtual arrangement unit 163 is used to make a fisted gesture before The selected track is played according to the command gesture. In addition, when the gesture recognition unit 161 recognizes that the gesture recognition unit 161 points out of the virtual stage through a stick or a finger tip, the virtual arrangement unit 163 stops the arrangement according to the command, and performs the existing MIDI music. Play the corresponding MIDI music according to the score of each track.

Next, referring to FIG. 16, when the gesture recognition unit 161 detects a gesture of lowering a palm, as in the gesture E15, the virtual arrangement unit 163 may lower the volume of the selected track and play it. In addition, like the gesture E16, when the gesture recognition unit 161 detects a gesture of lowering the palm of the hand, the virtual arrangement unit 163 may increase the volume of the selected track and play it.

Meanwhile, according to another embodiment of the present invention, a command gesture may be performed using both hands. According to an embodiment, when the gesture recognition unit 161 simultaneously recognizes the left and right hand command gestures, the virtual arrangement unit 163 adjusts the volume according to the recognized left hand gesture, and according to the recognized right hand gesture. You can adjust the time signature, tempo, and playing method. As an example, as shown in FIG. 17, the gesture recognition unit 161 detects a left-handed commanding gesture that maintains a fisted state, such as gesture B1, and at the same time, as a gesture B2, a right-handed commanding gesture When recognizing, the virtual arrangement unit 163 may maintain the volume according to the recognized gesture of the left hand, and adjust the beat, tempo, and playing method according to the recognized gesture of the right hand.

According to another embodiment, when the gesture recognition unit 161 simultaneously recognizes the left hand command gesture and the right hand select gesture, the virtual arrangement unit 163 selects a specific track according to the recognized right hand select gesture, and recognizes the recognized track. You can adjust the volume, beat, tempo, and playing method selected by the right hand according to the left hand's commanding gesture. As an example, as illustrated in FIG. 18, the gesture recognition unit 161 detects a selection gesture indicating a certain track image (Track 1) for a predetermined time or longer, such as gesture B4, and as gestures B5 or B6, When recognizing the selection gesture of lowering or raising the palm through the left hand, the virtual arrangement unit 163 selects the corresponding track (Track 1) according to the recognized selection gesture of the right hand, and is selected by the right hand according to the recognized gesture of the left hand. The volume of the selected track (Track 1) can be turned up or down to play.

Meanwhile, as a result of the determination in step S250, if the gesture recognized by the gesture recognition unit 161 is not a command gesture, the virtual arrangement unit 163 determines whether the gesture recognized by the gesture recognition unit 161 in step S270 is a button selection gesture. To discriminate. As a result of the determination in step S270, if the gesture recognized by the gesture recognition unit 161 is a button selection gesture, the virtual arrangement unit 163 performs a function assigned to the button selected according to the corresponding gesture in step S280. 19 and 20 show a method of performing a function assigned to a button according to a button selection gesture.

Referring to FIG. 19, when the gesture recognition unit 161 detects a button selection gesture of a user pointing a certain button BT1 over a predetermined time through a stick, as in the gesture F1, the virtual arrangement unit 163 corresponds A function assigned to the button BT1, that is, a fast forward function may be performed. The playback control functions include pause, rewind, and play.

In addition, as in the gesture F2, when the gesture recognition unit 161 detects a button selection gesture of a user pointing any one button BT2 through a stick for a predetermined time or longer, the virtual arrangement unit 163 applies the corresponding button BT2. You can play by changing the function assigned to, that is, the playing method to STACCATO.

Referring to FIG. 20, as in gesture F3, when the gesture recognition unit 161 detects a trajectory of a predetermined button selection gesture corresponding to a predetermined button, the virtual arrangement unit 163 is assigned to the corresponding button BT2. The function, that is, the playing method can be changed to STACCATO and played.

On the other hand, after performing steps S240, S260 and S280, the virtual arrangement unit 163 proceeds to step S290. In addition, as a result of the determination in step S270, if the gesture recognized by the gesture recognition unit 161 is not a button selection gesture, the virtual arrangement unit 163 proceeds to step S290.

In step S290, the virtual arrangement unit 163 determines whether playback of the MIDI music being played is finished. As a result of the determination in step S290, if the playback of the MIDI music has not ended, steps S220 to S280 described above are repeated. On the other hand, as a result of the determination in step S290, when the playback of the MIDI music is finished, the virtual arrangement unit 163 stores the MIDI music changed according to the gesture in step S300, that is, the arranged MIDI music in the storage unit 150. .

The method according to the embodiment of the present invention described above may be implemented in a form of a program readable through various computer means and recorded on a computer-readable recording medium. Here, the recording medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in computer software. For example, the recording medium includes magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic-optical media such as floptical disks. magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level language wires that can be executed by a computer using an interpreter, as well as machine language wires such as those produced by a compiler. Such a hardware device can be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

Although the present invention has been described above using some preferred examples, these examples are illustrative and not restrictive. As described above, those skilled in the art to which the present invention pertains will understand that various changes and modifications can be made according to the theory of equality without departing from the spirit of the present invention and the scope of the rights set forth in the appended claims.

10: MIDI music arrangement system 100: virtual arrangement device
110: projection unit 120: audio unit
130: sensor unit 140: communication unit
150: storage unit 160: control unit
161: gesture recognition unit 163: virtual arrangement unit
200: camera device 210: communication module
220: camera module

Claims (15)

A device for arranging MIDI music,
A communication unit that receives a gesture image of a user's gesture through a camera device;
A projection unit for projecting a virtual image visualizing a musical instrument image corresponding to a plurality of tracks of MIDI music;
A gesture recognition unit that converts the 3D coordinates of the gesture image into 3D coordinates of the virtual image, and then recognizes the user's gesture according to the converted 3D coordinates; And
And a virtual arranging unit for arranging a plurality of tracks of the MIDI music according to the recognized user's gesture while playing the MIDI music. According to claim 1,
The virtual image
A virtual stage, a plurality of track images disposed on the virtual stage, and corresponding to each of a plurality of tracks of the MIDI music, a whole track image corresponding to the entire plurality of tracks of the MIDI music, and the virtual stage Contains a button image that is placed on the border of,
The track image
And an instrument image representing an instrument playing the track, a volume image representing the size of the volume playing the track, and a selection image representing whether the track is selected. According to claim 2,
The gesture recognition unit
Recognizing the gesture of the user to select or deselect at least one track of the plurality of track images and the entire track image,
The virtual arrangement section
And selecting or deselecting at least one track according to the recognized user's gesture. According to claim 2,
The gesture recognition unit
Recognizing the number of bit points, which are inflection points that change the direction of the trajectory of the command gesture,
The virtual arrangement section
A device for arranging MIDI music, characterized in that the beat of the MIDI music being played is changed according to the number of bitpoints. According to claim 2,
The gesture recognition unit
Calculating the speed of a command gesture from the time it takes to complete one command gesture,
The virtual arrangement section
A device for arranging MIDI music, characterized in that the tempo of the MIDI music being played is changed and played according to the calculated speed. According to claim 2,
The gesture recognition unit
When the total length of the trajectory forming one command gesture is N, the point on the trajectory is x, and the gesture speed at point x is s(x),
The difference D between the speed at the bit point and the speed of the section other than the bit point among the trajectory of the command gesture
Equation

Calculating through,
The virtual arrangement section
A device for arranging MIDI music, characterized in that the playing method of the MIDI music being played is changed and played according to a difference in the calculated speed. According to claim 2,
The gesture recognition unit
If the size of the command gesture is calculated through the distance between two or more bit points among the plurality of bit points recognized in one command gesture,
The virtual arrangement section
The apparatus for arranging MIDI music, characterized in that the volume of the MIDI music being played is changed and played according to the calculated size. According to claim 1,
Further comprising a storage unit for storing a plurality of musical instrument images,
The virtual arrangement section
Extract musical instrument information for each of a plurality of tracks from the MIDI music,
It is determined whether or not there is a musical instrument image corresponding to musical instrument information of each track of the MIDI music among a plurality of musical instrument images in the storage unit,
If an instrument image corresponding to the instrument information of the track exists, the corresponding instrument image is selected,
If the instrument image corresponding to the instrument information of the track does not exist, the audio signal reproduced by playing the corresponding track and
Comparing the audio signal of the virtual instrument corresponding to the plurality of instrument images stored in the storage unit to select the instrument image of the virtual instrument having the highest similarity of the audio signal,
Apparatus for arranging MIDI music, characterized by generating a virtual image by visualizing a plurality of tracks using a selected instrument image. In the method for arranging MIDI music,
Generating a virtual image visualizing a musical instrument image corresponding to a plurality of tracks of MIDI music, and projecting the generated virtual image;
Playing the MIDI music;
Receiving a gesture image of a user's gesture through a camera device; And
Converting the three-dimensional coordinates of the gesture image into three-dimensional coordinates of the virtual image, recognizing the user's gesture according to the converted three-dimensional coordinates, and arranging the MIDI music according to the recognized user's gesture; Method for arranging MIDI music, characterized in that it comprises a. The method of claim 9,
Arranging the MIDI music is
A method for arranging MIDI music characterized by changing the beat of the MIDI music being played according to the number of bitpoints when recognizing the number of bitpoints, which are inflection points in which the direction of the trajectory of the command gesture is changed. The method of claim 9,
Arranging the MIDI music is
If the speed of the command gesture is calculated through the time required to complete one command gesture, the tempo of the playing MIDI music is changed and played according to the calculated speed. Way. The method of claim 9,
Arranging the MIDI music is
When the total length of the trajectory forming one command gesture is N, the point on the trajectory is x, and the gesture speed at point x is s(x),
The difference D between the speed at the bit point and the speed of the section other than the bit point among the trajectory of the command gesture
Equation

Calculating through,
A method for arranging MIDI music, characterized in that the playing method of the MIDI music being played is changed and played according to the difference in the calculated speed. The method of claim 9,
Arranging the MIDI music is
If the size of the commanding gesture is calculated through the distance between two or more bitpoints among a plurality of bitpoints recognized in one commanding gesture, a MIDI characterized in that the volume of the playing MIDI music is changed and played according to the calculated size. Method for arranging music. The method of claim 9,
The step of projecting the virtual image
Extracting musical instrument information for each of a plurality of tracks from the MIDI music;
Determining whether an instrument image corresponding to instrument information of each track of the MIDI music exists among a plurality of pre-stored instrument images;
As a result of the determination, if an instrument image corresponding to the instrument information of the track exists, the corresponding instrument image is selected. If the instrument image corresponding to the instrument information of the track does not exist as a result of the determination, the corresponding track is played. Comparing the audio signal reproduced by comparing the audio signal of the virtual instrument corresponding to a plurality of pre-stored instrument image to select the instrument image of the virtual instrument having the highest similarity of the audio signal;
And generating the virtual image by visualizing a plurality of tracks by using the selected instrument image. A computer-readable recording medium in which a program for performing a method for arranging MIDI music is recorded,
Generating a virtual image visualizing a musical instrument image corresponding to a plurality of tracks of MIDI music, and projecting the generated virtual image;
Playing the MIDI music;
Receiving a gesture image of a user's gesture through a camera device; And
Converting the three-dimensional coordinates of the gesture image into three-dimensional coordinates of the virtual image, recognizing the user's gesture according to the converted three-dimensional coordinates, and arranging the MIDI music according to the recognized user's gesture; A computer readable recording medium in which a program for performing a method for arranging MIDI music is recorded.

Images