JP2007322683A - Musical sound control device and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a musical sound control device and a program, capable of controlling a musical sound characteristics which is different for each sound range corresponding to each hand, and/or for each sound pitch corresponding to each finger, for touching a key of a keyboard. <P>SOLUTION: Player's key touching of the keyboard is imaged by two cameras installed near both ends of the keyboard, and a predetermined image recognition processing is performed on each image data which is converted into a digital data, and a position of each hand or each finger of the player in a predetermined direction. For example, when the position of each hand is recognized by a hand unit, the position of each hand in the predetermined direction is recognized independently with a left hand and a right hand. Based on the detected position for each hand, the musical characteristic allocated to the left hand and the musical characteristic allocated to the right hand are independently controlled. Meanwhile, when the position of each finger is recognized by a finger unit, the position of each finger is recognized independently with each finger, and based on the position for each detected finger, the musical sound characteristic allocated to each finger is independently controlled. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a musical tone control apparatus and program for detecting movement of a player's hand and / or finger on a keyboard and controlling musical tone characteristics in accordance with the detected movement of the hand and / or finger.

  2. Description of the Related Art Musical tone control devices that detect the movement of a player's hand on a keyboard and control musical tone characteristics in accordance with the detected hand movement are conventionally known.

As such a musical tone control device, the player's finger shape that touches the keys of the keyboard is imaged and the image of the captured finger is recognized to detect the shape of the player's finger, and the detected finger shape Some control the timbre parameters based on the above (see, for example, Patent Document 1). Further, Patent Document 1 captures images including a finger at many times, detects complicated finger movements such as changes in acceleration and movement, and controls timbres, sound volumes, and the like corresponding thereto. Is also described.
No. 7-53117

  However, the conventional musical tone control apparatus cannot control different musical tone characteristics for each musical tone range or pitch. For example, it is impossible to control different musical tone characteristics for the right and left hands operating the keyboard, or to control different musical tone characteristics for each finger operating the keyboard.

  The present invention has been made paying attention to this point, and it is possible to control different musical tone characteristics for each range corresponding to each hand operating the keyboard and / or for each pitch corresponding to each finger. An object of the present invention is to provide a musical tone control device and a program.

  In order to achieve the above object, the musical tone control device according to claim 1 is input by the input means for inputting the image data of the performer's hand and / or finger outputted from the imaging device, and input by the input means. And a recognition means for recognizing at least one position of the player's hand and / or finger on the keyboard in the vertical direction, the horizontal direction and the front-back direction based on the image data of the player's hand and / or finger. And, based on the position of at least one of the up-down direction, left-right direction, and front-rear direction of the player's hand and / or finger on the keyboard recognized by the recognition means, And a control means for controlling the tone characteristics independently for each corresponding pitch group and / or for each pitch.

  In order to achieve the above object, the program according to claim 2 can be realized by a technical idea similar to that of claim 1.

  According to the first or second aspect of the present invention, based on the input image data of the player's hand and / or finger, the player's hand and / or finger on the keyboard in the vertical direction, the horizontal direction, and Since at least one position in the front-rear direction is recognized, and the tone characteristics are controlled independently for each pitch group corresponding to the hand and / or finger and / or for each pitch based on the recognition result. Different musical tone characteristics can be controlled for each pitch group corresponding to each hand and / or for each pitch corresponding to each finger. For example, it is possible to control such that a vibrato is applied to a musical sound being pronounced based on the position of the right hand and a tremolo is applied to a musical sound being pronounced based on the position of the left hand. Alternatively, the degree of vibrato can be changed for each finger (every pitch). Note that “controlling different musical tone characteristics” includes not only the meaning of controlling different musical tone characteristics but also the meaning of controlling the same musical tone characteristics with different control amounts.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram showing a schematic configuration of a musical tone control apparatus according to an embodiment of the present invention.

  As shown in the figure, the musical tone control apparatus of the present embodiment includes a performance operator 1 including a keyboard 1a (see FIG. 2) for inputting pitch information, and a plurality of switches for inputting various types of information. Including a setting operator 2, a detection circuit 3 for detecting the operation state of the performance operator 1, a detection circuit 4 for detecting the operation state of the setting operator 2, and imaging for mainly imaging the hands and fingers of the performer. A device 5, a CPU 6 that controls the entire device, a control program executed by the CPU 6, a ROM 7 that stores various table data, and a RAM 8 that temporarily stores performance data, various input information, calculation results, and the like A timer 9 for measuring the interrupt time and various times in the timer interrupt processing, and a display device 10 for displaying various information, for example, including a liquid crystal display (LCD) and a light emitting diode (LED) The external storage device 11 that stores various application programs including the control program, various music data, various data, and the like, inputs MIDI (Musical Instrument Digital Interface) messages from the outside, and outputs MIDI messages to the outside. A MIDI interface (I / F) 12, a communication interface (I / F) 13 for transmitting / receiving data to / from a server computer (hereinafter abbreviated as “server”) 102 via a communication network 101, and a performance operation A tone generator circuit 14 for converting performance data input from the child 1 or preset performance data into a musical tone signal, an effect circuit 15 for applying various effects to the musical tone signal from the tone generator circuit 14, and the effect A musical sound signal from the circuit 15 is converted into sound, for example, DAC (Digital-to-Ana log converter), a sound system 16 such as an amplifier and a speaker.

  The above components 3 to 15 are connected to each other via a bus 17, a timer 9 is connected to the CPU 6, another MIDI device 100 is connected to the MIDII / F 12, and a communication network 101 is connected to the communication I / F 13. Is connected to the sound source circuit 14, and the sound system 16 is connected to the effect circuit 15. Here, the communication I / F 13 and the communication network 101 are not limited to the wired system but may be a wireless system. In addition, both types may be provided.

  Examples of the external storage device 11 include a flexible disk drive (FDD), a hard disk drive (HDD), a CD-ROM drive, and a magneto-optical disk (MO) drive. As described above, the external storage device 11 can also store a control program to be executed by the CPU 6. If the control program is not stored in the ROM 7, the control program is stored in the external storage device 11. By reading it into the RAM 8, it is possible to cause the CPU 6 to perform the same operation as when the control program is stored in the ROM 7. In this way, control programs can be easily added and upgraded.

  As described above, the communication I / F 13 is connected to a communication network 101 such as a LAN (Local Area Network), the Internet, or a telephone line, and is connected to the server 102 via the communication network 101. When the above programs and various parameters are not stored in the external storage device 11, the communication I / F 13 is used to download programs and parameters from the server 102. The musical tone control apparatus as a client transmits a command for requesting downloading of a program and parameters to the server 102 via the communication I / F 13 and the communication network 101. Upon receiving this command, the server 102 distributes the requested program and parameters to the musical tone control device via the communication network 101, and the musical tone control device receives these programs and parameters via the communication I / F 13. Thus, the download is completed by accumulating in the external storage device 11.

  In the present embodiment, the imaging device 5 includes two small cameras 5a and 5b installed near both ends of the keyboard 1a, as shown in FIG. FIG. 2 is a diagram showing an example of installation of the cameras 5a and 5b, and shows an external appearance in the vicinity of the keyboard including the keyboard 1a. FIG. 2A is a plan view thereof, FIG. 2B is a front view thereof, and FIG. 2C is a side view thereof.

  As shown in FIG. 2 (a), the optical axes of the cameras 5a and 5b are directed toward the center of the keyboard 1a, so that each of the cameras 5a and 5b can capture the entire area of the keyboard 1a. It should be noted that the cameras 5a and 5b may be ordinary ones that can image the performer's hands and fingers, but may be special ones such as infrared cameras. When an infrared camera is used, the performer's hands and fingers are heated, so the performer's hands and fingers are picked up from the surroundings, and the performer's hands and fingers are captured. The position detection accuracy can be further improved. In addition, in order to improve detection accuracy, the player's hand or finger may be marked with a specific color (for example, applying nail polish to the nail, attaching an attachment to the finger, wearing a bracelet, etc.) Good. The positions and number of the cameras 5a and 5b are not limited to the above positions and numbers. For example, a large number of cameras may be installed at intervals of one octave or the like, and by doing so, the accuracy of detecting the position of the hand or finger is further improved.

  The control process executed by the musical tone control apparatus configured as described above will be described first, and then will be described in detail with reference to FIGS.

  Each image data captured and converted into digital data by each camera 5a and 5b is stored in an image data storage area (not shown) secured at a predetermined position in the RAM 8 through the bus 17. . When the performer starts to press the keyboard 1a, first, the CPU 6 generates key-on event data corresponding to the key press and supplies it to the tone generator circuit. In response to this, the tone generator circuit 14 starts to generate a tone corresponding to the supplied key-on event data. Next, the CPU 6 performs a predetermined image recognition process on the two image data stored in the image data storage area, and performs a predetermined direction (vertical direction = vertical direction, horizontal direction) of each hand and each finger of the performer. = Recognize the position of at least one direction, preferably a plurality of directions, of the longitudinal (pitch) direction and the front-rear direction of the keyboard 1a = the shorter (depth) direction of the keyboard 1a. Here, the predetermined image recognition process means a known image recognition process as disclosed in, for example, Japanese Patent Application Laid-Open No. 2001-132340. The reason for adopting such a known image recognition process is that the feature of the present invention is not in the image recognition process itself but in the use of the image recognition process result. This is because the characteristics of the present invention are not changed.

  When the position of each hand and each finger of the performer is recognized in this way, the CPU 6 controls the characteristics of the musical sound being generated based on the recognized positions of each hand and each finger of the performer. Musical tone characteristic control parameters are generated and supplied to the tone generator circuit 14. In response to this, the tone generator circuit 14 controls the characteristics of the musical sound being sounded. The level of image recognition may be a finger unit or a hand unit. When image recognition is performed in units of fingers, different tone characteristics can be controlled for each finger (= each pitch), but high accuracy of image recognition is required. On the other hand, in the case of hand-based image recognition, different musical tone characteristics are controlled for each hand, but the accuracy of image recognition may not be so high.

  Next, when the performer releases the keyboard 1 a, the CPU 6 generates key-off event data corresponding to the key release and supplies it to the tone generator circuit 14. In response to this, the tone generator circuit 14 starts to mute the musical sound being generated.

  In the present embodiment, the image data output from the cameras 5a and 5b is stored in the image data storage area even when the image recognition is not performed. That is, when a key depression of the keyboard 1a is detected, it may be stored in the image data storage area.

  Next, this control process will be described in detail.

FIG. 3 is a flowchart showing the procedure of the musical tone characteristic control process executed by the musical tone control apparatus of the present embodiment, particularly the CPU 6. This musical sound characteristic control process is mainly
(1) Music sound generation start process according to key depression (step S2)
(2) Performer's hand and / or finger position recognition process (step S4)
(3) Musical sound characteristic control parameter generation / supply process (step S5) for generating a musical sound characteristic control parameter according to the recognition result of the position recognition process of (2) and supplying it to the tone generator circuit 14
(4) Mute processing according to key release (step S7)
I do.

  When the detection circuit 3 detects the key depression of the keyboard 1a, the CPU 6 advances the process to the tone generation start process corresponding to the key depression of the above (1), and the key-on event corresponding to the detected key depression. Data, specifically, key code data corresponding to the pitch of the depressed key, velocity data corresponding to the depressed key, and key-on event data including key-on data are generated and supplied to the tone generator circuit 14. In response to this, the tone generator circuit 14 starts to generate a tone signal corresponding to the supplied key-on event data, that is, starts to generate a tone (step S2).

  During key depression, the CPU 6 repeatedly performs the player's hand and / or finger position recognition process (2) and the tone characteristic control parameter generation / supply process (3).

  In the player's hand and / or finger position recognition process of (2) above, the predetermined image recognition process is performed on the captured image data, that is, two image data stored in the image data storage area of the RAM 8. By applying, the position of at least one direction among the up-down direction, left-right direction, and front-back direction of the hand and / or finger is recognized independently for each hand and / or finger (step S4).

  FIG. 4 is a diagram for explaining the player's hand and / or finger position recognition process of (2) above, and FIG. 4 (a) shows the position recognition in units of hands, and FIG. ) Indicates position recognition in units of fingers.

  In the position recognition process in units of hands, as shown in FIG. 4A, the position of at least one of the vertical direction, the horizontal direction, and the front-rear direction of the hand is recognized independently for the left hand and the right hand. In this way, when a position in a predetermined direction is detected for each hand, based on the detected positions for each hand, that is, the positions of the left hand and the right hand, the tone characteristics assigned to the left hand and the tone characteristics assigned to the right hand Can be controlled independently.

  In the position recognition process in units of fingers, as shown in FIG. 4B, each finger independently recognizes the position of at least one direction among the vertical direction, the horizontal direction, and the front-rear direction of the finger. As described above, when a position in a predetermined direction is detected for each finger, the tone characteristics assigned to each finger can be independently controlled based on the detected position for each finger.

  Further, a position in a predetermined direction may be detected for each hand and each finger by combining the position recognition process in units of hands and the position recognition process in units of fingers. In this case, the predetermined direction may be different between the direction with respect to the hand and the direction with respect to the finger. For example, only the position in the vertical direction is detected for the hand, and the position in each direction of the horizontal direction and the front-rear direction is detected for the finger.

  In the present embodiment, the position of the player's hand and / or finger at that time is recognized based on the two image data obtained from the cameras 5a and 5b. The player's hand and / or finger position recognized over time is stored as time-series data in a ring buffer (not shown) provided in the RAM 8, for example, and the player's hand is determined from the change in the time-series data. Alternatively, the speed and acceleration of the finger may be obtained, and the musical sound characteristics may be controlled based on such information.

Returning to FIG. 3, in the musical tone characteristic control parameter generating / supplying process (3), a musical tone characteristic control parameter is generated based on the recognized player's hand and / or finger position, The sound source circuit 14 is supplied. Hereinafter, the tone characteristic control parameters to be generated will be described.
(A) When position recognition is performed in units of hands: different types of tone characteristic control parameters are generated for each hand. For example, a control parameter for controlling the pitch is generated for the right hand, and a control parameter for controlling the volume is generated for the left hand. Alternatively, the same kind of tone characteristic control parameters are generated for each hand. For example, control parameters for controlling the pitch are generated for both the right hand and the left hand. However, even if the same type of tone characteristic control parameters are generated, the control amount can be determined independently for each hand. Further, the player may be able to select a control mode for generating different types of musical tone characteristic control parameters for each hand and a control mode for generating the same type of musical tone characteristic control parameters for each hand.
(B) When position recognition is performed in units of fingers: different types of tone characteristic control parameters are generated for each finger. For example, a control parameter for controlling the pitch is generated for the thumb, and a control parameter for controlling the volume is generated for the other fingers. Alternatively, the same kind of tone characteristic control parameters are generated for each finger. For example, a control parameter for controlling the pitch is generated for all fingers. However, even if the same type of tone characteristic control parameters are generated, the control amount can be determined independently for each finger. Further, the player may be able to select a control mode for generating different types of musical tone characteristic control parameters for each finger and a control mode for generating the same type of musical tone characteristic control parameters for each finger.
(C) When recognizing a position in a plurality of directions as a position in a predetermined direction: A different type of tone characteristic control parameter is generated for each direction. For example, a control parameter for controlling the pitch is generated for the left-right direction, and a control parameter for controlling the volume is generated for the front-back direction. Alternatively, the same kind of tone characteristic control parameters are generated for each direction. For example, control parameters for controlling the pitch are generated for all directions. However, even if the same kind of tone characteristic control parameters are generated, the control amount can be determined independently for each direction. Further, the player may be able to select a control mode for generating different types of musical tone characteristic control parameters for each direction and a control mode for generating the same type of musical tone characteristic control parameters for each direction.

  Examples of the types of musical sound characteristics to be controlled include volume, pitch, and tone color. In any of the cases (A) to (C), a plurality of characteristics may be controlled simultaneously by one operation of a hand or a finger. For example, the pitch and volume are controlled at the left and right positions. When both the hand position and the finger position are recognized, different types of musical tone characteristic control parameters may be generated for the recognized hand position and finger position. For example, a control parameter for controlling the sound volume is generated for the recognized hand position, and a control parameter for controlling the pitch is generated for the recognized finger position.

As a method for generating a musical sound characteristic control parameter, mainly,
(A) A method for generating a musical tone characteristic control parameter for directly controlling a target musical tone characteristic based on the recognized position of the hand or finger in a predetermined direction. (B) Based on the recognized position of the hand or finger in a predetermined direction. There can be mentioned two kinds of methods: a method of generating a musical tone characteristic control parameter for indirectly controlling a target musical tone characteristic.

  Specifically, in the generation method (a), the recognized position of the hand or finger in a predetermined direction is converted into a control parameter that directly controls the pitch, volume, tone color, etc. of the musical sound. On the other hand, in the generation method (b), the recognized position of the hand or finger in a predetermined direction is converted into a control parameter for controlling an LFO (Low Frequency Oscillator), an EG (Envelope Generator), or the like. Here, LFO, EG, and the like are devices that indirectly control the pitch, volume, timbre, etc. of musical tones, so that the position of the hand or finger in a predetermined direction and the types of control parameters that control LFO, EG, etc. If the type or amount of the control parameter is determined from the position of the recognized hand or finger in the predetermined direction and supplied to the tone generator circuit 14, the desired musical sound characteristic can be indirectly obtained. Can be controlled.

  The tone generator circuit 14 samples a musical sound generated by a real musical instrument when the real musical instrument is played with various musical performances such as a vibrato technique, that is, a musical sound whose musical characteristic is controlled to be changed. If a sound source that reads out and produces waveform samples corresponding to the specified performance method is adopted by specifying the type of performance during sound generation, it is based on the position of the recognized hand or finger in the specified direction. The musical performance type may be designated, and in response to this, a tone having a controlled tone characteristic may be generated from the tone generator circuit 14.

  Next, when the key release of the keyboard 1a is detected by the detection circuit 3, the CPU 6 advances the processing to the sound muting process according to the key release in (4), and the key off according to the detected key release. Event data, specifically key code data corresponding to the pitch of the key release key, key off event data including velocity data and key off data corresponding to the key release is generated and supplied to the tone generator circuit 14. In response to this, the tone generator circuit 14 mutes the tone being generated corresponding to the supplied key-off event data (step S7).

  Thus, in the present embodiment, different musical tone characteristics can be controlled for each pitch group corresponding to each hand and / or for each pitch corresponding to each finger, based on the captured image. it can. For example, it is possible to control such that a vibrato is applied to a musical sound being pronounced based on the position of the right hand and a tremolo is applied to a musical sound being pronounced based on the position of the left hand. Alternatively, the degree of vibrato can be changed for each finger (every pitch).

  A program in which a storage medium storing software program codes for realizing the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus is stored in the storage medium. It goes without saying that the object of the present invention can also be achieved by reading and executing the code.

  In this case, the program code itself read from the storage medium realizes the novel function of the present invention, and the program code and the storage medium storing the program code constitute the present invention.

  As a storage medium for supplying the program code, for example, a flexible disk, hard disk, magneto-optical disk, CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD + RW, magnetic A tape, a non-volatile memory card, a ROM, or the like can be used. Further, the program code may be supplied from a server computer via a communication network.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also the OS running on the computer based on the instruction of the program code performs the actual processing. It goes without saying that a case where the functions of the above-described embodiment are realized by performing part or all of the above and the processing thereof is included.

  Further, after the program code read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion is performed based on the instruction of the program code. It goes without saying that the CPU or the like provided in the board or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

It is a block diagram which shows schematic structure of the musical tone control apparatus which concerns on one embodiment of this invention. It is a figure which shows an example of installation of two cameras which comprise the imaging device in FIG. 2 is a flowchart showing a procedure of a musical tone characteristic control process executed by a musical tone control apparatus of FIG. 1, particularly a CPU. It is a figure for demonstrating the position recognition process of the player's hand and / or finger of step S4 of FIG.

Explanation of symbols

5 ... Imaging device, 6 ... CPU (input means, recognition means, control means), 8 ... RAM (input means)

Claims (2)

Input means for inputting image data of a player's hand and / or finger output from the imaging device;
Based on the player's hand and / or finger image data input by the input means, at least one of the up and down direction, left and right direction and front and back direction of the player's hand and / or finger on the keyboard Recognition means for recognizing the position of
Based on the position of at least one of the up and down direction, left and right direction, and front and back direction of the player's hand and / or finger on the keyboard recognized by the recognizing means, the player corresponds to the hand and / or finger. A musical tone control apparatus comprising control means for controlling musical tone characteristics independently for each pitch group and / or for each pitch. An input module for inputting image data of a player's hand and / or finger output from the imaging device;
Based on the player's hand and / or finger image data input by the input module, at least one of the up and down direction, left and right direction and front and back direction of the player's hand and / or finger on the keyboard A recognition module that recognizes the position of
The player's hand and / or finger on the keyboard recognized by the recognition module corresponds to the hand and / or finger based on the position in at least one of the vertical direction, left-right direction, and front-back direction. A program for causing a computer to execute a control module for controlling musical tone characteristics independently for each pitch group and / or for each pitch.

Images