JP4385860B2 - Performance learning apparatus and performance learning program - Google Patents

Description

  The present invention relates to a performance learning apparatus and a performance learning program.

  As a training device for performance operation using hand images, a photographed image obtained by photographing a player's own hands with a camera as the music progresses is combined with a hand image of a model performance reproduced from memory. In addition, there is a device for learning a performance operation method while confirming a performance position on the keyboard and a hand shape. (See Patent Document 1)

Japanese Patent Laid-Open No. 7-36446 (pages 1 to 7, FIGS. 1 to 12)

The conventional performance operation learning device imitates the sound of each hand so that the shape of the hand and the keyboard position are not mistaken by overlaying the hand image of the teacher performing the model performance with the hand image of the photographed player. The song progresses.
In such a learning device, the size of the teacher's hand performing the model performance may be significantly different from the size of the performer's hand, and it is unnatural if the player tries to fit the shape of the teacher's hand. There was a problem that it became a bill and, for example, the finger was stretched too much or the finger was warped, and the mobility of the finger was impaired.

In the invention which concerns on Claim 1, the imaging means arrange | positioned so that the hand of the player who operates the performance operator group which should be performed and the performance operator in the said performance operator group may be imaged, Display means for superimposing a determination image having a predetermined area on the position of an image indicating a performance operator to be operated in the image, and displaying the image on the display unit on the image picked up by the image pickup means; The image extracting means for extracting the image representing the player's hand operating the performance operator from the image captured by the imaging means, and the image representing the hand extracted by the image extracting means are Determining means for determining whether or not the image is included in the determination image area; and warning means for issuing a warning only when the determination means determines that the image representing the hand is not included in the determination image area. It is characterized by having.
With such a configuration, for example, when a beginner of a keyboard instrument learns a playing method, the shape of the hand during the performance operation is correct, or the palm and finger joints are extended. In order to determine whether or not the player is performing in a badly shaped state, a circular determination image area is set in advance based on the size of the player's own hand, and the captured player's hand image is this Judgment is made so that the hand image does not stick out of the judgment image area, and a warning is given when it is detected that the hand image sticks out of the judgment image. Can contribute to the improvement of musical instrument performance.

  The performance learning apparatus according to claim 2, wherein a matching means for determining whether or not the performance operator to be further operated matches the performance operator operated by the performer, and when the matching means detects a match Only the above-mentioned determination means may be controlled to make a determination.

  The performance learning apparatus according to claim 3 may further include determination image generation means for setting an area of the determination image so that enlargement / reduction is possible.

  The performance learning apparatus of the present invention, for example, when a beginner of a keyboard instrument learns a performance method, the palm or finger joints are stretched or the fingers are warped during performance of the music. If the player is warned that a bad shape is detected, the player can immediately understand the bad shape of the hand even during the performance, and can contribute to the improvement of the musical instrument performance.

  A warning is given when it is detected that the photographed player's hand has protruded from the area of the determination image.

FIG. 1 shows a block diagram of the overall configuration of the present invention in an embodiment of the present invention.
The CPU 101 is connected to each circuit and device, which will be described later, constituting the performance learning device via the bus line 111, and controls various functions by exchanging various data.

  The program ROM 102 stores a program for the CPU 101 to execute processing of various functions according to a predetermined procedure.

The work RAM 103 has a plurality of data areas for temporarily storing various data and a work area for storing various registers used for processing to be described later.
The song memory 104 stores song data for performance learning.
The SW unit 105 includes a plurality of switches, and includes a power switch, a mode switch, a start / stop switch for instructing start / end of music data reading, a switch operated when setting the shooting process, and the like.
The keyboard 106 is composed of a plurality of keys, and guide LEDs 107 are provided below these keys, and are turned on / off according to instructions from the CPU 101 based on the pitch and length of the music data.

The camera unit 108 is installed on the upper side so that the entire area of the keyboard 106 can be photographed, and photographs the bill of the performer performing the performance operation.
The camera unit 108 is composed of a CCD (Charge-Coupled Devices) camera, and image data photographed thereby is stored in the work RAM 103.
The display unit 109 is composed of, for example, a liquid crystal display device, and a determination image consisting of a circle of a predetermined size on an image of a key position to be pressed, which is photographed by the camera unit 108, and a performance operation performed on the key. The user's hand image and a warning of the performance operation are displayed.

  The sound source unit 110 includes a sound source, an amplifier, a speaker, and the like, and generates and generates a musical sound signal in response to a key operation on the keyboard 106.

Next, the data structure of song data stored in the song memory 104 will be described.
The song data is, for example, a practice song for a keyboard instrument, and “notes (0) to (n)” indicating pitch values and “time (0) to (n)” indicating pronunciation times correspond to musical notes. The music ends with “END” data stored alternately.

Next, basic processing of the present invention will be described with reference to FIG.
First, when a power switch provided in the SW unit 105 is turned on, the CPU 101 initializes each circuit of the entire performance learning apparatus and sets a start flag STF to “0” (S301).
Then, it is determined whether or not the mode flag MODE indicates “1” (S302).

Here, if MODE indicates “1”, a “setting process” for setting an appropriate bill size to be used by the performer is executed (S303), and MODE must indicate “0”. For example, the “performance processing” for the keyboard operation lesson is executed in accordance with the reproduction of the music data (S304).
When each of these “setting processing” and “performance processing” is completed, the processing returns to S302 again.

Next, the detailed operation of the “setting process” in S303 in FIG. 3 will be described with reference to the flowchart shown in FIG.
First, the CPU 101 determines whether or not a switch for driving the camera unit 108 is in an off state (S401).

If the drive switch of the camera unit 108 is not in the OFF state, the process proceeds to S403. If it is detected that the drive switch is in the OFF state, the drive switch of the camera unit 108 is turned on (S402). Then, it is determined whether or not a circle as a bill determination image is displayed on the screen of the display unit 109 (S403).
This circle display is a scale display showing a maximum shape defined by a circle with a radius R so that the player's hand during keyboard operation photographed by the camera unit 108 is within this circle.
If a circle is displayed on the screen of the display unit 109, the process proceeds to S405. If no circle is displayed on the screen, the performer uses a radius setting switch provided in the SW unit. Create a hand shape so that the hand playing the keyboard becomes an ideal hand shape that wraps the egg while taking into account the size of your hand, so that this captured hand image fits within the circle display Set the radius R of.
A state in which a circle is displayed by setting the radius R is shown in FIG.

The radius R of this circle is stored in the register R as circle display data (S404).
Then, the circle display data is read from the register R on the display data of the photographed image of the keyboard photographed by the camera unit 108 and synthesized (S405).
Next, whether or not the performer changes the value of the register R by finely adjusting the radius setting switch, whether the size of the circle display is an ideal bill size for playing the keyboard instrument for himself or herself. (S406).

If the value of the register R is to be changed, the process returns to the processing of S404 and the value of the circle display data is changed. However, if the value of the set register R is not changed, the performance SW for designating the performance mode is next selected. It is determined whether or not it is turned on (S407).
If the performance SW on state is not detected, the process returns to S401. However, if the performance SW on state is detected, the mode MODE is set to “0” indicating the “performance processing” mode, and this processing is performed. Is finished (S408).

  As described above, in the “setting process” of the present invention, the performer displays a bill that is generally ideal for playing a keyboard instrument, such as a palm and lightly wrapping an egg with five fingers. The circle display data is set so as to fit within the circle display on the screen, and this data is registered in the register R.

Next, the detailed operation of the performance process (S304) in FIG. 3 will be described with reference to FIGS.
First, it is determined whether or not a start / stop SW for starting performance operation training by turning on music data is turned on (S501).
If the start / stop SW is not in the on state, the process proceeds to step S601 in FIG. 6, but if the start / stop SW is in the on state, the start flag STF is inverted (S502).

  Then, it is determined whether or not the start flag STF is “1” indicating the start of the performance process. If the STF does not indicate “1”, all the guide LEDs that are lit are turned off ( In step S514, the driving of the camera unit 108 is turned off and then the process proceeds to step S601 in FIG. 6. If the STF indicates “1”, the driving of the camera unit 108 is turned on (S504). The head address of the song data stored in the song memory 104 is written into the address register AD (S505).

Then, the CPU 101 reads music data based on the address value indicated by the address register AD (S506).
It is determined whether the read music data is “note” or “time” indicating the sound length (S507).
Here, if the read music data is “time”, the data indicated by this “time” is stored in the register TIME, and then the process proceeds to S601 in FIG. If the data is “note”, the guide LED of the key number corresponding to this pitch is turned on to inform the performer of the key pressing position (S508).

Next, the display data of the radius R stored in the register R is read and circle display data for drawing on the screen of the display unit 109 is generated (S509).
Then, the data corresponding to the “note” of the music data is overlaid so that the data of the pitch position during key emission being photographed by the camera unit 108 and the central portion of the circle display data coincide (S510).

Next, after setting “1” indicating the key display state to the ON flag ONF (S511), the circle display data is displayed on the illuminated key photographed by the camera unit 108 on the screen of the display unit 109. Based on the above, a circle-displayed image having a radius R is displayed so as to overlap (S512).
Here, with reference to FIG. 9, a state in which an image displayed in a circle is superimposed on a lighted key image will be described.

  For example, when the “note” of the read music data indicates the pitch “re” corresponding to the key number n, the guide LED n provided below the keyboard is turned on. Then, an image of circle display data having a radius R is displayed superimposed on the lit keyboard image.

Returning to the description of the flowchart of FIG.
If the keyboard image and the circle display are displayed on the display unit 109, the memory address AD of the song data is incremented (S513), and the process returns to the processing of S506 to return the song data of the next address AD. Is read out, and the process proceeds to the sound generation time counting process based on the “time” of the sound length corresponding to the “note” that is key-emitted.

Next, when the song data read in the process of S507 is “time” indicating the sound length, whether or not the flag STF indicating the start of the performance procedure is “1” after the sound length is stored in the register TIME. It is determined whether or not (S601 in FIG. 6).
If this STF indicates “1”, it is determined whether or not the minimum unit time for engraving the sound duration is over (S602).

Here, when it is detected that the minimum unit time has elapsed, it is determined whether or not the ON flag ONF is “0” indicating the state before the start of sound generation (S603).
If this ONF indicates "0", the sound long time stored in the register TIME is decremented (S604).
Then, it is determined whether or not the sound duration of the register TIME has become zero due to the progress of the sound generation elapsed time (S605).

When the sound duration of the pronunciation indicated by TIME reaches “0”, the address AD is incremented to advance to the next music data (S606).
Then, the next music piece data is read based on this address AD (S607).
When the start flag STF does not indicate “1”, the minimum unit time has not elapsed, the on flag ONF does not indicate “0”, or the sound long time register. In any of these cases where the remaining time of TIME has not reached “0”, the process proceeds to the key scanning process (S701 in FIG. 7).

  Next, whether the next song data read in the process of S607 is “note”, “time” indicating the sound length, or “END” indicating the end of the song data. It is detected whether there is any (S608).

If the read song data is “note”, the guide LED of the key number corresponding to this pitch is turned on (S609), and the circle image of the circle display data is superimposed on this key and synthesized. (S610).
Then, the combined data of the circle display for measuring the size of the illuminated key and the bill taken by the camera unit 108 is displayed on the screen of the display unit 109 (S611).

Next, the ON flag ONF indicating the state in which the key is emitted and the state in which the composite data is displayed is set to “1”, and then the address AD is incremented to further advance the music (S606).
If the read song data is “time” indicating the sound length in the process of S608, the data of this “time” is stored in the register TIME, and then the key scanning process (S701 in FIG. 7). ), If the read song data is “END” indicating the end of the song, both the start flag STF and the ON flag ONF are set to “0” indicating the end of the performance process (S614). All the keys that are lit are extinguished (S615), the composite image of the key and the circle displayed on the display unit 109 is deleted (S616), and the driving of the camera unit 108 is turned off (S617). The process proceeds to the key scanning process (S701 in FIG. 7).

By this key scanning, it is detected whether any key is turned on, turned off, or not changed at all (S702).
If any key is pressed, the CPU 101 stores the key number in the register KEY (S703), and generates a tone having a pitch corresponding to the key number of the KEY so as to generate a sound. The unit 110 is instructed (S704).

Next, it is determined whether or not the ON flag ONF is “1” indicating the key emission state, that is, the key pressing instruction state (S705).
If the ONF indicates “1”, it is next determined whether or not the “note” of the song data is equal to the “KEY” of the key number pressed (S706).
Here, if it is detected that the key number operated by the key depression is different from the “note” of the music data, this is determined as a key depression error, as shown in FIG. 12 on the display screen of the display unit 109. A warning message “Miss key press. Check the key press position” is displayed (S715). If the pitches of both “Note” and “KEY” are equal, the key press position is correct. Evaluate bills for key press operations.

First, an image in a circle display displayed on the display unit 109 is extracted (S707), and whether or not a player's hand image exists in the image in the circle display, or a player's hand image is displayed in a circle. It is determined whether or not the image has protruded from the image (S708).
At this time, for example, as shown in FIG. 10, if it is detected that the hand image A exists without protruding outside the circle display, it is determined that the player is playing with the correct bill.

In order to proceed to the next step, the ON flag ONF is set to “0” (S716), the key emission corresponding to the key that has been instructed to be depressed is turned off (S717), and the setting SW changes the setting mode. It is determined whether or not it is in the ON state shown (S710).
If this setting SW indicates the on state, the mode flag MODE is set to “1” (S711), and this process is terminated. If this setting SW does not indicate the on state, the processing shown in FIG. Returning to the processing of S501, the ON state of the start / stop SW is detected.

  Further, as shown in FIG. 11, when it is detected that a part of five fingers protrudes outside the circle display as shown in FIG. As shown in FIG. 13, after displaying a warning “Please note that your hand is bad” (S709), the process proceeds to S710.

If it is detected in step S702 that the key has been turned off due to the key release, the key number that has been released is stored in the register KEY (S712).
Then, the sound source unit 110 is instructed to mute the sounding sound corresponding to the KEY pitch (S713), and if a warning is displayed on the screen of the display unit 109, this display is made. (S714), the process proceeds to S710 to check the ON state of the setting SW.

  It should be noted that in the case where no key depression or key release is detected as a result of the key scanning in the processing of S702, and in the case where it is detected that the ON flag ONF is not “1” indicating the key emission state in the processing of S705, both The process proceeds to S710 to check the ON state of the setting SW.

As described above, in the performance processing of the present invention, first, the screen of the display unit 109 is overlapped with the circle display data corresponding to the preset radius R at the position of the key number corresponding to the pitch of the read song data. Display above.
Then, the key of the key number corresponding to the “note” of the read music data is turned on to notify the performer of the key pressing position.
A performer performs a key pressing operation by holding the hand on the keyboard that emits the key. The key number pressed at this time is detected, and a bill for performing the performance is photographed by the camera of the camera unit 108. Then, it is determined whether or not the image of the bill is properly fitted at the correct pitch position without protruding from the circle display.

  When a mistake in the key pressing position is detected by this performance processing, or when it is detected that a part of the bill to be pressed is out of the circle even if the pitch position is correct, a warning is displayed. This alerts the performer.

As described in detail above, the performance learning device of the present invention is configured such that when a performer learns a performance method such as a keyboard instrument by himself, the shape of the hand to be performed is photographed and set in advance as the music data progresses. It is to learn that hand images always fit within the circle display.
With this configuration, you can play with the correct bill by paying attention to the performer so that the finger does not stretch or warp at any pitch position while the song is in progress. It is something to be attached to.

The embodiment of the present invention is not limited to the evaluation of the pitch position and the evaluation of the shape of the bill as described above, but evaluates other parameters such as the sound duration, performance tempo, and key-pressing strength. If a malfunction is found in the performance operation, a warning may be displayed.
Further, instead of this warning display, warning may be made using other means such as voice or warning sound.

  Furthermore, in the present embodiment, the maximum shape of the handprint to be played is set by the radius R of the circle, but the present invention is not limited to this, and the characteristics of the hand such as the finger length, palm size, finger thickness, etc. The parameters may be read from the captured image, and a hand image serving as a reference may be generated and displayed instead of the circle display image of the present embodiment.

Also, in the embodiment of the present invention, the circle display is displayed on the key of the key number that is lit based on the “note” of the song data, but the present invention is not limited to this, and the song data is not limited to this. The data of “fingering number” used for performance operation corresponding to “note” is stored, and based on this “note” and “fingering number”, according to the “note” and “fingering number” Alternatively, a circle display may be displayed in an overlapped position.
Further, symbol data indicating how to play such as staccato and legato may be stored in the music data, and the shape of the handprint may be regulated and evaluated according to the method of playing indicated by the symbol data.

  Moreover, it can implement by programming the operation | movement procedure demonstrated in the said embodiment, and making a computer perform. At that time, the computer program can be supplied to the computer through a disk-type recording medium such as a hard disk, various memories such as a semiconductor memory or a card-type memory, or various program recording media such as a communication network.

The block diagram of the whole structure of this invention is shown. The memory structure of the music data in the music memory 104 shown in FIG. 1 is shown. The basic structure of the performance learning apparatus of this invention is shown. The setting process shown by FIG. 3 is shown. The performance process shown by FIG. 3 is shown. (Part 1) The performance process shown by FIG. 3 is shown. (Part 2) The performance process shown by FIG. 3 is shown. (Part 3) A state in which the maximum shape of the bill is set by the radius R of the circle is shown. A state in which a circle display is superimposed on the key emission position of the keyboard n is shown. A correct performance operation state in which hand images are contained in a circle display is shown. Indicates a bad performance operation state when the bill protrudes from the circle display. Indicates a warning display when the key number of the pressed key is wrong. This shows a warning display when a part of the pressed hand image protrudes from the circle display.

Explanation of symbols

101 CPU
102 Program ROM
103 Work RAM
104 song memory 105 SW unit 106 camera unit 107 display unit 108 musical sound generation device

Claims (3)

An imaging means arranged to image a performance operator group to be played and a player's hand operating any performance operator in the performance operator group;
Display means for superimposing a determination image having a predetermined area on the position of an image indicating a performance operator to be operated in the image, and displaying the image on the display unit on the image picked up by the image pickup means; ,
Image extracting means for extracting an image representing a player's hand operating the performance operator from the images picked up by the image pickup means;
Determination means for determining whether an image representing the hand extracted by the image extraction means is included in the region of the determination image;
Warning means for giving a warning only when it is determined by the determination means that the image representing the hand is not included in the region of the determination image;
A performance learning apparatus characterized by comprising: The performance learning device further includes a matching means for determining whether or not the performance operator to be operated and the performance operator operated by the performer match, and the determination means only when the matching means detects a match. Determination control means for controlling to cause the determination to be performed;
The performance learning apparatus according to claim 1, further comprising: 2. The performance learning apparatus according to claim 1 , further comprising determination image generation means for setting an area of the determination image so that enlargement / reduction is possible.

Images