JP2002215139A - Method and device for recording playing operation - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize a method and a device for recording a playing operation, with which the natural playing operation of an acoustic keyboard instrument can be recorded as playing data. SOLUTION: A fingering picture and a fingertip picture obtained by picking up images of entire fingers of both hands and fingertips of a player are subjected to picture recognition processing, and fingers of both hands of the player are individually discriminated from the fingering picture and a key which is being operated is extracted from the fingering picture, and a finger which operates the key is extracted from the fingertip picture, and the key depression/ release operation is discriminated in accordance with the key surface position in the fingertip picture, and thus playing data indicating which key and by which finger is depressed or released is generated and recorded.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a performance operation recording device and a performance operation recording method for recording a performance operation of an acoustic keyboard instrument such as a piano as performance data.

[0002]

2. Description of the Related Art Conventionally, while detecting a pitch, a tone length and a velocity according to a key press / release operation, for example, a movement of a magnet attached to a player's finger has been detected, and based on which, a finger has been operated with any finger. There is known a technique of generating fingering data indicating the above and recording them as performance data.

[0003]

However, such a prior art is applied to a well-known electronic musical instrument, and a piano which cannot directly extract a pitch, a tone length and a velocity corresponding to a key press / release operation. This cannot be applied to acoustic keyboard instruments such as. Also,
When fingering data is generated by the conventional technique, it is necessary to attach some means to the finger of the player in order to detect the movement of the finger, and in such a form, it is unnatural and difficult to perform the playing operation. There is also a disadvantage that it becomes. The present invention has been made in view of such circumstances, and an object of the present invention is to provide a performance operation recording apparatus and a performance operation recording method capable of recording a natural performance operation of an acoustic keyboard instrument as performance data. And

[0004]

In order to achieve the above object, according to the first aspect of the present invention, there is provided a photographing means for acquiring image information of a performance operator and a performance operation of a player, Performance data recording means for generating and recording performance data based on the image information.

According to the second aspect of the present invention, there is provided a photographing means for acquiring image information of a performance operator and a performance operation of a player, and generating and recording performance data based on the image information acquired by the photographing means. Performance data recording means, and performance data recorded by the performance data recording means,
Determining means for comparing the model performance data recorded in advance with the performance data to determine whether the performance data is appropriate;

According to the third aspect of the present invention, an image pickup means for generating a fingering image and a fingertip image obtained by imaging the whole of both fingers and a fingertip of a player, and performing image recognition processing on the fingering image. Each of the performer's two fingers is identified, and which key is being operated is extracted. On the other hand, the fingertip image is subjected to image recognition processing to extract a key operated finger, and a key surface position in the fingertip image is extracted. And a performance data recording means for generating and recording performance data indicating which key has been pressed or released by which finger by determining a key press / release operation in response to the key operation.

According to a fourth aspect of the present invention, a photographing step of acquiring image information of a performance operation by a performance operator and a player, and generating and recording performance data based on the image information acquired in the photographing step. And performing the performance data recording process.

[0008] According to a fifth aspect of the present invention, a photographing step of acquiring image information of a performance operation by a performance operator and a player, and generating and recording performance data based on the image information acquired in the photographing step. Performance data recording process, and performance data recorded in the performance data recording process,
And comparing the model performance data recorded in advance with the model performance data to determine the suitability of the performance data.

According to the invention described in claim 6, the fingering image and the fingertip image are generated by imaging the entire fingers and fingertips of the performer, respectively, and an image recognition process is performed on the fingering image. Each of the performer's both fingers is identified, and which key is being operated is extracted. On the other hand, the fingertip image is subjected to image recognition processing to extract a key operated finger, and a key surface position in the fingertip image is extracted. A performance data recording step of generating and recording performance data indicating which key has been pressed or released by which finger by determining a key press / release operation in response to the key operation.

According to the present invention, image information of a performance operation performed by a performance operator and a player is acquired, and performance data is generated and recorded based on the image information, so that a natural performance operation on an acoustic keyboard instrument can be performed. It can be recorded as data. Specifically, image recognition processing is performed on each of the fingering image and the fingertip image obtained by imaging the entire fingers and the fingertips of the player, and both the fingers of the player are identified from the fingering images. By extracting whether the key is operated or not, on the other hand, extracting the finger to be operated from the fingertip image and determining the key press / release operation according to the key surface position in the fingertip image, To generate and record performance data indicating whether a key has been pressed or released.

[0011]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a performance operation recording apparatus according to an embodiment of the present invention will be described with reference to the accompanying drawings. (1) Configuration FIG. 1 is a block diagram showing the overall configuration of an embodiment according to the present invention. In FIG. 1, reference numeral 1 denotes an imaging unit for imaging a performance operation. As shown in FIG. 2, the imaging unit 1
A pedestal 1a mounted on the keyboard side of an acoustic keyboard instrument such as a piano, a reflection mirror 1b provided on a side of the pedestal 1a so as to face a key end, and an arm 1c extending upward from the pedestal 1a. , An imaging camera 1d disposed at the end of the arm 1c.

The imaging camera 1d has two imaging fields which are switched at predetermined intervals, and one of them is a reflection mirror 1b.
The image of the fingertip for the key operation is captured through the, and the other image of the entire finger of the player performing the key operation. The lateral direction of the reflection mirror 1b is set to a length corresponding to the performance key range width. An image recognition process is performed by a CPU 3 described later on a fingertip image obtained by imaging the fingertip of the performer by the imaging camera 1d and a fingering image obtained by imaging the entire finger of the performer. Performance data indicating which finger has been pressed is generated. Hereinafter, the principle of generating performance data from a fingertip image and a fingering image will be described with reference to FIGS.

FIG. 3 shows an example of a fingering image obtained by imaging the entire two fingers of a player with the imaging camera 1d. The fingering image is subjected to a well-known image recognition process using the length of each finger, the direction of each finger, and the like as a recognition parameter, thereby identifying both fingers of the player. As an identification result, identifiers F1 to F5 are assigned to the left little finger and the thumb, respectively, and identifiers F6 to F10 are assigned to the right thumb and the little finger. Further, as shown in FIG.
Finger widths W1 to W1 corresponding to the fingers each given 10
0 is measured as a feature parameter.

In the image recognition processing performed on the fingering image,
It also extracts which key is being operated. As a method for this, a key number is assigned to each key shown in the fingering image, and a key serving as a pitch reference (for example, a key corresponding to the C sound) is set from among them. The key operated from the fingering image, that is, the key whose upper surface is descending is detected, and the key operated according to the difference between the key number of the detected key and the key number of the reference key. Is determined. Whether the key operation is a key press or a key release is determined by image recognition processing using a fingertip image described later.

FIG. 4 shows an example of a fingertip image obtained by capturing an image of a player's fingertip by the imaging camera 1d. The fingertip image is subjected to image recognition processing, and the dimensions (width W and height H) of the finger for key operation are extracted as feature parameters. Key press /
The key release operation is determined based on the position of the key upper surface in the fingertip image. That is, when the position of the upper surface of the key is lowered by a predetermined amount from the reference position, the key is regarded as being pressed, and when the position of the upper surface of the key is raised than the pressed position, it is regarded that the key is released.

Simultaneously with the determination of the key press / release operation, it is determined which finger has been operated with the finger based on the characteristic parameter (dimension of finger). More specifically, as shown in FIG. 5, the characteristic parameters of the fingers of the performer, that is, the dimensions (width W and height H) of the fingers, which are the characteristic parameters, correspond to each of the identifiers F to F10 described above. Attached finger dimension table T
BL is provided, and the finger that has performed a key press operation or a key release operation is determined by comparing and referencing the contents of the table TBL and the characteristic parameters extracted from the fingertip image. The discrimination result is compared with the pitch of the key operated from the fingering image to generate performance data indicating which key has been pressed or released with which finger.

Next, the configuration of the embodiment will be described with reference to FIG. In FIG. 1, reference numeral 2 denotes a panel switch group having various switches and generating a switch event according to the operated switch. Although not shown, the panel switch group 3 includes a power switch for turning on / off the power, a mode switch for selecting an operation mode, and the like. Reference numeral 3 denotes a CPU that performs image recognition processing on the fingertip image and the fingering image supplied from the imaging unit 1 and generates performance data indicating which key has been pressed and released by which finger. The CPU 3 transmits the generated performance data to R
After recording in the AM5, the performance data is reproduced in accordance with the reproduction instruction to generate musical tones, and a function of evaluating the performance operation by comparing the model performance data stored in advance with the generated performance data is also provided. Prepare.

Reference numeral 4 denotes a ROM for storing various control programs and model performance data loaded on the CPU 3. Reference numeral 5 denotes a RAM, a work area WE for temporarily storing various register / flag data, and a finger size table TBL described above.
(See FIG. 5), a data area DE for storing performance data and the like generated by the CPU 3, and an image buffer for holding captured images of a plurality of frames. The performance data stored in the data area DE includes, as shown in FIG. 6, events EVT (1) to EVT (1) generated in response to a performance operation.
(N) and end data END indicating the end of the music. Each of the events EVT (1) to (n) is composed of a note-on event or a note-off event.

The note-on event includes a note-on NON indicating a sound, a note number NN indicating a pitch, a velocity VEL indicating a volume, and a tone length ONTYO indicating a sounding period.
U, YUBI representing a finger that has performed a key press operation, and a fingering image UNSI capturing a fingering state at that time. On the other hand, the note-off event is formed from a note-off NOF indicating silence and a note number NN indicating a pitch to be silenced.

Reference numeral 6 denotes a display unit composed of an LCD panel or the like, which evaluates the contents of the recorded performance data and the model performance data and the recorded performance data in accordance with a display control signal supplied from the CPU 3 described later. Display the results on the screen. Reference numeral 7 denotes a sound source configured by a well-known waveform memory reading method, which includes a plurality of time-divisionally operated sound channels. The sound source 7 stores waveform data of various timbres, reads out the waveform data designated by the CPU 5 from them, and forms a musical tone waveform W obtained by modifying the waveform data according to the performance data. 8 is a D / A conversion circuit,
The output of the sound source 7 is converted into an analog waveform signal and supplied to the next sound system 9. Sound system 9
For example, after performing filtering such as removing unnecessary noise on an analog waveform signal supplied from a preceding stage, the analog signal is amplified and sounded from a speaker SP.

(2) Operation Next, the operation of the embodiment will be described with reference to FIGS. Operation of Main Routine In response to power-on, the CPU 5 reads a predetermined control program from the ROM 6 and loads it into itself, executes the main routine shown in FIG. 2 and proceeds to step SA1, and is provided in the work area of the RAM 5. Performs initialization processing for initializing various register flags. Thereafter, in step SA2, the fingering image and the fingertip image are taken, a key number is assigned to each key shown in the fingering image, and a key serving as a pitch reference (for example, a key corresponding to sound C) ) Is set, and the finger size (width W and height H) of each finger F1 to F10 of the player extracted from the fingering image and the fingertip image is registered in the finger size table TBL.

Next, at step SA3, according to the operation of the mode switch, either a key press detection mode for generating performance data or an evaluation mode for comparing model performance data with performance data recorded according to the performance operation for evaluation. Execute the mode setting process to be set. If the key press detection mode is set in this mode setting process, a key press detection process for generating and recording performance data is executed in step SA4, while if the evaluation mode is set, At step SA5, an evaluation process for comparing and evaluating the model performance data and the performance data is executed. Thereafter, the processing according to the setting mode is executed until the power is turned off.

Operation of Initial Setting Processing Routine Next, the operation of the initial setting processing routine will be described with reference to FIG. When this routine is executed through step SA2 described above, the CPU 3 executes step S2 shown in FIG.
The process proceeds to B1, an initial fingering image is captured, and this is temporarily stored in the image buffer of the RAM 5 as the initial fingering image SUG. Note that the initial fingering image SUG referred to here refers to a fingering image captured in the performance key area with both fingers of the player in a performance start state.

Next, in step SB2, a key number is assigned to each key appearing in the initial fingering image SUG, and a key serving as a pitch reference (for example, a key corresponding to the C sound) is set from among them. Subsequently, in step SB3, a fingertip image is taken and temporarily stored in the image buffer of the RAM 5 as an initial fingertip image SYG, and the size of each finger of the player obtained by performing image recognition processing on this image SYG ( The width W and height H) correspond to the two fingers F1 to F10 of the player extracted from the initial fingering image SUG, and the finger dimension table TB.
Generate L.

Operation of Mode Setting Processing Routine Next, the operation of the mode setting processing routine will be described with reference to FIG. When this routine is executed via the above-described step SA3, the CPU 3 advances the process to step SC1 shown in FIG. 9, and generates an event according to the mode switch operation of the user (player). After step SC2, a flag corresponding to the switch event is set. That is, when an event to set the key press detection mode occurs, the determination result of step SC2 is “YES”, the process proceeds to the next step SC3, and the mode flag M is set to “0”. Set the key press detection mode. On the other hand, if an event to set the evaluation mode has occurred, the determination result of step SC4 is "YES", the process proceeds to the next step SC5, the mode flag M is set to "1", and the evaluation mode is set to "1". Set to.

Operation of Key Press Detection Processing Routine Next, the operation of the key press detection processing routine will be described with reference to FIGS. When this routine is executed through the above-described step SA4 (see FIG. 7), the CPU 3
Proceeds to step SD1 shown in FIG. 10, and determines whether or not the mode flag M is "0", that is, whether or not the key press detection mode is set. If the mode is set to the key press detection mode, the determination result is "YES" and the process proceeds to the next step SD2. However, if the mode is set to the evaluation mode, the determination result is "NO" and no processing is performed. This routine is completed without doing this.

In step SD2, the timer counter T is reset to zero. In step SD3, the initial fingering image SUG is set to the previous image LTG. Next, in step SD4, the value of the timer counter T is incremented and the step is advanced, and in the following step SD5, a fingering image is captured, and the present fingering image TUG (T) corresponding to the value of the timer counter T is taken. Is temporarily stored in the image buffer of the RAM 5. Next, in step SD6, the image of the fingertip is taken and temporarily stored in the image buffer of the RAM 5 as the current fingertip image TYG (T) corresponding to the value of the timer counter T.

In step SD7, the key surface shown in the previous fingering image LUG and the key surface shown in the current fingering image TUG (T) are compared, and the key surface of the key whose key surface descends according to the key depression is determined. Determine the presence or absence. If there is a key that descends the key surface, the determination result is “YES”, and the key depression confirmation processing after step SD8 is executed. If there is no key that the key surface descends, the determination result is “N”.
O ", and a key release confirmation process after step SD18 shown in FIG. 13 is executed. Hereinafter, the operation will be described separately for the key press confirmation process and the key release confirmation process.

(A) Key Press Confirmation Process When there is a key that descends the key surface, the CPU 3 proceeds to step SD8 to extract the key surface position from the current fingertip image TYG (T), and that position is determined by the predetermined key depression amount. Whether you have reached point A
That is, it is determined whether or not the key depression state has been started. Here, if the key surface position has not reached the predetermined key depression amount A point and has not reached the key depression start state, the determination result is “NO”, the process proceeds to step SD9, and the current fingering image TU
G (T) and the current fingertip image TYG (T) are updated to the previous fingering image LUG (T) and the previous fingertip image LYG (T), respectively. Thereafter, the process returns to step SD4 to increment the timer counter T, and then proceeds to steps SD5 to SD5.
After SD7, it is determined whether or not there is a key that descends again according to the updated image content.

When there is a key that descends the key surface and the key surface position reaches the predetermined key depression amount A, step SD8
Is determined as "YES", the process proceeds to step SD10, and the key pressing start time T is set in the register t1. Next, the process proceeds to step SD11 shown in FIG. 11, and it is determined whether or not the key surface position extracted from the fingertip image TYG (T) this time has reached a predetermined key depression amount B point (key depression start point).
If the key pressing start point has not been reached, the determination result is "NO", and the steps from step SD4 are repeated again. However, if the key pressing start point has been reached, the determination result is "YES", and the process proceeds to step SD12. . In step SD12, the pitch of the key operated is determined according to the difference between the key number assigned to the key that has reached the key depression start point and the key number of the reference key, and stored in the register NN. A note-on event indicating sound generation is set in the register NON.

Next, in step SD13, the key depression arrival time T is stored in the register NOTE ON (t), and in the following step SD14, the key depression arrival time T is stored in the register t2.
Store in Then, in step SD15, a velocity value corresponding to the required key pressing time t2-t1 is generated and set in the register VEL. Next, in step SD16, the present fingering image TUG (T) is registered as the fingering image UNSI (n) to be taken into the performance data. Thereafter, a fingering number search processing routine is executed via step SD17 to determine which finger has been pressed. Hereinafter, the operation of the fingering number search processing routine will be described with reference to FIG.

When the fingering number search processing routine is executed through step SD17, the CPU 3 proceeds to step SE1 shown in FIG. 12, and the fingertip image at the time when the predetermined key depression amount B point (key depression start point) is reached. TYG (T) is extracted, and in the following step SE2, the dimensions (width W and height H) of the finger pressed by the key are measured from the fingertip image TYG (T) to generate feature parameters YUBI (W, H). . Next, at step SE3, the characteristic parameter YUBI (W, H)
Is retrieved from the finger size table TBL (see FIG. 5) described above, and then in step SE4, the retrieved identifier F is stored in the register YUBI (n).

As described above, when a series of performance data corresponding to key depression is determined, the process proceeds to step SD18, and the contents of the registers NON, NN, VEL, YUBI (n) and UNSI (n) are stored in the performance data. RAM5 as
And temporarily completes the key press confirmation processing. Note that the note length ONTYOU indicating the sounding period during the note-on event is determined after a key release operation is performed, and this is generated in a key release confirmation process described later.

(B) Key release confirmation process If there is no key that descends in the key surface in step SD7 (see FIG. 10), the process proceeds to step SD19 shown in FIG. Judge. If there is no key whose key surface rises, the determination result is “NO”, and FIG.
After returning the process to step SD4 shown in FIG. 0 and incrementing the timer counter T, through steps SD5 to SD7,
Proceeding to step SD19, it is determined whether there is a key whose key surface rises again according to the updated image content. Then, when there is a key whose key surface rises, the determination result of step SD19 is “YE
S ", and the process proceeds to Step SD20. In step SD20, it is determined whether or not the key whose key surface rises is a key having a pitch stored in the register NN.

If the key is not a key corresponding to the pitch stored in the register NN, the determination result is "NO", and the process returns to step SD4 shown in FIG. If it is a key, the judgment result is “YE
S ", and proceeds to the next step SD21. In step SD21, it is determined whether or not the key has returned to a predetermined key release amount C (key release state).

If not restored, the determination result is "NO"
Then, the process proceeds to step SD25, and it is determined whether the amount is smaller than the return amount D, that is, whether key release is being performed. Not in the middle of key release,
If the return amount is less than D, the determination result is “YES”,
Proceed to step SD26. In step SD26, the current fingering image TUG (T) and the current fingertip image TYG (T) are respectively converted to the previous fingering image LUG (T) and the previous fingertip image LYG.
Update to (T). Thereafter, the process returns to step SD4 to increment the timer counter T, and then, through steps SD19 to SD21, it is determined whether or not there is a key whose key surface rises again according to the updated image content. On the other hand, if the key is being released and is not less than the return amount D, the determination result is “NO” and the process proceeds to Step SD21.

On the other hand, when the key release state is reached, the result of determination in step SD21 is "YES", and
The process proceeds to step S22, where the pitch corresponding to the released key is stored in the register NN, and a note-off event indicating mute is set in the register NOF.
The contents of N and NOF are stored in the data area DE of the RAM 5. Next, in step SD23, the key release time T is set in the register NOTE OFF (t), and in the following step SD24, the key press arrival stored in the register NOTE ON (t) in the previous step SD13 (see FIG. 11). From the time T, the above register NOTE OFF
The key length is calculated by subtracting the key release time T stored in (t), and is stored in the register ONTYOU to complete the present routine.

Operation of Evaluation Processing Routine Next, the operation of the evaluation processing routine will be described with reference to FIGS. When this routine is executed via the above-described step SA5 (see FIG. 7), the CPU 3 advances the processing to step SF1 shown in FIG. 14, and determines whether the mode flag M is "1", that is, whether the mode is set to the evaluation mode. Judge whether or not. If you are not in evaluation mode,
The determination result is "NO", and this routine is completed without performing any processing. However, if the evaluation mode is set, the determination result is "YES", and the process proceeds to the next step SF2. In steps SF2 and SF3, the model performance data and the performance data generated in the above-described key press detection processing routine are selected.

When the evaluation start switch is turned on, the result of determination in step SF4 becomes "YES",
Proceeding to step SF5, the model performance data and the leading event of the performance data selected in steps SF2 and SF3 are read. Next, in step SF6, both events are reproduced synchronously, and thereafter, in steps SF7 to SF
At 18, the contents of both events reproduced are compared. That is, in step SF7, it is checked whether or not the note numbers NN match, and if they match, the process proceeds to step SF8.
The evaluation result of "NOTE OK" is recorded in a predetermined area of the RAM 5 corresponding to the event number n, and if not, the evaluation result of "NOTE NG" corresponding to the event number n is stored in the RAM 5 at step SF9. Record in a predetermined area.

Next, in step SF10 shown in FIG. 15, it is determined whether or not the velocity VEL is within the allowable range. If it is within the allowable range, "VEL OK" is associated with the event number n in step SF11. The evaluation result is recorded in a predetermined area of the RAM 5, and if it is out of the allowable range, the evaluation result of “VEL NG” is recorded in a predetermined area of the RAM 5 in step SF12 in association with the event number n. Next, in step SF13, the sound length is turned on.
It is determined whether or not the TYOU falls within the allowable range. If the TYOU is within the allowable range, the evaluation result "ONTY OK" is recorded in a predetermined area of the RAM 5 in step SF14 in correspondence with the event number n, and If not, in step SF15, an evaluation result of "ONTY NG" is recorded in a predetermined area of the RAM 5 in association with the event number n.

Subsequently, at step SF16, the finger number YU
It is checked whether or not the BIs match. If they match, an evaluation result of "fingering OK" is recorded in a predetermined area of the RAM 5 in step SF17 in correspondence with the event number n. The evaluation result of “fingering NG” corresponding to the event number n is stored in the RAM in SF18.
5 in a predetermined area. Then, Step SF19
In step SF20, it is determined whether or not the content of the stepped event is END data indicating the end of the music. If it is not the end of the music, the determination result is "NO", and the process returns to step SF6 to reproduce the stepped event contents, and evaluates the suitability of the performance data with respect to the model performance data as described above. When the end of the song is reached, the above-described step SF
The determination result at 20 is "YES", and this routine is completed.

As described above, according to the present embodiment, the image recognition processing is performed on the fingering image and the fingertip image obtained by imaging the entire two fingers and the fingertip of the player, and the player's finger image is obtained from the fingering image. It identifies each finger of both hands and extracts which key is being operated, while extracting the finger to operate the key from the fingertip image and judges the key press / release operation according to the key surface position in the fingertip image By doing so, it is possible to generate performance data indicating which key has been pressed or released with which finger, so that a natural performance operation on an acoustic keyboard instrument can be recorded as performance data. .

In the above-described embodiment, the evaluation result obtained by comparing the model performance data and the performance data extracted from the performance operation for each event is recorded in the RAM 5, but the evaluation result recorded in the RAM 5 is described. Of the results, performance data corresponding to an erroneous performance operation may be displayed on the display unit 6 on a screen.

[0044]

According to the first and third aspects of the present invention, image information of a performance operation performed by a performance operator and a player is acquired, and performance data is generated and recorded based on the acquired image information. A natural performance operation of an acoustic keyboard instrument can be recorded as performance data. According to the second and fourth aspects of the present invention, image data of performance operation of a performance operator and a player is acquired, and performance data generated and recorded based on the acquired image information is recorded in advance. The performance data is compared with the model performance data to determine whether the performance data is appropriate, so that a natural performance operation on an acoustic keyboard instrument can be recorded as performance data.
The propriety of the recorded performance data can be determined. According to the third and sixth aspects of the present invention, image recognition processing is performed on the fingering image and the fingertip image obtained by imaging the entire fingers and the fingertips of the player, respectively. Each of the fingers is identified, and which key is being operated is extracted. On the other hand, the finger to be operated is extracted from the fingertip image, and the key press / release operation is determined according to the key surface position in the fingertip image. Thus, performance data indicating which key is pressed or released with which finger is generated and recorded, so that a natural performance operation on an acoustic keyboard instrument can be recorded as performance data.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of an embodiment according to the present invention.

FIG. 2 is a diagram showing a schematic structure of an imaging unit 1 mounted on a keyboard instrument.

FIG. 3 is a diagram for explaining the principle of generating performance data.

FIG. 4 is a diagram for explaining the principle of generating performance data.

FIG. 5 is a diagram for explaining the principle of generating performance data and the contents of a finger size table TBL.

FIG. 6 is a diagram showing a configuration of performance data.

FIG. 7 is a flowchart showing an operation of a main routine.

FIG. 8 is a flowchart illustrating an operation of an initialization processing routine.

FIG. 9 is a flowchart showing an operation of a mode setting processing routine.

FIG. 10 is a flowchart illustrating an operation of a key press detection processing routine.

FIG. 11 is a flowchart illustrating an operation of a key press detection processing routine.

FIG. 12 is a flowchart showing the operation of a fingering number search processing routine.

FIG. 13 is a flowchart showing the operation of a key press detection processing routine.

FIG. 14 is a flowchart illustrating an operation of an evaluation processing routine.

FIG. 15 is a flowchart illustrating an operation of an evaluation processing routine.

[Explanation of symbols]

 Reference Signs List 1 imaging unit 2 panel switch group 3 CPU 4 ROM 5 RAM 6 display unit 7 sound source 8 D / A conversion circuit 9 sound system

Claims (6)

[Claims] 1. A photographing means for acquiring image information of a performance operation element and a performance operation of a player, and a performance data recording means for generating and recording performance data based on the image information acquired by the photographing means. A performance operation recording device, comprising: 2. A photographing means for acquiring image information of a performance operation element and a performance operation of a player; a performance data recording means for generating and recording performance data based on the image information acquired by the photographing means; A performance operation recording apparatus, comprising: performance data recorded by the performance data recording means; and determining means for comparing the performance data recorded in advance with model performance data to determine whether the performance data is appropriate. 3. An imaging means for generating a fingering image and a fingertip image obtained by imaging the entire fingers and fingertips of a player, respectively; and performing an image recognition process on the fingering images to obtain both fingers of the player. In addition to identifying and extracting which key is being operated, the fingertip image is subjected to image recognition processing to extract a key to be operated, and a key press / release operation is performed according to a key surface position in the fingertip image. A performance data recording means for generating and recording performance data indicating which key has been pressed or released with which finger by determining which key has been pressed. 4. A photographing step of acquiring image information of a performance operation element and a performance operation of a player, and a performance data recording step of generating and recording performance data based on the image information acquired in the photographing step. A performance operation recording method, comprising: 5. A photographing step of acquiring image information of a performance operation element and a performance operation of a player; a performance data recording step of generating and recording performance data based on the image information acquired in the photographing step; A performance operation recording method, comprising: comparing performance data recorded in the performance data recording process with model performance data recorded in advance to determine whether the performance data is appropriate. 6. An imaging process for generating a fingering image and a fingertip image obtained by imaging the entire fingers and fingertips of a player, respectively; and performing an image recognition process on the fingering images so that the fingers of the player can be obtained. In addition to identifying and extracting which key is being operated, on the other hand, performing an image recognition process on the fingertip image to extract a finger to be operated, and performing a key press / release operation in accordance with the key surface position in the fingertip image A performance data recording step of generating and recording performance data indicating which key has been pressed or released with which finger by determining which key has been pressed.