JP4655455B2 - Performance recording / reproducing apparatus and program - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a technique for allowing a performer to visually confirm the contents of a performance performed using a musical instrument.
[0002]
[Prior art]
A performance using a musical instrument is realized by sequentially performing performance operations for generating a musical sound to be generated. For example, a device described in Patent Document 1 is known as a performance guidance device that assists in correctly performing the performance.
[0003]
In the performance guidance device, a figure showing a hand is displayed on a display device, and the player is supported by teaching the player how to move the hand through the figure. Also, whether or not the performance operation is appropriately performed from the image displayed on the display device by photographing the performance state of the performer, that is, the hand performing the performance operation and displaying the photographed hand superimposed on the figure. Or let the performer check.
[0004]
[Patent Document 1]
JP-A-7-36446
[0005]
[Problems to be solved by the invention]
During the performance, the hands are often moved in a complicated manner, and how to move the hands during the performance is not determined. Since the movement method is not determined, even if the performer performs an accurate performance, the photographed hand movement does not necessarily match that of the figure. For this reason, when displaying a photographed hand overlaid with a figure, even if the hand and the figure are displayed in an easily distinguishable form, the difference in performance operations existing between them is recognized. Is usually extremely difficult. Many performers with low performance skills cannot afford to see the display device, and thus cannot expect to recognize the difference. For these reasons, it is considered desirable to be able to confirm the content of the performance performed separately from the performance.
[0006]
The confirmation can be performed, for example, by recording an image of a photographed hand and reproducing it. However, simply reproducing does not always make it easy to determine whether or not the performance operation performed with the photographed hand is appropriate. Also, it may be difficult to accurately recognize the performance operation performed. If the recognition is difficult, it becomes more difficult to determine whether or not the performance operation performed is appropriate. For this reason, when shooting a performer for performance confirmation, it is desirable to reproduce the performance so that it can be easily confirmed whether or not the performance operation actually performed is appropriate. It is done.
[0007]
For example, in a keyboard instrument, it is necessary to be able to accurately grasp the key pressed, the finger used to press the key, and the timing when the key is pressed. It can be said that it is desirable to photograph the hand from above. However, in photographing from above, the movement of the finger when the key is simply pressed or released, and the change of the key accompanying the movement is relatively small. For example, when a so-called “fingering” is performed in which a thumb is passed under another finger and a key is pressed, the finger that has been penetrated is hidden by the other finger. For this reason, it is not always easy to accurately grasp the key pressed and the key pressing timing.
[0008]
An object of the present invention is to reproduce (display) an image obtained by photographing a performer performing a musical instrument so that it can be easily confirmed whether or not the performance operation actually performed is appropriate. To provide technology.
[0009]
[Means for Solving the Problems]
  The recording / reproducing apparatus of the present invention is an apparatus for recording image data of a performer who performed using a musical instrument and reproducing the recorded image data, and event data representing performance operations to be performed on the musical instrument. And performance data storage means for storing the operation timing,When the image data storage means, the mode designation means for designating one of the first and second modes, and the first mode is designated by the mode designation means,The event data stored in the performance data storage means is read based on the corresponding operation timing.WithImage data acquisition means for acquiring in real time image data of a performer performing with the instrument;TheReferring to the event data, it is determined whether or not the performance operation specified as performed by the performer is appropriate, and determination data acquisition means for acquiring determination data representing a determination result in real time, and the image data acquisition means acquire The determined image data and the determination data acquired by the determination data acquisition means are associated with the event data stored in the performance data storage means.In the image data storage meansRecordMakeImage data recordingcontrolMeans,When the second mode is specified by the mode specifying means, the event data stored in the performance data storing means is sequentially read based on the corresponding operation timing, and the read event data The image data and the determination data corresponding to the reading data are read from the image data storage unit, the determination result represented by the determination data read by the reading unit is determined, and the determination result is determined to be appropriate. If the determination result is determined to be inappropriate, the image data corresponding to the determination data is displayed on the display device as it is, and the image data corresponding to the determination data is colored. Display control means for displaying on the display device;It comprises.
[0012]
  The program of the present invention has a function for realizing the performance recording / reproducing apparatus of the present invention.
[0013]
In the present invention, image data of a performer performing using a musical instrument is acquired in real time, a performance operation performed by the performer during the performance, or a performance operation to be performed by the performer is specified in real time, Event data indicating the specified performance operation is associated with image data to be acquired and stored together as a performance record. By associating them in this way, it is possible to present to the performer in an easily understandable manner the performance operation performed by the performer or the performance operation to be performed by the performer when reproducing the image data.
[0014]
When it is determined whether or not the performance operation specified by the performer is appropriate and the determination data indicating the determination result is further stored in association with the image data, the performance operation is determined whether or not the performance operation is appropriate. It is possible to further present it in a form that is easy to understand.
[0015]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
<First Embodiment>
FIG. 1 is a diagram showing a configuration of a performance system constructed using a performance recording apparatus and a performance recording / reproducing apparatus according to the first embodiment.
[0016]
As shown in FIG. 1, the performance system is constructed by connecting the camera 20, the display device 30, and the electronic musical instrument 40 to the performance recording / reproducing apparatus 10 with cables. The performance recording / reproducing apparatus 10 captures and records an image of a user (performer) who plays the electronic musical instrument 40 with the camera 20, and reproduces the recorded image by displaying it on the display device 30. The performance recording apparatus and the performance recording / reproducing apparatus according to the present embodiment are realized by being mounted on the performance recording / reproducing apparatus 10. Hereinafter, the performance recording / reproducing apparatus 10 is abbreviated as “reproducing apparatus”.
[0017]
The electronic musical instrument 40 is provided with a keyboard 41. As shown in FIG. 4, the keys performed by the performer (user) among the keys constituting the keyboard 41, the fingers that performed the keys, and the timing thereof are easy to grasp from the viewpoint from above. For this reason, it is assumed that shooting by the camera 20 is performed on the performer's hand from above.
[0018]
FIG. 2 is a diagram showing a circuit configuration of the reproducing apparatus 10.
As shown in FIG. 2, the playback apparatus 10 includes a CPU 201 that controls the entire apparatus 10, a ROM 202 that stores programs executed by the CPU 201 and various control data, a work RAM 203 that the CPU 201 uses for work, A display controller 204 that displays an image instructed by the CPU 201 on the connected display device 30, a MIDI input / output unit 205 that can input MIDI data from the connected electronic musical instrument 40, and a playback switch (SW) shown in FIG. 11 and a switch (SW) circuit 206 having a practice switch (SW) 12, a sound source system 207 that generates a musical sound according to instructions from the CPU 201, and a camera controller 208 for controlling the connected camera 20. It is configured.
[0019]
The sound source system 207 includes, for example, a sound source that generates waveform data of a musical sound to be generated, a D / A converter that D / A converts the waveform data generated by the sound source into analog audio data, and amplification of audio data output from the D / A converter And a speaker that converts the amplified audio data into sound and emits the sound.
[0020]
The operation in the above configuration will be described.
The practice switch 12 shown in FIG. 1 is for instructing the user to record the performance using the electronic musical instrument 40 for practice, and the other reproduction switch 11 instructs the reproduction of the recorded contents. Is for. Here, the operation will be described focusing on the case where the user operates these switches 11 and 12.
[0021]
First, the operation when the user operates the practice switch 12 will be described.
In the present embodiment, when the switch 12 is operated, performance guidance (guidance) is performed in the form of teaching the key to be pressed via the display device 30. For this purpose, the user is allowed to designate a song to be played before operating the switch 12.
[0022]
The performance guide is performed with reference to music (performance) data representing the performance operation to be performed in the performance of the music and the timing to perform the performance operation. The music data is configured by adding time data indicating the timing for performing the performance operation to each event data (for example, MIDI data) indicating the performance operation (see FIG. 3). Although a detailed description is omitted, a plurality of song data is stored in the ROM 202, and the user selects a song data or causes the MIDI input / output unit 205 to receive the song data, so that the user can select the song data. Data is specified. The reception is performed when a user operates a switch (not shown).
[0023]
When the practice switch 12 is operated, the music data designated by the user is started to be reproduced without generating a musical sound in order to guide the performance. As a result, the key to be pressed and the period during which the key should be pressed are specified, and the user is informed of the key and the period.
[0024]
When the performance is guided as such, the user performs along the guide. For this reason, in the present embodiment, the camera 30 is photographed at the timing when the key should be pressed as a performance operation. The shooting is performed by the CPU 201 instructing the camera 30 to perform shooting via the camera controller 208.
[0025]
One camera 30 shoots according to the instruction and transmits image data obtained by the shooting. The CPU 201 receives the image data via the camera controller 208 and stores it in the area secured in the work RAM 203 (image data area in FIG. 3). The image data as shown in FIG. 4 is obtained in real time by the photographing.
[0026]
The image data is compressed by the camera 30 or when stored in the work RAM 203. However, for convenience of explanation, explanations on compression and decompression are omitted here.
In this way, in this embodiment, the state of the hand is intermittently captured as a still image at the timing when the user should press the key. This is because the total amount of data in a moving image is very large compared to a still image, the change between successive still images is relatively large, and it is easier to grasp the performance operation actually performed. This is because the tone generation start timing of the musical sound has a greater influence on the performance than the end timing. As a result, the performance content can be accurately confirmed while suppressing the amount of data. A moving image may be stored instead of the still image at the time of pressing the key, or the still image at the time of releasing the key may be stored together with the moving image.
[0027]
The stored image data is associated with event data in the song data. Thereby, using the music data, the stored image data can be reproduced in the order of photographing while maintaining the photographing interval. The reproduction is performed by displaying on the display device 30.
[0028]
However, even if the image data is reproduced in this way, it is not always possible to accurately and easily grasp whether or not the performance operation performed by the user during the performance is appropriate. For example, when shooting, the user may move his / her finger to press the key. Therefore, in the present embodiment, determination data indicating whether or not the performance operation performed by the user is appropriate is stored together with the image data. As a performance operation for determining whether or not it is appropriate, only the key depression is performed for the reason described above.
[0029]
The electronic musical instrument 40 transmits MIDI (event) data indicating the contents of the operation to the playback device 10 in real time every time the user performs a performance operation, that is, a key is pressed or released from the keyboard 41. The transmitted MIDI data is received by the MIDI input / output unit 205 mounted on the playback apparatus 10. From this, it is determined whether or not the performance operation performed by the user is appropriate. The performance operation to be performed specified from the music data being reproduced matches the performance operation indicated by the MIDI data received by the MIDI input / output unit 205. It is done by confirming whether or not to do.
[0030]
FIG. 3 is a diagram for explaining data stored in the work RAM 203.
The song data area in FIG. 3 is an area reserved for storing song data of a song for performing performance guidance by operating the practice switch 12. As described above, the music data is configured in a form in which data pairs including time data and event data are arranged in the reproduction order. The song data is stored in such a manner that it is overwritten every time the user designates a song.
[0031]
The image data area is an area reserved for storing image data transmitted from the camera 20. In the area, a sub-area for storing each image data (area indicating “(1)” in FIG. 3) is secured. For the sub-area for storing image data, the data stored in the sub-area indicates the result of the performance operation. Hereinafter, for convenience, it will be referred to as a “result data area”.
[0032]
As a result, in addition to the image data, data represented as “NGF” and “AD” in FIG. 3 are stored in the data area. NGF is determination data indicating whether or not the performance operation is properly performed, and AD is a timing to be performed at the time of photographing when the image data is obtained in the event data constituting the song data stored in the song data area. This is data indicating the event data. As the data, a value indicating an address where the event data is stored is stored. Here, the data indicating the location of the result data area is also referred to as “address”. Since NGF and AD are treated as variables, they will be denoted with variables.
[0033]
When the user operates the practice switch 12, the CPU 201 reproduces the music data stored in the music data area to perform performance guidance, and sequentially changes the result data area for storing data in the image data area, Image data obtained by causing the camera 20 to shoot, and further variables NGF and AD are stored in the result data area. In this way, the performance performed by the user is recorded.
[0034]
Next, an operation when the user operates the playback switch 11 will be described.
When the switch 11 is operated, the CPU 201 starts reproduction of the song data stored in the song data area, and reproduction (display) of the image data stored in the data area is ensured in the image data area. The event data associated with the variable AD stored in the area is processed at the timing of processing. Thereby, the reproduction of the image data is performed in accordance with the progress of the reproduction of the music data as in the case of shooting.
[0035]
Shooting by the camera 20 is performed at the timing when the key should be pressed. For this reason, if it can be recognized from the displayed image data that the key to be pressed is being pressed or if the key is to be pressed, it can be confirmed that the key is being pressed appropriately. . Otherwise, that is, if you can recognize that you are pressing a key that should not be pressed, or that you are not trying to press a key that should be pressed, press the key that should be pressed appropriately. It can be confirmed that it is not.
[0036]
However, in the display of still image data, the actual situation is that the forward / backward movement of the hand is difficult to understand. Therefore, in the present embodiment, the value of the variable NGF is reflected in the display of the image data. The reflection is performed, for example, by changing the display color of the key to be pressed (see FIG. 4) specified in the associated event data according to the value of the variable NGF. Thereby, it is possible to easily confirm whether or not an appropriate key depression has been performed.
[0037]
During the performance, the position of the electronic musical instrument 40 usually does not move. For this reason, such reflection is performed by, for example, performing photographing for the keyboard 41 in advance, extracting the range of each key in the photographed image by known image processing, and within the extracted range. This can be done by rewriting the value of the pixel data within the range corresponding to the event data. The image data is displayed on the display device 30 by sending image data to be displayed to the display controller 204.
[0038]
Note that another method may be used to reflect the value of the variable NGF in the image data. For example, it may be performed by changing the display color of a place other than a hand, a finger, or a key, or may be performed by displaying a graphic or a message corresponding to the value.
[0039]
The CPU 201 reads the image data associated with the event data indicating the key depression from the image data area every time it processes the event data indicating the key depression, and performs an operation corresponding to the value of the variable NGF stored together with the image data. And output to the display controller 204. In this way, the image data stored in the image data area is reproduced.
[0040]
Next, the operation of the playback apparatus 10 will be described in detail with reference to various flowcharts shown in FIGS.
FIG. 5 is a flowchart of the entire process. First, the entire process will be described in detail with reference to FIG. The overall processing is realized by the CPU 201 executing a program stored in the ROM 202, for example.
[0041]
When the power is turned on, first, in step 501, the apparatus 10 is initialized and set to a predetermined state. In the next step 502, switch information (SW) for receiving the operation information indicating the operation contents performed on the various switches including the reproduction switch 11 and the practice switch 12 from the switch circuit 206 and corresponding to the performed operation. Execute the process. Thereafter, the process proceeds to step 503.
[0042]
In step 503, a keyboard process for responding to a performance operation performed on the keyboard 41 of the electronic musical instrument 40 is executed. In the subsequent step 504, a practice process for responding to an operation on the practice switch 12 is executed. In the next step 505, a reproduction process for responding to an operation on the reproduction switch 11 is executed. In the next step 506, other processing for responding to output of MIDI data from the MIDI input / output unit 205 is executed. After the execution, the process returns to step 502. Thereby, the processing loop formed in steps 502 to 506 is repeatedly executed until the power is turned off.
[0043]
Hereinafter, a subroutine process executed in the overall process will be described in detail.
FIG. 6 is a flowchart of the switch process executed as step 502 described above.
[0044]
In the switch process, as shown in FIG. 6, first, a practice switch process for responding to the operation on the practice switch 12 is executed in step 601, and in order to respond to the operation on the playback switch 11 in the next step 602. The reproduction switch process is executed, and in the next step 603, the other switch process for responding to the operation of the other switch is executed, and then the series of processes is terminated. Here, the subroutine processing executed as steps 601 and 602 will be described in detail with reference to the flowcharts shown in FIGS.
[0045]
As described above, operation information indicating the contents of operations performed on various switches is received from the switch circuit 206. It is determined from the operation information whether or not the switch of interest has been operated. Saving the song data designated by the user in the song data area is realized by executing other switch processing in step 603.
[0046]
FIG. 7 is a flowchart of the regeneration switch process executed as step 602 above. In the subroutine process executed in the switch process, the playback switch process will be described in detail with reference to FIG.
First, in step 701, it is determined whether or not the regeneration switch 11 has been operated (turned on). If the operation information received from the switch circuit 206 indicates that the switch 11 has been operated, the determination is yes and the process moves to step 702; otherwise, the determination is no. Here, a series of processing ends.
[0047]
In step 702, a value of 1 indicating that playback is in progress is assigned to the playback management variable PLAYF. In the subsequent step 703, a value indicating the head address of the music data area (see FIG. 3) is substituted into the variable MAD. Thereafter, data (time data) is read from the address specified by the value of the variable MAD and substituted into the variable TIME (step 704), and the value of the address specifying the result data area positioned at the head in the image data area is set. Substituting into the variable GAD (step 705), canceling the prohibition of execution of the timer interrupt process (step 706), and ending the series of processes.
[0048]
The time data constituting the music data expresses time with a predetermined resolution. The timer interrupt process is a process executed to count the time represented by the time data with the resolution. In order to measure the time, the minimum time that can be expressed with the resolution (hereinafter referred to as “minimum unit time”) is executed as a cycle. To cancel the prohibition of execution, the interrupt signal generated in the cycle is made valid and executed by generation of the interrupt signal.
[0049]
FIG. 8 is a flowchart of the practice switch process executed as step 601 in the switch process shown in FIG. Next, the practice switch process will be described in detail with reference to FIG.
First, in step 801, it is determined whether or not the practice switch 12 has been operated (turned on). If the operation information received from the switch circuit 206 indicates that the switch 12 has been operated, the determination is yes and the process moves to step 802. Otherwise, the determination is no. Here, a series of processing ends.
[0050]
In step 802, 1 is assigned to the practice management variable STF, indicating a value of practice. In the subsequent step 803, a value indicating the head address of the music data area (see FIG. 3) is substituted into the variable AD. Thereafter, data (time data) is read from the address specified by the value of the variable AD and substituted into the variable TIME (step 804), and initialization is performed by clearing the data stored in the image data area (step 804). 805) After substituting the value of the address designating the result data area located at the head in the area into the variable GAD (step 806), and further canceling the prohibition of execution of the timer interrupt processing (step 807), The process ends.
[0051]
By executing the practice switch process in step 601, preparation for saving image data and the like as a performance record performed by the user is performed, and by performing the playback switch process in the other step 602, the performance recording is performed. As a result, preparation for reproducing the image data stored as is performed.
[0052]
Returning to the description of the subroutine processing executed in the overall processing shown in FIG.
FIG. 9 is a flowchart of the keyboard process executed as step 503 in the overall process. Next, the keyboard process will be described in detail with reference to FIG. Here, for convenience of explanation, the flowchart is shown on the assumption that one tone is emitted at the same time. The same applies to flowcharts of subroutine processing described below.
[0053]
The electronic musical instrument 40 transmits MIDI (event) data indicating a performance operation performed by the user to the keyboard 41 in real time, and the playback apparatus 10 receives the MIDI data at the MIDI input / output unit 205. For this reason, in the keyboard process, a process for the MIDI data received by the input / output unit 205 is performed. The input / output unit 205 stores the received MIDI data in an area secured in the work RAM 203, for example.
[0054]
First, in step 901, the MIDI data received by the MIDI input / output unit 205 is read and acquired from the area. In the subsequent step 902, a performance operation indicated by the MIDI data, that is, a key state change is determined. If the MIDI data cannot be acquired, it is determined that the state has not changed, and the series of processing ends here. If the acquired MIDI data is a note-off message indicating the key release (OFF), it is determined so, and the process proceeds to step 903 where the key number indicating the pitch of the MIDI data is set in the variable KEY. Further, after the MIDI data is sent to the sound source system 207 in step 904, a series of processing is terminated. If the acquired MIDI data is a note-on message indicating key depression (ON), it is determined to that effect and the process proceeds to step 905.
[0055]
In step 905, the key number indicating the pitch of the MIDI data is substituted into the variable KEY, and in the subsequent step 906, the MIDI data is sent to the sound source system 207. In step 907 which moves to thereafter, it is determined whether or not the value of the variable STF is 1, that is, whether or not the practice is started by the operation of the practice switch 12. If so, the determination is yes and the process moves to step 908. If not, the determination is no and the series of processes ends here.
[0056]
In the present embodiment, the variable NGF is “0” when the key to be pressed is pressed at the timing to press the key, and “1” when the key that should not be pressed is pressed at the timing to press the key. “2” is substituted when the key is depressed at a timing when the key should not be depressed, and “3” is substituted when there is a key to be depressed but the key is not depressed. As a result, in addition to whether or not the key was pressed properly, that is, whether or not the key to be pressed was performed at the timing to be pressed, why the key pressing that could not be performed properly is not appropriate. I'm trying to understand.
[0057]
Among the values assigned to the variable NGF, “0” to “2” are values that can be specified by the user's key depression. In step 908 and subsequent steps, processing for specifying a value to be substituted into the variable NGF and storing the value is performed.
First, in step 908, it is determined whether or not the value of the variable ONF is 1. The variable ONF is a variable prepared for determining the timing at which the key should be pressed, and 1 assigned to it indicates that it is the timing, and 0 indicates that it is not the timing. Therefore, if it is not the timing, the determination is NO and the process proceeds to step 913. Otherwise, the determination is YES and the process proceeds to step 909.
[0058]
In step 909, it is determined whether or not the key number of the variable NOTE to which the key number of the key to be pressed is assigned is not equal to that assigned to the variable KEY. If the user presses the key to be pressed, they match, so the determination is no, the process moves to step 910, and is designated by the value of the variable GAD in the result data area in the image data area. In the result data area, a variable NGF having a value of 0 is written. Further, in step 912, 1 is substituted into the variable OKF, and then a series of processing is terminated. If not, that is, if the user presses a key that should not be pressed, since they do not match, the determination is YES, the process moves to step 911, and is designated by the value of the variable GAD. After the variable NGF having a value of 1 is written in the result data area, the process proceeds to step 912.
[0059]
The variable OKF is a variable prepared for determining whether or not the key has been pressed after the key should be pressed, and 1 assigned to it indicates that the key has been pressed. , 0 indicates that the key was not pressed. Here, since the key was pressed, 1 is substituted for it.
[0060]
In step 913, where the determination in step 908 is NO and the process proceeds to step 913, the variable NGF having a value of 2 is written into the result data area designated by the value of the variable GAD in the result data area in the image data area. Thereafter, the process proceeds to step 912, and 1 is assigned to the variable OKF.
[0061]
Next, the practice process executed as step 504 in the overall process shown in FIG. 5 will be described in detail with reference to the flowcharts shown in FIGS. 10 and 11. As described above, the flowchart is based on the assumption that the musical sound to be generated simultaneously is one sound.
[0062]
First, in step 1001, it is determined whether the value of the variable STF is 1. During practice, that is, when performing a performance guide, since 1 is assigned to it, the determination is YES, and the routine proceeds to step 1002. Otherwise, the determination is no and the series of processing ends here.
[0063]
In step 1002, it is determined whether the minimum unit time has elapsed. If the value of the variable passed as an argument from the timer interrupt process executed to measure the time does not indicate the progress, the determination is no and the series of processes ends here. Otherwise, the determination is yes, and after decrementing the value of the variable TIME at step 1003, the routine proceeds to step 1004.
[0064]
In step 1004, it is determined whether or not the value of the variable TIME is 0 or less. When the time data assigned to it becomes the processing timing of the event data indicating the processing timing, the value is 0 or less, so the determination is YES and the process proceeds to step 1005. Otherwise, the determination is It becomes NO and a series of processes are complete | finished.
[0065]
In step 1005, the value of the variable AD is incremented. In the next step 1006, it is determined whether or not the value of the variable AD is equal to or less than the value of the address (final address) where the data located at the end of the music data is stored. If the reproduction of the song data has not ended, the value of the variable AD is equal to or less than the value of the final address, so the determination is YES and the process moves to step 1009. Otherwise, the determination is no, next, in step 1007, 0 is substituted into the variable STF, and further, in step 1008, the execution of the timer interrupt process is prohibited, and then the series of processes ends.
[0066]
In step 1009, data is read from the address specified by the value of the variable AD. In the next step 1010, the type of the data is determined. If the data is time data, it is determined so and the process proceeds to step 1011. After the data is substituted into the variable TIME, a series of processing ends. If the data is event (MIDI) data, the fact is determined and the process proceeds to step 1012 in FIG.
[0067]
In step 1012, it is determined whether or not the event (performance operation) indicated by the event data is a key depression (note on). If the event is not a key depression, the determination is no and the process proceeds to step 1019. If not, the determination is yes and the process proceeds to step 1013.
[0068]
In step 1013, the user is instructed to press the key to be pressed, which is indicated by the key number in the event data. Various methods can be used as the instruction method. For example, a keyboard figure representing the keyboard 41 is displayed on the display device 30, and the display color of the key to be pressed in the figure is changed. It is done by making it different from the key.
[0069]
In step 1014 following step 1013, 1 is substituted for variable ONF and 0 is substituted for variable OKF. In the next step 1015, the keyboard number indicated by the event data is assigned to the variable NOTE. Thereafter, the camera 20 is instructed to take a picture via the camera controller 208, and the controller 208 waits for the controller 208 to receive the image data transmitted by the instruction (step 1016), and the image data received by the controller 208 is received. Is stored in the result data area specified by the value of the variable GAD (step 1017), and the value of the variable AD (address value at which event data indicating key press (note-on) is stored) is further stored in the data area. Then (step 1018), the process returns to step 1005 in FIG.
[0070]
In step 1019 where the determination in step 1012 is NO and the process proceeds to step 1019, the display color is changed to another key to indicate that the key should be released in order to instruct the user to release the key to be released indicated by the key number in the event data. Change the display color of the key that was different from the previous one to the original display color. Thereafter, 0 is substituted into the variable ONF in step 1020, and then the process proceeds to step 1021.
[0071]
In step 1021, it is determined whether or not the value of the variable OKF is zero. If the user does not press the key before it is time to release the key, 0 is substituted for it, so the determination is YES, and the result data area specified by the value of the variable GAD is set to 3 in step 1022. After writing the variable NGF of the value of, return to step 1005 in FIG. Otherwise, the determination is no, 0 is substituted for variable OKF in step 1023, and the process returns to step 1005.
[0072]
In this way, in the result data area in which the image data is stored, any value from 0 to 3 is stored as the value of the variable NGF. Thereby, the user can easily understand the reason why the key depression becomes inappropriate.
12 and 13 are flowcharts of the reproduction process executed as step 505 in the overall process shown in FIG. Finally, the reproduction process will be described in detail with reference to FIGS.
[0073]
First, in step 1201, it is determined whether or not the value of the variable PLAYF is 1. During reproduction, that is, when displaying the image data stored in the image data area, since 1 is substituted for it, the determination is YES, and the routine proceeds to step 1202. Otherwise, the determination is no and the series of processing ends here.
[0074]
The processing contents of steps 1202 to 1211 are basically the same as those of steps 1002 to 1011 shown in FIG. 10 except that the variable name for address management for reading data from the song data area is different. Therefore, their description is omitted. The process proceeds to step 1212 in FIG. 13 when the data read in step 1209 is determined as event data in step 1210.
[0075]
In step 1212, it is determined whether or not the event (performance operation) indicated by the event data is a key press (note-on). If the event is not a key depression, the determination is NO, and then the event data is sent to the sound source system 207 in step 1013, and the process returns to step 1205 in FIG. Otherwise, the determination is yes, and the event data is sent to the sound source system 207 in step 1214, and then the process proceeds to step 1215.
[0076]
Thus, in this embodiment, the music data is also reproduced in parallel with the reproduction of the image data. As a result, the user can check the performance contents using visual and auditory senses.
That hearing makes it possible to easily recognize the appropriate sound generation start timing of the musical sound and the pitch of the musical sound. As a result, it is possible to compare those recognized from the auditory sense with those recognized from the visual sense, so the user can more easily confirm the timing of the key press performed by the user and whether or not the key pressed is appropriate. become able to.
[0077]
In step 1215, the value of the address designating the result data area located at the head in the image data area is substituted for the variable GAD. In the following step 1216, the value of the variable AD is read from the result data area specified by the value of the variable GAD. In the next step 1217, it is determined whether or not the value of the variable MAD matches the value of the variable AD. If the variable AD is read from the result data area in which the image data associated with the event data sent to the sound source system 207 is stored in step 1214, they match, so the determination is yes and the process moves to step 1219. To do. Otherwise, the determination is no, and in step 1218, the value of the variable GAD is incremented, for example, and then the process returns to step 1216. Accordingly, the processing loop formed in steps 1216 to 1218 is repeatedly executed until the result data area in which the image data associated with the event data transmitted to the sound source system 207 is stored is specified.
[0078]
In step 1219, image data is read from the result data area specified by the value of the variable GAD. In the next step 1220, the variable NGF is read from the data area as a result. In step 1221 which moves to after that, it is determined whether or not the value of the variable NGF is zero. If the value is 0, that is, if the user appropriately presses the key indicated by the event data, the determination is YES, and the image data read in step 1219 is sent to the display controller 204 as it is and sent to the display device 30. After the display, the process returns to step 1205 in FIG. Otherwise, the determination is no and the process moves to step 1223.
[0079]
In step 1223, the value of the variable NGF is determined. If the value of the variable NGF is 1, it is determined so and the process proceeds to step 1224, where the read image data is colored with the display color A. Similarly, if the value is 2, the image data is colored with the display color B in step 1225 to which the determination is made, and when the value is 3, In step 1226 where the determination is made and the process proceeds, the image data is colored with the display color C. In step 1227, the process proceeds after coloring in any one of the display colors A to C, the image data after the coloring is sent to the display controller 204 and displayed on the display device 30. Thereafter, the process returns to step 1205 in FIG. As described above, the coloring is performed on the key to be pressed (see FIG. 4) specified from the associated event data.
<Second Embodiment>
In the first embodiment, image data is reproduced (displayed) in accordance with the reproduction of music data. On the other hand, in the second embodiment, the reproduction of the image data is performed in accordance with a user instruction.
[0080]
The configuration of the playback device in the second embodiment is basically the same as that in the first embodiment. The operation is basically the same for the most part. For this reason, only the parts different from the first embodiment will be described while using the reference numerals in the description of the first embodiment as they are.
[0081]
In the second embodiment, as in the first embodiment, when the user operates the playback switch 11 shown in FIG. 1 in a state where the image data is not displayed, the display (playback) of the image data is performed. Start. However, after that, every time the user operates the switch 11, the image data to be displayed is sequentially switched and displayed from the top. As a result, it is possible to allow the user to check and study performance operations over time.
[0082]
Since the image data is reproduced as described above, the reproduction process is not executed as step 505 in the entire process shown in FIG. In the switch process executed as step 502, the regeneration switch process (see FIG. 6) executed as step 602 is greatly different from that of the first embodiment. Others are basically the same. For this reason, only the regeneration switch process executed in the second embodiment will be described in detail with reference to the flowcharts shown in FIGS.
[0083]
First, in step 1401, it is determined whether or not the regeneration switch 11 has been operated (turned on). If the operation information received from the switch circuit 206 indicates that the switch 11 has been operated, the determination is yes and the process moves to step 1402, otherwise the determination is no. Here, a series of processing ends.
[0084]
In step 1402, it is determined whether or not the value of the variable PLAYF is zero. As in the first embodiment, 1 is assigned to the image data during reproduction. From this, if the image data is being reproduced, the determination is no and the process proceeds to step 1407 in FIG. 15, and if not, the determination is yes and the process proceeds to step 1403.
[0085]
In step 1403, the value of the address designating the result data area located at the head in the image data area is substituted into the variable GAD. Thereafter, 1 is assigned to the variable PLAYF (step 1404), the image data is read from the result data area specified by the value of the variable GAD (step 1405), and the read image data is sent to the display controller 204 for display. After being displayed on the device 30 (step 1406), a series of processing is terminated.
[0086]
Each value of variables NGF and AD is also stored in the result data area. From this, the image data may be displayed with the display color of the key specified from the value of the variable AD in the display color corresponding to the value of the variable NGF, as in the first embodiment.
The determination NO in step 1402 means that the user has operated the playback switch 11 to switch the image data to be displayed. For this reason, in step 1407 and subsequent steps in FIG. 15 in which the determination is NO and the process proceeds, processing for switching is executed.
[0087]
In step 1407, the value of the variable GAD is incremented. In the subsequent step 1408, it is determined whether or not the value of the variable GAD is equal to or less than a value for designating the result data area storing the last image data. If all the image data is not displayed, the value of the variable GAD becomes equal to or less than the value that designates the data area as a result. Therefore, the determination is YES, and the process proceeds to step 1405 in FIG. Thereby, the image data to be displayed next is displayed. On the other hand, if this is not the case, that is, if the display of all the image data has been completed, the determination is no, 0 is substituted into the variable PLAYF (step 1409), and the image displayed on the display device 30 is erased. After that (step 1410), the series of processing is terminated.
<Third Embodiment>
In the first and second embodiments, all image data is reproduced (displayed). On the other hand, in the third embodiment, reproduction is performed by automatically setting a range to be reproduced among all image data.
[0088]
The configuration of the playback device in the third embodiment is basically the same as that in the first embodiment. The operation is basically the same for the most part. For this reason, like the second embodiment, only the parts different from the first embodiment will be described while using the reference numerals in the description of the first embodiment as they are.
[0089]
In confirming the performance operation, it can be said that it is more important to confirm where and how the inappropriate performance operation was performed, rather than confirming the appropriate performance operation. For this reason, in the third embodiment, the range from the first inappropriate performance operation (here, key pressing) to the last inappropriate performance operation is set as a playback target range, and shooting is performed within that range. Only the image data obtained by the above is reproduced from the beginning of the range. This makes it easier for the user to perform confirmation and examination focusing on inappropriate performance operations.
[0090]
Since the image data is reproduced as described above, in the entire process shown in FIG. 5, the switch process executed as step 502, more specifically, the playback switch process executed as step 602 in the switch process (FIG. 5). 6) and the reproduction processing executed as step 505 is different from that of the first embodiment. Others are basically the same. Therefore, only those subroutine processes will be described.
[0091]
16 and 17 are flowcharts of the regeneration switch process according to the third embodiment. First, the switch process will be described in detail with reference to FIGS. 16 and 17. In the switch process, a process for setting a range to be reproduced is added from the first embodiment.
[0092]
First, in step 1601, it is determined whether or not the regeneration switch 11 has been operated (turned on). If the operation information received from the switch circuit 206 indicates that an operation has been performed on the switch 11, the determination is yes and the process moves to step 1602, otherwise the determination is no. Here, a series of processing ends.
[0093]
In step 1602, 1 is substituted for the variable PLAYF, and 0 is substituted for the variables SF, SAD, and EAD. After the substitution, the value of the address designating the result data area located at the head in the image data area is substituted into the variable GAD (step 1603), and the variable NGF is read from the result data area designated by the value of the variable GAD. (Step 1604), the process proceeds to Step 1605. The variable SF is a variable for determining whether or not there is an inappropriate key press, and the variables SAD and EAD are variables for setting a range.
[0094]
In step 1605, whether or not the value of the read variable NGF (denoted as “NGF (GAD)” in the figure) is not 0, that is, the key depression at the time of shooting from which the image data stored therein is obtained is appropriate. It is determined whether it is not performed. If the key is properly pressed, the value of the variable NGF is 0. Therefore, the determination is no and the process proceeds to step 1610. Otherwise, the determination is yes and the process proceeds to step 1606. To do.
[0095]
In step 1606, it is determined whether or not the value of the variable SF is zero. 0 assigned as an initial value is a value indicating that improper key pressing is not performed. From this, if the value of the variable NGF read immediately before is a value other than 0, 0 is assigned to the variable SF, so the determination is YES, and in the next step 1607, the variable GAD is set to the variable SAD. After the value and 1 are assigned to the variable SF in the next step 1608, the process proceeds to step 1610. Otherwise, the determination is no, and in the next step 1609, the value of the variable GAD is assigned to the variable EAD, and then the process proceeds to step 1610.
[0096]
In this way, the value of the variable GAD when the variable NGF whose value is not 0 is first read is assigned to the variable SAD. After the assignment, the value of the variable GAD is changed each time the variable NGF whose value is not 0 is read. It is assigned to the variable EAD. As a result, variables SAD and EAD are substituted with values indicating the result data area in which image data obtained by photographing when inappropriate key pressing is performed first and last, respectively. .
[0097]
In step 1610, the value of the variable GAD is incremented. In the next step 1611, it is determined whether or not the value of the variable GAD is equal to or less than the value for designating the result data area in which the last image data is stored. If the value of the variable GAD is larger than that value, that is, if the confirmation of the values of all the variables NGF stored in the result data area is completed, the determination is no and the process proceeds to step 1612 in FIG. Otherwise, the determination is yes and the process returns to step 1604. Thus, the reproduction range focusing on the image data is specified until the focused result data area disappears.
[0098]
In step 1612 of FIG. 17, it is determined whether or not the value of the variable SAD is equal to the value of the variable EAD. If the value of the variable NGF is other than 0, any one of those values to which the same initial value is substituted is updated to a value other than the initial value, so that the values do not match. From this, when inappropriate key pressing has never been performed, the values are equal and the determination is YES, and after substituting 0 for the variable PLAYF in step 1613, the series of processing is terminated. Otherwise, the determination is no and the process moves to step 1614.
[0099]
By substituting 0 for the variable PLAYF, the image data is not reproduced, and the image data reproduction instruction made by the user is invalidated. In the third embodiment, such invalidation is used to notify the user that improper key pressing has never been performed instead of reproduction of image data.
[0100]
If reproduction is started from image data when an inappropriate key is pressed, the user cannot grasp the performance flow leading to the reproduction. Therefore, in the third embodiment, before and after the range specified as described above is reproduced according to the situation. The processing after step 1614 is performed in order to set the range including the front and back as a reproduction target. Since the reproduction of the image data is performed together with the reproduction of the song data, the range is set in such a manner that a reproduction range is set in the song data.
[0101]
First, in step 1614, it is determined whether or not the value obtained by subtracting the predetermined value α from the value of the variable SAD is smaller than the value indicating the result data area located at the head in the image data area. The predetermined value α is a value set in order to perform a performance of a length that seems necessary for grasping the flow. For this reason, when the key depression to be performed in the first part of the performance cannot be properly performed, the former value becomes smaller than the latter value, and the determination becomes YES. Next, in step 1615, the variable SAD is set to the latter value. After the value is substituted, the process proceeds to step 1617. Otherwise, the determination is no, and after the former value is substituted into the variable SAD in step 1616, the process proceeds to step 1617.
[0102]
In step 1617, the variable AD is read from the result data area specified by the value of the variable SAD, and in the subsequent step 1618, the value obtained by decrementing the value of the variable AD is substituted for the variable MAD. As a result, a value that designates data for starting reproduction in the music data is substituted into the variable MAD and stored so that a performance having a length that can grasp the flow is performed immediately before.
[0103]
In steps 1619 to 1623 following step 1618, a process for substituting a value for designating data for ending reproduction in the song data into the variable EAD so as to perform the performance having a length capable of grasping the flow last. Done. The processing of each of these steps is not only performed in the same flow as the above steps 1614 to 1618, but the processing content is basically determined based on the same concept. Therefore, the description thereof is omitted.
[0104]
In step 1624, which moves after executing the processing of step 1623, the data designated by the value of the variable MAD is read from the music data area (see FIG. 3) and substituted into the variable TIME. In the following step 1625, a value specifying the result data area located at the head in the image data area is substituted for the variable GAD. In subsequent step 1626, the prohibition of execution of the timer interrupt process is canceled. A series of processing ends after the execution.
[0105]
In this way, the range for reproducing the image data is set according to the range for reproducing the music data. The variables MAD and EAD into which the value indicating the range is substituted are transferred to a reproduction process (see FIG. 12) executed as step 505 in the overall process shown in FIG. Thereby, in the third embodiment, the variable EAD is newly transferred to the reproduction process.
[0106]
In the reproduction process, the variable MAD is used for managing data to be read from the music data, and there is no need to change the processing content in accordance with the value substituted for it. A variable EAD is substituted with a value indicating data to end reproduction in the music data. For this reason, in the third embodiment, in step 1206, it is changed to determine whether or not the value of the variable MAD is equal to or less than the value of the variable EAD. Other than that is basically the same as in the first embodiment, and the migration destination according to the determination result is not changed.
[0107]
In the first to third embodiments, the image data is reproduced by different methods. However, the reproduction may be performed by a plurality of methods. At the time of reproduction, a symbol or graphic indicating the pitch of the key may be displayed in order to specifically present the key to be pressed to the user. The shooting timing is the timing at which the key should be pressed specified from the song data, but may be the timing at which the user actually pressed the key.
[0108]
In the present embodiment (first to third embodiments), image data reflecting event data to be associated in song data is saved by shooting at the timing of key depression specified from song data. is doing. However, the event data associated with the image data may indicate a performance operation performed by the user, or may be associated with it.
[0109]
In order to associate with event data indicating a performance operation performed by the user, it is desirable to store data representing the performance operation performed by the user, for example, in the form of music data. By storing such data, not only can the image data captured and stored at the time of the actual key press be displayed at an appropriate timing, but the pitch of the key pressed can be displayed as a symbol or figure. It becomes possible to present by displaying. As a result, the performance content can be confirmed in a more desirable manner for the user.
[0110]
The present embodiment is a case where the performance recording device and the performance recording / reproducing device are mounted on one device 10, but they may be realized as separate devices. In such a case, the recorded data may be exchanged via a recording medium or a communication network.
[0111]
The program for realizing the operation of the performance recording apparatus or the performance recording / reproducing apparatus as described above may be distributed by being recorded on a recording medium such as a CD-ROM, DVD, or magneto-optical disk. Alternatively, part or all of the program may be distributed via a transmission medium used in a public network or the like. In such a case, the user can apply the present invention to the data processing apparatus by acquiring the program and loading it into a data processing apparatus such as a computer. Therefore, the recording medium may be accessible by a device that distributes the program.
[0112]
【The invention's effect】
As described above, the present invention obtains image data of a performer performing using a musical instrument in real time, and a performance operation performed by the performer during the performance or a performance operation to be performed by the performer. Is specified in real time, and event data indicating the specified performance operation is associated with the acquired image data and stored together as a performance record. Since they are associated with each other in this manner, the performance operation performed by the performer or the performance operation to be performed by the performer can be presented to the performer in an easily understandable manner during the reproduction of the image data.
[0113]
If it is determined whether or not the performance operation specified by the performer is appropriate, and the determination data indicating the determination result is further stored in association with the image data, whether or not the performance operation is appropriate Can be presented in a form that is easy to understand.
[0114]
Any of these presentations makes it easier to confirm whether or not the performance operation actually performed during the performance is appropriate. For this reason, the performer can easily perform the confirmation.
[Brief description of the drawings]
FIG. 1 is a diagram showing a configuration of a performance system constructed using a performance recording apparatus and a performance recording / reproducing apparatus according to a first embodiment.
FIG. 2 is a diagram showing a circuit configuration of a performance recording / reproducing apparatus equipped with a performance recording apparatus and a performance recording / reproducing apparatus according to the first embodiment;
FIG. 3 is a diagram illustrating data stored in a work RAM.
FIG. 4 is a diagram illustrating an image captured by a camera.
FIG. 5 is a flowchart of overall processing.
FIG. 6 is a flowchart of switch processing.
FIG. 7 is a flowchart of a regeneration switch process.
FIG. 8 is a flowchart of a practice switch process.
FIG. 9 is a flowchart of keyboard processing.
FIG. 10 is a flowchart of practice processing.
FIG. 11 is a flowchart of practice processing (continued).
FIG. 12 is a flowchart of a reproduction process.
FIG. 13 is a flowchart of playback processing (continued).
FIG. 14 is a flowchart of a regeneration switch process (second embodiment).
FIG. 15 is a flowchart of regeneration switch processing (second embodiment: continued);
FIG. 16 is a flowchart of a regeneration switch process (third embodiment).
FIG. 17 is a flowchart of playback switch processing (third embodiment: continued);
[Explanation of symbols]
10 Performance recording and playback device
11 Playback switch
12 Practice switch
20 cameras
30 Display device
40 Electronic musical instruments
41 keyboard
201 CPU
202 ROM
203 Work RAM
204 Display controller
205 MIDI input / output unit
206 Switch circuit
208 Camera controller

Claims (2)

An apparatus for recording image data of a performer who performed using a musical instrument and reproducing the recorded image data,
Event data representing performance operations to be performed on the musical instrument and performance data storage means for storing the operation timing;
Image data storage means;
Mode designation means for designating one of the first and second modes;
When the first mode is designated by the mode designating means, the event data Kioku the performance data SL billion unit reads out, based on the corresponding operation timing performed play using the instrument Image data acquisition means for acquiring the image data of the performer in real time;
Referring to event data to which the read, together with the player's performance operation identified and was performed to determine appropriate whether the determination data acquisition unit for acquiring determination data representing the judgment result in real time,
Image data to be recorded in the image data storage means in association with the event data stored in the performance data storage means, the image data acquired by the image data acquisition means and the determination data acquired by the determination data acquisition means Recording control means;
When the second mode is specified by the mode specifying means, the event data stored in the performance data storing means is sequentially read based on the corresponding operation timing, and the read event data Reading means for reading out the image data and determination data corresponding to the image data storage means;
While determining the determination result represented by the determination data read by the reading unit, and when it is determined that the determination result represents appropriate, the image data corresponding to the determination data is displayed on the display device as it is, If it is determined that the determination result represents inappropriate, display control means for coloring the image data corresponding to the determination data and displaying on the display device;
A performance recording / reproducing apparatus comprising: Performance data storage means for storing event data representing performance operations to be performed on the musical instrument and operation timing thereof, image data storage means, and mode specification means for specifying one of the first and second modes the a, a program executed by a computer used the image data recorded and recorded image data of the player that was played using the instrument as a device for reproducing,
When the first mode is designated by the mode designating means, the event data Kioku the performance data SL billion unit reads out, based on the corresponding operation timing performed play using the instrument An image data acquisition function that acquires the image data of the performer in real time;
Referring to event data to which the read, together with the player's performance operation identified and was performed to determine appropriate whether the determination data acquisition function of acquiring judgment data representing the judgment result in real time,
An image data recording control function for causing the image data storage means to record the acquired image data and determination data in association with the event data stored in the performance data storage means ;
When the second mode is specified by the mode specifying means, the event data stored in the performance data storing means is sequentially read based on the corresponding operation timing, and the read event data A read function for reading the image data and determination data corresponding to the image data from the image data storage means;
The determination result represented by the read determination data is determined, and if it is determined that the determination result indicates appropriate, the image data corresponding to the determination data is displayed as it is on the display device, and the determination result is not valid. When it is determined that it represents appropriate, a display control function for coloring the image data corresponding to the determination data and displaying it on the display device;
A program to realize

Images