JP2024046351A - PROGRAM, INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING METHOD, AND PERFORMANCE DATA PROCESSING SYSTEM - Google Patents

Abstract

[Problem] To make it possible to present images that match a musical performance with high accuracy. [Solution] The program of the present invention causes a computer to display on a display device images that match a musical piece corresponding to the musical performance, identified based on musical performance data generated in conjunction with the musical performance, and repeatedly derives the degree of match between the musical performance and the identified musical piece based on the time series of musical performance data generated in conjunction with successive musical performances, and causes the computer to re-search for a musical piece corresponding to the musical performance if the degree of match does not satisfy a criterion. [Selected Figure] Figure 5

Description

The present invention relates to a program, an information processing device, an information processing method, and a performance data processing system.

An image display device has been proposed that has the function of recognizing a piece of music that a user starts to play on an instrument or sing, and automatically displaying an image (video or still image) related to the recognized piece of music (see Patent Document 1).

There are many people who buy a musical instrument but give up halfway through because they find it difficult to practice. Techniques that visualize music performances and utilize visual effects are attracting attention in order to increase the motivation for practicing among people who are just starting to enjoy music.

Patent No. 4534926

In particular, with technology that displays images that match music with lyrics, it is difficult to select and present an appropriate image. For example, there are cases where it is preferable to change the image that matches the performance as the music progresses.

The object of the present invention is to provide a program, an information processing device, an information processing method, and a performance data processing system that can present images that match a performance with high accuracy.

In order to achieve the above object, a program according to an embodiment of the present invention causes a computer to display, on a display device, an image suitable for a piece of music corresponding to a performance, which is specified based on performance data generated in conjunction with a performance. An image display process to be displayed, a derivation process that repeatedly derives the degree of match between the performance and the identified song based on the time series of performance data generated in conjunction with the continuous performance, and a degree of match is the standard. This includes an instruction to execute a process of re-searching for a piece of music corresponding to the performance if the condition is not satisfied.

The present invention makes it possible to present images that match the performance with high accuracy.

FIG. 1 is a schematic diagram showing an example of a performance data processing system according to an embodiment. FIG. 2 is a diagram showing an example of the system configuration of the performance data processing system shown in FIG. 1. As shown in FIG. FIG. 3 is a block diagram showing an example of components constituting the performance data processing system according to the embodiment. FIG. 4 is a functional block diagram showing an example of the computer 200. FIG. 5 is a sequence diagram showing an example of a processing procedure in the performance data processing system according to the embodiment. FIG. 6 is a flowchart showing an example of the processing procedure of the music search processing section 11a. FIG. 7 is a flowchart showing an example of a processing procedure of the image display processing unit 11c in the first embodiment. FIG. 8 is a flowchart showing an example of a processing procedure of the alignment processing unit 11b. FIG. 9 is a diagram for explaining the calculation (derivation) of the degree of match using the DP matching method. FIG. 10 is a flowchart showing an example of the processing procedure of the image display processing section 11c in the second embodiment. FIG. 11 shows an example of a musical score. FIG. 12 is a diagram showing an example of the music data 13a. FIG. 13 is a flowchart showing another example of the processing procedure of the image display processing unit 11c in the second embodiment.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a schematic diagram showing an example of a performance data processing system according to an embodiment. The performance data processing system shown in FIG. 1 includes an input device, an application, and an output device. The input device is, for example, a digital keyboard (electronic device) 100, which generates performance data (for example, MIDI (Musical Instrument Digital Interface) data or a MIDI message string) from a user's performance and passes it to an application. The application is installed on a computer (information processing device) 200, such as a dedicated server, a notebook PC, or a tablet, and analyzes performance data. The image data acquired through the analysis is passed to an output device, and is displayed as an image on an image display device 300 such as a monitor or a projector as an output device according to display instructions from the computer 200. In this way, the performance data based on the user's (performer's) performance is processed by the application and output as a visual image.

FIG. 2 is a diagram showing an example of the system configuration of the performance data processing system shown in FIG. 1. As shown in FIG. Digital keyboard 100 is connected to computer 200 via MIDI interface 400. Computer 200 is connected to projector 300 via video interface 500.

FIG. 3 is a block diagram showing an example of elements constituting the performance data processing system. A performance data processing system has an application as its core, and is constructed by combining input devices, output devices, and various databases that can be accessed from the application.

In FIG. 3, a MIDI input device as an input device corresponds to the digital keyboard 100 in FIG. 1, and is a device for passing a user's performance as a MIDI message to an application. Further, the digital keyboard 100 includes at least one arithmetic processing unit such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor) that analyzes MIDI data and generates sound output data, and converts digital sound output data into analog. It is equipped with an output device such as a speaker that outputs sound to the outside. The application that receives the MIDI message accesses the music search database (DB), image DB, and alignment DB, selects an appropriate image from the MIDI message, and passes it to the output device. The output device displays an image using a display device.

Here, the application includes, for example, a music search processing section, an alignment processing section (derivation processing section), and an image display processing section as software modules. These modules and each database will be explained in more detail with reference to FIG.

FIG. 4 is a functional block diagram showing an example of the computer 200. The computer 200 includes a processor 11 , a ROM (Read Only Memory) 12 , a RAM (Random Access Memory) 13 , an internal bus 14 , an input/output interface 15 , an input section 16 , a communication section 17 , and an output section 21 .

The processor 11 is an arithmetic processing unit such as at least one CPU or MPU (Micro Processing Unit), and realizes various functions of the embodiment according to a program 12 a stored in a ROM 12 .
The RAM 13 functions as a working memory for the processor 11 and also stores music piece data 13a. The music piece data 13a is, for example, MIDI data, and is data that records information such as note positions, lengths, and beats for each of a plurality of songs.
The input unit 16 is connected to an operation unit 201 equipped with a keyboard and a pointing device, and accepts operations from the user.

The communication unit 17 is an interface connectable to the digital keyboard 100 and receives performance data (MIDI messages) from the digital keyboard 100 via the MIDI interface 400.
The output unit 21 transmits the image selected by the processor 11 to the projector 300 via the video interface 500, and causes the image to be visually output.

The ROM 12 is a nonvolatile storage device such as a flash memory, and stores a music search database 12b, an image database 12c, and an alignment database 12d in addition to the program 12a that controls the computer 200.
The music search database 12b is a database that is referred to during the music search process by the processor 11, and is a database in which information regarding a plurality of music pieces is registered in association with each other. The song search database 12b holds, for example, keys and titles for searching each song, information for image acquisition, and the like. By transmitting a query generated from the performance data to the music search database 12b as a search key, music information of the corresponding music piece is extracted.

The image database 12c is a database that holds images for displaying still images or moving images, and is a database in which image data of a plurality of images is registered in a format such as jpeg or PNG.
The alignment database 12d is a database in which melody information of the entire music piece for performing alignment processing is registered in association with each of a plurality of music pieces.

The processor 11 includes a music search processing section 11a, an alignment processing section 11b, and an image display processing section 11c as processing functions according to the embodiment. The music search processing section 11a, the alignment processing section 11b, and the image display processing section 11c are processing functions realized by the arithmetic processing of the processor 11 based on the program 12a. That is, the program 12a includes instructions for causing the processor 11 to execute the respective functions of the music search processing section 11a, the alignment processing section 11b, and the image display processing section 11c.

The music search processing unit 11a searches the music search database 12b based on the performance data (MIDI message string) generated when the user plays the digital keyboard 100, and identifies the music corresponding to the performance. . When a song is specified, the song search processing section 11a acquires information about the song from the song search database 12b.

The alignment processing section 11b repeatedly calculates the degree of matching between the music piece specified by the music search processing section 11a and the user's performance based on the time series of the performance data. In calculating the matching degree, both the performance data and the information held in the alignment database 12d are referred to. That is, the alignment processing section 11b executes alignment processing. Through the alignment process, it is also possible to obtain information such as which part of the song is currently being played at that point in time.

Incidentally, the technology for synchronizing a performance with a score is known as score alignment. Furthermore, synchronization between a performance and a sound source (PCM) is called audio-to-audio alignment, and synchronization between a performance and a sound source (MIDI) is called audio-to-MIDI alignment, etc.
Furthermore, if the degree of matching does not satisfy a predetermined standard during the performance, the alignment processing section 11b requests the music search processing section 11a to search again for a music piece corresponding to the current performance.

The image display processing section 11c selects an image corresponding to the song specified by the song search processing section 11a from the images held in the image database 12c, and sends it to the output section 21. As a result, an image (such as an illustration) corresponding to the performance is output from the projector 300.

Next, a plurality of embodiments will be described based on the above configuration.
[First embodiment]
FIG. 5 is a sequence diagram showing an example of a processing procedure in the performance data processing system according to the embodiment. In FIG. 5, MIDI messages generated when a performer plays the digital keyboard 100 are input to the music search processing section 11a and the alignment processing section 11b (processes p1 and p2).

The music search processing unit 11a generates a query for music search from the acquired time series of MIDI messages (MIDI message string) (process p3). Then, the song search processing unit 11a searches the song search database 12b using the generated query as a key (process p4), and acquires song information of the corresponding song (process p5). The acquired music information is passed to the alignment processing section 11b (process p6).

The alignment processing unit 11b searches the alignment database 12d using the song information as a key (process p7), and acquires the melody information of the entire song (process p8). The alignment processing unit 11b repeatedly calculates (derives) the degree of matching between the MIDI message string and the acquired melody information (process p9). That is, in process p9, the degree of matching between the music being played and the plurality of music stored in the music search database 12b is calculated (derived).

Even if the degree of matching meets a preset standard once during a performance, the degree of matching for each song may change if, for example, different songs are played one after the other without interrupting the performance (such as a medley). Also, if there is another song with a similar phrase to the song being played, and the degree of matching is calculated for this other song, the degree of matching may change as the performance progresses.

In this way, when switching songs such as in a medley, or during the course of playing the same song, the degree of matching may transition from a state in which the predetermined criterion is satisfied to a state in which it is not satisfied (for example, when it becomes lower than a predetermined threshold value). When the degree of matching transitions from a state in which the predetermined criterion is satisfied to a state in which it is not satisfied, the alignment processing unit 11b requests the song search processing unit 11a to search for the song again (process p10). In this case, the process returns to process p3, and the steps of processes p3 to p9 are repeated.

If the matching degree value satisfies the criteria, the song search processing unit 11a provides the song information to the image display processing unit 11c (process p11), and the alignment processing unit 11b applies the calculated matching degree information to the image display processing unit 11c. It is passed to the display processing unit 11c (process p12). The image display processing unit 11c selects and acquires an image (for example, a picture, photograph, video, etc. of "Akatonbo") corresponding to the lyrics information (for example, the lyrics "Akatonbo") included in the given song information from the image database 12c. (process p13). In response, the image database 12c returns the image data of the selected image to the image display processing section 11c (process p14). The image display processing unit 11c displays the received image data on the output device (process p15).

With reference to FIG. 6, the processes p3 and p4 in FIG. 5 will be described in detail. FIG. 6 is a flowchart showing an example of the processing procedure of the music search processing section 11a.
In FIG. 6, the music search processing unit 11a obtains Note On information from the MIDI message string obtained from the digital keyboard 100 (step S11). Next, the music search processing unit 11a checks whether a sufficient number of Note On information has been acquired to generate a query (step S12), and if it is insufficient (No), acquires Note On information again. repeat.

When a sufficient number of Note On messages have been acquired (Yes), the music search processor 11a generates a query for the music search database 12b from the pitch transitions contained in the Note On message sequence (step S13). Finally, the music search processor 11a searches the music search database 12b using the generated query (step S14).

With reference to FIG. 7, the processes p12 to p15 in FIG. 5 will be described in detail. FIG. 7 is a flowchart showing an example of the processing procedure of the image display processing section 11c in the first embodiment.
In FIG. 7, the image display processing unit 11c acquires matching degree information including a matching degree indicating the matching degree between the lyric information of the song information acquired in process p5 and the information of the image to be compared from the alignment processing unit 11b. (Step S21). Next, the image display processing unit 11c refers to the matching degree value and compares it with a predetermined threshold value (step S22). Here, if the matching degree is high (above the threshold value) (Yes), an image corresponding to the lyrics information in the music information acquired by the music search is acquired from the image database 12c (step S23).

Note that if the image database 12c holds a plurality of images corresponding to lyric information, the computer 200 may display two or more of the plurality of corresponding images sequentially or alternately during the performance. May be adjusted. Further, if the image is a moving image, the computer 200 may select a moving image with an optimal length according to the estimated remaining performance time and output it to the display device. Here, it is most preferable that the optimal length of the video be a video with a display time that is substantially equivalent to the estimated remaining performance time. Next, a video with a length longer than the estimated remaining performance time is preferable.

Note that even if the display time of a video is shorter than the estimated remaining performance time, the computer 200 can instruct the display device to display the video repeatedly or slow down the display speed on the display device so that the display time of the video is shorter than the estimated performance time. It may be adjusted so that it is substantially equivalent to .

On the other hand, if the matching degrees of all images held in the image database 12c are all low (below the threshold), the image display processing unit 11c acquires a general-purpose image prepared in advance from the image database 12c. (Step S24). A general-purpose image should be an image that corresponds to any musical idea, such as an abstract image or another image similar in touch or image to the previously displayed image. . In particular, since the degree of matching will inevitably be low immediately after the performance starts and until the first search is completed, it is advisable to set images in advance that will reasonably match any song.

Next, the image display processing unit 11c instructs the display device (projector 300) to output and display the acquired image (step S25). Then, the image display processing unit 11c determines whether the performance has ended by counting up the elapsed time since the last key press/release and comparing the count value with a predetermined value (step S26). When it is determined that the performance has ended (Yes), the image display processing section 11c stops displaying the image (step S27) and ends the process.

On the other hand, if it is determined that the performance is continuing (No in step S26), the image display processing unit 11c determines whether to display the next image, that is, whether it is time to switch images (step S28). If it is not time to switch images (No), the processing procedure loops back to step S26 with the display on the display device continuing.

If it is determined in step S28 that it is time to switch images (Yes), the effect processing (e.g., fade) to be performed when switching images is set appropriately (step S29), and the processing procedure returns to step S22, where the procedure is repeated from the selection of the next image.

Processes p6 to p9 in FIG. 5 will be described in detail with reference to FIG. FIG. 8 is a flowchart showing an example of the processing procedure of the alignment processing section 11b.
In FIG. 8, the alignment processing unit 11b acquires music information of the music that is hit as a result of the music search by the music search processing unit 11a (step S41). Furthermore, the alignment processing section 11b acquires Note information included in the MIDI data string generated in the performance up to the present time (step S42). Based on this information, the alignment processing unit 11b searches the alignment database 12d (step S43) and acquires melody information of the song.

Then, the alignment processing unit 11b calculates the degree of match between the acquired note information and melody information (step S44) and calculates (derives) the degree of match information. Note that immediately after the performance starts, a low value may be returned as the degree of match information. The obtained degree of match information is passed to the image display processing unit 11c (step S45). Here, if the degree of match is less than the threshold (Yes in step S46), the alignment processing unit 11b requests the song search processing unit 11a to search for the song again (step S47).

FIG. 9 is a diagram for explaining the calculation of the matching degree in step S44 of FIG. 8. For example, it is possible to calculate the degree of matching between music information and a melody line using the DP matching method.

In FIG. 9, the horizontal axis is note information of the performance, and a, b, and c indicate pitches. The vertical axis is melody information acquired from the alignment database 12d, and similarly, a, b, and c indicate pitch. The state transitions from the starting state at the bottom left to the final state at the top right. The cumulative value of each state is set for each path to adjacent states.

In Figure 9, where a penalty is added when the vertical and horizontal axes represent different states (pitches), 2 points are added if the states are the same in a horizontal transition, and 20 points are added if they are different. In vertical transitions, 2 points are added if the states are the same, and 20 points are added if they are different. In diagonal transitions, 1 point is added if the states are the same, and 10 points are added if they are different.

If the boxed area on the right side of Figure 9 (7th frame) is taken as the current time, the minimum score for the 7th note is 14, and this value basically becomes smaller the more similar the performance and melody information are. Therefore, by normalizing by the number of frames as in equation (1), the degree of match can be calculated easily. In addition, the position on the vertical axis where the minimum value is recorded can be used to find out the approximate position (location) where the song is being played.

As described above, in the first embodiment, a query is generated from performance data generated during a performance, and music information is acquired by searching a music search database. Furthermore, melody information corresponding to the song information is acquired from the alignment database, and the degree of matching between the song information and the melody line is continuously calculated through alignment processing. When the degree of matching decreases, the music search database is searched again to identify the music that matches the performance again. In this way, as long as the performance continues, the degree of matching between the performance and the song is constantly calculated, so image data that matches the song can be obtained at all times.

As mentioned above, if the degree of match becomes low, you can take into account the degree of match in previous performances to obtain an overall matching song, or if the degree of match becomes extremely low from a certain phrase. , the matching degree of the performance up to now may be reset, and the matching degree may be calculated (derived) from the point at which it becomes low. In the latter case, even if the songs change one after another like a medley without interrupting the performance, the appropriate image will always be selected and displayed.

With existing technology, once a song is identified, it cannot be corrected midway, which can lead to an unnatural image corresponding to the song. In other words, once a search is completed and a song with a high degree of matching is determined, illustrations related to the song with a high degree of matching may continue to be displayed even if a different song is subsequently played. In contrast, with the technology of the first embodiment, there is no risk of such a mismatch occurring. In other words, with the first embodiment, if the performance and the displayed image no longer match, the mismatch is immediately corrected, and an image that does not make the mismatch noticeable can be selected. Furthermore, since there is another song with a similar phrase to the song being played and the degree of matching is calculated for this other song, the same effect as described above can be achieved even if the degree of matching changes as the performance progresses.

For these reasons, according to the first embodiment, images that match the performance can be presented with high accuracy, thereby providing a program, an information processing device, and an information processing method that further enhance the enjoyment of playing. , and a performance data processing system.

Second Embodiment
In the first embodiment, an example was described in which, after the degree of matching is determined to be equal to or greater than a threshold value, if the degree of matching decreases during the continued performance, image data matching the performance is searched again from the image database 12c. In the first embodiment, for example, when the performance of the song "Akatonbo" is first recognized, the image display device displays a picture related to Akatonbo. Also, when the performance of the song "Yuki no Furu Machi wo" is first recognized, the image display device displays an image of, for example, a cityscape with snow falling. It is also possible that the same pattern of image will continue to be displayed until the performance ends.

However, as the phrase progresses with the performance, the content of the lyrics and the melody (including the mood of the song, such as brightness or darkness, and whether it is in a minor or major key) may change. In other words, even if the song is the same, the desired image may change depending on the content of the lyrics or the melody. Even in the performance of the song ``Akatonbo'' or the song ``Yuki no Furu Machi wo,'' the desired image may change as the phrase progresses. Even in such a case, it is preferable to display an image that is more suitable for the phrase being played.

Therefore, in the second embodiment, the display image is switched to a display image that matches the lyric information and the melody at an appropriate timing during the performance of the same song, in response to changes in the lyric information including the lyric content in the song data and changes in the melody. Realize. In the second embodiment, descriptions of parts similar to those in the first embodiment will be omitted as appropriate.

FIG. 10 is a flowchart showing an example of the processing procedure of the image display processing section 11c in the second embodiment. In FIG. 2, the image display processing unit 11c extracts phrases one by one from the plurality of phrases that make up the music in the music information acquired in process p5, in the order of scheduled performance. Then, the image display processing unit 11c sequentially receives matching degree information including a matching degree indicating the degree of matching between the image and at least one of the lyrics information and melody information of one extracted phrase from the alignment processing unit 11b. Acquire (step S21). Here, the lyrics information includes, for example, digital value data according to the content of the lyrics. The melody information includes digital value data corresponding to the melody.

Next, the image display processing unit 11c refers to the matching degree value for each phrase and compares it with a predetermined threshold value (step S22). Here, if there is an image in the image database 12c that has a high matching degree (above the threshold) for each phrase constituting the song in the song information obtained by the song search (Yes), the image display processing unit 11c Images corresponding to the phrase are sequentially acquired from the image database 12c (step S30).

Here, if there are multiple image data in the image database 12c that closely match a certain phrase, the computer 200 may instruct the computer 200 to display an image randomly selected from the image database. Alternatively, the computer 200 may instruct the computer 200 to read the past display history and select and display an image from among images excluding the most recently displayed image.

On the other hand, in step S22, if all the matching degrees are low (below the threshold), the image display processing unit 11c acquires a general-purpose image prepared in advance from the image database 12c (step S24). The image display processing unit 11c then instructs the display device (projector 300) to output and display the image acquired in step S30 or step S24 (step S25). Thereafter, the processing procedure of steps S26 to S29 is executed as in the first embodiment.

FIG. 11 shows an example of the musical score of Twinkle Little Star, and in the second embodiment, music data representing this musical example has a structure as shown in FIG. 12, for example.

FIG. 12 is a diagram showing an example of the music data 13a. The music data 13a (FIG. 4) corresponding to the musical score example in FIG. 11 is recorded in association with the pitch and length of the note and a tick indicating subtle timing for each beat of the measure. There is. In addition, tempo and time signature are included as meta-events, and a phrase number is also embedded.

In the second embodiment, the image number (identifier) of the image to be displayed while the phrase is being played is further added to the area corresponding to the phrase number of the music data 13a in accordance with at least one of the lyrics information and melody information for each phrase. Add and register. Then, a matching image is searched from the image database 12c using the image number of the position corresponding to the progress of the performance as a key. As a result of the search, an image based on the image data specified is displayed on the display device during the performance and at a timing matching the image number (timing of switching between a phrase and the next phrase).

13 is a flowchart showing an example of a processing procedure of the image display processing unit 11c in the second embodiment. Only the parts that are different from FIG.
The same procedure as in the first embodiment is carried out up to step S26 in Fig. 13. If it is determined in step S26 that the performance is continuing (No), the image display processing unit 11c determines whether to display the next image, that is, whether it is time to switch to an image suitable for the next phrase when moving to the next phrase (step S31). The image display processing unit 11c executes the determination of step S31 in association with the progress of the music data 13a of the searched music.

That is, by associating the MIDI data strings that are generated one after another by the performance with the music data 13a, it is possible to determine, for example, that (the position is now approaching the beginning of the first beat of the seventh measure). Referring to FIG. 12, image number G4 is associated with this position along with phrase P4. Therefore, before playing phrase P4, the computer 200 obtains an image corresponding to image number G4 from the image display processing unit 11c (step S32). Then, after applying the necessary effect processing to the image (step S29), the display device starts displaying the image corresponding to image number G4 in time with the start of the performance of phrase P4. In other words, by identifying the music being played, it is possible to start displaying images corresponding to at least one or more phrase units in this music at the timing when at least one or more phrase units are switched during the performance.

As explained above, in the second embodiment, the image switching timing and the identifier (number, file name, etc.) indicating the illustration to be displayed are stored in advance in the music data 13a, and the input performance data and the music are stored in advance. The timing is compared with the data 13a, and when the timings match, the image is switched. By doing this, it becomes possible to switch the optimal image at the optimal timing.

Most lesson data that is also used as music data has phrase data defined, as shown in FIG. 12. Therefore, by embedding the image data number in the music data along with the phrase data, as in the second embodiment, it is possible to switch to an appropriate image when the phrase changes.

Also, lyrics often have content that is divided according to the phrase position. Therefore, by including image switching information in the music data 13a, it becomes possible to display images optimally in accordance with the precise timing of the lyrics. Furthermore, if you want to specify even more precise timing for image switching, it is possible to use, for example, undefined control changes.

As a result, the second embodiment also makes it possible to present images that match the performance with a high degree of accuracy, thereby making it possible to provide a program, information processing device, information processing method, and performance data processing system that further enhance the enjoyment of playing.

Note that the present invention is not limited to specific embodiments. For example, in the embodiment, an example has been disclosed in which the processing unit (hardware) and functions (application (software)) according to the embodiment are implemented in the computer 200. Instead of the computer 200, the digital keyboard 100 may be equipped with processing units and functions according to the embodiment. Similarly, if the computer 200 has a display means, the computer 200 can be used as a display device instead of the image display device 300. Further, if the digital keyboard 100 has a display means, the digital keyboard 100 can be applied as a substitute for the image display device 300.

In the above embodiment, the degree of match is derived by calculation, but this is not the only possible method. For example, the degree of match can be derived using a table that can determine whether the music being played matches any of the multiple music pieces stored in the music search database 12c, that is, a table that automatically extracts matching music pieces by inputting a data value that indicates the content of the music being played.

Furthermore, the technical scope of the present invention includes various modifications and improvements within the scope of achieving the object of the present invention, and this will be clear to those skilled in the art from the description of the claims. be.

11...Processor 11a...Music search processing section 11b...Alignment processing section 11c...Image display processing section 12a...Program 12b...Song search database 12c...Image database 12d...Alignment database 13...RAM 13a...Song data 14...Internal bus 15...Input Output interface 16...Input unit 17...Communication unit 21...Output unit 100...Digital keyboard 200...Computer 201...Operation unit 300...Projector 400...MIDI interface 500...Video interface.

Claims (10)

to the computer,
an image display process that causes a display device to display an image that is specified based on performance data generated in conjunction with the performance and that is suitable for a piece of music that corresponds to the performance;
a derivation process of repeatedly deriving the degree of match between the performance and the identified music piece based on a time series of the performance data generated in conjunction with the continuously performed performances;
A program including an instruction for executing a process of re-searching for a piece of music corresponding to the performance when the degree of matching does not satisfy a criterion. The method further includes a music search process for searching for and identifying a music piece corresponding to the performance based on the performance data generated in association with the performance,
2. The program according to claim 1, wherein the music search process identifies the music by searching a music search database in which related information is registered for each of a plurality of music pieces. The program according to claim 1, wherein the image display process obtains image data for the image to be displayed from an image database in which multiple image data are registered. The program according to claim 3, wherein the image display process acquires an image identifier, and acquires image data corresponding to a timing according to the identifier from the image database before the timing according to the identifier. The program according to claim 4, wherein the image display process causes the display device to display an image of the image data corresponding to the timing according to the identifier. 2. The program according to claim 1, wherein the derivation process derives a degree of match between the melody information acquired from an alignment database in which melody information of the music piece is registered in association with each other for each of a plurality of music pieces, and the performance data. The program according to claim 1, wherein the image display process causes the display device to display a general-purpose image prepared in advance in a state where the matching degree does not meet a criterion or a state where the song is not specified. . an interface connectable to an electronic device that generates performance data;
a processor;
The processor,
searching for and identifying a piece of music corresponding to the performance based on the performance data generated in association with the performance;
selecting an image corresponding to the identified piece of music and displaying it on a display device;
Iteratively calculating a degree of match between the performance and the identified piece of music based on a time series of the performance data;
When the degree of match does not satisfy a predetermined standard, the information processing device searches again for a piece of music corresponding to the performance. In an information processing method of an information processing device,
displaying on a display device an image that is specified based on performance data generated in association with the performance and that matches the music piece corresponding to the performance;
repeatedly deriving a degree of matching between the performance and the identified piece of music based on a time series of the performance data generated in association with the successive performances;
When the degree of match does not satisfy a criterion, a search is again made for a piece of music corresponding to the performance. A performance data processing system including an information processing device and a display device,
The information processing device includes:
an interface connectable to a device that generates performance data;
an image display processing unit that displays, on the display device, an image that is specified based on the performance data generated in association with the performance and that matches the music piece corresponding to the performance;
a derivation processing unit that repeatedly derives a degree of matching between the performance and the identified piece of music based on a time series of the performance data generated in association with the performances that are continuously performed,
The performance data processing system, wherein the derivation processing unit re-searches for a piece of music corresponding to the performance if the degree of match does not satisfy a criterion.