JP7467907B2 - Program, method, electronic device, and performance data display system - Google Patents

Description

The embodiments relate to a program, a method, an electronic device, and a performance data display system.

For example, a computer graphics moving image generation system is known that can move image objects such as dancers in sync with the performance of music such as MIDI (Musical Instrument Digital Interface) pieces, and generate moving images that not only match the ideas of the music, but also change in unison with the progression of the musical notes (see Patent Document 1).

Japanese Patent Application Laid-Open No. 11-224084

The computer graphics moving image generation system disclosed in Patent Document 1 has the problem of lacking expressiveness, as it simply dynamically creates characters representing dancers during a performance.

The present invention has been made in consideration of these circumstances, and aims to provide, for example, a program, a method, an electronic device, and a performance data display system that have excellent visual expressiveness of a performance.

According to the program of the embodiment, the computer of the information processing device determines whether the input performance data is input from at least a first device or a second device, and if it is determined that the device is the first device, displays first image data contained in a first image group according to information that can be determined based on the input performance data, and if it is determined that the device is the second device, displays second image data contained in a second image group and having a different aspect from the first image data, even if the information that can be determined based on the input performance data is the same as when the device is determined to be the first device, and the first image group and the second image group may contain the same image data .

The present invention can provide, for example, a program, a method, an electronic device, and a performance data display system that provide excellent visual expression of a performance.

FIG. 1 is a diagram illustrating an example of a performance data display system including a program according to an embodiment of the present invention. FIG. 2 is a diagram showing an overview of the performance data display system according to the embodiment. FIG. 3 is a diagram showing hardware of a control system for an electronic musical instrument according to an embodiment. FIG. 4 is a diagram illustrating hardware of the information processing apparatus according to the embodiment. FIG. 5 is a diagram showing an example of a table stored in the ROM. FIG. 6 is a diagram showing an example of a display of an image (character). FIG. 7 is a diagram showing an example of a trajectory pattern of an image (character) in a real-time image displayed on the display unit of a display device. FIG. 8 is a diagram showing a spiral line as an example of a trajectory pattern generated by the image determination unit. FIG. 9 is a table showing connection cases of musical instruments that can be connected to an information processing device. FIG. 10 is a flowchart showing the operation of the information processing device. FIG. 11 shows a real-time image when the first user is playing a melody.

FIG. 12 shows a post-performance image after the first and second users have finished playing the song. FIG. 13 is a diagram showing an example of a real-time image when four musical instruments are connected to an information processing device. FIG. 14 is a diagram showing an example of a real-time image when three or more musical instruments are connected to the information processing device. FIG. 15 is a flowchart showing the operation of the information processing device. FIG. 16 is a diagram showing an example of a real-time image related to FIG.

Below, an embodiment (hereinafter also referred to as "embodiment") relating to one aspect of the present invention will be described with reference to the drawings. However, the embodiment described below is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, in implementing the present invention, a specific configuration according to the embodiment may be appropriately adopted. Note that while the data appearing in the embodiment is described in natural language, more specifically, it is specified in pseudo-language, commands, parameters, machine language, etc. that can be recognized by a computer.

<0> Application Example First, an example of a situation in which the present invention is applied will be described with reference to Fig. 1. Fig. 1 shows a schematic example of a performance data display system including a program according to the application example. The performance data display system is intended to stimulate the user's motivation to practice an electronic musical instrument by drawing images (pictures) in real time in sync with the performance (music) of the user (performer).

<0-1> Overview of application examples
Before describing the configuration of the performance data display system, an overview of the performance data display system will be described. The performance data display system analyzes performance data received from, for example, an electronic musical instrument capable of outputting performances by multiple users as performance data (e.g., MIDI data), and converts the performances by the multiple users into images. For example, the performance data does not include note name information or chord type information, and the note name information and chord type information can be determined from the note number information included in the performance data.

<0-2> Configuration of the Application Example The configuration of the application example will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of a performance data display system according to the application example.

As shown in FIG. 1, the performance data display system includes a first musical instrument, a second musical instrument, an information processing device, and a display device.

The first musical instrument converts the performance of the first user into performance data (e.g., MIDI data). The first musical instrument then outputs the performance data to the information processing device.

More specifically, the first instrument generates first performance data including a first identifier (e.g., a first MIDI channel number) capable of identifying the first instrument in response to a performance operation by the first user, and outputs the generated first performance data to the information processing device. The first instrument is, for example, an electronic instrument such as a keyboard, an electronic piano, an electric guitar, or an electronic violin. However, the first instrument is not limited to those described above, and may be appropriately selected depending on the embodiment.

The second musical instrument converts the performance of the second user into performance data (e.g., MIDI data). The second musical instrument then outputs the performance data to the information processing device.

More specifically, the second instrument generates second performance data including a second identifier capable of identifying the second instrument in response to a performance operation by the second user, and outputs the generated second performance data to the information processing device. The second instrument is, for example, an electronic instrument, a vocal microphone, or an acoustic piano. However, the second instrument is not limited to those described above, and may be appropriately selected depending on the embodiment.

The information processing device determines whether the input performance data is input from at least a first instrument or a second instrument. If the information processing device determines that the input performance data is from the first device, it outputs first image (character) data according to information that can be determined based on the input performance data (e.g., note name, chord type, etc.). If the information processing device determines that the input performance data is from the second device, it outputs second image data (character) in a manner different from the first image data, even if the information that can be determined based on the input performance data (e.g., note name, chord type, etc.) is the same as the information that can be determined based on the first performance data. The information processing device is, for example, a PC (personal computer). Note that "setting" or "registration" basically has the same meaning as "storing".

The display device displays an image based on the performance of the first user and an image based on the performance of the second user on the display unit. As described above, the image based on the performance of the first user is generated based on the first image group, and the image based on the performance of the second user is generated based on the second image group. Therefore, it is possible to distinguish between the image based on the performance of the first user and the image based on the performance of the second user.

<0-3> Operation of the Application Example Next, an example of the operation of the performance data display system according to the application example will be described.

When the first user starts playing using the first musical instrument, the information processing device determines from the information included in the input performance data that the data is performance data from the first musical instrument (first performance data). The information processing device then performs a musical analysis on the performance of the first musical instrument. Specifically, the information processing device determines an image (e.g., a beetle character) included in a first image group (e.g., a character group including multiple insect characters) according to the note name (or chord) included in the first performance data, and draws the image in real time along a trajectory according to the set first condition (e.g., display according to chord progression). As a result, the image (character) included in the first image group is displayed in real time on the display device. Note that "real time" here does not necessarily mean that the timing of the performance and the timing of the display of the image (character) are the same. In addition, an image in which an image (character) is displayed in accordance with the performance may be described as a real-time image, for example. In other words, multiple images (characters) are drawn in the real-time image. The method of drawing the real-time image will be described later.

When the second user starts playing using the second instrument, the information processing device determines from information included in the input performance data that the performance data is from the second instrument (second performance data). The information processing device then performs a musical analysis on the performance of the second instrument. Specifically, the information processing device determines an image (e.g., a giraffe character) included in a second image group (e.g., a character group including multiple animal characters) according to the note name (or chord) included in the second performance data, and draws it in real time along a trajectory according to the set first condition (e.g., display according to a chord progression). As a result, the image (character) included in the second image group is displayed in real time on the display device.

In this way, the information processing device draws the real-time image so that it is possible to distinguish between an image based on the performance of the first user and an image based on the performance of the second user. After the first and second users finish playing the musical instruments, the information processing device displays the image data on the display device according to the set second condition (e.g., a spiral display). Such an image may be described as, for example, a post-performance image. The post-performance image is an image in which an image (character) analyzed from the beginning to the end of the performance of the first and second users is drawn according to the second condition (e.g., a spiral display). At this time, similar to the real-time image, the information processing device draws the post-performance image so that it is possible to distinguish between an image based on the performance of the first user and an image based on the performance of the second user.

<0-4> Effects of the Application Example As described above, the performance data display system according to the application example can determine the performances of the first and second users and change the group of images to be displayed. In other words, the performance data display system according to the application example can appropriately configure settings (drawing settings) related to images while the first and second users are playing, without performing specific operations unrelated to the performance. Therefore, the performance data display system can generate highly expressive images while the first and second users remain focused on their performance.

<1> Embodiment <1-1> Configuration <1-1-1> Performance Data Display System Hereinafter, a performance data display system according to an embodiment will be described with reference to FIG.

Figure 2 shows an overview of the performance data display system according to the embodiment.

As shown in FIG. 2, the performance data display system includes, for example, an information processing device (electronic device) 1, an electronic musical instrument 2 (e.g., 2(0) and 2(1)), a speaker 3, a display device 4, a MIDI patcher (MIDI patch bay) 5, a vocal microphone 6, an acoustic piano 7, and an audio interface 8.

Note that in FIG. 2, two electronic musical instruments 2, a vocal microphone 6, and an acoustic piano 7 are shown as an example, but this is just an example. At least two musical instruments are required. In this embodiment, the electronic musical instruments 2, the vocal microphone 6, and the acoustic piano 7 are collectively treated as musical instruments.

<1-1-2> Electronic Musical Instrument The electronic musical instrument according to the embodiment will now be described with reference to Figures 2 and 3. Figure 3 is a diagram showing the hardware of a control system for the electronic musical instrument according to the embodiment.

As shown in Figures 2 and 3, the electronic musical instrument 2 to which the information processing device 1 of this embodiment is connected is, for example, an electronic keyboard instrument such as an electronic piano, synthesizer, or electronic organ.

The electronic musical instrument 2 includes a plurality of keys 11, a display unit 20, an operation unit 30, and a sound generating unit 40.

As shown in FIG. 3, the electronic musical instrument 2 also includes a key press detection unit 50, a memory unit 60, a control unit 70, and a communication unit 80.

The key 11 is an operator that allows the performer to specify a pitch, and when the performer presses and releases the key 11, the electronic musical instrument 2 produces and mutes a sound corresponding to the specified pitch.

The display unit 20 has, for example, a liquid crystal monitor with a touch panel, and is a part that displays messages in response to the player's operation of the operation unit 30.

In this embodiment, the display unit 20 has a touch panel function, so the display unit 20 can function as part of the operation unit 30.

The operation unit 30 has operation buttons that allow the performer to perform various settings, such as adjusting the volume.

The sound output unit 40 is a part that outputs sound, and has an SP amplifier 41 (speaker amplifier), a speaker 42, an HP amplifier 43 (headphone amplifier), an HP jack 44 (headphone jack) into which the speaker 3 (or headphones) is plugged, and an HP jack insertion detection unit 45 that detects that the speaker 3 (or headphones) is plugged into the HP jack 44.

When the speaker 3 (or headphone) plug is inserted into the HP jack 44, the HP jack insertion detection unit 45 detects the insertion and sound is output to the HP jack side, and conversely, when the HP jack insertion detection unit 45 does not detect the insertion of a plug, sound is output to the speaker 42 side.

In addition, a sound generating unit 40 may be used instead of the speaker 3.

The key press detection unit 50 is a part that detects the pressing and releasing of the keys 11. For example, a rubber switch is provided on the underside of each key 11, and the pressing and releasing of the keys are detected by the rubber switch.

The memory unit 60 includes a ROM 61, which is a read-only memory unit, and a RAM 62, which is a readable and writable memory unit.

The ROM 61 stores, for example, control programs executed by the control unit 70 (e.g., programs for lesson mode programs, etc.) and various data tables, etc.

RAM 62 is a storage area in which data such as musical tone data and song data corresponding to each key 11 is stored, and also functions as a temporary storage area for expanding data and control programs generated by the control unit 70 during performance.

The control unit 70 is responsible for controlling the entire electronic musical instrument 2. The control unit 70 is, for example, an LSI (Large Scale Integration).

The control unit 70 then performs control such as, for example, causing the sound generation unit 40 to generate a musical tone in response to pressing of the key 11, and controlling the sound generation unit 40 to mute a musical tone in response to releasing of the key 11.

The communication unit 80 includes a wireless unit and a wired unit for communicating with external devices. In this embodiment, the information processing device 1 is connected via a wired connection, and data is transmitted to the information processing device 1, as described later.

However, the communication unit 80 does not need to be limited to only performing a transmission function, and may also have a receiving function.

The above-described components (display unit 20, operation unit 30, sound output unit 40, key press detection unit 50, memory unit 60, control unit 70, and communication unit 80) are connected via bus 100 for communication, allowing necessary data to be exchanged between the components.

<1-1-3> MIDI Patcher Next, the MIDI patcher (MIDI patch bay) 5 will be described.

MIDI Patcher 5 is a device that changes the path of MIDI signals.

Specifically, the MIDI Patcher has multiple pairs of MIDI IN and MIDI OUT terminals, each of which is numbered. The MIDI Patcher can freely switch between multiple MIDI devices by operating switches, etc.

In the case of FIG. 2, when two electronic musical instruments 2 are connected, the MIDI patcher 5 transfers the MIDI signals of the two electronic musical instruments 2 to the information processing device 1.

<1-1-4> Vocal Microphone Next, the vocal microphone 6 will be described.

The vocal microphone 6 is a device for collecting vocals and the sound of live musical instruments. More specifically, the vocal microphone 6 converts sound (air vibrations) into an electrical signal. Examples of vocal microphones 6 include a dynamic microphone (no power supply required) and a condenser microphone (power supply required).

<1-1-5> Acoustic Piano Next, the acoustic piano 7 will be described.

Acoustic piano 7 is an instrument that does not use electric speakers. When the player presses the keys, the hammers strike the strings, causing them to vibrate. The vibrations are then transmitted to the soundboard and then to the entire body, producing sound. Although not shown, the sound of acoustic piano 7 needs to go through a device similar to vocal microphone 6 to be converted into an electrical signal.

<1-1-6> Audio Interface Next, the audio interface 8 will be described.

When sounds such as vocals or acoustic piano (called analog signals) are handled by the information processing device 1, they need to be converted into digital signals. The audio interface 8 is a device that can convert analog signals into digital signals (AD conversion). Therefore, by using the audio interface 8, the analog signal from the vocal microphone 6 or acoustic piano 7 can be converted into a digital signal and supplied to the information processing device 1.

<1-1-7> Information Processing Device Next, the information processing device will be described with reference to Fig. 2 and Fig. 4. Fig. 4 is a diagram showing hardware of the information processing device according to the embodiment.

In this embodiment, the information processing device 1 is a PC as shown in FIG. 2, and an application for displaying images on the display device 4, as described below, is installed on the information processing device 1.

However, the information processing device 1 does not need to be limited to a PC and may be a tablet-type mobile terminal, etc.

The information processing device 1 may also include a sequencer that receives MIDI codes from the electronic musical instrument 2 and plays back music data.

As shown in FIG. 4, the information processing device 1 mainly includes an operation unit 101, a display unit 102, a communication unit 103, a sound output unit 104, a control unit 105, and a memory unit 106.

The operation unit 101 includes, for example, switches such as a power switch for turning the power of the information processing device 1 on and off.

The display unit 102 has a liquid crystal monitor and can display images. The display unit 102 may have a touch panel function. In that case, the display unit 102 can function as part of the operation unit 101. The display unit 102 may be used instead of the display device 4.

The communication unit 103 includes a wireless unit or a wired unit for communicating with other devices. The communication unit 103 receives data from, for example, the electronic musical instrument 2 or the audio interface 8. The communication unit 103 is also connected by wire to, for example, the communication unit of the display device 4, and transmits data to the display device 4.

The sound output unit 104 is the part that outputs sound and has a speaker, earphone jack, etc.

The memory unit 106 includes a RAM (Random Access Memory) 1061, which is a read-only memory unit that can be read and written, and a ROM (Read Only Memory) 1062.

RAM 1061 includes volatile memory and is a storage area in which data necessary to operate the control program is stored, and also functions as a temporary storage area for storing data generated by the control unit 105, data sent from the electronic musical instrument 2 (performance information including pitch information), and applications.

ROM 1062 is a non-volatile semiconductor memory such as a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (registered trademark) (Electrically Erasable Programmable ROM), and serves as a so-called secondary storage device or auxiliary storage device.

The ROM 1062 stores a program 10621 and a table 10622. The program 10621 is used by the control unit 105 to perform various processes. The table 10622 is a table that links performance information with images.

The control unit 105 is a part that controls the entire information processing device 1. The control unit 105 has, for example, a CPU (Central Processing Unit). The control unit 105 centrally controls the information processing device 1 by reading out a program 10621 stored in the ROM 1062 of the storage unit 106 and using the RAM 1061 of the storage unit 106 as a work area.

Next, the functional configuration of the control unit 105 of the information processing device 1 according to the embodiment will be described. The control unit 105 reads and executes a program 10621 stored in the ROM 1062, thereby functioning as a performance judgment unit 1051, an image determination unit 1052, an image information output unit 1053, a condition judgment unit 1054, a performance end judgment unit 1055, and a connected device judgment unit 1056. Note that the performance judgment unit 1051, the image determination unit 1052, the image information output unit 1053, the condition judgment unit 1054, the performance end judgment unit 1055, and the connected device judgment unit 1056 may each be realized by a circuit configuration rather than a program.

The connected device determination unit 1056 determines whether the performance data input from the outside (electronic musical instrument 2 or audio interface 8) is input from at least a first device (e.g., electronic musical instrument 2) or a second device (e.g., electronic musical instrument 2 or audio interface 8).

MIDI data is transmitted through a port. This port has, for example, 16 "channels" available. A sound source can play one part per channel. An electronic musical instrument 2 can specify a channel to transmit MIDI data. For example, if two electronic musical instruments 2 specify different channels, the connected device determination unit 1056 can determine the two electronic musical instruments 2 from the channel of the MIDI data.

In addition, the digital signal from the audio interface 8 and the MIDI signal are different. Therefore, the connected device determination unit 1056 can determine the performance data from the electronic musical instrument 2 and the performance data from the audio interface 8.

The performance determination unit 1051 determines the key of the song (e.g., 24 types from C major to B minor), note name (e.g., Do, Re, Mi...), chord type (e.g., Major, Mimor, Sus4, Aug, Dim, 7th, etc.), velocity value (a number that indicates the strength of a sound in the MIDI standard), note length, syncopation, chord function, and chord progression based on the performance information (performance data) received from the electronic musical instrument 2 via the communication unit 103. Note that any method may be used to determine the key of the song, note name, chord type, velocity value, note length, syncopation, chord function, and chord progression.

The performance determination unit 1051 also extracts pitch from the digital signal received from the audio interface 8, and performs music analysis based on MIDI information obtained by converting the pitch information. The performance determination unit 1051 also performs multi-tone analysis on the digital signal received from the audio interface 8, and performs music analysis based on MIDI information obtained by converting the results of the multi-tone analysis. Note that since the MIDI created by multi-tone analysis may contain noise due to misrecognition, a mechanism may be included to ignore short notes that are likely to occur due to misrecognition. The performance determination unit 1051 then determines the tonality, note names, chord types, velocity values, note lengths, syncopation, chord functions, and chord progressions of the song, just like the signal received from the electronic musical instrument 2.

The image determination unit 1052 can determine from which instrument the data was received based on information from the connected device determination unit 1056.

Each time performance information is received from the electronic musical instrument 2, the image determination unit 1052 determines one of the image data (characters) included in the image group (character group) based on the information determined by the performance determination unit 1051.

Here, the relationship between the information determined by the performance determination unit 1051 and the image data will be described with reference to FIG. 5. FIG. 5 shows an example of a table stored in table 10622 of ROM 1062.

As shown in FIG. 5, table 10622 stores a first image group and a second image group in which images (characters) are set corresponding to note names (here, examples are C, D, E, F, G, L, and S). The image determination unit 1052 assigns, for example, the first image group to a first instrument and the second image group to a second instrument. Furthermore, table 10622 is not limited to this, and images corresponding to other note names (e.g., C#) and chord types may also be stored.

For example, when the image determination unit 1052 determines that the note name included in the first performance information received from the first instrument is "C", it determines the image (character) of "butterfly" in the first image group of the table 10622. When the image determination unit 1052 determines that the note name included in the second performance information received from the second instrument is "Mi", it determines the image (character) of "monkey" in the second image group of the table 10622.

Here, a method for displaying an image (character) will be described with reference to FIG. 6. FIG. 6 is a diagram showing an example of how an image (character) is displayed.

As shown in FIG. 6, the image (character) stored in table 10622 is, for example, a basic shape. The image determination unit 1052 determines the size of the image (character) to be displayed on the display device 4 based on the velocity value included in the performance information. The size of the basic shape is determined as height HB and width WB. Then, for example, based on the velocity value, the height HB and width WB are changed to height HC and width WC by changing the percentage. The image determination unit 1052 may also randomly rotate the basic shape of the image (character) by R degrees, flip it left and right, or change the color. Although not described in detail here, the image determination unit 1052 may also change the size, rotation, flip, or color as appropriate based on information not shown here.

When displaying a real-time image, the image determination unit 1052 determines a trajectory pattern (first condition) for arranging an image (character) on the display unit of the display device 4 based on performance information (e.g., chord progression). The trajectory pattern (first condition) is stored, for example, in the ROM 1062.

Now, referring to FIG. 7, an example of a trajectory pattern when displaying an image (character) will be described. FIG. 7 is a diagram showing an example of a trajectory pattern of an image (character) in a real-time image displayed on the display unit 401 of the display device 4. FIG. 7 shows a real-time image based on two instruments (a first instrument or a second instrument).

As shown in FIG. 7, an example of a trajectory pattern for a first instrument is where images (characters) are arranged in order on the upper half of the real-time image (see 1st (U)) and on a virtual arrow along arrow A1. An example of a trajectory pattern for a second instrument is where images (characters) are arranged in order on the lower half of the real-time image (see 1st (L)) and on a virtual arrow along arrow A2.

In addition, each image (character) is positioned so that at least a portion of each image (character) overlaps with the virtual arrows (A1, A2) of the trajectory pattern.

The image determination unit 1052 generates a trajectory pattern (second condition) of a post-performance image to be displayed on the display device 4 after the user's performance. The image determination unit 1052 generates a trajectory pattern (second condition) in which an image (character) associated with each performance information is arranged. The image determination unit 1052 determines the length of the performance trajectory pattern (second condition) based on the performance time calculated by the performance judgment unit 1051. For example, for a first length which is the length of the trajectory pattern (second condition) when the performance time is a first performance time length and a second length which is the length of the trajectory pattern (second condition) when the performance time is a second performance time length, if the first performance time length is shorter than the second performance time length, the first length will be shorter than the second length.

An example of a trajectory pattern (second condition) generated by the image determination unit 1052 will be described with reference to FIG. 8. FIG. 8 is a diagram showing a spiral line, which is an example of a trajectory pattern (second condition) generated by the image determination unit 1052.

When generating the post-performance image, the image determination unit 1052 places the images (characters) on a virtual spiral line along the arrows as shown in FIG. 8 in order from the "beginning of the performance" to the "end of the performance." In particular, the image (character) relating to the first instrument is placed on the side of the arrow B1, and the image (character) relating to the second instrument is placed on the side of the arrow B2.

For example, the "start of performance" and "end of performance" positions are fixed, and the length of the spiral trajectory pattern (second condition) increases as the performance time increases.

The image information output unit 1053 generates a real-time image or a post-performance image by placing an image (character) associated with the performance information on the trajectory pattern (second condition) generated by the image determination unit 1052, and generates image information, which is information for displaying the image.

The performance end determination unit 1055 determines whether or not the performance has ended based on at least one of the following: not accepting input of performance information for a certain period of time; and accepting information indicating the end of the performance via the communication unit 103.

The condition determination unit 1054 can determine whether or not the first condition is satisfied, or whether or not the second condition is satisfied, based on the performance data acquired via the communication unit 103, or on information from the performance end determination unit 1055. The condition determination unit 1054 then notifies the image determination unit 1052 of the result. Note that this condition is set (stored) in, for example, the RAM 1061.

The image determination unit 1052 generates a real-time image or a post-performance image based on information from the condition determination unit 1054.

The above-described units (operation unit 101, display unit 102, communication unit 103, sound output unit 104, control unit 105, and memory unit 106) are communicatively connected via bus 107, allowing necessary data to be exchanged between the units.

When a user presses a key 11 on the electronic musical instrument 2 to play the instrument, the key press detection unit 50 detects the pressing of the key 11.

When pressing of a key 11 is detected, the control unit 70 of the electronic musical instrument 2 generates performance information (e.g., note-on including pitch information (such as note number and velocity value)) corresponding to the pressed key 11. The control unit 70 of the electronic musical instrument 2 then controls the speaker 3 (or the sound generation unit 40) to produce a sound corresponding to the performance information.

On the other hand, when the user releases the key 11 that was pressed, the key press detection unit 50 detects the release of the key 11.

When the release of a key 11 is detected, the control unit 70 of the electronic musical instrument 2 generates performance information (e.g., note off) corresponding to the released key 11. The control unit 70 of the electronic musical instrument 2 then controls the speaker 3 (or the sound generation unit 40) to mute the sound corresponding to the performance information.

The performance information generated by the control unit 70 of the electronic musical instrument 2 is then transmitted from the communication unit 80 of the electronic musical instrument 2 to the communication unit 103 of the information processing device 1.

<1-1-8> Speaker The speaker 3 is a part that outputs sound from the sound generation unit 40. When the speaker 3 is connected to the HP jack 44 of the electronic musical instrument 2, it receives sound signals from the HP jack 44 and outputs sound.

<1-1-9> Display Device The display device 4 has a liquid crystal monitor as a display unit 401, and can display an image. The display device 4 is connected to the communication unit 103 of the information processing device 1 by wire, and receives data from the information processing device 1. Then, the display device 4 displays an image based on the data on the display unit 401.

<1-1-10> Combinations of Instruments Combinations of instruments in this embodiment will be described with reference to FIG.

As shown in Figure 9, there are approximately eight possible combinations of instruments.

<1-1-10-1> Case 1
Case 1 is a case in which only an electronic musical instrument 2 ( 0 ) (digital single-tone musical instrument) is connected to the information processing device 1 .

In this case, for example, single-note performance data is supplied to the information processing device 1 via the electronic musical instrument 2 (single-note input).

<1-1-10-2> Case 2
Case 2 is a case in which only an electronic musical instrument 2 ( 1 ) (digital chord instrument) is connected to the information processing device 1 .

In this case, for example, chord performance data is supplied to the information processing device 1 via the electronic musical instrument 2 (chord input).

<1-1-10-3> Case 3
Case 3 is a case in which an electronic musical instrument 2(0) (digital monophonic instrument) and an electronic musical instrument 2(1) (digital monophonic instrument) are connected to the information processing device 1.

In this case, single note performance data and chord performance data are supplied to the information processing device 1 via, for example, the MIDI patcher 5. This is the case, for example, when one person plays the accompaniment and another person plays the melody.

<1-1-10-4> Case 4
Case 4 is a case in which an electronic musical instrument 2 ( 1 ) (digital harmony instrument) and a vocal microphone 6 (vocal or acoustic monophonic instrument) are connected to the information processing device 1 .

In this case, for example, chord performance data is supplied to the information processing device 1 via the electronic musical instrument 2, and single note performance data is supplied to the information processing device 1 via the audio interface 8. This is the case, for example, when singing along with musical accompaniment.

<1-1-10-5> Case 5
Case 5 is a case in which a vocal microphone 6 (vocal or acoustic single-instrument) is connected to the information processing device 1 .

In this case, single-note performance data is supplied to the information processing device 1 via, for example, the audio interface 8. This is the case, for example, when there is only singing.

<1-1-10-6> Case 6
Case 6 is a case in which an acoustic piano 7 (an acoustic harmony instrument or a karaoke accompaniment sound source) is connected to the information processing device 1 .

In this case, for example, chord performance data is supplied to the information processing device 1 via the audio interface 8. This is the case, for example, when there is only an acoustic chord instrument or a karaoke accompaniment sound source.

<1-1-10-7> Case 7
Case 7 is a case in which a vocal microphone 6 (vocals or an acoustic monophonic instrument) and an acoustic piano 7 (acoustic harmonic instrument or karaoke accompaniment sound source) are connected to the information processing device 1 .

In this case, single note performance data and chord performance data are supplied to the information processing device 1 via, for example, the audio interface 8. This is the case, for example, when singing along with an acoustic chord instrument or a karaoke accompaniment sound source.

<1-1-10-8> Case 8
Case 8 is a case in which an electronic musical instrument 2(0) (digital monophonic instrument) and an acoustic piano 7 (acoustic chord instrument, or karaoke accompaniment sound source) are connected to the information processing device 1.

In this case, for example, single note performance data is supplied to the information processing device 1 via the electronic musical instrument 2, and chord performance data is supplied to the information processing device 1 via the audio interface 8. This is the case, for example, when playing a melody along with a karaoke accompaniment sound source.

The information processing device 1 according to this embodiment is capable of generating real-time images and post-performance images in all of the above-mentioned cases.

Note that the above case is just an example, and three or more musical instruments may be connected to the information processing device 1.

<1-2> Operation <1-2-1> Flowchart Next, the operation of the information processing device 1 will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the operation of the information processing device 1.

In the following explanation, it is assumed that the communication unit 103 of the information processing device 1 and the communication unit 80 of the electronic musical instrument 2 are connected. In addition, as will be explained below, it is assumed that the control unit 105 of the information processing device 1 is running an application for displaying an image on the display device 4.

<S1001>
As shown in Fig. 10, the control unit 105 (connected device determination unit 1056) of the information processing device 1 checks the musical instrument connected to the information processing device 1. For example, the case of connecting a musical instrument to the information processing device 1 is as shown in Fig. 9. Then, the control unit 105 (image determination unit 1052) of the information processing device 1 assigns a different image group to each musical instrument connected to the information processing device 1. As a result, the control unit 105 (image determination unit 1052) of the information processing device 1 performs drawing settings related to real-time images and post-performance images.

<S1002>
10, the control unit 105 of the information processing device 1 determines whether or not the performance information transmitted from the musical instrument is received by the communication unit 103 of the information processing device 1 and whether or not the performance information has been input (step S1002). If the performance information has not been input (step S1002: NO), the control unit 105 of the information processing device 1 proceeds to step S1006.

<S1003>
Then, when the performance information transmitted from the instrument is received by the communication unit 103 of the information processing device 1 (step S1002: YES), the control unit 105 of the information processing device 1 executes a reception process to accept the input of the performance information, and executes a performance judgment process (which may also be described as music analysis or music analysis process) of step S1003.

Specifically, the control unit 105 (performance determination unit 1051) of the information processing device 1 performs the performance determination process of step S1003 by determining, for example, the key, note names, chord types, velocity values, note lengths, syncopation, chord functions, and chord progressions of the song based on the received performance information.

However, as the performance judgment process of step S1003, the control unit 105 (performance judgment unit 1051) of the information processing device 1 may judge any one of the key, note name, chord type, velocity value, note length, syncopation, chord function, and chord progression of the song.

Details about what is determined in the performance determination process in step S1003 will be explained in the next step, S1004.

<S1004>
When the control unit 105 (image determination unit 1052) of the information processing device 1 finishes the performance determination process in step S1003, the process proceeds to step S1004, where the control unit 105 (image information output unit 1053) of the information processing device 1 generates a real-time image corresponding to the performance information in accordance with the timing at which the performance information was received. Then, the control unit 105 (image information output unit 1053) of the information processing device 1 executes a real-time image output process for outputting and displaying the image on the display device 4.

Please note that the above phrase "in accordance with the timing at which the performance information is received" does not mean simultaneously, but rather each time the performance information is received.

In this embodiment, a base image stored in the storage unit 106 of the information processing device 1 is processed using computer graphics (CG) to create a real-time image, which is then displayed on the display device 4. However, the real-time image displayed on the display device 4 does not need to be limited to one obtained by such a method, and may be any real-time image that corresponds to the performance information.

<S1005>
The control unit 105 (performance end determination unit 1055) of the information processing device 1 performs a performance end determination process when the performance information transmitted from the musical instrument is not received by the communication unit 103 of the information processing device 1 (step S1002: NO), or when it is determined that the performance data does not satisfy the set condition (S1004: NO), or after step S1004. Specifically, the control unit 105 of the information processing device 1 determines whether or not a predetermined time has passed without input of performance information. If the control unit 105 of the information processing device 1 determines that the predetermined time has not passed without input of performance information (step S1005: NO), it proceeds to step S1002 again and determines whether or not performance information has been input.

<S1006>
The control unit 105 (play end determination unit 1055) of the information processing device 1 determines that the performance has ended if a predetermined time has elapsed without input of performance information (step S1005: YES), that is, if the reception of performance information is not detected for a predetermined time. If the control unit 105 (image determination unit 1052 and image information output unit 1053) of the information processing device 1 determines that the performance has ended, it proceeds to step S1006, and performs a post-play image output process of outputting and displaying a post-play image on the display device 4, and ends the entire process.

In addition, when this embodiment is applied to a case where accompaniment data is automatically played and the performer is allowed to specify a melody, it may be determined that the performance has ended when the automatic playback of the accompaniment data has ended, and post-performance image output processing may be performed. In other words, the control unit 105 of the information processing device 1 may determine that the performer has ended their performance for any reason, as long as the control unit 105 of the information processing device 1 determines that the performer has ended their performance, the post-performance image output processing may be performed.

On the other hand, if the predetermined time has not elapsed without input of performance information (step S1005: NO), the control unit 105 of the information processing device 1 again proceeds to step S1002 and determines whether or not performance information has been input.

<1-2-2> Specific Example Next, a specific example of the operation of the information processing device 1 will be described.

As a specific example, we will explain the case where two users play "Furusato" and image groups are assigned to the two users.

<1-2-2-1> Example of drawing of real-time images Next, a state in which different image groups are drawn when two users play "Hometown" will be described with reference to Fig. 11. Fig. 11 shows real-time images when a first user and a second user are playing a melody.

As shown in FIG. 11, when the first user plays a melody on the electronic musical instrument 2(0), the control unit 105 of the information processing device 1 causes the display device 4 to display, as a real-time image, an image (insect character) included in the first image group. At this time, for example, the image (insect character) is displayed along the arrow A1 in the upper half of the real-time image (see 1st (U)) (see C1 and D1 to D6 in the figure).

Also, as shown in FIG. 11, when the second user plays accompaniment on the electronic musical instrument 2(1), the control unit 105 of the information processing device 1 causes the display device 4 to display images (animal characters) included in the second image group as real-time images. At this time, for example, the images (animal characters) are displayed along the arrow A2 in the lower half of the real-time image (see 1st(L)) (see C2 and E1 to E6 in the figure).

<1-2-2-5> Example of Drawing Post-Performance Image Next, a post-performance image after two users finish playing "Hometown" will be described with reference to Fig. 12. Fig. 12 shows a post-performance image after two users finish playing the song.

When the control unit 105 of the information processing device 1 determines that the two users have finished playing, it causes the display device 4 to display a post-play image.

As shown in FIG. 12, the control unit 105 of the information processing device 1 places a first group of images (insect characters) in the locations corresponding to the performance of the first user (see C1), and places a second group of images (animal characters) in the locations corresponding to the performance of the second user (see C2).

This allows you to achieve the same effect as above.

<1-3> Effects According to the embodiment described above, in the performance data display system that draws images in accordance with a user's performance (music), when multiple performances are input, different expressions are output for each user.

For example, a performance data display system that draws images in time with a user's performance (music) can motivate users to practice playing an electronic musical instrument. However, when multiple users are performing together, unless the image is changed for each user, the result is the same as one person's performance with an increased number of notes. In this case, the correspondence between the image and each performance becomes unclear.

However, according to the above-described embodiment, it is possible to assign a different drawing representation to each of multiple users. As a result, the relationship between the user's performance and the image becomes clearer, providing greater enjoyment. As a result, it is possible to further stimulate the user's motivation to practice electronic musical instruments.

In addition, in the above-described embodiment, by using an audio interface and performing music analysis in the information processing device 1, even raw sound can be converted into an image.

<2> Modifications In the above-described embodiment, the control unit 105 of the information processing device 1 switches the image group when changing the drawing settings in step S1001 of FIG. 10. However, the present invention is not limited to this, and may be, for example, "display position of image (character)", "change of brightness", "change of image (character) size", "change of background color (background color of real-time image or post-performance image) or color table of image (character)", "change of type of image (bonus character) displayed when a specific condition is satisfied", or "change of image (character) movement". The background color (background color of real-time image or post-performance image) and color table of image (character) are stored in, for example, the ROM 1062. The image (bonus character) displayed when a specific condition is satisfied is, for example, an image displayed in the real-time image when the control unit 105 of the information processing device 1 determines that the rhythm of the user's performance is stable. The image (character) may move with animation or the like. The same image (character) may be included in multiple image groups. Also, if the instrument is "vocals", the "vocals" may be displayed large and prominently in the center of the real-time image and the post-performance image. Also, low-pitched instruments may be displayed in a rounded manner, and high-pitched instruments in a sharp manner.

In addition, in the above-described embodiment, an example of drawing images relating to two musical instruments has been shown, but this is not limiting. For example, as shown in FIG. 13, four musical instruments (piano, bass guitar, electric guitar, electronic instrument, etc.) may be connected to the information processing device 1. Furthermore, as shown in FIG. 13, for example, a real-time image or a post-performance image may be divided into four, and a musical instrument may be assigned to each. In this way, who is drawing which image (character) can be known from the location of the screen divided into four.

Also, as shown in FIG. 14, the real-time image or post-performance image may be drawn as one image without being clearly divided. However, since a different image (character) is used for each instrument, it is possible to determine who is drawing which image (character).

In addition, if the performances of multiple users satisfy a certain condition (for example, matching timing), a special image (character) such as a ribbon may be drawn in the real-time image or the post-performance image.

Below is a simple flowchart explaining how to draw these special images (characters).

Steps S2001 to S2004, S2007, and S2008 shown in FIG. 15 are similar to steps S1001 to S1006 in FIG. 10. Therefore, steps S2005 and S2006 will be explained.

<S2005>
The control unit 105 (condition determination unit 1054) of the information processing device 1 determines whether or not the performance data satisfies a set condition. As an example of determining whether or not the set condition is satisfied, it may be determined whether or not the timing of performances by multiple users coincides.

<<S2006>>
When the control unit 105 (image determination unit 1052 and image information output unit 1053) of the information processing device 1 determines that the performance data satisfies the set conditions (S2005: YES), it changes the drawing settings of the real-time image (and the post-performance image). Specifically, the control unit 105 (image determination unit 1052 and image information output unit 1053) of the information processing device 1 displays a dedicated image (character) after the performance data determined to satisfy the set conditions. Specifically, as shown in FIG. 16, it draws an image (character) of a ribbon.

The control unit 105 of the information processing device 1 may determine whether a note is a "chord" or a "single note" manually (e.g., by selecting a MIDI channel) or automatically.

In addition, the control unit 105 of the information processing device 1 converts the results of pitch extraction and multi-tone analysis into MIDI format, but it does not have to be in MIDI format.

Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims.

The above embodiment includes the following additional notes:

<Appendix 1>
The computer of the information processing device
determining whether the input performance data is input from at least a first device (e.g., a first electronic musical instrument) or a second device (e.g., a second electronic musical instrument or an audio interface);
When it is determined that the input device is the first device, the first image data is displayed according to information that can be determined based on the input performance data (e.g., note name, chord type. The performance data does not include note name information or chord type information, but note name information and chord type information can be determined from note number information included in the performance data).
When the device is determined to be the second device, even if information (e.g., note name, chord type) that can be determined based on the input performance data is the same as when the device is determined to be the first device, second image data in a form different from the first image data is displayed.
program.

<Appendix 2>
The source from which the performance data is input is determined based on information included in the input performance data (e.g., MIDI channel information).
2. The program according to claim 1.

<Appendix 3>
The first image data is included in a first image group;
The second image data is included in a second image group;
The first image group and the second image group may include the same image data.
3. The program according to claim 1 or 2.

<Appendix 4>
The information that can be determined includes at least one of pitch name information (for example, pitch name information is not included in the performance data, and pitch name information can be determined from note number information included in the performance data) and chord type information (for example, similar to pitch name information).
4. The program according to claim 1 ,

<Appendix 5>
the first device includes one of a first electronic musical instrument and an audio interface;
the second device includes one of a second electronic musical instrument and an audio interface;
5. The program according to claim 1 .

<Appendix 6>
The computer of the information processing device
determining whether the input performance data is input from at least a first device (e.g., a first electronic musical instrument) or a second device (e.g., a second electronic musical instrument or an audio interface);
When it is determined that the input device is the first device, the first image data is displayed according to information that can be determined based on the input performance data (e.g., note name, chord type. The performance data does not include note name information or chord type information, but note name information and chord type information can be determined from note number information included in the performance data).
When the device is determined to be the second device, even if information (e.g., note name, chord type) that can be determined based on the input performance data is the same as when the device is determined to be the first device, second image data in a form different from the first image data is displayed.
Method.

<Appendix 7>
A display device;
a processor, the processor comprising:
determining whether the input performance data is input from at least a first device (e.g., a first electronic musical instrument) or a second device (e.g., a second electronic musical instrument or an audio interface);
When the device is determined to be the first device, instruct the display device to display first image data according to information that can be determined based on the input performance data (e.g., note name, chord type. The performance data does not include note name information or chord type information, but note name information and chord type information can be determined from note number information included in the performance data);
When the device is determined to be the second device, even if information (e.g., note name, chord type) that can be determined based on the input performance data is the same as that when the device is determined to be the first device, instruct the display device to display second image data in a format different from that of the first image data.
Electronics.

<Appendix 8>
A first device, a second device, and a display device,
The first device is
generating first performance data including a first identifier (e.g., a first MIDI channel number) capable of identifying the first device in response to a performance operation by a first user, and outputting the generated first performance data to the display device;
The second device is
generating second performance data including a second identifier capable of identifying the second device in response to a performance operation by a second user, and outputting the generated second performance data to the display device;
The display device includes:
displaying first image data according to information (e.g., note name, chord type) that can be determined based on the input first performance data;
Even if information (e.g., note name, chord type) that can be determined based on the input second performance data is the same as information that can be determined based on the first performance data, the second image data is displayed in a manner different from the first image data.
Performance data display system.

1...information processing device 2...electronic musical instrument 3...speaker 4...display device 5...MIDI patcher 6...vocal microphone 7...acoustic piano 8...audio interface 11...keys 20...display unit 30...operation unit 40...sound generation unit 41...SP amplifier 42...speaker 43...HP amplifier 44...HP jack 45...HP jack insertion detection unit 50...key press detection unit 60...storage unit 61...ROM
62...RAM
70...Control unit 80...Communication unit 100...Bus 101...Operation unit 102...Display unit 103...Communication unit 104...Sound output unit 105...Control unit 106...Storage unit 107...Bus 401...Display unit 1051...Performance judgment unit 1052...Image determination unit 1053...Image information output unit 1054...Condition judgment unit 1055...Performance end judgment unit 1056...Connected device judgment unit 1061...RAM
1062...ROM
10621... program 10622... table

Claims (7)

The computer of the information processing device
determining whether the input performance data is input from at least the first device or the second device;
When the device is determined to be the first device, the first image data included in the first image group is displayed in accordance with information that can be determined based on the input performance data;
When the device is determined to be the second device, even if information that can be determined based on the input performance data is the same as that when the device is determined to be the first device, second image data that is included in a second image group and has a different aspect from the first image data is displayed ;
The first image group and the second image group may include the same image data.
program. The source from which the performance data is input is determined based on information included in the input performance data.
The program according to claim 1. The information that can be determined includes at least either pitch name information or chord type information.
3. The program according to claim 1 or 2 . the first device includes one of a first electronic musical instrument and an audio interface;
the second device includes one of a second electronic musical instrument and an audio interface;
The program according to any one of claims 1 to 3 . The computer of the information processing device
determining whether the input performance data is input from at least the first device or the second device;
When the device is determined to be the first device, the first image data included in the first image group is displayed in accordance with information that can be determined based on the input performance data;
When the device is determined to be the second device, even if information that can be determined based on the input performance data is the same as that when the device is determined to be the first device, second image data that is included in a second image group and has a different aspect from the first image data is displayed ;
The first image group and the second image group may include the same image data.
Method. A display device;
a processor, the processor comprising:
determining whether the input performance data is input from at least the first device or the second device;
When the device is determined to be the first device, instruct the display device to display first image data included in a first image group in accordance with information that can be determined based on the input performance data;
when it is determined that the device is the second device, even if information that can be determined based on the input performance data is the same as that when it is determined that the device is the first device, instruct the display device to display second image data that is included in a second image group and has a different aspect from the first image data ;
The first image group and the second image group may include the same image data.
Electronics. A first device, a second device, and a display device,
The first device is
generating first performance data including a first identifier capable of identifying the first device in response to a performance operation by a first user, and outputting the generated first performance data to the display device;
The second device is
generating second performance data including a second identifier capable of identifying the second device in response to a performance operation by a second user, and outputting the generated second performance data to the display device;
The display device includes:
displaying first image data included in a first image group according to information that can be determined based on the first performance data input;
displaying second image data in a manner different from that of the first image data , the second image data being included in a second image group, even if information that can be determined based on the second performance data inputted is the same as information that can be determined based on the first performance data ;
The first image group and the second image group may include the same image data.
Performance data display system.