JP2021099459A - Program, method, electronic apparatus, and musical performance data display system - Google Patents

Abstract

To provide a program, a method, an electronic apparatus, and a musical performance data display system that can express a musical performance well visually.SOLUTION: The program causes a computer of an information processor to: determine whether input musical performance data is musical performance data input from at least one of a first apparatus and a second apparatus; display first image data according to information that can be determined according to the input musical performance data if the input musical performance data is determined to be input from the first apparatus; and display second image data in a format different from that of the first image data even if information that can be determined on the basis of input musical performance data is the same as the one when the input musical performance data is determined to be input from the first apparatus in a case where the input musical performance data is determined to be input from the second apparatus.SELECTED DRAWING: Figure 1

Description

Embodiments relate to programs, methods, electronic devices, and performance data display systems.

For example, an image object such as a dancer (dancer) is moved in synchronization with the performance of music such as MIDI (Musical Computer Interface) music, and is integrated with this not only according to the musical composition but also according to the progress of the musical sound. A computer graphics moving image generation system capable of generating a changing moving image is known (see Patent Document 1).

Japanese Unexamined Patent Publication No. 11-224084

The computer graphics moving image generation system disclosed in Patent Document 1 has a problem that it lacks expressive power because it merely dynamically causes a character indicating a dancer to appear during a performance.

The present invention has been made in view of such circumstances, and an object of the present invention is, for example, to provide a program, a method, an electronic device, and a performance data display system having excellent visual expressiveness of performance.

According to the program according to the embodiment, the computer of the information processing apparatus is made to determine whether the input performance data is at least the performance data input from the first device or the second device, and the first device. When it is determined that the first image data is displayed according to the information that can be determined based on the input performance data, and when it is determined that the second device is used, the determination is made based on the input performance data. Even if the information that can be generated is the same as when it is determined to be the first device, the second image data in a mode different from that of the first image data is displayed.

According to the present invention, for example, it is possible to provide a program, a method, an electronic device, and a performance data display system having excellent visual expressiveness of performance.

FIG. 1 is a diagram schematically illustrating an example of a performance data display system including a program according to an application example. FIG. 2 is a diagram showing an outline of the performance data display system according to the embodiment. FIG. 3 is a diagram showing the hardware of the control system of the electronic musical instrument according to the embodiment. FIG. 4 is a diagram showing the hardware of the information processing device according to the embodiment. FIG. 5 is a diagram showing an example of a table stored in the ROM table. FIG. 6 is a diagram showing a display example of an image (character). FIG. 7 is a diagram showing an example of a trajectory pattern of an image (character) in a real-time image displayed on a display unit of a display device. FIG. 8 is a diagram showing a spiral winding which is an example of an orbital pattern generated by the image determining unit. FIG. 9 is a table showing connection cases of musical instruments connected to the information processing device. FIG. 10 is a flowchart showing the operation of the information processing device. FIG. 11 shows a real-time image when the first user is playing the melody.

FIG. 12 shows post-performance images after the first and second users have finished playing the song. FIG. 13 is a diagram showing an example of a real-time image when four musical instruments are connected to an information processing device. FIG. 14 is a diagram showing an example of a real-time image when three or more musical instruments are connected to an information processing device. FIG. 15 is a flowchart showing the operation of the information processing device. FIG. 16 is a diagram showing an example of a real-time image with respect to FIG.

Hereinafter, embodiments according to one aspect of the present invention (hereinafter, also referred to as “embodiments”) will be described with reference to the drawings. However, the embodiments described below are merely examples of the present invention in all respects. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. That is, in carrying out the present invention, a specific configuration according to the embodiment may be appropriately adopted. The data appearing in the embodiment are described in natural language, but more specifically, the data is specified in a pseudo language, a command, a parameter, a machine language, or the like that can be recognized by a computer.

<0> Application Example First, an example of a situation in which the present invention is applied will be described with reference to FIG. FIG. 1 schematically illustrates an example of a performance data display system including a program according to an application example. The performance data display system is supposed to stimulate the user's motivation to practice an electronic musical instrument by drawing an image (picture) in real time according to the performance (music) of the user (performer).

<0-1> Outline of application example
Before explaining the configuration of the performance data display system, the outline of the performance data display system will be described. The performance data display system analyzes performance data received from an electronic musical instrument or the like that can output performances of a plurality of users as performance data (for example, MIDI data), and converts the performances of the plurality of users into images. For example, the performance data does not include the note name information and the chord type information, and the note number information and the chord type information can be determined from the note number information included in the performance data.

<0-2> Configuration of Application Example The configuration of the application example will be described with reference to FIG. FIG. 1 is a diagram showing an example of the configuration of a performance data display system according to an application example.

As shown in FIG. 1, the performance data display system includes a first musical instrument, a second musical instrument, an information processing device, and a display device.

The first musical instrument converts the performance of the first user into performance data (for example, MIDI data). Then, the first musical instrument outputs the performance data to the information processing device.

More specifically, the first musical instrument generates first performance data including a first identifier (for example, the first channel number of MIDI) that can identify the first musical instrument according to a performance operation by the first user. , The generated first performance data is output to the information processing device. The first musical instrument is, for example, an electronic musical instrument such as a keyboard, an electronic piano, an electric guitar, or an electronic violin. However, the first musical instrument is not limited to the one described above, and may be appropriately selected according to the embodiment.

The second musical instrument converts the performance of the second user into performance data (for example, MIDI data). Then, the second musical instrument outputs the performance data to the information processing device.

More specifically, the second musical instrument generates the second performance data including the second identifier that can identify the second musical instrument according to the performance operation by the second user, and also informs the generated second performance data. Output to the processing device. The second musical instrument is, for example, an electronic musical instrument, a vocal microphone, an acoustic piano, or the like. However, the second musical instrument is not limited to the one described above, and may be appropriately selected according to the embodiment.

The information processing device determines whether the input performance data is at least the performance data input from the first musical instrument or the second musical instrument. Further, when the information processing device determines that the input performance data is the performance data from the first device, the information processing device provides information (for example, sound name, chord type, etc.) that can be determined based on the input performance data. The first image (character) data is output accordingly. Further, when the information processing device determines that the input performance data is the performance data from the second device, the information processing device can determine the information (for example, sound name, chord type, etc.) based on the input performance data. , Even if the information is the same as the information that can be determined based on the first performance data, the second image data (character) having a mode different from that of the first image data is output. The information processing device is, for example, a PC (personal computer). Note that "setting" or "registration" basically has the same meaning as "memory".

The display device displays an image based on the performance of the first user and an image based on the performance of the second user on the display unit. As described above, the image based on the performance of the first user is generated based on the first image group, and the image based on the performance of the second user is generated based on the second image group. Therefore, it is possible to determine an image based on the performance of the first user and an image based on the performance of the second user.

<0-3> Operation of application example Next, an example of operation of the performance data display system according to the application example will be described.

When the first user starts playing using the first instrument, the information processing device determines that the performance data is from the first instrument (first performance data) from the information included in the input performance data. Then, the information processing device performs music analysis on the performance of the first musical instrument. Specifically, the information processing device has an image (for example, for example) included in the first image group (for example, a character group including a plurality of insect characters) according to a sound name (or chord) included in the first performance data. The character of the beetle) is determined and drawn in real time along the trajectory according to the set first condition (for example, the display according to the code progress). As a result, the image (character) included in the first image group is displayed on the display device in real time. The "real time" here does not necessarily mean that the timing of the performance and the timing of displaying the image (character) are the same. Further, an image in which an image (character) is displayed according to the performance may be described as, for example, a real-time image. That is, a plurality of images (characters) are drawn in the real-time image. The drawing method of the real-time image will be described later.

When the second user starts playing using the second instrument, the information processing device determines that the performance data is from the second instrument (second performance data) from the information included in the input performance data. Then, the information processing device performs music analysis on the performance of the second musical instrument. Specifically, the information processing device has an image (for example, for example) included in the second image group (for example, a character group including a plurality of animal characters) according to a sound name (or chord) included in the second performance data. , Kirin character) is determined and drawn in real time along the trajectory according to the set first condition (for example, display according to the code progress). As a result, the image (character) included in the second image group is displayed on the display device in real time.

In this way, the information processing apparatus draws the real-time image so that the image based on the performance of the first user and the image based on the performance of the second user can be determined. Further, after the first user and the second user finish playing the musical instrument, the information processing device causes the display device to display the image data according to the set second condition (for example, spiral display). In addition, such an image may be described as, for example, a post-performance image. The post-performance image is an image in which the images (characters) analyzed from the beginning to the end of the performance of the first user and the second user user are drawn according to the second condition (for example, spiral display). At this time, similarly to the real-time image, the information processing apparatus draws the image based on the performance of the first user and the image based on the performance of the second user as the post-performance image so that it can be determined.

<0-4> Effect of Application Example As described above, according to the performance data display system according to the application example, it is possible to determine the performance of the first user and the second user and change the group of images to be displayed. In other words, according to the performance data display system according to the application example, during the performance of the first user and the second user, settings (drawing settings) related to the image are appropriately performed without performing a peculiar operation unrelated to the performance. You can do things. Therefore, the performance data display system can generate a highly expressive image while the first user and the second user concentrate on the performance.

<1> Embodiment <1-1> Configuration <1-1-1> Performance data display system The performance data display system according to the embodiment will be described below with reference to FIG.

FIG. 2 is a diagram showing an outline of the performance data display system according to the embodiment.

As shown in FIG. 2, the performance data display system includes, for example, an information processing device (electronic device) 1, an electronic musical instrument 2 (for example, 2 (0) and 2 (1)), a speaker 3, and a display device 4. , MIDI patcher (MIDI patch bay) 5, vocal microphone 6, acoustic piano 7, and audio interface 8.

Note that FIG. 2 shows two electronic musical instruments 2, a vocal microphone 6, and an acoustic piano 7 as an example, but this is an example. You only need at least two instruments. Further, in the present embodiment, the electronic musical instrument 2, the vocal microphone 6, and the acoustic piano 7 are collectively treated as musical instruments.

<1-1-2> Electronic Musical Instrument Hereinafter, the electronic musical instrument according to the embodiment will be described with reference to FIGS. 2 and 3. FIG. 3 is a diagram showing the hardware of the control system of the electronic musical instrument according to the embodiment.

As shown in FIGS. 2 and 3, the electronic musical instrument 2 to which the information processing device 1 of the present embodiment is connected is, for example, an electronic keyboard instrument such as an electronic piano, a synthesizer, or an electronic organ.

The electronic musical instrument 2 includes a plurality of keys 11, a display unit 20, an operation unit 30, and a sounding unit 40.

Further, as shown in FIG. 3, the electronic musical instrument 2 includes a key pressing detection unit 50, a storage unit 60, a control unit 70, and a communication unit 80.

The key 11 is an operator whose pitch is specified by the performer, and when the performer presses and releases the key 11, the electronic musical instrument 2 produces and mutes the sound corresponding to the designated pitch. I do.

The display unit 20 has, for example, a liquid crystal monitor with a touch panel, and is a part that displays a message or the like accompanying the operation of the operation unit 30 of the performer.

In this embodiment, since the display unit 20 has a touch panel function, the display unit 20 can play a part of the operation unit 30.

The operation unit 30 has operation buttons for the performer to perform various settings and the like, and is a part for performing various setting operations such as volume adjustment.

The sounding unit 40 is a part that outputs sound, and is an HP jack 44 (headphones) into which the plugs of the SP amplifier 41 (speaker amplifier), the speaker 42, the HP amplifier 43 (headphone amplifier), and the speaker 3 (or headphones) are inserted. It has a jack) and an HP jack insertion detection unit 45 that detects that the plug of the speaker 3 (or headphones) is inserted into the HP jack 44.

When the plug of the speaker 3 (or headphones) is inserted into the HP jack 44, the HP jack insertion detection unit 45 detects the insertion, and the sound is output to the HP jack side, and conversely, the HP jack insertion detection is performed. When the unit 45 does not detect the insertion of the plug, the sound is output to the speaker 42 side.

The sounding unit 40 may be used instead of the speaker 3.

The key press detection unit 50 is a portion that detects the key press and release of the key 11. For example, a rubber switch is provided under each key 11, and the key press and release are detected by the rubber switch. To.

The storage unit 60 includes a ROM 61 which is a read-only storage unit and a RAM 62 which is a readable and writable storage unit.

The ROM 61 stores, for example, a control program (for example, a program for a lesson mode program) executed by the control unit 70, various data tables, and the like.

The RAM 62 is a storage area for storing data such as music sound data and music data corresponding to each key 11, and a temporary storage area for developing data generated by the control unit 70 and a control program during performance. Also works as.

The control unit 70 is a part that controls the entire electronic musical instrument 2. The control unit 70 is, for example, an LSI (Large Scale Integration).

Then, the control unit 70 controls, for example, to make the musical sound sound from the sounding unit 40 in response to the key pressed by the key 11, and to mute the musical sound sounded from the sounding unit 40 in response to the release of the key 11. Do.

The communication unit 80 includes a wireless unit and a wired unit for communicating with an external device. In the present embodiment, the information processing device 1 is connected by wire, and as will be described later, information processing is performed. Data is transmitted to the device 1.

However, the communication unit 80 does not have to be limited to one that performs only a transmission function, and may also have a reception function.

Then, each unit (display unit 20, operation unit 30, sound generation unit 40, key press detection unit 50, storage unit 60, control unit 70, and communication unit 80) described above is communicably connected by the bus 100. Therefore, necessary data can be exchanged between each department.

<1-1-3> MIDI Patcher Next, the MIDI patcher (MIDI patch bay) 5 will be described.

The MIDI patcher 5 is a device that changes the path of a MIDI signal.

Specifically, the MIDI patcher has a plurality of MIDI IN terminals and MIDI OUT terminals as a pair, and each of them has a number. The MIDI patcher can freely switch the connection of a plurality of MIDI devices by operating it with a switch or the like.

In the case of FIG. 2, the MIDI patcher 5 transfers the MIDI signals of the two electronic musical instruments 2 to the information processing device 1 when the two electronic musical instruments 2 are connected.

<1-1-4> Vocal Microphone Next, the vocal microphone 6 will be described.

The vocal microphone 6 is a device for collecting sounds of vocals and live musical instruments. More specifically, the vocal microphone 6 converts sound (vibration of air) into an electric signal. Examples of the vocal microphone 6 include a dynamic microphone (power supply not required), a condenser microphone (power supply required), and the like.

<1-1-5> Acoustic Piano Next, the acoustic piano 7 will be described.

The acoustic piano 7 is an instrument that does not use an electric speaker, and when a player presses a keyboard, a hammer strikes a string and vibrates, and the vibration is transmitted to the soundboard and the entire body to produce a sound. Although not shown, when converting the sound of the acoustic piano 7 into an electric signal, it is necessary to use the same equipment as the vocal microphone 6.

<1-1-6> Audio Interface Next, the audio interface 8 will be described.

By the way, when the information processing device 1 handles sounds such as vocals and acoustic pianos (called analog signals), it is necessary to convert them into digital signals. The audio interface 8 is a device capable of converting an analog signal into a digital signal (AD conversion). Therefore, by using the audio interface 8, the analog signal from the vocal microphone 6 or the acoustic piano 7 can be converted into a digital signal and supplied to the information processing device 1.

<1-1-7> Information Processing Device Next, the information processing device will be described with reference to FIGS. 2 and 4. FIG. 4 is a diagram showing the hardware of the information processing device according to the embodiment.

In the present embodiment, the information processing device 1 is a PC as shown in FIG. 2, and an application for displaying an image as described later on the display device 4 is installed.

However, the information processing device 1 does not have to be limited to a PC, and may be a tablet-type mobile terminal or the like.

Further, the information processing device 1 may include a sequencer or the like that receives a MIDI code from the electronic musical instrument 2 and reproduces the song data.

As shown in FIG. 4, the information processing device 1 mainly includes an operation unit 101, a display unit 102, a communication unit 103, a sound output unit 104, a control unit 105, and a storage unit 106. There is.

The operation unit 101 includes, for example, switches such as a power switch for turning on / off the power of the information processing device 1.

The display unit 102 has a liquid crystal monitor and can display an image. The display unit 102 may have a touch panel function. In that case, the display unit 102 can play one end of the operation unit 101. The display unit 102 may be used instead of the display device 4.

The communication unit 103 includes a wireless unit and a wired unit for communicating with other devices and the like. The communication unit 103 receives data from, for example, the electronic musical instrument 2 or the audio interface 8. Further, the communication unit 103 is connected by wire to, for example, the communication unit of the display device 4, and transmits data to the display device 4.

The sound output unit 104 is a part that outputs sound, and has a speaker, an earphone jack, and the like.

The storage unit 106 includes a RAM (Random Access Memory) 1061 which is a read / write storage unit which is a read-only storage unit, and a ROM (Read Only Memory) 1062.

The RAM 1061 is a storage area that includes a volatile memory and stores data necessary for operating a control program, data generated by the control unit 105, and data sent from the electronic instrument 2 (pitch information). It also functions as a temporary storage area for expanding applications) and performance information including.

The ROM 1062 is a non-volatile semiconductor memory such as a flash memory, an EPROM (Erasable Program Read Only Memory), or an EEPROM (registered trademark) (Electrically Erasable Program ROM), and serves as a so-called secondary storage device or an auxiliary storage device. ..

ROM 1062 stores program 10621 and table 10622. Program 10621 is used by the control unit 105 to perform various processes. Table 10622 is a table in which performance information and images are linked.

The control unit 105 is a part that controls the entire information processing device 1. The control unit 105 includes, for example, a CPU (Central Processing Unit). The control unit 105 collectively controls the information processing device 1 by reading the program 10621 stored in the ROM 1062 of the storage unit 106 and using the RAM 1061 of the storage unit 106 as a work area.

Next, the functional configuration of the control unit 105 of the information processing device 1 according to the embodiment will be described. By reading and executing the program 10621 stored in the ROM 1062, the control unit 105 reads and executes the performance determination unit 1051, the image determination unit 1052, the image information output unit 1053, the condition determination unit 1054, and the performance end determination unit 1055. And the connected device determination unit 1056. The performance determination unit 1051, the image determination unit 1052, the image information output unit 1053, the condition determination unit 1054, the performance end determination unit 1055, and the connected device determination unit 1056 are not programs but circuit configurations. May be realized by.

In the connected device determination unit 1056, the performance data input from the outside (electronic musical instrument 2 or audio interface 8) is at least the first device (for example, electronic musical instrument 2) and the second device (for example, electronic musical instrument 2 or audio interface 8). ) To determine which of the input performance data is used.

By the way, MIDI data is transmitted through the port. And, for example, 16 "channels" are prepared in this port. The sound source can reproduce one part per one channel. Then, the electronic musical instrument 2 can transmit MIDI data by designating a channel. For example, when the two electronic musical instruments 2 specify different channels, the connected device determination unit 1056 can determine the two electronic musical instruments 2 from the MIDI data channel.

Further, the digital signal from the audio interface 8 and the MIDI signal are different. Therefore, the connected device determination unit 1056 can determine the performance data from the electronic musical instrument 2 and the performance data from the audio interface 8.

Based on the performance information (performance data) received from the electronic musical instrument 2 via the communication unit 103, the performance determination unit 1051 determines the tonality of the song (for example, 24 types from C major to B minor) and the sound name (for example, 24 types in B minor). Do, Re, Mi ...), chord type (for example, Major, Mimor, Sus4, Aug, Dim, 7th, etc.), velocity value (numerical value indicating the strength of the sound in the MIDI standard), tone length, syncopation, chord function, and Judge the chord progression. Any method of determining the tonality, note name, chord type, velocity value, note length, syncopation, chord function, and chord progression of the song may be used.

Further, the performance determination unit 1051 performs pitch extraction from the digital signal received from the audio interface 8 and performs music analysis based on the MIDI information obtained by converting from the pitch information. Further, the performance determination unit 1051 performs multiple sound analysis on the digital signal received from the audio interface 8, and performs music analysis based on the MIDI information obtained by converting the multiple sound analysis from the result. Note that MIDI created by multiple sound analysis may contain noise due to erroneous recognition, so a mechanism may be added to ignore short notes that are likely to occur due to erroneous recognition. Then, the performance determination unit 1051 determines the tonality, note name, chord type, velocity value, note length, syncopation, chord function, and chord progression of the song, similarly to the signal received from the electronic musical instrument 2.

The image determination unit 1052 can determine from which musical instrument the data was received based on the information from the connected device determination unit 1056.

Each time the image determination unit 1052 receives performance information from the electronic musical instrument 2, any image data (character) included in the image group (character group) is selected based on the information determined by the performance determination unit 1051. decide.

Here, with reference to FIG. 5, the relationship between the information determined by the performance determination unit 1051 and the image data will be described. FIG. 5 is a diagram showing an example of a table stored in the table 10622 of the ROM 1062.

As shown in FIG. 5, in the table 10622, for example, a first image (character) in which an image (character) is set corresponding to a note name (here, an example of do, re, mi, fa, so, la, shi) is set. The group and the second image group are stored. In the image determination unit 1052, for example, the first musical instrument is assigned the first image group, and the second musical instrument is assigned the second image group. Further, the table 10622 is not limited to this, and other note names (for example, do # etc.) and images according to the chord type may be stored.

For example, when the image determination unit 1052 determines that the note name included in the first performance information received from the first instrument is "do", the image determination unit 1052 determines that the image (character) of the "butterfly" in the first image group of the table 10622 is displayed. decide. Further, when the image determination unit 1052 determines that the note name included in the second performance information received from the second musical instrument is "mi", the image determination unit 1052 determines that the image (character) of the "monkey" in the second image group of the table 10622 is displayed. decide.

Here, a method of displaying an image (character) will be described with reference to FIG. FIG. 6 is a diagram showing a display example of an image (character).

As shown in FIG. 6, the image (character) stored in the table 10622 is, for example, a basic form. The image determination unit 1052 determines the size of the image (character) to be displayed on the display device 4 based on the velocity value included in the performance information. The size of the basic shape is fixed as height HB and width WB. Then, for example, the height HC and the width WC are changed by changing the percentages of the height HB and the width WB based on the velocity value. Further, the image determination unit 1052 may randomly rotate the basic shape of the image (character) by R degree, invert it left and right, or change the color. Although not described in detail here, the image determination unit 1052 may appropriately change the size, rotation, inversion, or color based on the information not shown here.

When displaying a real-time image, the image determination unit 1052 determines a trajectory pattern (first condition) for arranging an image (character) on the display unit of the display device 4 based on performance information (for example, chord progression). The orbital pattern (first condition) is stored in, for example, ROM 1062.

Here, an example of an orbital pattern when displaying an image (character) will be described with reference to FIG. 7. FIG. 7 is a diagram showing an example of a trajectory pattern of an image (character) in a real-time image displayed on the display unit 401 of the display device 4. FIG. 7 shows a real-time image based on two musical instruments (first musical instrument or second musical instrument).

As shown in FIG. 7, as an example of the trajectory pattern related to the first musical instrument, the upper half of the real-time image (see 1st (U)) and the image (character) are arranged in order on the virtual arrow along the arrow A1. Will be done. Further, as an example of the trajectory pattern relating to the second musical instrument, the images (characters) are sequentially arranged on the lower half of the real-time image (see 1st (L)) and on the virtual arrow along the arrow A2.

Each image (character) is arranged so that at least a part of each image (character) overlaps on the virtual arrows (A1, A2) of the trajectory pattern.

The image determination unit 1052 generates a trajectory pattern (second condition) of the post-performance image to be displayed on the display device 4 after the user's performance. The image determination unit 1052 generates a trajectory pattern (second condition) for arranging an image (character) associated with each performance information. The image determination unit 1052 determines the length of the performance trajectory pattern (second condition) based on the performance time calculated by the performance determination unit 1051. For example, the length of the first length, which is the length of the orbital pattern (second condition) when the playing time is the first playing time length, and the length of the orbital pattern (second condition), which is the length of the orbital pattern (second condition) when the playing time is the second playing time length. When the first playing time length is shorter than the second playing time length, the first length is shorter than the second length.

An example of the trajectory pattern (second condition) generated by the image determination unit 1052 will be described with reference to FIG. FIG. 8 is a diagram showing a spiral winding which is an example of an orbital pattern (second condition) generated by the image determining unit 1052.

When the image determination unit 1052 generates an image after the performance, the images (characters) are sequentially arranged on the virtual spiral winding along the arrow as shown in FIG. 8 from the "beginning of the performance" to the "end of the performance". Be placed. In particular, the image (character) relating to the first musical instrument is provided on the arrow B1 side, and the image (character) relating to the second musical instrument is provided on the arrow B2 side.

For example, the "start of performance" position and the "end of performance" position are fixed, and the length of the spiral orbital pattern (second condition) increases as the performance time increases. ..

The image information output unit 1053 arranges an image (character) associated with the performance information on the trajectory pattern (second condition) generated by the image determination unit 1052 to generate a real-time image or a post-performance image, and generates an image. Generates image information, which is information for display.

The performance end determination unit 1055 determines whether or not the performance has ended, at least by not accepting the input of the performance information for a certain period of time or by receiving the information indicating the end of the performance via the communication unit 103. To do.

The condition determination unit 1054 determines whether or not the first condition is satisfied or whether or not the second condition is satisfied based on the performance data acquired via the communication unit 103 or the information from the performance end determination unit 1055. Can be determined. Then, the condition determination unit 1054 notifies the image determination unit 1052 to that effect. This condition is set (stored) in, for example, RAM 1061.

The image determination unit 1052 generates a real-time image or a post-performance image based on the information from the condition determination unit 1054.

Each unit (operation unit 101, display unit 102, communication unit 103, sound output unit 104, control unit 105, and storage unit 106) described above is communicably connected by the bus 107, and is required between the units. Data can be exchanged.

When the user presses the key 11 of the electronic musical instrument 2 for playing, the key pressing detection unit 50 detects the key pressing of the key 11.

Then, when the key 11 is detected, the control unit 70 of the electronic musical instrument 2 obtains performance information (for example, pitch information (note number, velocity value, etc.)) corresponding to the key 11 pressed. Generate note-on) including. Then, the control unit 70 of the electronic musical instrument 2 controls the speaker 3 (or the sounding unit 40) to produce a sound according to the performance information.

On the other hand, when the user releases the key 11 that has been pressed, the key release detection unit 50 detects the release of the key 11.

Then, when the release of the key 11 is detected, the control unit 70 of the electronic musical instrument 2 generates performance information (for example, note-off) corresponding to the released key 11. Then, the control unit 70 of the electronic musical instrument 2 controls the speaker 3 (or the sounding unit 40) to mute the sound according to the performance information.

Then, the performance information generated by the control unit 70 of the electronic musical instrument 2 is transmitted from the communication unit 80 of the electronic musical instrument 2 to the communication unit 103 of the information processing device 1.

<1-1-8> Speaker In the speaker 3, the sounding unit 40 is a portion that outputs sound. The speaker 3 receives a sound signal from the HP jack 44 by connecting to the HP jack 44 of the electronic musical instrument 2, and outputs the sound.

<1-1-9> Display device The display device 4 has a liquid crystal monitor as a display unit 401 and can display an image. The display device 4 is wiredly connected to the communication unit 103 of the information processing device 1 and receives data from the information processing device 1. Then, the display device 4 displays an image based on the data on the display unit 401.

<1-1-10> Combination of musical instruments With reference to FIG. 9, the combination of musical instruments in the present embodiment will be described.

As shown in FIG. 9, approximately eight combinations of musical instruments are assumed.

<1-1-10-1> Case 1
Case 1 is a case in which only the electronic musical instrument 2 (0) (digital monophonic instrument) is connected to the information processing device 1.

In this case, single-note performance data is supplied to the information processing device 1 via, for example, the electronic musical instrument 2 (single-note input).

<1-1-10-2> Case 2
Case 2 is a case in which only the electronic musical instrument 2 (1) (digital Japanese music instrument) is connected to the information processing device 1.

In this case, chord performance data is supplied to the information processing device 1 via, for example, the electronic musical instrument 2 (chord input).

<1-1-10-3> Case 3
Case 3 is a case in which the electronic musical instrument 2 (0) (digital monophonic instrument) and the electronic musical instrument 2 (1) (digital monophonic instrument) are connected to the information processing device 1.

In this case, for example, the performance data of a single note and the performance data of a chord are supplied to the information processing device 1 via the MIDI patcher 5. This is the case, for example, when one person plays the accompaniment and the other plays the melody.

<1-1-10-4> Case 4
Case 4 is a case in which the electronic musical instrument 2 (1) (digital Japanese musical instrument) and the vocal microphone 6 (vocal or acoustic monophonic instrument) are connected to the information processing device 1.

In this case, for example, chord performance data is supplied to the information processing device 1 via the electronic musical instrument 2, and single note performance data is supplied to the information processing device 1 via the audio interface 8. This is the case, for example, when singing along with the accompaniment.

<1-1-10-5> Case 5
Case 5 is a case in which the vocal microphone 6 (vocal or acoustic monophonic instrument) is connected to the information processing device 1.

In this case, single-note performance data is supplied to the information processing device 1 via, for example, the audio interface 8. This is the case, for example, for songs only.

<1-1-10-6> Case 6
Case 6 is a case in which an acoustic piano 7 (acoustic Japanese music instrument or karaoke accompaniment sound source) is connected to the information processing device 1.

In this case, chord performance data is supplied to the information processing device 1 via, for example, the audio interface 8. This is the case, for example, only with an acoustic Japanese music instrument or a karaoke accompaniment sound source.

<1-1-10-7> Case 7
Case 7 is a case in which a vocal microphone 6 (vocal or acoustic monophonic instrument) and an acoustic piano 7 (acoustic Japanese musical instrument or karaoke accompaniment sound source) are connected to the information processing device 1.

In this case, for example, the performance data of a single note and the performance data of a chord are supplied to the information processing device 1 via the audio interface 8. This is the case, for example, when singing along with an acoustic Japanese music instrument or an accompaniment sound source of karaoke.

<1-1-10-4> Case 4
Case 4 is a case in which the electronic musical instrument 2 (0) (digital monophonic instrument) and the acoustic piano 7 (acoustic Japanese musical instrument or karaoke accompaniment sound source) are connected to the information processing device 1.

In this case, for example, single-tone performance data is supplied to the information processing device 1 via the electronic musical instrument 2, and chord performance data is supplied to the information processing device 1 via the audio interface 8. This is a case where, for example, a melody is played according to an accompaniment sound source of karaoke.

According to the information processing apparatus 1 according to the present embodiment, it is possible to generate a real-time image and a post-performance image in all the cases described above.

The above-mentioned case is an example, and three or more musical instruments may be connected to the information processing device 1.

<1-2> Operation <1-2-1> Flow chart Next, the operation of the information processing apparatus 1 will be described with reference to FIG. FIG. 10 is a flowchart showing the operation of the information processing device 1.

In the following description, it is assumed that the communication unit 103 of the information processing device 1 and the communication unit 80 of the electronic musical instrument 2 are connected. Further, as will be described below, it is assumed that the control unit 105 of the information processing device 1 has started an application for displaying an image on the display device 4.

≪S1001≫
As shown in FIG. 10, the control unit 105 (connected device determination unit 1056) of the information processing device 1 confirms the musical instrument connected to the information processing device 1. For example, the case of connecting the musical instrument to the information processing device 1 is as shown in FIG. Then, the control unit 105 (image determination unit 1052) of the information processing device 1 assigns a different image group to each musical instrument connected to the information processing device 1. As a result, the control unit 105 (image determination unit 1052) of the information processing device 1 sets drawing for the real-time image and the post-performance image.

≪S1002≫
As shown in FIG. 10, the control unit 105 of the information processing device 1 determines whether or not the performance information transmitted from the instrument is received by the communication unit 103 of the information processing device 1 and the performance information is input (step S1002). ). The control unit 105 of the information processing device 1 proceeds to step S1006 when there is no input of performance information (step S1002: NO).

≪S1003≫
Then, when the performance information transmitted from the musical instrument is received by the communication unit 103 of the information processing device 1 (step S1002: YES), the control unit 105 of the information processing device 1 executes a reception process for receiving the input of the performance information. Then, the performance determination process (may be described as music analysis or music analysis process) of step S1003 is executed.

Specifically, the control unit 105 (performance determination unit 1051) of the information processing device 1, as the performance determination process in step S1003, is based on the received performance information, for example, the tonality of the song, the note name, the chord type, and the velocity. Judges the value, note length, syncopation, chord function, and chord progression.

However, as the performance determination process in step S1003, the control unit 105 (performance determination unit 1051) of the information processing device 1 uses the tonality of the song, the note name, the chord type, the velocity value, the note length, the syncopation, the chord function, and the chord. Any of the determination of progress may be made.

The details of what is specifically determined as the performance determination process in step S1003 will be described in the next step S1004.

≪S1004≫
When the control unit 105 (image determination unit 1052) of the information processing device 1 finishes the performance determination process in step S1003, the process proceeds to step S1004 and generates a real-time image according to the performance information in accordance with the timing of receiving the performance information. To do. Then, the control unit 105 (image information output unit 1053) of the information processing device 1 executes a real-time image output process of outputting and displaying the image information to the display device 4.

It should be noted that the above-mentioned "according to the timing at which the performance information is received" does not mean simultaneous time, but means each time the performance information is received.

In the present embodiment, the basic image stored in the storage unit 106 of the information processing device 1 is processed by computer graphics (CG) to create a real-time image and displayed on the display device 4. The real-time image to be displayed on the display device 4 does not have to be limited to such a method, and may be a real-time image according to the performance information.

≪S1005≫
The control unit 105 (performance end determination unit 1055) of the information processing device 1 sets the performance data when the performance information transmitted from the instrument is not received by the communication unit 103 of the information processing device 1 (step S1002: NO). When it is determined that the above conditions are not satisfied (S1004: NO), or after step S1004, the performance end determination process is performed. Specifically, the control unit 105 of the information processing device 1 determines whether or not a predetermined time elapses without inputting performance information. If the predetermined time has not elapsed without inputting the performance information (step S1005: NO), the control unit 105 of the information processing device 1 proceeds to step S1002 again and determines whether or not the performance information has been input. To do.

≪S1006≫
The control unit 105 (performance end determination unit 1055) of the information processing device 1 has passed a predetermined time without inputting the performance information (step S1005: YES), that is, when the reception of the performance information is not detected for the predetermined time. It is determined that the performance has ended. When the control unit 105 (image determination unit 1052 and image information output unit 1053) of the information processing device 1 determines that the performance has been completed, the control unit 105 proceeds to step S1006 and outputs the image after the performance to the display device 4. Image output processing is performed after the performance to be displayed, and the entire processing is completed.

Further, when the present embodiment is applied when the accompaniment data is automatically reproduced and the melody is specified by the performer, it is determined that the performance is completed when the automatic reproduction of the accompaniment data is completed, and the image after the performance is performed. Output processing may be performed. That is, the reason why the control unit 105 of the information processing device 1 determines the end of the performance by the performer may be any reason, and the control unit 105 of the information processing device 1 determines the end of the performance by the performer. At that point, post-performance image output processing may be executed.

On the other hand, if the predetermined time has not elapsed without inputting the performance information (step S1005: NO), the control unit 105 of the information processing device 1 proceeds to step S1002 again to determine whether or not the performance information has been input. To judge.

<1-2-2> Specific Example Next, a specific example of the operation of the information processing device 1 will be described.

In a specific example, for example, when two users play "hometown", a case where an image group is assigned to the two users will be described.

<1-2-2-1> Example of drawing a real-time image Next, using FIG. 11, a state in which different image groups are drawn by two users playing "hometown" will be described. .. FIG. 11 shows a real-time image when the first user and the second user are playing the melody.

As shown in FIG. 11, when the first user plays a melody on the electronic musical instrument 2 (0), the control unit 105 of the information processing device 1 uses an image (insect character) included in the first image group as a real-time image. ) Is displayed on the display device 4. At this time, for example, the image (insect character) is displayed along the arrow A1 in the upper half of the real-time image (see 1st (U)) (see C1 and D1 to D6 in the figure).

Further, as shown in FIG. 11, when the second user plays the accompaniment on the electronic musical instrument 2 (1), the control unit 105 of the information processing device 1 uses an image (animal) included in the second image group as a real-time image. Character) is displayed on the display device 4. At this time, for example, the image (animal character) is displayed along the arrow A2 in the lower half of the real-time image (see 1st (L)) (see C2 and E1 to E6 in the figure).

<1-2-2-5> Example of Drawing Post-Performance Image Next, with reference to FIG. 12, a post-performance image after the two users have finished playing in their "hometown" will be described. FIG. 12 shows a post-performance image after the two users have finished playing the song.

When the control unit 105 of the information processing device 1 determines that the two users have finished the performance, the control unit 105 causes the display device 4 to display the image after the performance.

As shown in FIG. 12, the control unit 105 of the information processing device 1 arranges an image (insect character) of the first group at a place corresponding to the performance of the first user (see C1), and arranges the image (character of the insect) of the second user. An image (animal character) of the second group is placed at a place corresponding to the performance (see C2).

Thereby, the same effect as the above-mentioned effect can be obtained.

<1-3> Effect According to the above-described embodiment, in a performance data display system that draws an image according to a user's performance (music), when a plurality of performances are input, different expressions are output for each user. ..

For example, a performance data display system that draws an image according to a user's performance (music) can stimulate the user's motivation to practice an electronic musical instrument. However, when playing by a plurality of users, unless the image is changed for each user, only the same expression as when the number of sounds of one performance is increased can be obtained. In this case, the correspondence between the image and each performance becomes unclear.

However, according to the above-described embodiment, the drawing expression can be assigned to each of a plurality of users. As a result, the relationship between the user's performance and the image becomes clear, and more enjoyment can be provided. As a result, the user's willingness to practice the electronic musical instrument can be further stimulated.

Further, in the above-described embodiment, even raw sound can be converted into an image by using an audio interface and performing music analysis with the information processing device 1.

<2> Modification Example In the above-described embodiment, the control unit 105 of the information processing device 1 switches the image group when the drawing setting is changed in step S1001 of FIG. However, the present invention is not limited to this, for example, "display position of image (character)", "change of brightness", "change of size of image (character)", "background color (real-time image, or background of image after performance). Color), change the color table of the image (character), "change the type of image (bonus character) displayed when certain conditions are met", or "change the movement of the image (character)", etc. But it's okay. The background color (the background color of the real-time image or the image after the performance) and the color table of the image (character) are stored in, for example, ROM 1062. Further, the image (bonus character) displayed when a specific condition is satisfied is a real-time image when, for example, the control unit 105 of the information processing device 1 determines that the rhythm of the user's performance is stable. This is the image to be displayed. Further, the image (character) may be moved by animation or the like. Further, the same image (character) may be included in a plurality of image groups. Further, when the musical instrument is "vocal", the "vocal" may be large so as to stand out and may be displayed at the center of the real-time image or the post-performance image. In addition, the low musical instrument may be expressed in a round shape, and the high musical instrument may be expressed in a spiky manner.

Further, in the above-described embodiment, an example of drawing an image relating to two musical instruments has been shown, but the present invention is not limited to this. For example, as shown in FIG. 13, four musical instruments (piano, bass guitar, electric guitar, electronic musical instrument, etc.) may be connected to the information processing device 1. Further, as shown in FIG. 13, for example, a real-time image or a post-performance image may be divided into four parts, and musical instruments may be assigned to each of them. As a result, who is drawing which image (character) can be seen from the location of the screen divided into four.

Further, as shown in FIG. 14, the real-time image or the image after the performance may not be clearly divided and may be drawn as one image. However, since the image (character) is changed for each musical instrument, it is possible to determine who is drawing which image (character).

Further, when the performances of a plurality of users satisfy a predetermined condition (for example, matching of timings), a special image (character) such as a ribbon may be drawn on a real-time image or a post-performance image.

The flowchart for drawing the special image (character) will be briefly described below.

Steps S2001 to S2004, S2007, and S2008 shown in FIG. 15 are the same as steps S1001 to S1006 of FIG. Therefore, steps S2005 and S2006 will be described.

≪S2005≫
The control unit 105 (condition determination unit 1054) of the information processing device 1 determines whether or not the performance data satisfies the set condition. As an example of determining whether or not the set condition is satisfied, it may be determined whether or not the performance timings of a plurality of users match.

≪S2006≫
When the control unit 105 (image determination unit 1052 and image information output unit 1053) of the information processing device 1 determines that the performance data satisfies the set conditions (S2005: YES), the real-time image (and the post-performance image) ) Change the drawing settings. Specifically, the control unit 105 (image determination unit 1052 and image information output unit 1053) of the information processing device 1 displays a dedicated image (character) after the performance data determined to satisfy the set conditions. Display it. Specifically, as shown in FIG. 16, an image (character) of the ribbon is drawn.

Further, the control unit 105 of the information processing device 1 may manually (for example, select a MIDI channel) or automatically recognize the "chord" and "single note".

Further, although the control unit 105 of the information processing device 1 converts the results of pitch extraction and multiple sound analysis into MIDI format, it does not have to be in MIDI format.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

The above embodiment includes the following additional notes.

≪Appendix 1≫
To the computer of the information processing device
It is made to determine whether the input performance data is the performance data input from at least the first device (for example, the first electronic musical instrument) or the second device (for example, the second electronic musical instrument or the audio interface).
Information that can be determined based on the input performance data (for example, note name, chord type. The performance data does not include note name information or chord type information, and the performance data) when it is determined to be the first device. The first image data is displayed according to the note number information included in (the note name information and the chord type information can be determined).
Even if the information (for example, sound name, chord type) that can be determined based on the input performance data when it is determined to be the second device is the same as when it is determined to be the first device. Displaying the second image data in a mode different from that of the first image data.
program.

≪Appendix 2≫
Which of the above is the input performance data is determined from the information contained in the input performance data (for example, MIDI channel information).
The program described in Appendix 1.

≪Appendix 3≫
The first image data is included in the first image group,
The second image data is included in the second image group, and is included in the second image group.
The same image data may be included in the first image group and the second image group.
The program described in Appendix 1 or 2.

≪Appendix 4≫
The information that can be determined is at least note name information (for example, note name information is not included in the performance data, and note number information can be determined from note number information included in the performance data) and chord type information (for example, sound). Includes any of (similar to name information),
The program according to any one of Appendix 1 to 3.

≪Appendix 5≫
The first device includes either a first electronic musical instrument or an audio interface.
The second device includes either a second electronic musical instrument or an audio interface.
The program according to any one of Appendix 1 to 4.

≪Appendix 6≫
To the computer of the information processing device
It is made to determine whether the input performance data is the performance data input from at least the first device (for example, the first electronic musical instrument) or the second device (for example, the second electronic musical instrument or the audio interface).
Information that can be determined based on the input performance data (for example, note name, chord type. The performance data does not include note name information or chord type information, and the performance data) when it is determined to be the first device. The first image data is displayed according to the note number information included in (the note name information and the chord type information can be determined).
Even if the information (for example, sound name, chord type) that can be determined based on the input performance data when it is determined to be the second device is the same as when it is determined to be the first device. Displaying the second image data in a mode different from that of the first image data.
Method.

≪Appendix 7≫
Display device and
The processor, including the processor,
It is determined whether the input performance data is the performance data input from at least the first device (for example, the first electronic musical instrument) or the second device (for example, the second electronic musical instrument or the audio interface).
Information that can be determined based on the input performance data (for example, note name, chord type. The performance data does not include note name information or chord type information, and the performance data) when it is determined to be the first device. The note number information included in the data can be used to determine the note name information and the code type information), and the display of the first image data on the display device is instructed.
Even if the information (for example, sound name, chord type) that can be determined based on the input performance data when it is determined to be the second device is the same as when it is determined to be the first device. Instructing the display of the second image data in a mode different from that of the first image data on the display device.
Electronics.

≪Appendix 8≫
A first device, a second device, and a display device are provided.
The first device is
The first performance data including the first identifier (for example, the first channel number of MIDI) that can identify the first device according to the performance operation by the first user is generated, and the generated first performance data is used as described above. Output to the display device
The second device is
The second performance data including the second identifier that can identify the second device is generated according to the performance operation by the second user, and the generated second performance data is output to the display device.
The display device is
The first image data is displayed according to the information (for example, note name, chord type) that can be determined based on the input first performance data.
Even if the information that can be determined based on the input second performance data (for example, sound name, chord type) is the same as the information that can be determined based on the first performance data, the information can be determined with the first image data. Displays the second image data of a different aspect,
Performance data display system.

1 ... Information processing device 2 ... Electronic musical instrument 3 ... Speaker 4 ... Display device 5 ... MIDI patcher 6 ... Vocal microphone 7 ... Acoustic piano 8 ... Audio interface 11 ... Key 20 ... Display unit 30 ... Operation unit 40 ... Sound unit 41 ... SP Amplifier 42 ... Speaker 43 ... HP amplifier 44 ... HP jack 45 ... HP jack insertion detection unit 50 ... Key press detection unit 60 ... Storage unit 61 ... ROM
62 ... RAM
70 ... Control unit 80 ... Communication unit 100 ... Bus 101 ... Operation unit 102 ... Display unit 103 ... Communication unit 104 ... Sound output unit 105 ... Control unit 106 ... Storage unit 107 ... Bus 401 ... Display unit 1051 ... Performance determination unit 1052 ... Image determination unit 1053 ... Image information output unit 1054 ... Condition determination unit 1055 ... Performance end determination unit 1056 ... Connected device determination unit 1061 ... RAM
1062 ... ROM
10621 ... Program 10622 ... Table

Claims (8)

To the computer of the information processing device
It is made to judge whether the input performance data is at least the performance data input from the first device or the second device.
When it is determined to be the first device, the first image data is displayed according to the information that can be determined based on the input performance data.
Even if the information that can be determined based on the input performance data when it is determined to be the second device is the same as when it is determined to be the first device, it is different from the first image data. Display the second image data of the aspect,
program. Which of the above is the performance data input is determined from the information contained in the input performance data.
The program according to claim 1. The first image data is included in the first image group,
The second image data is included in the second image group, and is included in the second image group.
The same image data may be included in the first image group and the second image group.
The program according to claim 1 or 2. The information that can be determined includes at least one of note name information and chord type information.
The program according to any one of claims 1 to 3. The first device includes either a first electronic musical instrument or an audio interface.
The second device includes either a second electronic musical instrument or an audio interface.
The program according to any one of claims 1 to 4. To the computer of the information processing device
It is made to judge whether the input performance data is at least the performance data input from the first device or the second device.
When it is determined to be the first device, the first image data is displayed according to the information that can be determined based on the input performance data.
Even if the information that can be determined based on the input performance data when it is determined to be the second device is the same as when it is determined to be the first device, it is different from the first image data. Display the second image data of the aspect,
Method. Display device and
The processor, including the processor,
It is determined whether the input performance data is at least the performance data input from the first device or the second device.
When it is determined to be the first device, the display of the first image data on the display device is instructed according to the information that can be determined based on the input performance data.
Even if the information that can be determined based on the input performance data when it is determined to be the second device is the same as when it is determined to be the first device, it is different from the first image data. Instructing the display of the second image data of the embodiment on the display device,
Electronics. A first device, a second device, and a display device are provided.
The first device is
The first performance data including the first identifier that can identify the first device is generated according to the performance operation by the first user, and the generated first performance data is output to the display device.
The second device is
The second performance data including the second identifier that can identify the second device is generated according to the performance operation by the second user, and the generated second performance data is output to the display device.
The display device is
The first image data is displayed according to the information that can be determined based on the input first performance data.
Even if the information that can be determined based on the input second performance data is the same as the information that can be determined based on the first performance data, the second image data in a mode different from that of the first image data. To display,
Performance data display system.

Images