JP2007086572A - Image display device and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To automatically display an image matching music that a user begins to play or sing. <P>SOLUTION: This image display device has musical piece data of many musical pieces stored in a musical piece database D and also has image data of a moving picture, a plurality of still pictures, etc., prepared in an image database K according to each musical piece. When the user begins to play some music on a musical instrument or sing it, performance contents are input in the form of MIDI data (A to C), musical piece data having a tone series pattern matching the tone series pattern of the MIDI data are extracted from the musical piece database D to recognize the musical piece that the user beings to play or sing, the tempo, progress position (measure line timing etc.), etc., of the MIDI data are detected (E). Then image data related to the recognized musical piece are automatically selected from the image database K to display images matching the MIDI data one after another according to the detected tempo and progress position timing (L). <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an electronic music system capable of recognizing a song and sequentially outputting information that matches the song when the user starts playing a song or a musical instrument, more specifically, recognizing a song that has started playing. The present invention relates to an image display system that displays an image suitable for music.

An electronic music apparatus such as an automatic performance apparatus having an image display function for displaying an image as a performance background is known from Patent Document 1, for example. In such an image display device, it is necessary to designate and select a song of an image to be displayed, or to designate display start by the user, and the operation is troublesome.
JP 2003-99035 A

  In view of such circumstances, an object of the present invention is to provide an image display device capable of automatically displaying an image suitable for a song that a user starts to play or sing.

  According to the main feature of the present invention, image data storage means (K) for storing image data in association with music, and performance data input means (AC) for inputting performance data (MIDI data) based on user performance. And music recognition means (E; R1 to R8) for recognizing music based on the performance data input by the performance data input means (A to C), and music recognized by the music recognition means (E). Image display device (electronic music device, computer) comprising display means (L) for selecting the selected image data from the image data storage means (F) and displaying an image based on the selected image data 1] is provided, and a computer (electronic music apparatus) provided with image data storage means (K) storing image data in association with music is performed on the basis of a user performance. Performance data input steps (A to C) for inputting data (MIDI data), and music recognition steps (E; R1 to R8) for recognizing music based on the performance data input at the performance data input steps (A to C) And a display step (L) for selecting image data associated with the music recognized in the music recognition step (E) from the image data storage means (F) and displaying an image based on the selected image data. An image display program (Claim 3) for executing a procedure consisting of: Note that the parentheses indicate reference symbols, terms, and the like of the examples added for convenience of understanding, and the same applies to the following.

  After the image data is selected, the display means (G) of the image display device according to the present invention automatically starts displaying an image in accordance with the progress of the input performance data (MIDI data) (R6) [Billing Item 2] can be configured as follows.

  In the image display system according to the main feature of the present invention (Claims 1 and 3), image data is stored in advance in the image data storage means (K) in association with music, and when the user starts playing the music, Performance data (MIDI data) is input (A to C), and the music is first recognized based on the input performance data (E; R1). Here, when the user generates a sound by playing a song or generates an instrument sound by playing a musical instrument, the sound signal (sound / sound signal) based on the performance sound of the song or the musical instrument can be recognized. MIDI, etc., which is converted into performance data such as MIDI, is used as a system input (A, B), and the user can perform recognition processing generated based on the performance operation when the user performs an operation with the electronic performance operator. The performance data is input as it is (C). Next, image data associated with the music recognized corresponding to the input performance data is automatically selected from the image data storage means (K), and an image is displayed based on the selected image data. (L). Therefore, according to the present invention, it is possible to automatically select an image suitable for a song that has just started to be played by the user simply by starting to sing a song by singing or starting to play a song by an instrument. Is no longer necessary.

  The image display device according to the present invention is configured to automatically start image display (R6) in accordance with the progress of the input performance data (MIDI data) after the image data is selected (claim 6). Item 2). For example, after image data related to the recognized music is selected, image display is automatically started from a predetermined position (for example, a musical break position such as a measure) on the progress of the music [FIG. The fade-in control is performed from immediately after the image data is selected to a predetermined position, and the image display is automatically started [FIG. 6 (b)]. Therefore, according to the present invention, it is possible not only to save the trouble of instructing the display start by automatically starting the image display, but also to match the music by starting the image display from a predetermined position or starting the image display by fade-in. You can start displaying images naturally.

[Outline of electronic music device]
FIG. 1 is a block diagram showing the hardware configuration of an electronic music apparatus according to an embodiment of the present invention. As this electronic music apparatus, an electronic musical instrument or a music information processing apparatus having a music information processing function equivalent to that of an electronic musical instrument, such as a personal computer (PC) provided with a performance operation unit or a musical tone signal generation unit, is used. This electronic music apparatus includes a central processing unit (CPU) 1, a random access memory (RAM) 2, a read only memory (ROM) 3, an external storage device 4, a performance operation detection circuit 5, a setting operation detection circuit 6, and a display circuit 7. A sound source circuit 8, an effect circuit 9, a sound data input interface (I / F) 10, a communication interface (I / F) 11, a MIDI interface (I / F) 12, and the like. Are connected to each other.

  The CPU 1 constitutes a data processing circuit DP together with the RAM 2 and the ROM 3 and executes various music information processing including music recognition processing using a clock by the timer 14 in accordance with a predetermined control program. The RAM 2 is used as a work area for temporarily storing various data necessary for these processes. The ROM 3 stores various control programs, control data, performance data, and the like necessary for executing these processes.

  The external storage device 4 is not only a built-in storage medium such as a hard disk (HD) but also a compact disk read only memory (CD-ROM), a flexible disk (FD), a magneto-optical (MO) disk, a digital multipurpose disk ( Various data including a control program can be stored in any external storage device 4 including various portable external recording media such as a small memory card such as a DVD and a smart media (registered trademark). Further, a music database, accompaniment database, music information database, image database, and the like can be constructed using a predetermined external storage device (for example, HD) 4.

  The performance operation detection circuit 5 detects the performance operation content of the performance operator 15 such as a keyboard, and introduces MIDI performance data (referred to as MIDI data) corresponding thereto into the data processing circuit DP. The setting operation detection circuit 6 detects the setting operation content of the setting operation element 16 such as a key switch or a mouse, and introduces setting data corresponding thereto into the data processing circuit DP. For example, in addition to the function of setting conditions for generating musical tone signals in the sound source circuit 8 and the effect circuit 9, the setting operator 16 includes a mode switch for setting a mode such as a music recognition mode, and a music recognition mode. Output selection switch for selectively instructing output such as accompaniment, song information display, image display, fade-in switch for instructing “start immediately with fade-in” to start output of accompaniment, etc., song information There are display selection switches and the like for selectively specifying the display of musical scores, lyrics, chord names, etc. at the time of display output. The display circuit 7 controls display / lighting contents of a display (display device) 17 such as a screen display LCD and various indicators (not shown) in accordance with commands from the CPU 1, and provides display assistance to the operations of the operators 15 and 16. I do.

  The tone generator circuit 8 and the effect circuit 9 function as a musical tone signal generation unit (also called a tone generator unit), and in accordance with the conditions set by the setting operator 16, the tone generator circuit 8 converts actual performance MIDI data from the performance operation detection circuit 5. The corresponding musical tone data is generated, and a predetermined effect is imparted to the musical tone data by the effect circuit 9 having the effect imparting DSP to generate a musical tone signal for actual performance. The tone signal generators 8 and 9 generate an accompaniment tone signal according to the accompaniment data determined in accordance with the actual performance MIDI data during the music recognition process, or storage means ( It is also possible to generate a musical tone signal for automatic performance according to performance data from 3, 4), etc. The sound system 17 following the effect circuit 9 includes a D / A converter, an amplifier, and a speaker, and generates a musical sound based on a musical sound signal from the effect circuit 9.

  The sound data input I / F 10 is connected to a sound input device 30 including a sound signal generating device such as a microphone and an electric guitar and a sound signal processing circuit. The sound input device 30 digitizes an input sound signal from a microphone or sound signal generating device by a sound signal processing circuit and converts it into sound data. The sound data thus digitized is converted into a data processing circuit through the sound data input I / F 10. Introduced into DP. Note that the sound data introduced into the data processing circuit DP is reconverted into an output sound signal through the effect circuit 9 and emitted from the sound system 17 in order to amplify the input sound signal from the microphone or the sound signal generating device. be able to.

  A communication network 40 such as the Internet or a local area network (LAN) is connected to the communication I / F 11, and a control program is downloaded from an external server computer 50 or the like, and performance data is stored in the storage device 4 to store the electronic music. Can be used in the device.

  An external MIDI device 60 having the same MIDI music information processing function as the electronic music device is connected to the MIDII / F12, and MIDI data is exchanged between the electronic music device and the external MIDI device 60 through the MIDII / F11. be able to. Also, the MIDI data from the external MIDI device 60 based on the operation of the performance operator in the external MIDI device 60 is the same as the actual performance MIDI data based on the operation of the performance operator 15 of the electronic music apparatus, and the musical tone signal generator 8 , 9 can be used to generate musical tone signals for actual performance.

[Embodiments]
An electronic music apparatus according to an embodiment of the present invention executes a music recognition process when a music recognition mode is set by a mode switch operation of a setting operator, and starts playing when a user starts playing a song or melody. Automatically recognizes songs and melody songs, and outputs information that matches the songs in sequence. In the first embodiment of the electronic music apparatus, the electronic music apparatus operates as an automatic accompaniment apparatus that automatically plays the accompaniment in accordance with the progress of the music. In the second embodiment, the score, lyrics, and chord names are synchronized with the progress of the music. The third embodiment operates as an image display device that sequentially displays images suitable for the music in progress as the music progresses. 2 to 4 are schematic functional block diagrams for explaining the functions of the electronic music apparatus according to the first to third embodiments of the present invention.

[First Embodiment]
The automatic accompaniment device according to the first embodiment functions when the accompaniment output is selected by the output selection switch of the setting operator, recognizes the music that the user starts to sing or starts playing with the instrument, and relates to the recognized music Can be performed automatically. When this automatic accompaniment apparatus is schematically represented by functional blocks, as shown in FIG. 2, a voice / sound input unit A, a MIDI unit B, a MIDI input unit C, a music database D, a music recognition unit E, an accompaniment database F, and The accompaniment control unit G and the function units A to C constitute performance data input means for inputting performance data (MIDI data) in MIDI format that can be recognized by the music recognition unit E.

  The voice / sound input unit A corresponds to the functions of the sound input device 30 and the sound data input I / F 10, and when the user plays a melody with a musical instrument such as a song or humming, for example, A user's performance sound is input from the microphone of the sound input device 30, and a sound signal representing a sound waveform generated by voice or instrument performance of singing or humming is digitized and digitized by the sound signal processing circuit of the sound input device 30. The sound data is input to the data processing circuit DP via the sound data input I / F 10. The MIDI conversion unit B corresponds to the MIDI conversion function unit of the data processing circuit DP, and performs the MIDI conversion by detecting the pitch, timing, length, etc. from the sound data input from the voice / sound input unit A. The sound data is converted into MIDI data.

  The MIDI input unit C corresponds to the functions of the performance operator 15 and the performance operation detection circuit 5 or the external MIDI device 60 and MIDII / F12. MIDI data generated by a user operation of the performance operator 15 or an external MIDI device The MIDI data received from 60 is directly input to the data processing circuit DP.

  The music database D corresponds to the music database construction unit of the external storage device 4 and stores music data related to a large number of music. Each piece of music data includes, for example, song name data representing the name of the song, song identification data such as a song ID for specifying the song, reference tempo data representing the reference tempo value at the time of playing the song, and each musical tone constituting the song. “Pitch + pitch” information that represents pitch and pitch in pitch-time coordinates. Pitch + pitch sequence data (simply referred to as “pitch sequence data”) arranged in time series along the time axis. Etc.

  The music recognizing unit E corresponds to the music recognizing function unit of the data processing circuit DP, and is supplied with MIDI data converted from the sound data by the MIDI conversion unit B and MIDI data from the MIDI input unit C. In addition, as shown in the figure, the present invention is not limited to the one having both the MIDI data input sequence composed of the voice / sound input unit A and the MIDI conversion unit B and the MIDI data input sequence by the MIDI input unit B, but only one of the MIDI data. An input sequence may be provided.

  The music recognition unit E first converts the supplied MIDI data into a format similar to the tone length + pitch sequence data included in the music data stored in the music database D according to the music recognition processing program (“ "Sound length + pitch sequence"). Next, in the MIDI data, the music length + pitch sequence pattern (simply referred to as “pitch sequence pattern”) represented by the pitch length + pitch sequence data of the first predetermined section (for example, about the first few measures) Matching processing is performed with the music data in the database D to recognize which music the MIDI data represents. That is, the music data having the tone length + pitch sequence pattern having the highest matching degree with respect to the initial tone length + pitch sequence pattern of the MIDI data represents the music played by the user corresponding to the MIDI data. As a result, it is extracted from the music database D.

  The music recognizing unit E performs the pattern matching process of the tone length + pitch sequence data without considering the tempo value and the pitch value, so the tone length + sound between the input MIDI data and the determined song data. The tempo and transposition amount of the MIDI data are determined (detected) by comparing the pitch sequence and pitch sequence of the high sequence data. For example, the pitch length of the matching data and the time length of the entire pitch string data are compared to detect the tempo ratio or difference, and the detected tempo ratio or difference and the reference tempo of the music data are detected. Based on this, the tempo of the MIDI data is determined. In addition, the (average) pitch difference between the MIDI data and the music data is detected from the pitch of each “pitch + pitch” information included in the pitch + pitch sequence data, and based on the detected pitch difference. Thus, the transposition amount of the MIDI data with respect to the music data is determined. The music recognition unit E further determines (detects) a progress position on the music such as a beat position and a bar position based on the tempo of MIDI data and the elapsed performance time.

  Then, the music recognizing unit E stores the music identification data of the music data extracted from the music database D, the data indicating the tempo, transposition amount and progress position of the MIDI data obtained from the music data, and the setting operator 16 An output start command based on whether or not the fade-in switch is operated is output to the accompaniment control unit G as control data. Such control data is also output to output control units J and L of second and third embodiments (FIGS. 5 and 6) described later.

  FIG. 5 shows an input example of MIDI data, and each white block Pa to Pe is “sound length + pitch” that constitutes pitch length + pitch sequence data that occupies a predetermined portion at the beginning of the input MIDI data. Represents information. Each hatched block Sa to Se represents “sound length + pitch” information constituting the tone length + pitch sequence data of the music data corresponding to the MIDI data, and the tone length of the input MIDI data as shown in the figure. The + pitch string patterns Pa to Pe match the tone length + pitch string patterns Sa to Se of the reference music data.

  The example of FIG. 5A shows the time length component (sound length string) of the tone length of both data + pitch string pattern when MIDI data is input at a faster tempo than the music data. The time length t0 to tp of the tone length sequence represented by the tone length information of the portion, that is, the entire time lengths t0 to tp of the patterns Pa to Pe, is the total time of the tone length sequence represented by the total tone length information of the corresponding portion of the music data, that is, the patterns Sa to Se. The overall time length is shorter than t0 to ts. When the tempo of the two data is different from each other, for example, the input is performed by (reference tempo value of music data) × (time length tp of input sound length sequence / time length ts of reference sound length sequence) [however, t0 = 0] The tempo value of data can be obtained.

  FIG. 5B shows an example of the pitch length + pitch sequence pattern of both data when the MIDI data is input in a transposed state with respect to the music data. The pitch of the pitch row represented is indicated by the vertical position of the tone length + pitch row pattern Pa to Pe, and the pitch of the pitch row represented by the pitch information of the corresponding portion of the music data is also the sound. It is indicated by the vertical position of the long + pitch row pattern Sa to Se. In this example, the pitches of the input MIDI data patterns Pa to Pe are lower than the reference music data patterns Sa to Se. If there is a pitch difference between the two data in this way, the pitch difference between the input pitch row and the reference pitch row (average pitch difference if the pitch row patterns of both data do not match completely). The transposition amount corresponding to is obtained.

  The accompaniment database F corresponds to the accompaniment database construction unit of the external storage device 4 and stores a large number of accompaniment data associated with each piece of music data in the music database D. Accompaniment data may be stored on a one-to-one basis with each piece of music data, or one piece of accompaniment data may be shared for a plurality of pieces of music data. The accompaniment data stored on a one-to-one basis like the former is accompaniment data created only for a specific piece of music, and can be MIDI data of an accompaniment part for one normal piece of music. The accompaniment data shared by a plurality of music data like the latter is general accompaniment style data. When using a general accompaniment style for accompaniment data, each piece of music should have separate chord progression data and section switching data (data indicating when to switch each accompaniment section such as intro, main, fill-in, and ending). This is preferable because accompaniment suitable for each music piece can be performed.

  The accompaniment control unit G corresponds to the accompaniment control function unit of the data processing circuit DP, and automatically selects the accompaniment data associated with the recognized music according to the control data from the music recognition unit E and is selected. The accompaniment playback starts automatically according to the accompaniment data. That is, the accompaniment data associated with the recognized music is selectively read out from the accompaniment database F in accordance with the music identification data of the music data determined to correspond to the MIDI data in the music recognition unit E, and is sent to the musical sound generation units 8 and 9. An accompaniment tone signal is generated to generate an accompaniment sound suitable for the MIDI data input by the user through the sound system 14. In this case, accompaniment starts very naturally from a predetermined musical break position in the performance progress according to the output start command included in the control data from the music recognition unit E, and the tempo, transposition amount and progress position ( The accompaniment sound tempo, transposition amount and progress position are controlled in accordance with the section switching position.

  FIG. 6 shows an output start example of accompaniment and the like by the output control unit (G, J, L) such as the accompaniment control unit G, and FIG. 6A shows an output when the fade-in switch (16) is not operated. FIG. 6B shows an example in which output of accompaniment and the like is started from a predetermined musical break position after the music recognition unit E recognizes the music in accordance with the start command. FIG. 6B shows a case where the fade-in switch is operated. This is an example in which after the music recognition by the music recognition unit E according to the output start command, fade-in is performed immediately and output of accompaniment and the like is started (referred to as “start immediately with fade-in”). In each figure, the upper row shows the input MIDI data whose progress position has been determined, separated by bar line timing, the middle row shows the music recognition period representing the time required for music recognition, and the lower row shows the accompaniment. Etc. are shown separated by bar line timing.

  The accompaniment control unit G starts accompaniment from a predetermined position in a normal setting, and after the time t1 when the music recognition period has elapsed from the time t0 when the input of MIDI data is started, as in the example of FIG. 6A. Accompaniment starts at the next bar line timing t2. On the other hand, when “start immediately with fade-in” is set, as shown in FIG. 6B, the accompaniment starts immediately at the time t1 when the music recognition period elapses from the input start time t0. From t1 to the next bar timing t2, the accompaniment output volume is faded in by a volume coefficient value that gradually increases from “0” to “1”, and a more natural accompaniment start is performed.

  As described above, in the automatic accompaniment apparatus according to the first embodiment, music data is stored in the music database D for a large number of music pieces, and accompaniment data such as accompaniment styles are prepared in the accompaniment database F corresponding to each music piece. . When a user starts playing a song with a musical instrument or singing, the content of the performance is input as MIDI data (A to C), and song data having a tone string pattern that matches the tone string pattern of the MIDI data is stored from the song database D. The extracted music is recognized by the user, and the tempo, transposition amount, progress position (bar line timing, etc.) of the MIDI data are detected (E). Then, accompaniment data suitable for the recognized music is automatically selected from the accompaniment database F, and automatic accompaniment according to the detected tempo and transposition amount and in accordance with the progress position timing is executed (G). In addition, this automatic accompaniment can be started naturally by starting from a predetermined position (bar line timing or the like) or by fading in.

[Second Embodiment]
The music information display device according to the second embodiment functions when the music information display output is selected by operating the output selection switch (16), and recognizes and recognizes the music that the user starts to sing or starts playing with an instrument. It has a function of displaying a musical score and / or lyrics and / or chord (name) related to the music. When this music information display device is schematically represented by functional blocks, as shown in FIG. 3, a voice / sound input unit A, a MIDI conversion unit B, a MIDI input unit C, a music database D, a music recognition unit E, a music information database H and the music information display control unit J, and the functions of the performance data input means A to C and the music recognition means D and E are the same as in the first embodiment, but the output execution means are the music information database H and the music information display control. It consists of part J.

  The song information database H corresponds to the song information database construction unit of the external storage device 4 and stores a large number of song information display data associated with each song data in the song database D. The song information display data includes score data, lyrics data, and chord data. The song information database H may be stored as a set of score + lyric + code, or score + lyric, lyrics + code, score + Even a set of chords may be stored only with a score, only with lyrics, or with only chords.

  The musical score data stored in the song information database H may be musical score image data such as a bitmap, or may be logical musical score data indicating a display position or generation timing of a musical score symbol. MIDI performance data may be used. The lyrics data may be image data or text data including page breaks, lyrics timing data, and the like. Furthermore, the code data is preferably text data.

  The music information display control unit J corresponds to the music information display control function unit of the data processing circuit DP, and according to the control data from the music recognition unit E, music information display data (display selection associated with the recognized music) The switch can automatically select at least one of musical score data, lyrics data, and chord data), and music information display is automatically started according to the selected music information display data. That is, the music information display data associated with the recognized music is selectively read from the music information database H according to the music identification data of the music data determined to correspond to the MIDI data in the music recognition unit E, and is displayed on the display circuit 7. The music information of the music corresponding to the MIDI data input by the user is displayed on the display 17. When music information is displayed, display control is performed in accordance with the tempo, transposition amount or progress position detected by the music recognition unit E, and optimal music information is sequentially displayed in accordance with the progress of the MIDI data. For example, the wipe speed for each piece of music information display is changed according to the tempo, the display of the score and the chord name is changed according to the transposition amount, and the page is switched according to the progress position.

  In addition, about the start aspect of music information display, like the automatic accompaniment apparatus of 1st Embodiment, a display may be started from the predetermined timing after music recognition is completed like Fig.6 (a), The display may be started immediately after the music recognition is completed. Furthermore, when displaying immediately after music recognition as in the latter case, such a fade-in is not used even if the displayed fade-in is used as shown in FIG. 6B "Start immediately with fade-in". May be. The music information display according to the second embodiment may be performed independently, or may be accompanied by the automatic accompaniment of the first embodiment by simultaneous selection of the output selection switch.

  As described above, in the song information display device of the second embodiment, song data for a large number of songs is stored in the song database D, and the music score, lyrics, Song information display data for displaying song information such as a chord name is prepared in the song information database H. When a user starts playing a song with a musical instrument or singing, the content of the performance is input as MIDI data (A to C), and song data having a tone string pattern that matches the tone string pattern of the MIDI data is stored from the song database D. The extracted music is recognized by the user, and the tempo, transposition amount, progress position (bar line timing, etc.) of the MIDI data are detected (E). Then, song information display data corresponding to the recognized song is automatically selected from the song information database H, and song information such as a score, lyrics, and chord name is displayed in accordance with the detected tempo and progress position timing. (J).

[Third Embodiment]
The image display device according to the third embodiment has a function of recognizing music that the user has started playing or singing with an instrument and automatically displaying an image (moving image or still image) related to the recognized music. When this automatic accompaniment apparatus is schematically represented by functional blocks, as shown in FIG. 4, a voice / sound input unit A, a MIDI unit B, a MIDI input unit C, a music database D, a music recognition unit E, an image database K, and The performance data input means A to C and the music recognition means D and E have the same functions as the first or second embodiment, but the output execution means is the image database K and the image display control part L. Consists of.

  The image database K corresponds to the image database construction unit of the external storage device 4 and stores a large number of image data associated with each piece of music data in the music database D. The image data may be a video of a music artist, a background video of karaoke, an animation suitable for the music, a picture of the music artist, a background image suitable for the music, a picture book or picture-story show suitable for the music (for example, If the song is a nursery rhyme “Taro Urashima”, it may be a plurality of still images that can be sequentially switched and displayed, such as a picture-story show of Taro Urashima). The song information database H may store image data on a one-to-one basis with each song, or may share one image data for a plurality of songs.

  The display control unit L corresponds to the image display control function unit of the data processing circuit DP, and automatically selects and selects image data associated with the recognized music according to the control data from the music recognition unit E. The image display is automatically started according to the image data. That is, the image data associated with the recognized music is selectively read out from the image database K and sent to the display circuit 7 according to the music identification data of the music data determined to be compatible with the MIDI data by the music recognition unit E, and sent to the display circuit 7. An image suitable for the music corresponding to the MIDI data by the performance is displayed on the display 17. At the time of image display, display control is performed according to the tempo and progress position (progress timing) detected by the music recognition unit E, and optimal images are sequentially displayed in accordance with the progress of MIDI data. For example, the moving image playback speed is controlled according to the tempo, and the page switching of the still image is controlled according to the progress position.

  The transposition amount detected by the music recognition unit E is not used for image display control. Furthermore, about the start mode of music information display, like the automatic accompaniment apparatus of 1st Embodiment, you may start a display from the predetermined timing after music recognition is completed like FIG. 6 (a), The display may be started immediately after the completion of the music recognition. In the latter case, the fade-in is not used even if the displayed fade-in is used as shown in FIG. Also good.

  Further, the image display according to the third embodiment may be performed independently, or may be accompanied by the automatic accompaniment of the first embodiment and the music information display according to the second embodiment by simultaneous selection of the output selection switch. Good. When displaying a picture book or picture of a picture-story show, the voice data of the narrative may be stored and the voice data of the narrative may be reproduced in addition to the reproduction of the image.

  As described above, in the image display device according to the third embodiment, music data is stored in the music database D for a large number of music pieces, and image data such as a moving image or a plurality of still images is stored in the image database corresponding to each music piece. Prepared for K. When a user starts playing a song with a musical instrument or singing, the content of the performance is input as MIDI data (A to C), and song data having a tone string pattern that matches the tone string pattern of the MIDI data is stored from the song database D. The music that is extracted and recognized by the user is recognized, and the tempo and progress position (bar line timing, etc.) of the MIDI data are detected (E). Then, image data related to the recognized music is automatically selected from the image database K, and images suitable for the MIDI data are sequentially displayed in accordance with the detected tempo and progress position timing (L).

[Example of processing flow]
FIG. 7 is a flowchart showing music recognition processing according to one embodiment of the present invention. In this music recognition processing flow, the music recognition mode is set in the electronic music device based on the operation of the mode switch of the setting operator 16 by the user, the output selection switch is operated, and accompaniment, music information display, image display, etc. Is started and is executed by the music recognition unit E.

  In the first step R1, the CPU 1 converts the MIDI data inputted by the performance data input means A to C into tone length + pitch sequence data (pitch length + pitch sequence), and this tone length + pitch sequence. For the predetermined part at the beginning of the data, the music data in the music database D is matched with the tone length + pitch sequence pattern (differences in tempo or transposition are allowed), and the tone length + pitch sequence of the predetermined part Music data having a tone length + pitch sequence pattern that most closely matches the pattern is extracted from the music database D, and is recognized as the music indicated by the music identification data of the music data.

  In the next step R2, the tempo of the input MIDI data is determined in accordance with the time ratio between the tone length sequence of the predetermined portion in the input MIDI data and the tone length sequence of the corresponding portion in the extracted music data [FIG. In the subsequent step R3, the transposition amount of the input MIDI data is determined in accordance with the pitch difference between the input MIDI data pitch sequence and the reference music data pitch sequence [FIG. 5 (b)].

  In step R4, the music identification data in the extracted music data and the data representing the tempo and transposition determined in steps R2 and R3 are designated as control data corresponding to the selection operation of the output selection switch. Supply to output control units G, J, and L. For example, when the accompaniment is instructed by operating the output selection switch, these control data are supplied to the accompaniment control unit G specified in response to this instruction, and when the music information display is instructed, the music information display unit J The control data is supplied to the image control unit L, and when the image display is instructed, the control data is supplied to the image control unit L.

  Next, in step R5, for the input MIDI data, the progress position in the music currently recognized in step R1 is obtained, it is determined whether or not the obtained current progress position is a predetermined position, and the input MIDI data While the advance position does not reach the predetermined position (R5 → NO), the process returns to step R5 and waits until it reaches the predetermined position. When the progress position of the user performance reaches a predetermined position in the music (R5 → YES), the process proceeds to step R6, and an output start command is sent to the output control units G, J, L designated by the output selection switch. Instructs the start of information output. For example, when the accompaniment is instructed by operating the output selection switch, the accompaniment control unit G is instructed to start accompaniment, and when the music information display is instructed, the data is displayed on the music information display unit J (score, lyrics or lyrics). Code name display) is instructed, and when image display is instructed, the image control unit L is instructed to start image display.

  Note that if “start immediately with fade-in” is instructed by the operation of the fade-in switch (16), as shown by a broken line, control data such as music identification data, tempo data, transposition amount data, etc., as shown by a broken line. Is immediately supplied to the accompaniment control unit G, and the process immediately proceeds to step RA. As control data for the accompaniment control unit G and the other output control units J and L, the “start immediately with fade-in” described with reference to FIG. Supply an output start command to indicate

  After instructing the start of information output in step R6 or step RA, the process proceeds to step R7, and the MIDI data input from the performance data input means A to C is sequentially added to the tone length + pitch sequence tone length + pitch sequence. The detected tempo and progress position of the input MIDI data are detected, and data representing the detected tempo and progress position is supplied as control data to designated output control units G, J, and L. In subsequent step R8, it is determined whether or not the progress position of the input MIDI data has reached the end of the music. If the progress position of the input MIDI data does not reach the end of the music (R8 → NO), the process returns to step R7. Until the progress position reaches the end of the music (R8 → YES), the control data is sequentially supplied to the designated output control units G, J, and L.

  As a result, the output control unit, that is, the accompaniment control unit G, the music information display control unit J, or the image display control unit L starts in a predetermined manner according to the output start command based on the supplied control data, and inputs MIDI Accompaniment, music information display (score, lyrics or chord name display) or image display in accordance with the tempo of the data and the progress position timing is automatically executed. When the progress position of the input MIDI data reaches the end of the music (R8 → YES), the music recognition is terminated.

[Various Embodiments]
The preferred embodiment of the present invention has been described in detail with reference to the drawings. However, this is merely an example, and various modifications can be made without departing from the spirit of the present invention. For example, in the embodiment, the input MIDI data is converted into a sound string pattern and compared with the sound string pattern of the music data stored in the music database D, but the music is recognized. However, the music data in the music database D may be stored in the MIDI data format, and the music may be recognized by directly comparing with the input MIDI data.

  In addition, the music data in the music database D is stored in a data format other than the sound train pattern and MIDI data (for example, waveform feature data indicating the characteristics of the musical sound waveform), and the input MIDI data or sound data is stored in the data. It may be converted into a format (waveform feature data, etc.) for music recognition. In short, any data format can be used as long as the input MIDI data and sound data are compared with the data format stored in the music database D and compared to recognize music.

  Also, the sound input device 30 is not limited to digitalizing the input signal and inputting it to the sound data input 1 / F10 as digital sound data. On the sound input device 30 side, a sound signal processing circuit for digitization is provided. Instead, the analog sound signal may be input to the sound data input 1 / F10 and then converted to digital sound data by a sound signal processing circuit for digitization provided on the electronic music apparatus side.

  In addition, after music recognition is completed, not only automatic accompaniment, music information display, and image display are automatically performed, but automatic accompaniment data, music information display data, and image data are selected. The automatic accompaniment, music information display, and image display may be started according to the start instruction. In that case, the part of the user performance and singing input for music recognition is not limited to the beginning of the music but can be any part.

  As for the database, the music data base, the accompaniment database, the music information database, and the image database are configured as separate databases, but not limited thereto, the music data, the accompaniment data, the music information display data, and You may comprise as a single database with which image data was matched.

It is a block diagram showing the hardware structural example of the electronic music apparatus by one Example of this invention. It is a block diagram showing the functional structural example of the electronic music apparatus (automatic accompaniment apparatus) by 1st Embodiment of this invention. It is a block diagram showing the functional structural example of the electronic music apparatus (music information display apparatus) by 2nd Embodiment of this invention. It is a block diagram showing the functional structural example of the electronic music apparatus (image display apparatus) by 3rd Embodiment of this invention. It is a figure showing the input example of the MIDI data in each embodiment of this invention. It is a figure showing the start example of the accompaniment and display in each embodiment of this invention. It is a flowchart showing the music recognition process by one Example of this invention.

Explanation of symbols

A to C voice / sound input unit, MIDI unit and MIDI input unit (performance data input means),
D, E music database and music recognition unit (music recognition means),
G, J, L Accompaniment control unit, song information display control unit and image display control unit (output control unit),
Pa ~ Pe MIDI data "sound length + pitch" pattern,
Sa-Se “sound length + pitch” pattern of music data extracted from the music database D.

Claims (3)

Image data storage means for storing image data in association with music;
Performance data input means for inputting performance data based on a user performance;
Music recognition means for recognizing music based on performance data input by the performance data input means;
An image display apparatus comprising: display means for selecting image data associated with the music recognized by the music recognition means from the image data storage means and displaying an image based on the selected image data. .   2. The image display device according to claim 1, wherein the display means automatically starts displaying an image in accordance with the progress of the input performance data after the image data is selected. In a computer comprising image data storage means for storing image data in association with music,
A performance data input step for inputting performance data based on a user performance;
A music recognition step for recognizing music based on the performance data input in the performance data input step;
An image display program for executing a procedure including a display step of selecting image data associated with a song recognized in the song recognition step from image data storage means and displaying an image based on the selected image data.

Images