JP5754449B2 - Music code score generator - Google Patents

Description

  The present invention relates to a music chord score generation apparatus for generating a chord score for playing a musical instrument.

  For example, chord scores representing fingering positions on each string are widely used as musical scores for musical instruments such as guitars and bass guitars. As a technique for creating this chord score, for example, the technique described in Patent Document 1 is already known. In this prior art, when a guitar player plays a song, the player's fingering for each of the plurality of chords of the song is detected by a contact detection type sensor, and based on the detection result, the chord score data of the song Is generated.

JP 2005-266728 A

  In the above prior art, the player's fingering position is detected by the contact detection method. For example, for the purpose of improving detection efficiency and detection accuracy, the player's fingering position is detected by non-contact by photographing the player's fingering position. It is possible to do.

  However, in the above case, in the case of a stringed instrument, generally, the performer performs fingering while holding the palm with the palm inward, so that shooting is performed from the back side of the hand. In this case, even if the codes are different from each other, if the difference in fingering is slight, the difference is difficult to understand in the captured image. As a result, the actual chord performed by the performer may be different from the chord calculated based on the captured image data, and an incorrect chord score may be generated. There is.

  An object of the present invention is to provide a music chord score generation apparatus capable of generating an accurate chord score based on detection of fingering playing a musical instrument.

  To achieve the above object, according to a first aspect of the present invention, there is provided storage means for storing music data associated with chord information comprising a plurality of chord progressions, and reading the music data stored in the storage means. Chord display means for displaying the progression of the plurality of chords of the corresponding chord information, and for each chord displayed by the chord display means, a plurality of fingering position candidate data corresponding to the chord in a predetermined stringed instrument A fingering candidate data generating unit for generating a finger, a photographing unit for photographing a fingering to the stringed instrument by a musical instrument operator corresponding to the display of the chord on the chord display unit, and generating corresponding image data; Based on the image data generated by the photographing unit, for each code, the plurality of fingering position candidate data generated by the fingering candidate data generation unit That is, the fingering position specifying means specifies the fingering position specifying means for specifying actual fingering position data corresponding to the actual fingering position by the instrument operator, and the fingering position specifying means for each of the plurality of codes of the code information. And chord data generation means for generating chord data related to the stringed musical instrument of the music data by associating the actual fingering position data.

  In the present invention, the chord score data of the music is generated using the photographed image. That is, the fingering position of the stringed musical instrument is different for each chord, and the posture and shape of the instrument operator's hand are different for each chord. Using this, in the present invention, the fingering of the musical instrument operator for each of the plurality of chords of the music is photographed by the photographing means, and corresponding image data is generated.

  And in this invention, both the said image data and the chord information showing the chord progression of a music are used. That is, chord information is associated with music data in advance and stored in the storage means. When the music data is read from the storage means, the progress of a plurality of chords of the corresponding chord information is displayed by the chord display means. The operator performs fingering in response to the displayed code, so that the corresponding image data is generated by the imaging unit.

  On the other hand, at this time, the fingering candidate data generating means generates a plurality of fingering position candidate data corresponding to the code for each code. Then, the fingering position specifying means uses the image data to determine which data (actual fingering position data) corresponds to the actual fingering position among the plurality of fingering position candidate data. Identify by code. The identified actual fingering position data is associated with each chord of the chord information by the chord score data generating means, thereby generating chord score data.

  As described above, in the present invention, chord score data is generated by narrowing down a plurality of fingering position candidate data generated for each chord using image data. This makes it possible to create an accurate chord score with higher accuracy than when chord score data is generated only from image data.

  In a second aspect based on the first aspect, the fingering position specifying means is a position of the fingertip of the musical instrument operator appearing in the image data or the musical instrument operator estimated from the image data. The actual fingering position data is specified using a fingertip position.

  As described above, the fingering position of the stringed instrument is different for each chord. In particular, the position of the fingertip of the instrument operator is always different for each chord. Therefore, in the present invention, the fingering position specifying means uses the fingertip position of the musical instrument operator appearing in the image data (or the fingertip position of the musical instrument operator estimated from the image data), and the actual fingering position. Identify the data. Thereby, an actual fingering position can be specified reliably.

  According to a third invention, in the first or second invention, the fingering candidate data generation means generates the plurality of fingering position candidate data for respectively sounding a plurality of sounds constituting a chord of each chord. It is characterized by doing.

  In the present invention, fingering candidate data used to specify actual fingering position data is created in consideration of a plurality of sounds constituting the chord of each chord. For example, it may be created for each of a plurality of sounds, or may be created for each scale. Thereby, in the fingering position specifying means, the actual fingering position can be specified with high accuracy.

4th invention has a karaoke music reproduction | regeneration means which reads and reproduce | regenerates the karaoke music data containing the performance part of the said stringed instrument as said music data in any one of the said 1st thru | or 3rd invention, The chord display means displays the progression of the plurality of chords of the corresponding chord information according to the reproduction of the karaoke music data by the karaoke music reproduction means, and the chord data generation means The chord score of the stringed instrument performance part of the karaoke music data is obtained by associating the actual fingering position data specified by the fingering position specifying means with respect to each of the plurality of chords of the chord information associated with the data. It generates data, the karaoke music song playback means, the karaoke song data when playing next In accordance with the reproduction of the karaoke song data, a code Notation the generated, and displaying on the code display unit.

  In the present invention, karaoke song data is read from the storage means and reproduced by the karaoke song reproducing means. Along with the reproduction of the karaoke song data, the chord display means displays the progress of a plurality of chords related to the karaoke song data. Corresponding image data is generated by the photographing means by the fingering of the operator corresponding to this display. Using the generated image data, the fingering position specifying means narrows down a plurality of fingering position candidate data, and the actual fingering position data is specified. The identified actual fingering position data is associated with each code of the chord information of the karaoke music data by the chord data generating means, and chord data is generated.

  Thereby, the musical score operator can easily create a chord score of the karaoke music piece by fingering each chord in accordance with the reproduction of the karaoke music piece data. Further, when the karaoke music piece is reproduced next time, for example, a general user can easily perform a guitar performance by playing while watching the generated chord score.

  ADVANTAGE OF THE INVENTION According to this invention, the exact chord score based on the detection of the fingering which plays a musical instrument can be produced | generated.

It is explanatory drawing which shows the main structures of the karaoke apparatus concerning one Embodiment of this invention. It is a functional block diagram which shows the main structures of the control system of the control apparatus with which the karaoke apparatus was equipped. It is explanatory drawing which shows an example of the data structure of the reception data from the server containing music data. It is explanatory drawing which shows an example of the display content displayed on a monitor at the time of reproduction | regeneration of music data. It is a flowchart which shows the processing content performed by CPU at the time of reproduction | regeneration of music data. It is explanatory drawing showing the state in which the musical instrument operator is performing the fingering of a chord, and explanatory drawing showing the image data of the fingering state produced | generated from the imaging | photography result of the said fingering demonstration. It is explanatory drawing explaining the method of narrowing down fingering position candidate data using image data, specifying actual fingering position data, and producing | generating chord data. It is a flowchart which shows the processing content performed by CPU at the time of preparation of chord score information. It is a flowchart which shows the processing content performed by CPU for the preparation of image recognition. In preparation for image recognition, an explanatory diagram showing how the instruction line video is displayed on the monitor, an explanatory diagram showing the neck portion of the guitar displayed on the monitor, and the instruction line video and the guitar neck portion overlap It is explanatory drawing showing a mode that it is displayed.

  Hereinafter, an embodiment in which the present invention is applied to a karaoke apparatus will be described with reference to the drawings. This embodiment demonstrates the case where a user mainly plays an electric guitar himself with the performance of a karaoke apparatus.

<Main configuration>
FIG. 1 is an explanatory diagram showing the main configuration of the karaoke apparatus according to the present embodiment. As shown in FIG. 1, the karaoke apparatus 10 has a monitor television (hereinafter, abbreviated as a monitor) that displays a lyrics telop indicating lyrics, a background video displayed on the background of the lyrics telop, a video showing a music selection number, etc. on the CRT. 13 (corresponding to chord display means), a small monitor 14 (corresponding to chord display means), a musical instrument connection board 8 for connecting various electronic musical instruments including stringed instruments such as electric guitar 4 and electric bass, Request signal indicating the selection of music for karaoke composed of musical instrument performance parts (karaoke music; hereinafter, abbreviated as “music” as appropriate), music selection control such as reservation of music playback, and transmission request for the selected music Communication to the server and communication control such as reception of music data (corresponding to karaoke music data) corresponding to the music indicated by the request signal, received music A control device 20, such as to specify the same type of instrument sound source and an instrument connected to the instrument connection board 8 of the tone type specified information serving MIDI data included in the chromatography data, are provided.

  Further, in this example, the karaoke apparatus 10 includes a mixing circuit 9 that mixes a musical instrument performance signal input from the musical instrument connection board 8, an audio signal input from the microphones 17 and 18, and a music reproduction signal. An amplifier 16 that performs volume balance between sound and music, echo adjustment, delay adjustment, amplification of mixing signal, pitch control (key control) of played music, control of high and low sounds (tone control), and the amplifier 16 A pair of speakers 11, 11 for reproducing the amplified signal output from the sound as sound, a pair of speakers 12, 12 for hanging the ceiling, a remote controller 30 for remotely operating the control device 20, and a camera 25 (Details will be described later).

  Although not shown, the musical instrument connection board 8 has input terminals (phone jacks) for connecting the output terminals (phone plugs) of the electric guitar 4 and an electronic drum, keyboard, electric bass, etc. (not shown), and input terminals. And an output terminal for outputting performance signals input to the. The musical instrument connection board 8 automatically detects that each musical instrument is connected and the type of the musical instrument by connecting the output terminal of each musical instrument to the input terminal, and selects one performance signal of the connected musical instrument. The selected performance signal can be output to the input terminal 44 (see FIG. 2 described later) of the control device 20. For example, a performance signal output from the electric guitar 4 is input to the musical instrument connection board 8 via a phone jack 4 a (see FIG. 2 described later) for connecting a phone plug of the electric guitar 4, and is output from the output terminal of the musical instrument connection board 8. The output performance signal is input to the input terminal 44 of the control device 20.

<Control system>
The configuration of the control system of the control device 20 will be described with reference to FIG. FIG. 2 is a functional block diagram showing the main configuration of the control system of the control device 20. The control device 20 includes a communication terminal 40 that connects the LAN line 15 to the device housing, an input terminal 44 that is connected to the output terminal of the musical instrument connection board 8, and an audio output terminal 41 that is connected to the audio input terminal of the amplifier 16. The video output terminal 42 connected to the video input terminal of the monitor 13 and the video output terminal 43 connected to the video input terminal of the monitor 14 are provided. The control device 20 is provided with a CPU 45 that executes various controls according to a control program. The CPU 45 includes a RAM 46 for temporarily storing data transmitted from the remote controller 30, music selection number data indicating the music selection number of the selected music, music selection number data of the reserved music, a program executed by the CPU 45, and the like. A ROM 47 in which necessary data tables are stored is connected.

  Further, the CPU 45 has a video RAM 48 in which character video data for displaying lyrics telop and various message videos on the monitors 13 and 14 and music data and chord score information transmitted from the server 58 (details will be described later). A LAN board 50 for receiving lyrics data and video data via the communication terminal 40, and a RAM 49 (corresponding to storage means) for temporarily storing music data and lyrics data received by the LAN board 50 And are connected.

  Further, the CPU 45 is connected to a MIDI sound source board 51 that inputs MIDI data included in the song data read from the RAM 49 and outputs a sound source signal from a sound source specified by the input MIDI data. . Further, the CPU 45 inputs the output sound source signal and converts it into a signal that can be amplified by the amplifier 16, and a CD-ROM player 53 that reads background video data indicating a general background video. Is connected.

  The CPU 45 also receives general background video data read from the CD-ROM player 53, song-specific background data read from the RAM 49, lyrics telop data included in the song data, and the like. A video control circuit 54 that performs video control for creating a video in which the lyrics telop is superimposed on the background video displayed on the display screens 13 and 14 and changing the color of the lyrics telop as the music progresses. It is connected.

  Further, the CPU 45 includes a conversion circuit 55 that converts an optical signal from the remote controller 30 received by the light receiving unit 38 of the control device 20 into a digital signal, and various buttons and keys 39 (tenkeys) provided on the housing of the control device 20. A display circuit 56 that outputs a display signal to the LED 37 that is turned on when the music selection button is pressed, and an input circuit 57 that inputs a switching signal generated when the various buttons and key 39 are pressed. Yes.

  In the karaoke apparatus 10 having the above basic configuration, when a user selects karaoke music including a performance part of the electric guitar 4 by the remote controller 30 and receives music data transmitted from the server 58 corresponding to the music selection, the music data Is played. When the user plays the electric guitar 4 in accordance with the performance of the music data including the performance part of the electric guitar 4, the performance signal output from the electric guitar 4 is input to the input terminal of the control device 20 via the instrument connection board 8. 44. Then, in the sound control circuit 52, the sound source signal is output to the sound output terminal 41. In the case of the so-called minus one playback state, a part of the music part of the electric guitar 4 of the sound source signal is replaced with the performance signal of the electric guitar 4 and output.

  At this time, in this karaoke apparatus 10, chord score information is created in advance for performance support for music data including the guitar performance part of the electric guitar 4 received from the server 58 together with the lyrics data and video data (details will be described later). The chord score information is displayed together with the lyrics data and the video data by the monitors 13 and 14 when the music data is reproduced. Thereby, the user who is a performer can easily perform the performance part of the electric guitar 4 by himself using the chord score information displayed on the monitors 13 and 14 according to the karaoke music to be reproduced. You can practice the electric guitar 4 and enjoy an ensemble of the entire karaoke song.

<Data structure including chord score information>
FIG. 3 is an explanatory diagram showing a data structure of received data including music data received from the server 58 in the present embodiment. In the example shown in FIG. 3, four bars of karaoke music pieces of data received from the server are shown. The received data includes music data of a plurality of instrument performance parts (in this example, a guitar part (corresponding to the electric guitar 4), a bass part, and a drum part) corresponding to the progress of the performance of four bars, and a chord representing a chord name. Information, chord score information indicating fingering for, for example, a guitar for playing the chord, lyric data (lyric telop data), and video data.

  The music data is reproduced from the first bar (hereinafter simply referred to as “first bar” or the like) in the data shown in FIG. 3 together with the lyrics data and the video data, and the second bar, the third bar, Playback proceeds to the fourth measure. Corresponding to the reproduction, display based on the lyrics data and the video data also proceeds.

  The chord information and the chord score information are associated with music data. In this example, the first bar in the figure is chord Em, the second bar is chord Bm, the third bar is chord C, and the fourth bar is chord. Bm.

<Display example of chord information and chord information>
FIG. 4 is an explanatory diagram showing an example of display contents displayed on the monitors 13 and 14 during reproduction of music data. In this example, a display example corresponding to the music data for four bars shown in FIG. 3 is shown. That is, in this example, on the monitors 13 and 14, the lyrics for the four bars are “unless you say that word”, and the chord information for the four bars attached to the lower side of the lyrics (first bar: Em, 2nd bar: Bm, 3rd bar: C, 4th bar: Bm) are displayed.

  Further, below the chord information, chord information representing an example of fingering when the chord represented by the chord information is played with a guitar is displayed. In this example, the fingering for the chord Em in the first bar is 0 fret for the 1st string (do not hold the fret as an open string, the same applies below), the 0th fret for the 2nd string, the 0th fret for the 3rd string, The 2nd fret, the 2nd fret for the 5th string, and the 0th fret for the 6th string are designated.

  The fingering for chord Bm in the second bar is 2 frets for 1 string, 3 frets for 2 strings, 4 frets for 4 strings, 4 frets for 4 strings, 2 frets for 5 strings, 2 frets for 6 strings, Is specified.

  The fingering for chord C in the third measure is 0 fret for 1st string, 1st fret for 2nd string, 0th fret for 3rd string, 2nd fret for 4th string, 3rd fret for 5th string, 0th fret for 6th string, Is specified.

  The fingering for the chord Bm of the 4th bar is the same as the 2nd bar, 2 frets for 1 string, 3 frets for 2 strings, 4 frets for 3 strings, 4 frets for 4 strings, 2 frets for 5 strings In the 6th string, 2 frets are specified.

  The user of the electric guitar 4 looks at the chord score displayed on the monitors 13 and 14 as described above and plays the chords of the electric guitar 4 in the order of chord progression. You can easily play the electric guitar part.

<Basic processing when playing karaoke music>
Processing contents executed by the CPU 45 of the karaoke apparatus 10 when the user plays the electric guitar while reproducing the karaoke music as described above will be described with reference to the flowchart of FIG.

  In FIG. 5, this flow is started when, for example, the user presses the power button of the control device 20 of the karaoke device 10 to turn on the power of the control device 20. In conjunction with the rising of the power supply of the control device 20, the power supply of the amplifier 16, the musical instrument connection board 8 and the monitors 13, 14 is started up. As described above, in this example, a case where the user performs with the electric guitar 4 connected to the musical instrument connection board 8 will be described. When the user connects the phone plug of the electric guitar 4 to the input terminal (phone jack) 4a of the musical instrument connection board 8, the musical instrument connection board 8 detects that the electric guitar 4 is connected.

  In FIG. 5, first, in step S10, the CPU 45 determines whether or not the music selection by the user has been completed. In the music selection, for example, when the user inputs the music selection number of the music that he / she wants to play the electric guitar 4 with the numeric keypad of the remote controller 30 and presses the music selection button, the music selection ends. Until the music selection is completed, the determination in step S10 is not satisfied (step S10: NO), and a loop standby is performed. When the music selection is completed, the determination in step S10 is satisfied (step S10: YES), and the process proceeds to step S15.

  In step S15, the CPU 45 temporarily stores the music selection number data indicating the music selection number in the RAM 46 corresponding to the music selection result in step S10, and transmits the music data corresponding to the music selection number via the LAN board 50. A request signal to be requested is transmitted to the server 58 via the LAN line 15. Thereby, the server 58 stores the music data corresponding to the music selection number indicated in the request signal, the chord information corresponding to the music data, the chord score information, the lyrics data, and the data from a storage device (not shown). The video data is searched and read out, and the read music data and the like are transmitted to the control device 20 via the LAN line 15. When step S15 ends, the process proceeds to step S20.

  In step S20, the CPU 45 receives music data (including the chord information and the chord information) transmitted from the server 58 via the LAN line 15 via the LAN board 50. Thereafter, the process proceeds to step S25.

  In step S25, the CPU 45 temporarily stores the music data received in step S20 in the RAM 49. Thereafter, the process proceeds to step S30.

  As processing of step S15, step S20, and step S25, music data, code score information, and the like associated with the music selection number are processed on demand from the server 58 for each music selection. In addition, a large-capacity auxiliary storage means (not shown) is provided in the control device 20 of the karaoke apparatus 10, and music data and the like are stored in advance in the auxiliary storage means, and each song selection is associated with a song selection number. The obtained information group may be read from the auxiliary storage unit to obtain target information.

  In step S30, the CPU 45 starts reading the MIDI data stored in the RAM 49 in step S25 and writes the read MIDI data to the MIDI tone generator board 51. When step S30 ends, the process proceeds to step S35.

  In step S35, the CPU 45 outputs a control signal to the MIDI sound source board 51 and causes the MIDI sound source board 51 to output a sound source signal corresponding to the MIDI data written in step S30. The sound source signal output from the MIDI sound source board 51 is output to the sound control circuit 52 and converted into a music signal that can be amplified by the amplifier 16, and the converted music signal is transmitted from the sound output terminal 41 to the amplifier 16. Is output. Also, the audio signal input from the microphones 17 and 18 is mixed with the music signal in the mixing circuit 9 built in the amplifier 16. At this time, the performance signal output from the output terminal of the electric guitar 4 is input to the mixing circuit 9 via the musical instrument connection board 8 and the control device 20, and is mixed with the audio signal and the music signal. The mixed signal thus mixed is amplified by an amplifier circuit (not shown) built in the amplifier 16 and then output to the speaker 11 and the speaker 12 and reproduced by both speakers. In addition, the procedure of this step S35 functions as a karaoke music reproducing means described in each claim. When step S35 ends, the process proceeds to step S40.

  In step S40, the CPU 45 outputs a control signal to the monitors 13 and 14, and causes the monitors 13 and 14 to display the chord information, chord score information, lyrics, etc. stored in step S25. Thereby, the user who is a player of the electric guitar 4 plays the electric guitar 4 according to the chord information and chord score displayed on the monitors 13 and 14, so that the chord progression of the reproduced karaoke music is adjusted. Electric guitar performance part can be played easily. Thereafter, the process proceeds to step S80.

  In step S80, the CPU 45 determines whether or not the reproduction of the music data has been completed (in other words, whether or not the reading of the MIDI data in step S30 has been completed up to the last MIDI data of the music data). If the reproduction of the music data is finished, the determination is satisfied (step S80: YES), and this flow is finished. Until the reproduction of the music data is completed, the determination is not satisfied (step S80: NO), the process returns to step S30, and the procedure of steps S30 to S80 is repeated.

<Features of this embodiment>
As described above, the feature of the present embodiment lies in a method for newly creating chord score information used by the user as described above (before being used by the user). Hereinafter, the details will be described in order.

<Utilization of camera images>
That is, in the present embodiment, the chord score data of the karaoke music is generated using an image captured by the above-described camera 25 (corresponding to the imaging means). That is, the fingering position of the stringed instrument such as the electric guitar 4 is different for each chord, and the posture and shape of the hand of the instrument operator is different for each chord. By utilizing this, in the present embodiment, the luck of the musical instrument operator (a different karaoke service provider or a person entrusted by the worker who is different from the above user) for each of a plurality of chords of karaoke music. A finger is photographed by the camera 25, and corresponding image data (which may be moving image data or still image data) is generated.

  That is, at the time of generating the chord score data, at least the chord score information is not created in the data structure shown in FIG. For example, in this state, music data, chord information, lyrics data, and video data are reproduced and chord information is displayed on the monitors 13 and 14. The instrument operator looks at the displayed chord information and actually presses the chord on the fret of the neck of the electric guitar 4 so that the fingering is photographed by the camera 25. At this time, it is not always necessary for the musical instrument player to actually perform and pronounce the sound, and at least the chord must be pressed in appearance. In addition, the musical instrument performance part (guitar part, bass part, drum part in this example) and the lyric data and video data (at least at this time) are not necessarily required in the music data. For example, only the chord information is displayed on the monitors 13 and 14. May be displayed. Moreover, it is not necessary to be a karaoke music piece, and the music piece may be only a musical instrument performance that is not a singing piece.

<Image data generation>
In the example shown in FIG. 6 (a), the chord information (code Bm) of the second measure is displayed on the monitors 13 and 14 for the karaoke piece shown in FIG. This represents a state where Bm's fingering is being demonstrated. In this example, as shown in the figure, the index finger is present from the 1st to 6th strings of the 2nd fret (fingertip position is around 6th string) by the fingering of the musical instrument operator, and the middle finger from the 1st to 2nd strings of the 3rd fret. (The fingertip position is near the 2nd string), the ring finger is present from the 1st to 4th string of the 4th fret (the fingertip position is near the 4th string), and the little finger is present from the 1st to 3rd string of the 4th fret (the fingertip position is (Near the third string).

  Then, the image of the fingering shown in FIG. 6A is photographed by the camera 25, so that corresponding image data is generated as shown in FIG. 6B. In this example, the image data is generated as finger area (see shaded area in the figure) data including the figure of each finger.

<Possibility of false detection>
However, in the stringed musical instrument such as the electric guitar 4 described above, generally, the musical instrument operator performs fingering with the palm held inside, so that shooting is performed from the back side of the hand (see FIG. 6A). . In this case, even if the codes are different from each other, if the difference in fingering is slight, the difference is difficult to understand in the captured image. As a result, if an attempt is made to create a chord score from only the image data, there is a possibility that the actual chord performed by the musical instrument operator and the chord calculated based on the photographed image data may be different. Incorrect chord score may be generated.

<Combination of fingering position candidate data based on code information>
Therefore, in the present embodiment, the code information of the karaoke music described above is used together with the image data. That is, for each of a plurality of chords included in the chord progression represented by the chord information, a plurality of fingering position candidate data corresponding to the chord is generated. Then, using the image data, it is specified for each code which data (actual fingering position data) corresponds to the actual fingering position among the plurality of fingering position candidate data.

  As a specific example of the above technique, processing related to the second measure of the code Bm will be described with reference to FIG. As described above, the image data represented by the shaded area as shown in FIG. 7A is generated by the performance of the musical instrument operator. At this time, based on the chord information (code Bm) of the second measure, the chords constituting the chord Bm are “shi”, “le”, and “fa #”. Data is generated.

  FIG. 7B shows fingering position candidate data that can pronounce the above “shi”. In this example, 1 string 7 frets, 2 strings 0 frets, 2 strings 12 frets, 3 strings 4 frets, 4 strings 9 frets, 5 strings 2 frets, 6 strings 7 frets, a total of 7 Fingering position candidate data is generated. Of these seven fingering position candidate data, the three-string four-fret and the five-string two-fret that overlap with the shaded area of the image data shown in FIG. And extracted and specified as actual fingering position data.

  FIG. 7C shows fingering position candidate data that can pronounce the above-mentioned “re”. In this example, 1 string 10 frets, 2 strings 3 frets, 3 strings 7 frets, 4 strings 0 frets, 4 strings 12 frets, 5 strings 5 frets, 6 strings 10 frets, a total of 7 Fingering position candidate data is generated. Of these seven fingering position candidate data, the 2-string 3-fret that overlaps the shaded area of the image data shown in FIG. 7A is the actual fingering position data (relative to the image data). Extracted and identified.

  FIG. 7D shows fingering position candidate data that can pronounce “Fa #”. In this example, there are 6 fingering position candidate data in total: 1st string 2nd fret, 2nd string 7th fret, 3rd string 11th fret, 4th string 4th fret, 5th string 9th fret, 6th string 2nd fret Generated. Of these six fingering position candidate data, the 1st string 2nd fret, 4th string 4th fret, and 6th string 2nd fret which overlap with the shaded area of the image data shown in FIG. 7A. Are extracted and specified as actual fingering position data (for the image data).

  Then, the 4th fret of the 3rd string and the 2nd fret of the 5th string extracted as the actual fingering position data in FIG. 7B, and the 2nd string extracted as the actual fingering position data in FIG. 7C. The 3rd fret and the 1st string 2nd fret, 4th string 4th fret, and 6th string 2nd fret extracted as actual fingering position data in FIG. 7 (d) are all synthesized (FIG. 7 (e)). )) And is associated with the code Bm of the second measure as the final actual fingering position data. Thereby, chord score data for the chord Bm is generated.

<Control procedure when creating chord data (1)>
The processing contents executed by the CPU 45 of the karaoke apparatus 10 when the musical instrument operator creates chord data using the above-described karaoke apparatus 10 will be described with reference to the flowchart of FIG. The same steps as those in FIG. 5 described above are denoted by the same reference numerals, and description thereof will be omitted or simplified as appropriate.

  In FIG. 8, this flow is started when, for example, the musical instrument operator presses the power button of the control device 20 of the karaoke apparatus 10 to turn on the power supply of the control device 20 as described above.

  In FIG. 8, first, after steps S10 and S15 similar to those in FIG. 5, the process proceeds to step S20A corresponding to step S20. In step S20A, as in step S20, the CPU 45 receives the music data and the like transmitted from the server 58 via the LAN line 15 via the LAN board 50. However, at this time, as described above, the received data includes the music data and the chord information corresponding to the music data, but does not include the chord score information.

  The music data, the chord information, and the like may be acquired from the server 58 as in the description of FIG. 5, or may be acquired from auxiliary storage means incorporated in the control device 20.

  When step S20A is completed, the process proceeds to step S40 newly provided through steps S25, S30, and S35 similar to those in FIG.

  In step S40, the CPU 45 moves to a new chord based on the music data read out in step S30 and reproduced in step S35 (when the chord is switched from the previous chord to a different chord, Including the case where the first code is started after being newly started). When the transition to a new chord has not been made (when the performance of the same chord continues), the determination at step S40 is not satisfied (S40: NO), and the routine proceeds to step S80 described later. If the code has been transferred to a new code, the determination in step S40 is satisfied (S: YES), and the process proceeds to step S55.

  In step S55, the CPU 45 outputs a control signal to the monitors 13 and 14, and causes the monitors 13 and 14 to display the chord information stored in step S25 and corresponding to the reproduction of the music data in steps S30 and S35. . As a result, the instrument operator changes the actual fingering state corresponding to the chord information displayed on the monitors 13 and 14 (the chord Bm in the example of FIG. 6A) with the camera 25 as will be described later. Demonstration using electric guitar 4 (so that it can be photographed). The fingering demonstration performed in this way is photographed by the camera 25 in steps S65 and S70 described later, and the corresponding image data is generated by a known image recognition technique. The generated image data is input to the CPU 45. The image recognition process may be performed by the camera 25 or the CPU 45 side.

<Details of image recognition procedure>
An image recognition preparation process, which is a process performed for the preparation of the image recognition method, will be described with reference to a flow executed by the CPU 45 shown in FIG. 9 and an explanatory diagram of FIG. That is, in order to perform image recognition with high accuracy, the musical instrument operator previously sets the position of each fret of the electric guitar 4 in the field of view photographed by the camera 25. The image recognition preparation process is performed upstream of step S65. For example, if it is executed before the music data reproduction (step S35), the electric guitar instrument operator can play the music at a position where the CPU 45 can specify his fingering position as fingering position data. Electric guitar operation can be started with the start of performance.

  That is, in FIG. 9, first, in step S410, the CPU 45 outputs a control signal to the monitors 13 and 14 to display an instruction line image for performing the positioning. As shown in FIG. 10A, the instruction line image is a substantially rectangular figure simulating the appearance of a part of the neck of the electric guitar 4 (in this example, from the 1st fret to the 12th fret).

  Thereafter, in step S420, the CPU 45 starts obtaining a captured video output from the camera 25. The image at this time includes the appearance itself of a part (or all) of the neck of the electric guitar 4 as shown in FIG. The appearance image can be adjusted in size and position by the instrument operator changing the position or position of the electric guitar 4 in front of the camera 25.

  Thereafter, in step S430, the CPU 45 outputs a control signal to the monitors 13 and 14, and displays the substantially rectangular instruction line video in step S410 and the video from the camera 25 in step S420. At this time, as described above, the musical instrument operator appropriately adjusts the size and position of the image from the camera 25 with respect to the instruction line image which is a fixed image, as shown in FIG. In addition, the outer circumference of the fret of the electric guitar 4 and the substantially rectangular instruction line image are aligned (in other words, the instruction line image is set in advance so that it can be perfectly matched). Thereafter, the process proceeds to step S440.

  In step S440, the CPU 45 determines whether or not the outer circumference of the fret of the electric guitar 4 and the substantially rectangular instruction line image substantially coincide with each other within a predetermined range. This determination may be automatically performed by the CPU 45 by a known method such as edge detection, or may be performed based on whether or not the operator has made an operation input indicating that a substantially coincidence has been visually confirmed. The loop waits while the determination is not satisfied (S440: NO), and when the determination is satisfied (S440: YES), the process proceeds to step S450.

  In step S450, the CPU 45 considers that the preparation process for recognizing the performance of the musical instrument operator is completed, for example, an appropriate notification (completion display on the monitors 13 and 14 and lighting of lamps at appropriate locations on the karaoke apparatus 10). Etc.) to finish this flow.

<Control procedure when creating chord data (2)>
Returning to FIG. 8, in step S55, it is not always necessary to actually play and pronounce the electric guitar 4 as described above. Therefore, even if the electric guitar 4 and the musical instrument connection board 8 are not connected. good. In this sense, an acoustic guitar that is a non-electronic musical instrument may be used instead of the electric guitar 4. After step S55, the process proceeds to step S60.

  In step S60, the CPU 45 generates fingering position candidate data corresponding to the code represented by the code information displayed in step S55. In this example, fingering position candidate data is generated for each chord constituting the chord as described above. This step S60 functions as fingering candidate data generation means described in each claim. Thereafter, the process proceeds to step S65.

  In step S <b> 65, the CPU 45 acquires the image data (see FIG. 6B in the above example) based on the shooting result of the fingering performed by the camera 25. Thereafter, the process proceeds to step S70.

  In step S70, the CPU 45 extracts data corresponding to the image data acquired in step S65 from the plurality of fingering position candidate data generated in step S60, and specifies it as the actual fingering position data. (See FIGS. 7A to 7D in the above example). This step S70 functions as a fingering position specifying means described in each claim. Thereafter, the process proceeds to step S75.

  In step S75, the actual fingering position data identified in step S70 (in the above example, the 1st string 2nd fret, 2nd string 3rd fret, 3rd string 4th fret, 4th string 4th fret, 5th string 2nd) The chord data for the chord Bm is generated by associating the fret and the six chord 2 frets) with the corresponding chord information (the chord Bm in the above example). 7 (e)). This step S75 functions as chord data generating means described in each claim. Thereafter, the process proceeds to step S80.

  In step S80, as in FIG. 5, the CPU 45 finishes reproducing the music data (target code creation target) (in other words, reading out the MIDI data in step S30 is completed up to the last MIDI data of the music data). Determine whether or not). If the reproduction of the music data is finished, the determination is satisfied (step S80: YES), and this flow is finished. Until the reproduction of the music data is completed, the determination is not satisfied (step S80: NO), the process returns to step S30, and the procedure of steps S30 to S80 is repeated.

  As described above, the chord score data based on the fingering demonstration by the musical instrument operator is generated for all chords included in the chord progression of the music data to be chorded. As a result, the chord score information for the music data is completed, and when the karaoke music is reproduced from the next time onward, the general user performer is created and displayed on the monitors 13 and 14 as described above. It is possible to easily perform the electric guitar 4 or the like by performing while watching the chord score.

  As described above, in this embodiment, chord score data is generated by narrowing down a plurality of fingering position candidate data generated for each chord of music data using image data obtained by photographing an instrument operator's performance. To do. This makes it possible to create an accurate chord score with higher accuracy than when chord score data is generated only from image data.

  In the present embodiment, in particular, when fingering candidate data is generated, a plurality of fingering position candidate data for generating a plurality of sounds constituting the chord of each chord is generated (FIG. 7B). ) To (d)). Thereby, an actual fingering position can be specified with high accuracy. As in the example of the chord Bm described above, it may be created for each of a plurality of sounds (“shi”, “le”, “fa #”), or may be created for each scale.

  In the above embodiment, when the fingering position candidate data is narrowed down using the image data to generate the actual fingering position data, each finger that performs the fingering as shown in FIG. 6 and FIG. 101 (a). Although the fingering position candidate data included in the finger region data including the figure is extracted and extracted, the present invention is not limited to this. That is, as described above, the fingering position of the stringed instrument is different for each chord, but in particular, the position of the fingertip of the instrument operator is always different for each chord. Therefore, when the position of the fingertip of the musical instrument operator clearly appears in the image data (for example, in the example of the code Bm shown in FIG. 6B, the fingertip of the index finger located on the 6th string of the 2nd fret, The fingertip position of the instrument operator can be estimated from the fingertip of the middle finger located on the 2nd string, the fingertip of the ring finger located on the 4th string of the 4th fret, the fingertip of the little finger located on the 3rd string of the 4th fret, etc.) In this case, the actual fingering position data may be specified using the fingertip position. In this case, the actual fingering position can be specified with higher accuracy.

  In the above, the flowcharts shown in FIGS. 5, 8, 9, etc. are not intended to limit the present invention to the procedures shown in the above-mentioned flow, but to the extent that they do not depart from the spirit and technical idea of the invention. You may delete or change the order.

  In addition to those already described above, the methods according to the above-described embodiments and modifications may be used in appropriate combination.

  In addition, although not illustrated one by one, the present invention is implemented with various modifications within a range not departing from the gist thereof.

4 Electric guitar (stringed instrument)
10 Karaoke device (music code score generator)
13 Monitor (code display means)
14 Monitor (code display means)
25 Camera (photographing means)
49 RAM (storage means)

Claims (4)

Storage means for storing music data associated with chord information consisting of a plurality of chord progressions;
Code display means for reading the music data stored in the storage means and displaying the progress of the plurality of chords of the corresponding chord information;
For each chord displayed by the chord display means, fingering candidate data generating means for generating a plurality of fingering position candidate data corresponding to the chord in a predetermined stringed instrument;
Photographing means for photographing fingering to the stringed instrument by a musical instrument operator corresponding to the display of the code on the code displaying means, and generating corresponding image data;
Based on the image data generated by the photographing unit, an actual fingering position by the instrument operator among the plurality of fingering position candidate data generated by the fingering candidate data generation unit for each code. Fingering position specifying means for specifying actual fingering position data corresponding to
Chord data generation means for generating chord data relating to the stringed musical instrument of the music data by associating the actual fingering position data specified by the fingering position specifying means with each of the plurality of chords of the chord information When,
A music chord score generation apparatus characterized by comprising: The music chord score generator according to claim 1, wherein
The fingering position specifying means includes
The actual fingering position data is specified using the fingertip position of the musical instrument operator appearing in the image data or the fingertip position of the musical instrument operator estimated from the image data. A music chord generation device. In the music chord score generating device according to claim 1 or 2,
The fingering candidate data generation means includes:
A music chord score generating apparatus for generating a plurality of fingering position candidate data for generating a plurality of sounds constituting a chord of each chord. In the music chord score generating device according to any one of claims 1 to 3,
As the music data, karaoke music data reproduction means for reading out and reproducing karaoke music data including the performance part of the stringed instrument from the storage means,
The code display means includes
In response to the playback of the karaoke song data by the karaoke song playback means, the progress of the plurality of chords of the corresponding chord information is displayed,
The chord score data generating means includes:
For each of the plurality of chords of the chord information associated with the karaoke song data, by associating the actual fingering position data specified by the fingering position specifying means, the stringed instrument performance part of the karaoke song data is related to the generated code score data, the karaoke music song playback means, the karaoke song data when playing next, in accordance with the reproduction of the karaoke song data, a code notation the generated, displayed on the code display unit A music chord score generation apparatus characterized by:

Images