JP3921817B2 - Automatic performance device and fingering recording device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an automatic performance apparatus capable of displaying a player's animation in accordance with an automatic performance, and a fingering recording apparatus for recording fingering data for animating a fingering that is a finger usage of the performer. .
[0002]
[Prior art]
2. Description of the Related Art Conventionally, an automatic performance apparatus that automatically plays music by reading performance data for automatic performance in time series and generating a musical sound by driving a sound source with the read data has been put to practical use.
[0003]
[Problems to be solved by the invention]
However, the conventional automatic performance device automatically generates the musical tone of the part written in the performance data. What kind of musical instrument is currently being played, what kind of musical instrument combination band (ensemble) ), And there is a problem that the user cannot freely change the combination.
[0004]
The present invention relates to an automatic performance apparatus capable of displaying an animation of a performer performing a performance part in accordance with an automatic performance, an automatic performance apparatus capable of changing the performance part, and a player's fingering for animation. An object of the present invention is to provide a fingering recording apparatus capable of recording.
[0005]
[Means for Solving the Problems]
  The invention according to claim 1 is an automatic performance means for automatically performing music based on performance data;Multiple types of instrumentsMaterial storage means for storing the performer's animation material,Select a performer of the instrument whose tone is specified by the performance data, andAn animation creating means for creating an animation of a performer who plays the music piece by synthesizing an animation material based on at least the pitch of the performance data.
[0007]
  Claim2The invention ofIn the invention of claim 1, the material storage means isStores animation material including shape information and performance information of performers' fingersThe animation creating meansInput fingering data instructing which finger the performance operation was performed in synchronization with the performance data,At least the pitch of the performance data andBy synthesizing the animation material based on the fingering data,Performers of musical instruments of the tone specified by the performance data and theirAn animation for displaying the fingering of the performer is created.
[0009]
  Claim3The invention ofIn the invention of claim 1,Each has multiple parts, the same melody, and multiple different chords and rhythmsVersionReading means for reading performance data in parallel;Select and select any part from the multiple versions,A selection means for inputting only the data of the selected part into the sound source;
  Selection changing means for inputting selection change data for instructing the selection means to change the selection while the performance data is being read by the reading means;Performance data input means for inputting the data of the part selected by the selection means to the animation creating means as performance data;TheMoreIt is characterized by having.
[0011]
  Claim4The invention of claim3In this invention, when the selection of the part is changed by the selection changing means, the animation creating means creates an animation of the player of the newly selected part instead of the animation of the player of the part that is not selected. It is a means.
[0012]
According to the present invention, an automatic performance is performed based on performance data, and the timbre organization of the automatic performance is displayed as an animation. Most of the musical sounds generated during automatic performance simulate the musical sound waveform of the musical instrument played by the performer. At the same time, it is possible to enjoy the automatic performance only by hearing in the past, and the automatic performance can be further enhanced.
[0013]
  The performer's animation displays the form of the player's limbs and musical instrument, and is a video that moves the performer's limbs and instrument so that the player performs the performance according to the sounding and muting. In the method of recording individual animation video data, record animation video data of multiple performance data.UtoIn this case, a huge amount of storage device is required.In general performance data, music is played in a plurality of parts, and the combination of the parts is not constant depending on the performance data.Therefore, in the invention of claim 1,Storing the animation material of the performers of multiple parts in advance, selecting the performer based on the tone specified by the performance data,An animation is created by synthesizing this animation material based on information such as the pitch of the music input from the performance data. Thereby, even if it is an apparatus which plays many music like a karaoke apparatus, since an animation can be created each time, it is not necessary to memorize | store individual animation video data. The animation material is, for example, graphic data for drawing the form of a player or a musical instrument, vector data for instructing an operation mode at the time of performance, or the like.Furthermore, it is possible to create an animation regardless of the performance data of any part configuration.
[0015]
  Also, in actual musical instrument performance, the fingers move the fastest and finely, such as pressing down the keyboard, pressing the fret, playing the strings, etc. If you reproduce the fingering that is the movement of this finger , Animation becomes extremely real. In addition, if accurate fingering is animated, it can be used for music learning. However, determination of fingering is not easy even for a skilled player, and is determined in consideration of the connection between the front and rear sounds and the required sound quality. Therefore, it is not easy to determine an accurate fingering on the spot based on the pitch acquired from the performance data. Therefore, the claim2In this invention, the fingering animation for creating the fingering operation is input in synchronization with the performance data to create a fingering animation. Thereby, an accurate fingering animation can be easily created.
[0016]
  Also armThe animation of the limbs and limbs is created based on the performance data as in the first aspect of the invention, and the animation of the fingers is created based on the fingering data.TheBy combining these, it is possible to create an animation of a performer performing with accurate fingering.
[0019]
  Claims3The present invention relates to an automatic performance apparatus capable of changing the part composition of automatic performance during performance. The reading means reads in parallel a plurality of pieces of performance data having different arrangements (chords, rhythms, etc.) with the same melody music (that is, the same music). Each piece of performance data consists of a plurality of parts, and the selecting means selects an appropriate part from these and inputs it to the sound source. As a result, a plurality of arrangement parts can be mixed and played, and an unprecedented arrangement can be played. Also, since all of the data is part data produced for this song, there is no monotony like the arrangement change using a general-purpose pattern.
[0020]
In addition, since a plurality of performance data is read out in parallel, the part to be sounded can be changed by changing the selection even during reading (during performance). Organization changes can be made.
[0021]
  Also onThe animation can be displayed in parallel with the automatic performance of organizing parts in any combination. This makes it easy to visually understand any part organization. And claims4In the present invention, when the part selection is changed during the performance, the animation of the newly selected player is created and displayed in place of the player who is not selected in the animation. As a result, the result of the part change and the band organization change is displayed in a visually easy-to-understand manner. At this time, in the animation, it will be more visually interesting if the person who has left the game leaves and the newly selected person appears.
To do.
[0022]
DETAILED DESCRIPTION OF THE INVENTION
An automatic performance apparatus according to an embodiment of the present invention will be described with reference to the drawings. This automatic performance device is connected to a display device such as a karaoke device, and has a function of displaying the animation of a player (player) performing the performance on the display device in parallel with the automatic performance.
[0023]
FIG. 1 is a functional block diagram of the automatic performance apparatus. The song data storage unit 1 stores performance data that is data for automatically playing a song (karaoke song or the like). The performance data is configured as shown in FIG. 4A, and includes a header in which bibliographic information such as the genre of the song and the title of the song is written, a tone track for generating tones of a plurality of tones (parts), It consists of an audio control track for instructing the timing of back chorus (audio data) generation, an effector control track for controlling an effector that gives effects to musical sounds, a lyrics track for instructing lyrics to be displayed on the monitor, and the like. .
[0024]
Each track is described in MIDI format sequence data. As shown in FIG. 5C, event data for instructing an event such as musical sound production / mute is combined with timing data for instructing the event occurrence timing. Are stored in chronological order. As the event data, in the case of a musical sound track, there are the above-mentioned musical sound pronunciation / mute, after touch, various control information, and the like.
[0025]
The song data storage unit 1 stores performance data of a plurality of songs. The automatic performance unit 2 selects one of them and reads the event data while counting the timing data with the tempo clock. This event data is input to the sound source 3 and the like. Although only the sound source 3 is shown in the functional block diagram of FIG. 1, an audio playback unit that plays back chorus, an effector that gives an effect to the generated musical sound, and a display unit that displays lyrics are also connected to this automatic performance unit 2 Has been.
[0026]
On the other hand, the animation unit 4 is a functional unit that creates an animation of the player in accordance with the automatic performance of the automatic performance unit 2. The performance data has tone tracks of various parts (tones) as described above, and the sound source 3 generates the tone for the parts, but the animation unit 4 is a player that plays the tone instruments of each part of the tone track. Create an animation. In this player animation, each player is caused to perform an operation of actually playing a musical sound that is generated by the automatic performance of the performance data. The player animation created by the animation unit 4 is displayed by the display unit 5. This player animation is displayed in a manner as shown in FIG. 2, for example.
[0027]
As a result, it is possible to visually understand what part (sound color) is being generated during automatic performance, and even in automatic performance, it is possible to obtain a sense of reality that is similar to the performance of an actual natural instrument. Can increase your enjoyment.
[0028]
FIG. 1B is a detailed view of the animation unit 4. The animation unit 4 includes a player selection / placement unit 10, a performance operation calculation unit 11, an animation generation unit 12, a player database 13, an effect database 14, and a display mode control unit 15. The player selection / arrangement unit 10 and the performance operation calculation unit 11 determine the shape (function value) of each player with reference to the player database 13 and the effect database 14 according to the input musical sound track and effect data. The animation generation unit 12 renders an actual animation image based on the determined shape.
[0029]
  As shown in FIG. 5A, the player database 13 has player polygon data 130, which is 3D polygon data of the player's limbs of each part and the musical instrument to be played, and how to move these limbs and musical instruments during performance. The performance operation data 131 is the operation information. Part of the rock band part such as keyboard, electric guitar, electric bass, drumsHoOr, a full orchestra part such as a violin, cello, flute, horn, etc. is stored.
[0030]
The player polygon data 130 is a 3D polygon figure (function) for each of the player's head, neck, torso, waist, upper arm, forearm, hand, leg, and foot as shown in FIG. 3D polygon figures shown, and basic arrangement data (relative coordinates) of these polygon figures. By rendering these polygon figures in a three-dimensional space and rendering them, a player animation is generated. This rendering is performed by the animation generation unit 12.
[0031]
In this embodiment, different player polygon data 130 is stored for each player so that even if players of each part perform side by side, they change in different body shapes, but are common to all players. One player polygon data may be stored, and all players may be generated based on the polygon data. In this case, the shape of each player may be changed by expanding / compressing the polygon.
[0032]
The performance operation data 131 is data indicating the operation of the player when playing the musical instrument, and is stored for each tone generation mode of musical tone such as pitch and volume according to the characteristics of each musical instrument. For example, in the case of the illustrated electric guitar, the pitch is determined by the selection of the string by the left hand (finger) and the position of the fret, so the performance data indicates the left hand movement corresponding to the pitch of the note-on event data. To do. Further, since the sound generation and volume control are performed by picking the string with the right hand, the performance data indicates the speed and magnitude of the right hand motion corresponding to the velocity of the note-on event data.
[0033]
By changing the frets to be pressed, you can produce a musical tone with the same pitch on different strings. If the strings change, the overtone structure of the musical tone changes, and the sound quality (especially brightness and volume) changes. You may make it select a string by (contents of control change data).
[0034]
In addition, picking near the bridge of the string will produce a hard tone with many overtones, and picking a place far from the bridge will produce a solid round tone with a fundamental tone, so the picking position of the string may be changed according to sound quality instructions. .
[0035]
Also, a performance technique peculiar to an electric guitar such as vibrato or choking may be expressed by animation. Vibrato is a performance technique that vibrates the pitch of the musical sound by finely vibrating the left hand (finger) holding the string. Choking is a technique for vibrating the string by pushing the string sideways with the left hand (finger) holding the string. This is a performance technique that increases the pitch by increasing the tension. Vibrato may be animated according to the controller data associated with the LFO among the control change data, and choking may be animated according to the control change data of the pitch vendor.
[0036]
Information on motion vectors and motion positions for animating the performance technique as described above is stored as performance motion data.
[0037]
FIGS. 6A, 6B, and 6C show examples of player animation frames (still images obtained by cutting out moments of motion) rendered based on the player polygon data 130 and performance motion data 131. FIG. This figure shows an example of an electric guitar player. In FIGS. 5A, 5B, and 5C, the attitude of holding the guitar is different, but even with the same electric guitar, the style of playing the guitar differs depending on the genre, so the animation is realistic. In order to do this, the guitar is held differently depending on the genre. FIG. 6A shows a rock band system, FIG. 5B shows a group sound system, and FIG. 4C shows a solo musician system, for example. Such holding variations may be stored in performance data described later, or a separate player database may be provided for each genre.
[0038]
The animation shown in FIGS. 6A to 6C shows an operation in which only both arms are moved based on the performance data from the basic posture shown in FIG. 5B. 6 (D) and (E) are performed by moving the entire body. However, how to match the mood of the song varies depending on the song. Therefore, effect data as shown in FIG. 4B is stored in association with each karaoke piece, and the movement of the player's body is controlled using this effect data. This effect data is also stored in the song data storage unit 1.
[0039]
The production data is composed of a header and various tracks in the same manner as the performance data. The header is an area in which the attribute of the data is written, and is associated with the performance data by the music number written in the header. The track includes a player operation track, a camera work control track, and a lighting control track. The player operation track is a track for instructing what kind of performance operation the player of each part performs, and an instruction of a performance mode such as “gender”, “flashy”, “stand play”, etc. for each player is event data. As written. How each player actually moves the entire body according to the performance mode is stored in the effect database 14 of the automatic performance device.
[0040]
In addition, at the beginning of the header or the player operation track, data for instructing each player's costume and makeup is also written, and a specific costume shape / pattern, makeup pattern, etc. corresponding to this costume / make instruction, etc. Is stored in the production database 14.
[0041]
The camera work control track is a track for instructing the viewpoint of the player animation as camera work. As the event data, the camera position, camera angle (direction), zooming, panning, etc. are instructed. The animation generation unit 12 renders the 3D animation of each player at an angle and size as captured by this camera work.
[0042]
The lighting control track is a track for instructing the lighting color of the entire stage, the direction and color of a pin spot that illuminates a specific player, and the like. The animation generation unit 12 forms a highlight part and a shadow part in the animation to be rendered according to the lighting instruction. Also, stage effect data such as smoke and snowstorms may be written in this lighting control track. Such a stage production track may be provided separately.
[0043]
As described above, in this embodiment, as shown in FIGS. 4A and 4B, the performance data and the effect data are stored as separate data, and are associated with each other by a code such as a song number. . By doing in this way, each can be downloaded separately and updated, and maintenance becomes easy. On the other hand, the effect data may be stored as a part of the performance data. This reduces the number of files and makes management easier.
[0044]
Further, when there is no production data, production content (production data) may be generated based on the contents of the musical tone track and the effect control track of the performance data.
[0045]
In the animation unit 4 having the database as described above, the following operations are performed when a song is played.
[0046]
The player selection / arrangement unit 10 receives a header of performance data and a head portion of a musical sound track. The character selection / arrangement unit 10 determines the genre and part organization of the song from this data, selects a player, and determines a layout according to the performance style of the genre. The polygon data of the selected player is read from the player database 13 and arranged in the determined layout. Also, costume data and makeup data are read from the production database, and the polygon data is transformed accordingly. The costume and makeup are determined based on the production data, but may be determined based on the genre data.
[0047]
The performance operation calculation unit 11 inputs basic shape data and layout data of the player from the player selection / arrangement unit 10 and performs an operation of moving each player in accordance with the performance operation based on the music track data and effect data. . In the case of the above-mentioned electric guitar player, both arms are moved in accordance with the musical track data, and the upper body, lower body, head, etc. are moved in accordance with the production data. Specific movements of both arms are described in the performance data of the player database, and movements of the upper body, lower body, head, etc. are described in the production database 14. The performance operation calculation unit 11 outputs to the animation generation unit 12 function data representing the shape (posture) of each player for each frame (1/30 second in the case of NTSC) by the above calculation.
[0048]
The display mode control unit 15 calculates the viewpoint direction of the screen, the light direction / color, and the like based on the data of the camera work control track and the lighting control track of the effect data, and outputs them to the animation generation unit 12.
[0049]
  The animation generation unit 12 is based on the function data input from the performance operation calculation unit 11 and the data such as the direction of the viewpoint of the screen, the direction of light, and the color input from the display mode control unit 15.animationRender the image. This image data is output to the display unit 5 and displayed to the user. The display mode control unit 12 may use a high-speed drawing device with a built-in DSP, and a drawing device may be provided in parallel for each player.
[0050]
FIG. 2 is a diagram showing an example of an animation generated by this automatic performance device. In this way, a plurality of players are arranged on the stage, and an animation playing each instrument is displayed on the screen.
[0051]
FIG. 3 shows a configuration example when the animation unit 4 is configured in an object-oriented software manner. In this configuration, each player module knows how to play its musical instrument, and when a note is received, an animation for playing the note is generated as a polygon. Also, the stage manager receives instructions from the stage manager on how to improve the visual effect. At the start of the song, the director creates an effect from the song data if there is no effect data in the song, creates a timetable, and passes it to the stage manager. The stage manager arranges players on the stage based on timbre information, and instructs the stage equipment based on the genre of the music. When the song starts, the writing / camera work instructions are given to the light / camera staff from the timetable.
[0052]
The costume / makeup staff decides the costume / makeup of each player based on the genre of the music and adds the costume and makeup to each player module. The light / camera staff and the stage equipment staff perform lighting and smoke on the stage to effectively produce the music according to instructions from the stage manager.
[0053]
In actual performance of natural musical instruments, each finger moves very finely. However, in the case of a band (ensemble) in which a plurality of players play together, each player is displayed in a small size, and the 3D of the plurality of players is displayed. Since the calculation amount of the animation rendering process is enormous, the operation of the finger is omitted.
[0054]
Next, a player animation when performing a solo performance in the automatic performance device will be described. In the case of solo, since there is only one player, that player is displayed large, and sometimes the fingers are displayed in close-up. Therefore, in the case of solo, a realistic animation cannot be generated unless the fine movement of the finger is reproduced. Therefore, in the case of a solo performance, unlike the case of the band performance described above, a player animation accompanied by a fingering animation for moving a finger is generated in the same manner as an actual instrument player. This fingering animation can be used for learning a piano or a guitar because finger usage is realistically reproduced.
[0055]
It should be noted that the animation of the solo performance may be only finger animation (fingering animation) as shown in FIG. 7 (A), and the player's whole body animation is accompanied by fingering as shown in FIG. 7 (B). May be. In the case of the whole-body animation, the animation up to the wrist may be generated based on the music track data using the player database data shown in FIG. FIG. 7 shows an example of a pianist, but any other musical instrument such as a guitar or a wind instrument can perform fingering animation.
[0056]
Data for performing a fingering animation is also stored in the player database 13 as a fingering database as shown in FIG. The fingering database is described for each player of each part in the same manner as the normal player database, and includes finger polygon data representing the shape of the fingers and performance operation data for instructing the movement of each finger during performance. The finger polygon data is a set of polygon data representing the shapes of the palm and the first to fifth fingers, as shown in FIG. In this embodiment, each player stores a unique finger shape (finger polygon data). However, only one finger polygon data may be stored in common for all players.
[0057]
When the automatically played music is a solo music, the polygon data 130 of the player's limb is read from the player database 13 of the instrument, and the finger polygon data 136 is read from the fingering database 135. Then, as in the case of band performance, costumes and the like are determined according to the genre, and an animation of the performance action is generated according to the music.
[0058]
When this animation is for production such as karaoke, the whole body of the player is mainly displayed as shown in FIG. 7B, and when this animation is used for learning such as a piano or guitar, FIG. As shown in (A), the animation is such that the fingering can be seen in a plane. This distinction may be written in the production data of the song, or the user may be able to select either.
[0059]
Note that when only fingering is displayed for learning, there is no need for the whole body animation of the player. In that case, only the fingering animation is generated based only on the finger polygon data and the fingering motion data. .
[0060]
  Here, as shown in FIG. 8C, the fingering data is individually stored for each song (solo performance song). The fingering data is stored in the music data storage unit 1 in association with the performance data. As a method of obtaining this fingering data, fingering of actual performanceTheThere are a method of recording, a method of generating a fingering using rules of skilled performers based on musical notes (musical track sequence data).
[0061]
FIG. 9 is a diagram showing a fingering data recording apparatus for recording fingering data when an actual performer performs. This fingering data recording device also serves as a performance recording device for recording so-called performance data. The keyboard 20 includes a key switch for detecting key-on / key-on and a touch-intensity switch for detecting touch intensity, as well as a touch contact 20a provided on the surface of each key and a pulse signal input to each touch contact. A pulse switch 22 is provided. When the performer plays this keyboard, the performance data (note-on data, note-off data, touch intensity data) is detected by the note sensor 21 and input to the finger determination unit 24. When note-off data and touch intensity data are input, the finger determination unit 24 inputs the data to the recording unit 27 as it is. However, when note-on data is input, the finger determination unit 24 performs the following operation.
[0062]
When note-on is detected, the finger determination unit 24 turns on a pulse switch corresponding to the key to input a pulse signal to the touch contact of the key. Then, the pulse signal generated by the pulse generator 23 flows into the touch contact formed on the surface of the key turned on via the pulse switch, and the pulse signal is transmitted to the finger pressing the key at that time.
[0063]
As shown in the figure, finger terminals 25 are connected to each finger of the performer, and the sensor 26 detects from which finger terminal the pulse signal is received. The sensor 26 inputs the detected finger numbers (R1 to R5, L1 to L5) to the finger determination unit 24. The finger determination unit inputs the note-on data and the finger number data into the recording unit 27 in association with each other.
[0064]
The recording unit 27 records the input performance data and finger number data together with timing data from the timer 28. This performance can be reproduced by reading the recorded data according to the timing data.
By recording performance data using this apparatus, finger usage (fingering) of the player who actually played the song can be recorded as data.
[0065]
On the other hand, there are performance data in which a solo performance is recorded by a normal performance recording device and so-called in-game performance data in which event data is input step by step, and there is no corresponding fingering data in such performance data. . Therefore, in order to perform fingering animation during automatic performance of such performance data, it is necessary to generate fingering data by some method. Hereinafter, a method for generating fingering data by rule-based inference will be described.
[0066]
FIG. 10 is a block diagram of a fingering data inference apparatus that generates fingering data based on performance data. This device is a device for inferring fingering data of performance of a keyboard instrument. The performance data (note on / off data) 30 of the solo performance is input to the inference unit 31. A rule storage unit 32 and a fingering table 33 are connected to the inference unit 31. When note-on (key-on) data is input, based on the rule or table, it is inferred which finger should be turned on, taking into account the key that was turned on at the time and the key that was turned on immediately before . This inference rule may be a simulation of the fingering decision rule of a skilled keyboard player (pianist). In addition, the fingering table 33 stores basic fingering patterns, so that the effort of inference is saved in re-determination of simple fingering. The fingering data determined by this inference is recorded in the fingering data recording unit 35. By associating the fingering data with the performance data, a fingering animation can be displayed during automatic performance of the performance data.
[0067]
Inference rules used by the inference unit 31 include the following.
[0068]
Rule A: If there is no key that is currently turned on and there is no key that was turned off immediately before, a table is used to determine which finger on the left or right hand is turned on based on the key number turned on this time.
[0069]
B rule: When there is a key that is already turned on or there is a key that was turned off immediately before, the key pressing finger is determined as follows based on the key-on pattern at that time.
[0070]
B-1: When the sound is the highest, the key is turned on with the finger of the right hand, and all the keys that are turned on with the finger of the right hand at that time are sorted in ascending order, and R1 (right Thumb numbers R2 (right index finger), R3 (right middle finger), R4 (right ring finger), and R5 (right little finger) are output as fingering data.
[0071]
B-2: When the sound is the lowest, the key is turned on with the finger of the left hand, and then all the keys that are turned on with the finger of the left hand are sorted from the highest to the least, and in order L1 (left Thumb numbers L2 (left index finger), L3 (left middle finger), L4 (left ring finger), and L5 (left little finger) are output as fingering data.
[0072]
B-3: When the sound is within one octave from the highest sound, the sound is turned on with the finger of the right hand, and a free finger is assigned. If there is no free finger, or if the free finger does not reach, sort again to change the finger assignment.
[0073]
B-4: When the sound is within one octave from the lowest sound, the sound is turned on with the finger of the left hand, and a free finger is assigned. If there is no free finger, or if the free finger does not reach, sort again to change the finger assignment.
[0074]
At the time of sorting, not only the finger numbers are assigned in order, but also attention may be paid to those that are spaced apart from each other, and the finger numbers may be skipped and assigned so that the intervals can be crossed.
[0075]
The fingering data may be generated at any time. For example, it may be generated in real time during the performance of the song. When a song is requested (performance instruction), the fingering data may be generated first and then the automatic performance may be started. It may also be performed in advance regardless of the performance.
[0076]
Further, if the fingering data once generated is stored in correspondence with the performance data, automatic performance and fingering animation can be immediately displayed using the data thereafter. However, when the storage capacity of the music data storage unit 1 is limited, the fingering data may be discarded for each performance and regenerated each time the performance is performed.
[0077]
FIG. 11 is a flowchart showing the operation of the automatic performance device. This operation is executed when there is an automatic performance instruction (a request for a karaoke device). First, the performance data of the instructed music is read (s1), and it is determined whether the music is a band (ensemble) performance or a solo performance based on the configuration of the header and the music track (s2). ). In the case of a band performance song, the performance part is read based on the tone of the musical tone track (s3), the polygon data of the player of that part is read from the player database (s4), its layout is determined and placed on the stage. (S5). At the same time, the player's costumes / makeup and stage effects are performed.
[0078]
When performance preparation is completed in this way, automatic performance is started (s6), and player animation is displayed according to note data input in parallel with the performance of the music (s7).
[0079]
On the other hand, in the case of solo performance data, it is determined whether fingering data already exists (s10). This fingering data may be recorded at the time of performance by the fingering data recording device of FIG. 9, or may be determined by the fingering data inference device of FIG. If there is fingering data, it is read out (s11), the player is selected (s4), displayed on the stage (s5), the performance is started (s6), and the solo performance animation is displayed. Display (s7). In this case, a fingering animation is displayed.
[0080]
On the other hand, if there is no fingering animation, the musical tone track data is passed to the inference device of FIG. 10 prior to automatic performance, and after generating fingering data, the process proceeds to s4.
[0081]
In this way, when performing fingering animation, rough body movements are based on note-on / off data, and fine finger movements are accurately determined by determining fingering data in addition to note-on / off data. It can be animated and facilitates real-time processing.
[0082]
It should be noted that a robot is prepared for player animation and fingering animation of a specific part, and when this part is designated (by performance data), the robot may be moved to perform a performance operation. Good.
[0083]
A karaoke apparatus having the animation function will be described below. FIG. 12 is a block diagram of the karaoke apparatus. This karaoke device outputs the player's animation as a video signal in the same way as the automatic performance device when playing karaoke music. In addition, this karaoke device has two automatic performance functions (sequencer) and one or two sound sources, and synchronizes two performance data of the same song with different arrangements (gorgeous, techno, etc.) in parallel. By reading and inputting the parts selected from both performance data to the sound source, it is possible to play the music by mixing (morphing) the performance parts of different arrangements. This morphing change can be made freely during the performance of the song. The player animation is adapted to the performance mode such that the player of the part that has been pronounced by morphing appears and the player of the part that is no longer pronounced leaves.
[0084]
In FIG. 12, two sequencers 41 and 42 sharing a performance clock signal read out song data version 1 and song data version 2 which are songs of the same song and different arrangements in parallel. The song data version 1 and the song data version 2 each have a plurality of tracks (tone colors), and the data of each track is read in parallel according to the performance clock. The data of each track read by the sequencer 41 and the sequencer 42 is input to the mute controller 45. The mute controller selects a track to be cut off and a track to be passed out of the two performance data tracks, and inputs only the data of the track to be passed to the sound source 46.
[0085]
The morphing controller 50 instructs the mute controller 45 which part of which performance data is to be passed. The morphing control unit 50 reads the morphing sequence data 51 in parallel with the automatic performance of the sequencers 41 and 42, and controls the part to be muted based on this. The morphing sequence data 51 is stored in the storage unit 40 in association with song data version 1 and song data version 2. When there is no morphing sequence data 51, the part to be muted is controlled based on the operation data (generated by the user's operation) input from the morphing operation unit 52.
[0086]
FIG. 13 shows an aspect of morphing. In this example, version 1 song data consists of 7 parts, part 1 to part 7, and version 2 song data also consists of 7 parts, part 1 to part 7. Each part may be, for example, a part indicating functions such as “melody, bass, rhythm, obligato, arrange key, chord, etc.”. “E guitar 1, E guitar 2, keyboard, E bass, drum set, brass 1” , Brass 2 ”and other parts divided by musical instruments (timbres). In the morphing sequence data, morphing parameters that change every moment are written as event data as shown in the lower part of FIG.
[0087]
In this example, at the start of performance, all version 1 parts are input to the sound source 46, and all version 2 parts are muted (morphing parameter = 0). Then, the morphing control unit 50 reads the event data of the morphing sequence data 51, and when the morphing parameter is not 0, the version 1 part is muted according to the numerical value, and the version 2 part is input to the sound source 46.
[0088]
In this example, the versions are switched from part 1 to part 7 in order, but this order may be other than this. Instead of switching between parts of the same type, switching may be performed between different parts (for example, part 1 of version 1 and part 7 of version 2).
[0089]
On the other hand, the animation generation unit 52 receives the animation data 53 stored in the storage unit 40. The animation data 53 includes the animation database and the production database. The animation generation unit 52 can create animations of both the version 1 song player and the version 2 song player based on the animation data 53, but based on the control information input from the morphing control unit 50. Only the animation of the part player that the sound source 46 produces is generated and output as a video signal. This video signal is displayed on a monitor.
[0090]
In this embodiment, the morphing parameters are changed by the operation of the morphing sequence data 51 or the morphing operation unit 52. However, the morphing parameters are determined according to the singing situation of the singer input from the microphone (not shown). You may make it fluctuate. Further, the morphing parameters may be automatically changed by a specific algorithm.
[0091]
Part morphing in the player animation is replaced by morphing of the animation in which the player who is playing is gradually deformed and becomes a new part player in addition to the replacement due to the leaving / appearance of the player. Also good.
[0092]
For example, the animation morphing when the performance arrangement is morphed from a full orchestra to a band can be as follows.
1. Performance by full orchestra
2. Contrabass is replaced by morphing to electric bass
3. Percussion instruments are replaced by morphing drum sets
4). The trumpet team leaves with an interlude and a guitar player appears
5. The string instrument player leaves and a keyboard player appears instead.
[0093]
Further, the performance parts may be replaced by fade-in / fade-out in addition to the method of performing cut-in / cut-out. In this case, the mute controller 45 does not suddenly cut off the data of the track to be faded out, but gradually rewrites the velocity so as to output it to the sound source 46, and finally when the velocity becomes zero, the data. The output is stopped. Conversely, the data of the track that fades in is output to the sound source 46 while gradually increasing from velocity = 0, and is increased to 100 percent (the velocity originally written in the music data).
[0094]
In this case, the sound pan control is performed in parallel with the fade-in / fade-out, and the animation that the player who fades out disappears in the right (left) direction and the player who fades in appears from the left (right) direction is displayed. May be.
[0095]
Further, in the case of completely shifting from the performance of version 1 to the performance of version 2 in accordance with a predetermined scenario (morphing sequence data), it is also possible to shift parts with the following effects.
The musical sound of a new arrangement is generated as a reverb sound, faded in and gradually changed to a dry sound, overwhelming the musical sound of the previous arrangement, and the transition is completed.
Replace with sound field control using sound space effector.
In any case, in the case of these part transfer methods, animation is also transferred by a method corresponding to the part transfer method.
[0096]
【The invention's effect】
As described above, according to the present invention, an animation of a player (player) can be created by synthesizing animation materials based on performance data, thereby displaying an animation without preparing video data in advance. It becomes possible.
[0097]
  As for the fingering of the performer, by preparing fingering data describing the correct fingering in advance, it is possible to create and display an accurate and realistic animation.wear.
[0098]
Further, since fingering data can be recorded in parallel with recording of performance data, this can be used as it is in an automatic performance device to display automatic performance and fingering animation.
[0099]
Also, performance data parts with different arrangements can be mixed and played automatically, and the selection of that part can be switched in the middle of the automatic performance, so it is possible to switch to another arrangement during the performance. Of course, a new arrangement that has never been possible will be possible.
[0100]
At that time, since an animation corresponding to the part selection is displayed, there is an advantage that the user can visually easily understand what part selection is currently automatically performed.
[Brief description of the drawings]
FIG. 1 is a block diagram of an automatic performance apparatus according to an embodiment of the present invention.
FIG. 2 is a view showing an example of a player animation displayed by the automatic performance device
FIG. 3 is a diagram for explaining functions of an animation unit of the automatic performance device
FIG. 4 is a view showing an example of karaoke performance data and effect data used in the automatic performance device
FIG. 5 is a diagram showing an example of a player database used in the automatic performance device
FIG. 6 is a diagram showing an example of a player animation created by the automatic performance device
FIG. 7 is a diagram showing an example of a fingering animation
FIG. 8 is a diagram showing an example of a fingering database
FIG. 9 is a configuration diagram of a fingering recording apparatus according to an embodiment of the present invention.
FIG. 10 is a block diagram of a fingering reasoning apparatus.
FIG. 11 is a flowchart showing the operation of the automatic performance device.
FIG. 12 is a block diagram of an automatic performance device according to an embodiment of the present invention.
FIG. 13 is a diagram showing an example of part morphing of the automatic performance device
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 ... Song data storage part, 2 ... Automatic performance part, 3 ... Sound source, 4 ... Animation part, 5 ... Display part, 10 ... Player selection and arrangement | positioning part, 11 ... Performance motion calculation part, 12 ... Animation production | generation part, 13 ... Player database 130... Player polygon data 131. Performance data 135. Fingering database 136. Finger polygon data 137 performance data 14. Production database 15 display mode control unit
DESCRIPTION OF SYMBOLS 20 ... Keyboard, 20a ... Touch contact, 21 ... Note sensor, 22 ... Pulse switch, 23 ... Pulse generator, 24 ... Finger determination part, 25 ... Finger terminal, 26 ... Sensor, 27 ... Recording part, 28 ... Timer,
30 ... Music data storage unit, 31 ... Inference unit, 32 ... Inference rule storage unit, 33 ... Fingering table, 35 ... Fingering data storage unit,
DESCRIPTION OF SYMBOLS 40 ... Memory | storage part, 41, 42 ... Sequencer, 43, 44 ... Performance data, 45 ... Mute controller, 46 ... Sound source, 50 ... Morphing control part, 51 ... Morphing sequence data, 52 ... Morphing operation part, 53 ... Animation generation part 54 animation data

Claims (4)

Automatic performance means for automatically playing music based on performance data;
Material storage means for storing animation materials of performers of multiple types of instruments ,
An animation creating means for creating an animation of the performer who plays the music piece by selecting a performer of a musical instrument designated by the performance data and synthesizing an animation material based on at least the pitch of the performance data; ,
An automatic performance device characterized by comprising: The material storage means stores animation material including shape information and performance information of the performer's fingers ,
The animation creating means inputs fingering data indicating which finger performs the performance operation in synchronization with the performance data,
Synthesizing the animation material based on at least the pitch and fingering data of the performance data, thereby creating an animation displaying the performer of the musical instrument specified by the performance data and the fingering of the player; The automatic performance device according to claim 1 . Reading means that each has a plurality of parts, reads out in parallel a plurality of versions of performance data having the same melody and different chords and rhythms,
A selection means for selecting and selecting an arbitrary part from the plurality of versions, and inputting only the data of the selected part into the sound source;
Selection changing means for inputting selection change data for instructing the selection means to change the selection while the performance data is being read by the reading means;
Performance data input means for inputting the data of the part selected by the selection means to the animation creating means as performance data;
The automatic performance device according to claim 1, further comprising: The animation creation means is means for creating an animation of a player of a newly selected part instead of an animation of a player of a part that is not selected when the selection of the part is changed by the selection changing means. Item 4. The automatic performance device according to Item 3 .

Images