JP2007122085A - Device and method for performing synchronous playback of voice data and performance data - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a recording device, a playback device, a recording method, a playback method, and a program, capable of synchronously playing back the playback data with the correct timing with respect to the voice data of the same musical piece but recorded at a different speed. <P>SOLUTION: A controller part 6 records on an SMF a MIDI data of a performance which is carried out to a playback of a music CD using a piano 31. In this case, the controller part 6 makes a comparison between the values shown by components of 100Hz or lower and 1Hz or lower of the voice data of the music CD, and records an event for management produced, as a result to the SMF. Following this, the controller part 6 performs reproduction of the MIDI data, recorded on the SMF in synchronization with the playback of the music CD. In that case, the controller part 6 produces the event for management from the voice data of the music CD, and adjusts the timing for reproducing the MIDI data, by comparing the timing for management recorded on the SMF with the timing for management to be reproduced. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an apparatus and method for reproducing performance data including information related to music performance control in synchronization with reproduction of audio data.

  As means for reproducing music, there is an apparatus that reads audio data from a storage medium such as a music CD (Compact Disc) and generates and outputs audio from the read audio data. Further, as another means for reproducing the music, data including information related to performance control of the music is read from a storage medium such as FD (Floppy (registered trademark) Disk), and the read data is used for the sound source device. Some devices perform automatic performance by controlling pronunciation. As data including information related to music performance control, there is MIDI data created according to the MIDI (Musical Instrument Digital Interface) standard.

  Recently, a method has been proposed in which automatic performance using MIDI data is synchronized with reproduction of audio data recorded on a music CD. One of them is a method using a time code recorded on a music CD (see, for example, Patent Document 1 and Patent Document 2). Hereinafter, this method will be described.

  First, audio data and a time code of a music CD are reproduced by the music CD reproducing device. The audio data is output as sound, and the time code is supplied to the recording device. Here, the time code is data associated with a certain unit of audio data, and each time code represents an elapsed time from the start of the music to the reproduction timing of the audio data corresponding to the time code. ing. The musical instrument is played in accordance with the reproduction of the music CD, and MIDI data is sequentially supplied from the musical instrument to the recording device. When the recording device receives the MIDI data from the musical instrument, the recording device records the MIDI data on a recording medium together with time information indicating the reception timing. Further, when the recording device receives the time code from the music CD reproducing device, the recording device records it on the recording medium together with time information indicating the reception timing. As a result, a file in which time code and MIDI data are mixed is created on the recording medium. In this file, each time code and MIDI data is accompanied by time information indicating the elapsed time from the music playback start time to each playback time.

  When the MIDI data and the time code are recorded on the recording medium in this way, when the audio data of the same music is subsequently reproduced from the music CD, the MIDI data is read from the recording medium in synchronism with this and the automatic performance is performed. It can be carried out. The operation is as follows.

  First, audio data and time code are reproduced from the music CD by the music CD reproducing device. The audio data is output as sound, and the time code is supplied to a MIDI data reproducing apparatus. At the same time, the playback device reads the MIDI data recorded in the file in accordance with the time information recorded together and sequentially transmits it to the musical instrument capable of automatic performance using the MIDI data. At that time, the playback device adjusts the time lag between the playback of the audio data of the music CD and the playback of the MIDI data based on the time code received from the music CD playback device and the time code read from the file together with the MIDI data. As a result, the synchronized playback of the audio data of the music CD and the MIDI data is realized.

Japanese Patent Application No. 2002-7872 Japanese Patent Application No. 2002-7873

  However, for music CDs with different time codes even for the same music, synchronized playback of audio data and MIDI data of the music CD cannot be realized by a method using the time code of the music CD.

  Currently, there are many different versions of music CDs for the same song. Music CDs usually store audio data that has been edited such as adding reverberation or other acoustic effects to the master data of the music. In such different versions of music CDs, the offset time until the beginning of the music may be different. Also, the clock speed used when recording the edited audio data may not exactly match the clock speed used when the master data is recorded. As a result, time codes having slightly different time intervals may be attached to different versions of music CDs that record differently edited audio data. In such a case, if MIDI data for synchronized performance created by a technique using a conventional time code is used for a music CD of a different version of the same music piece, the music CD and the MIDI data are used at the beginning of the music piece. Even if the performance is out of sync or the performance of the music CD and the MIDI data is synchronized at the beginning of the music, the music CD and the performance of the MIDI data are gradually shifted over time.

  Therefore, according to the conventional technology using time codes, even for music CDs that record audio data of the same music, according to variations in time intervals of time codes attached to the respective music CD versions. Therefore, there is a problem that MIDI data for different synchronized performances must be prepared.

  In view of the above-described situation, the present invention can be applied to a plurality of versions of audio data recorded at different time intervals because of different clock speeds used for recording, even if the audio data is the same music. On the other hand, an object of the present invention is to provide a recording device, a reproducing device, a recording method, a reproducing method, and a program for performance data such as MIDI data that can be synchronized and reproduced.

  In order to solve the problems described above, the present invention provides first receiving means for receiving sound data indicating a sound waveform of music, second receiving means for receiving control data for instructing performance control, and the sound data. First generation means for generating a first index indicating a change tendency of a predetermined period of time, second generation means for generating a second index indicating a change tendency of the audio data for a period longer than the predetermined period, and the control data And the first time data indicating the reception timing of the control data are recorded, and the first index and the second index are compared to detect a sudden change in the audio data, There is provided a recording apparatus comprising: recording means for recording second time data indicating the timing of abrupt change.

  The present invention also provides performance data comprising control data for instructing performance control and first time data for instructing execution timing of the performance control, and second time data indicating the timing of a sudden change in the sound waveform; First receiving means for receiving the sound, second receiving means for receiving the sound data indicating the sound waveform of the music, first generating means for generating a first index indicating a change tendency of the sound data over a predetermined period, The second generation means for generating a second index indicating a change tendency of the audio data for a period longer than the predetermined period is compared with the first index and the second index to detect a sudden change in the audio data. And adjusting the first time data on the basis of the second time data and the time information indicated by the third time data, the third generating means for generating the third time data indicating the timing of the rapid change. And adjusting means, at a timing according to the first hour data adjusted by said adjustment means, to provide a reproducing apparatus, characterized in that it comprises a transmission unit that transmits the control data.

  Further, the present invention relates to a first reception process for receiving audio data indicating the audio waveform of a song, a second reception process for receiving control data for instructing performance control, and a change tendency of the audio data over a predetermined period. A first generation process for generating a first index to be shown, a second generation process for generating a second index indicating a change tendency of the audio data over a period longer than the predetermined period, and reception timing of the control data and the control data The performance data including the first time data indicating the first time data is recorded, the first index is compared with the second index, a sudden change in the audio data is detected, and the timing of the sudden change is indicated. And a recording process for recording the second time data.

  The present invention also provides performance data comprising control data for instructing performance control and first time data for instructing execution timing of the performance control, and second time data indicating the timing of a sudden change in the sound waveform; A first reception process for receiving a sound, a second reception process for receiving sound data indicating a sound waveform of a song, a first generation process for generating a first index indicating a change tendency of the sound data over a predetermined period, and A second generation process for generating a second index indicating a change tendency of audio data over a period longer than the predetermined period is compared with the first index and the second index, and a sudden change in the audio data is detected. And adjusting the first time data based on a third generation process for generating third time data indicating the timing of the sudden change, and time information indicated by the second time data and the third time data. And adjustment process, the timing according to the adjusted first hour data by adjusting means, to provide a reproducing method characterized by comprising a transmission process for transmitting the control data.

  The present invention also provides a program for causing a computer to execute processing using these recording methods and reproducing methods.

  The present invention also includes performance data comprising control data for instructing performance control, first time data for instructing execution timing of the performance control, and second time data indicating the timing of abrupt change in the sound waveform. A recorded recording medium is provided.

  When the apparatus, method, program, and recording medium having such a configuration are used, time data for time adjustment is generated and recorded at the timing when the sound waveform of the music changes suddenly at the time of recording and playback of the performance data. The transmission timing of the control data is adjusted by comparing the time data generated sometimes and the time data generated during reproduction. As a result, synchronized reproduction of audio data and control data is realized.

The recording apparatus according to the present invention further includes third receiving means for receiving a time code indicating the reproduction timing of the audio data, and the recording means is configured to receive the first time based on time information indicated by the time code. The data and the second time data may be generated.
The playback apparatus according to the present invention further includes third receiving means for receiving a time code indicating the playback timing of the audio data, and the third generating means is configured to receive the third code based on time information indicated by the time code. Time data may be generated, and the transmission unit may transmit the control data based on time information indicated by the time code.

  When the recording device and the playback device having such a configuration are used, the time data according to the time code is also measured for the audio data played back by the playback device having a bias in the playback speed. Is called.

Further, in the recording apparatus according to the present invention, the first generation unit performs a filtering process for extracting a component in a frequency band equal to or lower than a first frequency on the voice waveform represented by the voice data, thereby obtaining the first index. And the second generation means performs a filtering process to extract a component of a frequency band equal to or lower than a second frequency lower than the first frequency, on the voice waveform represented by the voice data, thereby obtaining the second index. It is good also as composition to generate.
In the playback apparatus according to the present invention, the first generation unit generates the first index by performing a filtering process for extracting a component in a frequency band equal to or lower than the first frequency on the voice waveform represented by the voice data. Then, the second generation means generates the second index by performing a filtering process for extracting a component in a frequency band equal to or lower than the second frequency lower than the first frequency with respect to the voice waveform represented by the voice data. It is good also as a structure.

  When the recording apparatus and reproducing apparatus having such a configuration are used, time data for time adjustment having a suitable time interval and accuracy can be obtained.

Further, in the recording apparatus according to the present invention, the recording means is arranged such that the second time data only when a sudden change is not detected in a certain period in the past from the timing at which the sudden change is detected. May be recorded.
Further, in the playback apparatus according to the present invention, the third generation means is configured so that the third generation means is provided only when a sudden change is not detected in the audio data for a certain period in the past from the timing when the sudden change is detected. It is good also as a structure which produces | generates time data.

  When the recording device and the playback device having such a configuration are used, even when the voice waveform frequently shows abrupt changes, the time interval of the time data for time adjustment is kept moderate, and confusion between adjacent time data is obtained. Does not occur.

  Further, in the playback device according to the present invention, the adjusting means may perform the performance based on a value indicated by a regression line or a regression curve obtained by performing regression analysis on the second time data and the third time data. A configuration may be employed in which the data reproduction timing is adjusted.

  When the reproduction apparatus having such a configuration is used, the error in the time data for time adjustment is adjusted as a whole, so that the accuracy of the transmission timing of the control data in the synchronous reproduction increases.

  As described above, according to the present invention, any one of different versions of audio data that is reproduced at a slightly different speed even if the same music is recorded due to reasons such as being recorded based on clocks of different frequencies. On the other hand, the performance data can be synchronously reproduced at the correct timing. Therefore, it is not necessary to prepare different performance data for different versions of the same music piece, and the creation and management of data is simplified.

[1] First Embodiment [1.1] Configuration, Function, and Data Format [1.1.1] Overall Configuration FIG. 1 shows the configuration of a synchronous recording / reproducing apparatus SS according to the first embodiment of the present invention. FIG. The synchronous recording / reproducing apparatus SS includes a music CD drive 1, an FD drive 2, an automatic performance piano unit 3, a sound generation unit 4, an operation display unit 5, and a controller unit 6.

  The music CD drive 1, the FD drive 2, the automatic performance piano unit 3, the sound generation unit 4, and the operation display unit 5 are connected to the controller unit 6 through communication lines. Further, the automatic performance piano unit 3 and the sound generation unit 4 are directly connected by a communication line.

[1.1.2] Music CD Drive The audio data recorded on the music CD includes audio data indicating audio information and a time code indicating the reproduction timing of the audio data. The music CD drive 1 is a device that reads audio data from a loaded music CD in accordance with an instruction from the controller unit 6 and sequentially outputs audio data included in the read audio data. The music CD drive 1 is connected to the communication interface 65 of the controller unit 6 by a communication line.

  The audio data output from the music CD drive 1 is digital audio data consisting of left and right channels with a sampling frequency of 44100 Hz and a quantization bit number of 16. The data output from the music CD drive 1 does not include a time code. Since the configuration of the music CD drive 1 is the same as that of a general music CD drive capable of digitally outputting audio data, the description thereof is omitted.

[1.1.3] FD Drive The FD drive 2 is a device that records SMF (Standard MIDI File) on the FD, reads the SMF recorded on the FD, and transmits the read SMF. The FD drive 2 is connected to the communication interface 65 of the controller unit 6 through a communication line. Since the configuration of the FD drive 2 is the same as that of a general FD drive, the description thereof is omitted.

[1.1.4] MIDI events and SMF
The SMF is a file including a MIDI event that is performance control data according to the MIDI standard, and a delta time that is data indicating the execution timing of each MIDI event. The MIDI event and SMF format will be described with reference to FIGS.

  FIG. 2 shows a note-on event, a note-off event, and a system exclusive event as examples of the MIDI event. The note-on event is a MIDI event for instructing the pronunciation of a musical sound. The note-on event is 9 nH (n is a channel number, H is a hexadecimal number, and so on), the note number is a pitch, and the strength of the sound ( Or velocity indicating the speed of keystroke). Similarly, the note-off event is a MIDI event for instructing the muting of a musical sound, and is composed of 8nH indicating muting, a note number indicating pitch, and a velocity indicating the strength at the time of muting (or the speed at which the key is released). . On the other hand, the system exclusive event is a MIDI event for transmitting / receiving or recording data in a format freely determined by the manufacturer of the product or software. F0H indicating the start of the system exclusive event, data length, data, and the end of the system exclusive event It consists of F7H which shows. As described above, the MIDI event does not have time information and is used for the purpose of performing tone generation, muting, and other control in real time.

  FIG. 3 shows an outline of the SMF format. The SMF consists of a header chunk and a track chunk. The header chunk includes control data related to the format and time unit of data included in the track chunk. The track chunk includes a MIDI event and a delta time indicating the execution timing of each MIDI event.

In SMF, the delta time is a method of expressing the relative time for the immediately preceding MIDI event in units of time called a clock, and the absolute time from the beginning of the music is called hours, minutes, seconds and frames. There is a method of expressing by a combination of time units. In the following description, for ease of explanation, the delta time is an absolute time from the reference time point, and its unit is expressed in seconds.
In this specification, MIDI data is a generic name for data created in accordance with the MIDI standard.

[1.1.5] Automatic Performance Piano Unit The automatic performance piano unit 3 generates an acoustic piano sound and a piano sound by electronic musical tone synthesis in response to a key operation and pedal operation by a user of the synchronous recording / reproducing apparatus SS. It is a musical sound generator that outputs. The automatic performance piano unit 3 generates a MIDI event in response to a key operation and a pedal operation by the user, and transmits the generated MIDI event. Further, the automatic performance piano unit 3 receives a MIDI event, and performs an automatic performance with an acoustic piano sound and a piano sound by electronic musical sound synthesis in accordance with the received MIDI event.

  The automatic performance piano unit 3 includes a piano 31, a key sensor 32, a pedal sensor 33, a MIDI event control circuit 34, a sound source 35, and a drive unit 36.

  The key sensor 32 and the pedal sensor 33 are disposed on each of the plurality of keys and the plurality of pedals of the piano 31, respectively, and detect the positions of the keys and the pedals, respectively. The key sensor 32 and the pedal sensor 33 transmit the detected position information to the MIDI event control circuit 34 together with the identification number and detection time information corresponding to each key and pedal.

  The MIDI event control circuit 34 receives the key and pedal position information from the key sensor 32 and the pedal sensor 33 together with the key and pedal identification information and time information. This is a circuit that generates a MIDI event such as an off event and outputs the generated MIDI event to the controller unit 6 and the sound source 35. Further, the MIDI event control circuit 34 has a function of receiving a MIDI event from the controller unit 6 and transferring the received MIDI event to the sound source 35 or the driving unit 36. Whether the MIDI event control circuit 34 transfers the MIDI event received from the controller unit 6 to either the sound source 35 or the drive unit 36 depends on an instruction from the controller unit 6.

  The sound source 35 is a device that receives a MIDI event from the MIDI event control circuit 34 and outputs sound information of various musical instruments as digital audio data of two left and right channels based on the received MIDI event. The sound source 35 electronically synthesizes the digital audio data of the pitch designated by the received MIDI event and transmits it to the mixer 41 of the sound generator 4.

  The drive unit 36 is disposed on each key and each pedal of the piano 31 and includes a solenoid group that drives them and a control circuit that controls the solenoid groups. When receiving the MIDI event from the MIDI event control circuit, the control circuit of the drive unit 36 adjusts the amount of current supplied to the solenoid arranged in the corresponding key or pedal, and controls the magnetic force generated by the solenoid, Realizes key or pedal operation in response to MIDI events.

[1.1.6] Sound Generation Unit The sound generation unit 4 is a device that receives audio data from the automatic performance piano unit 3 and the controller unit 6, converts the received audio data into sound, and outputs the sound. The sound generation unit 4 includes a mixer 41, a D / A converter 42, an amplifier 43, and a speaker 44.

  The mixer 41 is a digital stereo mixer that receives a plurality of digital audio data consisting of left and right channels and converts them into a set of left and right digital audio data. The mixer 41 receives the digital audio data from the sound source 35 of the automatic performance piano unit 3 and simultaneously receives the digital audio data read from the music CD by the music CD drive 1 via the controller unit 6. The mixer 41 arithmetically averages the received digital audio data and transmits the digital audio data to the D / A converter 42 as a pair of left and right digital audio data.

  The D / A converter 42 receives the digital audio data from the mixer 41, converts the received digital audio data into an analog audio signal, and outputs the analog audio signal to the amplifier 43. The amplifier 43 amplifies the analog audio signal input from the D / A converter 42 and outputs it to the speaker 44. The speaker 44 converts the amplified analog audio signal input from the amplifier 43 into sound. As a result, the sound data recorded on the music CD and the sound data generated by the sound source 35 are output from the sound generator 4 as stereo sound.

[1.1.7] Operation Display Unit The operation display unit 5 is a user interface used when the user of the synchronous recording / reproducing apparatus SS performs various operations of the synchronous recording / reproducing apparatus SS.
The operation display unit 5 has a keypad to be pressed when the user gives an instruction to the synchronous recording / reproducing apparatus SS, a liquid crystal display for the user to confirm the state of the synchronous recording / reproducing apparatus SS, and the like. When the user presses the keypad, the operation display unit 5 outputs a signal corresponding to the pressed keypad to the controller unit 6. When the operation display unit 5 receives bitmap data including character and graphic information from the controller unit 6, the operation display unit 5 displays characters and graphics on the liquid crystal display based on the received bitmap data.

[1.1.8] Controller Unit The controller unit 6 is a device that controls the entire synchronous recording / reproducing apparatus SS. The controller unit 6 includes a ROM (Read Only Memory) 61, a CPU (Central Processing Unit) 62, a DSP (Digital Signal Processor) 63, a RAM (Random Access Memory) 64, and a communication interface 65. These components are connected to each other by a bus.

  The ROM 61 is a non-volatile memory that stores various control programs. The control program stored in the ROM 61 includes a program for causing the CPU 62 to execute processing in SMF recording and playback operations, which will be described later, in addition to a program for performing general control processing. The CPU 62 is a microprocessor capable of executing general-purpose processing, reads a control program from the ROM 61, and performs control processing according to the read control program. The DSP 63 is a microprocessor capable of processing digital audio data at a high speed. Under the control of the CPU 62, the DSP 63 receives digital audio data received by the controller unit 6 from the music CD drive 1 or FD drive 2 in a management event generation process described later. Necessary processing such as filter processing is performed, and data obtained as a result is transmitted to the CPU 62. The RAM 64 is a volatile memory, and temporarily stores data used by the CPU 62 and the DSP 63. The communication interface 65 is an interface capable of transmitting / receiving digital data of various formats, and is digital data transmitted / received among the music CD drive 1, the FD drive 2, the automatic performance piano unit 3, the sound generation unit 4, and the operation display unit 5. The necessary format conversion is performed, and data is relayed between these devices and the controller unit 6.

[1.2] Operation Next, the operation of the synchronous recording / reproducing apparatus SS will be described.
[1.2.1] Recording Operation First, the user of the synchronous recording / reproducing apparatus SS performs the piano in accordance with the reproduction of the commercially available music CD, and the synchronous recording / reproducing when the performance information is recorded on the FD as MIDI data. The operation in the device SS will be described. Note that a music CD used in a recording operation described below is referred to as a music CD-A in order to distinguish it from a music CD used in a reproduction operation described later.

[1.2.1.1] Recording start operation The user sets the music CD-A in the music CD drive 1 and the blank FD in the FD drive 2. Subsequently, the user presses the keypad of the operation display unit 5 corresponding to the start of performance data recording. The operation display unit 5 outputs a signal corresponding to the pressed keypad to the controller unit 6.

  When the CPU 62 of the controller unit 6 receives a signal corresponding to the start of recording of performance data from the operation display unit 5, it transmits a music CD playback command to the music CD drive 1. In response to this playback command, the music CD drive 1 sequentially transmits audio data recorded on the music CD-A to the controller unit 6. The controller unit 6 receives one set of left and right audio data from the music CD drive 1 every 1/44100 seconds. Hereinafter, the value of one set of left and right data is represented as (R (n), L (n)), and is generated from the value of this one set or from this one set of data in the management event generation process described later. The value of each data is called “sample value”. R (n) and L (n) represent the values of the data of the right channel and the data of the left channel, respectively, and are integers from −32768 to 32767. n is an integer representing the order of the audio data, and increases in the order of 0, 1, 2,.

[1.2.1.2] Transmission of Audio Data to Sound Generation Unit When the CPU 62 receives a sample value from the music CD drive 1, the CPU 62 starts measuring time at that time as 0 seconds. At the same time, the CPU 62 sequentially transmits the received sample values to the sound generator 4. When the sound generation unit 4 receives the sample value from the controller unit 6, it sequentially converts it into a sound and outputs it. As a result, the user can listen to the music recorded on the music CD-A.

[1.2.1.3] Generation of Management Event The CPU 62 transmits the received sample values to the sound generation unit 4 and simultaneously records the received sample values in the RAM 64. When the CPU 62 records a predetermined number of sample values in the RAM 64, the CPU 62 transmits an execution command for performing management event generation processing described below to the DSP 63. The number of sample values recorded in the RAM 64 before the CPU 62 transmits an execution command for management event generation processing to the DSP 63 depends on the characteristics of the filter used in the processing described below. The longer the delay time of the delay units used in the filter, the larger the number. In the following description, as an example, it is assumed that the CPU 62 transmits an execution instruction for the management event generation process when sample values for one second, that is, 44100 sample values are recorded in the RAM 64.

The management event generating process is a filter process for extracting a component in a frequency band below a certain frequency from a sound waveform indicated by the sound data recorded in the RAM 64, and a filter for extracting a component in a frequency band lower than that frequency. This is a process for generating a management event by performing a process and comparing and determining values obtained by the two filter processes. The management event is a flag for creating time information used in timing adjustment processing described later.
Hereinafter, the management event generation process will be described with reference to FIG.

  When the DSP 63 receives the execution instruction of the management event generation process from the CPU 62, 44100 samples from the top of the sample values recorded in the RAM 64, that is, (R (0), L (0)) to (R (44099), L (44099)) is read out (step S1). Hereinafter, a series of 44100 sample values will be referred to as “raw audio data”, and when it is necessary to distinguish the raw audio data, the raw data ending with (R (n), L (n)) is used. The voice data is expressed as “raw voice data (n)”. That is, the raw audio data that is first read by the DSP 63 in step S1 is raw audio data (44099).

  Subsequently, the DSP 63 converts the stereo data into monaural data by arithmetically averaging the left and right values of each sample value of the read raw audio data (step S2). This monaural conversion process is a process for reducing the load on the DSP 63 in the process after this step.

  Subsequently, the DSP 63 obtains an absolute value for the series of sample values converted to monaural (step S3). The processing in this step is processing for obtaining an alternative value for the power of each sample value. Since the absolute value is smaller than the square value indicating power and is easy to process, the absolute value is used as a substitute value for the square value of each sample value in this embodiment. Therefore, when the processing capacity of the DSP 63 is high, a square value may be calculated instead of the absolute value of each sample value in this step.

  Subsequently, the DSP 63 further performs filter processing using a low-pass filter on the series of sample values converted into absolute values in step S3 (step S4). Hereinafter, a value obtained as a result of the processing of Steps S1 to S4 is referred to as “medium term index”, and a medium term index corresponding to (R (n), L (n)) is referred to as “medium term index (n)”. Hereinafter, as an example, the frequency of the low-pass filter used in step S4 is 100 Hz. The medium term index (n) indicates the tendency of medium term change in n of the speech waveform indicated by the series of sample values. That is, the voice waveform indicated by the series of sample values changes vertically in the short term, but the series of sample values subjected to the filtering process by the low-pass filter is suppressed from being changed by a plurality of preceding sample values. As a result, the short-term fluctuation element is removed from the speech waveform indicated by the series of medium-term indicators, and only the medium-term and long-term fluctuation elements remain. Through the processing in step S4, a series of medium-term indices ending with the medium-term index (n), that is, a medium-term index (n-2), a medium-term index (n-1), and a medium-term index (n) are created. The DSP 63 records these medium-term indices in the RAM 64 (step S5).

  Subsequently, the DSP 63 performs filter processing using a comb filter on the series of medium-term indices obtained in step S4 (step S6). The processing in this step is processing for extracting a component in a low frequency band below a certain frequency from the speech waveform indicated by the series of medium-term indicators. This step is the same as the processing using the low-pass filter having a frequency lower than that of the low-pass filter used in step S4. However, the comb filter usually has a smaller load on the DSP 63 than the low-pass filter. In the form, the low-pass filter is replaced by a comb filter.

  FIG. 5 shows the configuration of an example of a comb filter that can be used in step S6. In FIG. 5, processing indicated by a rectangle indicates delay processing, and k in z−k means that the delay time in the delay processing is (sampling period × k). As described above, since the sampling frequency of the music CD is 44100 Hz, the sampling period is 1/444100 seconds. On the other hand, processing indicated by a triangle indicates multiplication processing, and a value indicated in the triangle indicates a multiplication coefficient. Hereinafter, as an example, k = 222050 is set, and as a result, components in a frequency band higher than 1 Hz are roughly removed by the filtering process in step S6. In other words, the speech waveform indicated by the series of values obtained as a result of the processing in step S6 is obtained by removing only the long-term variation element from the speech waveform indicated by the series of medium-term indicators by removing the medium-term variation factor.

  Subsequently, the DSP 63 multiplies each of the series of values obtained by the filtering process in step S6 by a positive constant h (step S7). The multiplication process by h is a process for adjusting the frequency at which a positive result is obtained in the comparison determination process in the next step S8. Generally, when the value of h is small, a positive result in the comparison determination process is obtained. The time interval to be narrowed. If this time interval is too wide, the time interval for generating the management event in the following step S11 becomes wide, and as a result, the accuracy in the timing adjustment processing described later becomes low. On the other hand, if the time interval at which the positive result in the comparison determination process is obtained is too narrow, the positive result is canceled one after another by the process in the following step S10. The interval is widened, and as a result, the accuracy in the timing adjustment process is lowered. Therefore, as the value of h, a value that generates a management event at an appropriate frequency is empirically used.

  The value obtained as a result of the processing in step S7 is hereinafter referred to as “long-term index”, and the long-term index corresponding to (R (n), L (n)) is referred to as “long-term index (n)”. Through the processing in step S7, a series of long-term indicators ending with the long-term indicator (n), that is, a long-term indicator (n-2), a long-term indicator (n-1), and a long-term indicator (n) are created. The DSP 63 records these long-term indices in the RAM 64 (step S8).

  Subsequently, the DSP 63 reads out the medium-term index (n) and the long-term index (n) from the RAM 64, and compares and determines whether or not the medium-term index (n) is equal to or greater than the long-term index (n) (step S9). The medium-term index and long-term index first read by the DSP 63 in step S9 are the medium-term index (44099) and the long-term index (44099). This comparison determination process indicates that the speech waveform indicated by the raw speech data has greatly fluctuated in the medium term at the time corresponding to the sample value of (R (n), L (n)). That is, in the sound waveform of the music, when the volume of the sound included in the frequency band of 1 Hz to 100 Hz rapidly increases, the value of the medium term index exceeds the value of the long term index, and a positive result is obtained from the comparison determination process in step S9. (Hereinafter referred to as “Yes”).

  When Yes is obtained by the comparison determination process in step S <b> 9, the DSP 63 records the current time in the RAM 64. At the same time, the DSP 63 reads from the RAM 64 the record at the time when Yes was obtained in the comparison determination process performed in the past in step S9, and determines whether or not there is a record at which Yes was obtained in the past period τ. S10). The determination process in this step is a process for preventing the management event from being generated in the same short time interval in the next step S11 when the result of the comparison determination process in Step S9 is Yes in a short time interval. . If a management event is generated at a short time interval, it is possible to correctly associate the recorded management event with the management event generated from newly obtained audio data in the timing adjustment process described later. It becomes difficult. By the processing in step S10, it is avoided that management events are generated in close proximity at time intervals equal to or shorter than the period τ. As the value of τ, a value at which the generated management event has an appropriate time interval is empirically used. In the first determination process in step S10, the determination result is No because there is no preceding step S9.

  When a negative result (hereinafter referred to as “No”) is obtained in the determination process in step S10, the DSP 63 determines that the voice waveform indicated by the raw voice data is (R (n)) as a result of the series of processes described above. , L (n)), a management event indicating that a predetermined condition has been satisfied is transmitted to the CPU 62 (step S11).

  If the comparison determination result in step S9 is No, if the determination result in step S10 is Yes, and if the processing in step S11 is terminated, the DSP 63 causes the CPU 62 to read a new sample value from the music CD drive 1. That is, (R (n + 1), L (n + 1)) is received, and the process waits until the sample value is recorded in the RAM 64. When a new sample value is recorded in the RAM 64 (step S12), the DSP 63 returns the process to step S1, and a series of sample values ending with the sample value newly recorded in the RAM 64, that is, raw audio data (n + 1) ) For the above-described step S1 and subsequent steps.

  The processes in steps S1 to S12 are continued until the user gives an instruction to end recording of performance data while increasing the value of n by 1. The above is the description of the management event generation process. FIG. 6 is a diagram showing a state of management event generation when management event generation processing is performed on actual audio data. In the creation of this figure, a one-stage IIR (Infinite Impulse Response) filter is used as the low filter in step S4. Further, 4 is used as the constant h in step S7, and 0.55 seconds is used as the period τ in step S10.

  In FIG. 6, B and C are the time points when the medium-term index exceeds the long-term index immediately after the time point A when the first management event is generated, but the management event is generated only in A. This is because B and C have not passed a predetermined period from A, that is, 0.55 seconds, so that the result of determination in step S10 is Yes and the process in step S11 is not performed.

[1.2.1.4] Generation of MIDI Event While the management event generation process by the DSP 63 described above is performed, the user starts a performance using the piano 31. That is, the user performs keystrokes and pedal operations on the piano 31 according to the music while listening to the music on the music CD-A output from the sound generation unit 4.

  Information on performance using the user's piano 31 is detected as key and pedal movements by the key sensor 32 and the pedal sensor 33, converted into MIDI events by the MIDI event control circuit 34, and then transmitted to the CPU 62 of the controller unit 6. Sent.

[1.2.1.5] Recording Event to SMF As described above, the CPU 62 receives management events from the DSP 63 during playback of the music CD-A, and from the MIDI event control circuit 34 of the automatic performance piano unit 3. Receive a MIDI event. FIG. 7 is a schematic diagram showing a temporal relationship between generation of management events and generation of MIDI events. The CPU 62 manages the timing after starting the time measurement, that is, when the time of 0.13 seconds, 0.81 seconds, 1.45 seconds,... Has elapsed since the first sample value was received from the music CD drive 1. Receive events. Further, the CPU 62 receives the MIDI event when the time of 0.49 seconds, 1.23 seconds, 2.18 seconds,.

  When the CPU 62 receives the management event, the CPU 62 generates a system exclusive event indicating the management event, and records the generated system exclusive event in the RAM 64 with a delta time indicating the elapsed time when the management event is received. To do. Similarly, when the CPU 62 receives a MIDI event, the CPU 62 records the received MIDI event in the RAM 64 with a delta time indicating the elapsed time at the time of receiving the MIDI event.

  When the reproduction of the music CD-A is finished and the performance by the user using the piano 31 is finished, the user presses the keypad of the operation display unit 5 corresponding to the end of the recording of the performance data. The operation display unit 5 transmits a signal corresponding to the pressed keypad to the controller unit 6. When the CPU 62 receives a signal indicating the end of recording of performance data from the operation display unit 5, it transmits a music CD playback stop command to the music CD drive 1. In response to this playback stop command, the music CD drive 1 stops playback of the music CD-A.

  Subsequently, the CPU 62 reads the system exclusive event and the MIDI event indicating the management event recorded in the RAM 64 together with the corresponding delta time, and combines these data to generate an SMF track chunk. Further, the CPU 62 adds a header chunk corresponding to the created track chunk to generate an SMF. FIG. 8 is a diagram showing an outline of the SMF generated by the CPU 62. When completing the generation of the SMF, the CPU 62 transmits the generated SMF to the FD drive 2 together with a write command. When the FD drive 2 receives the write command and the SMF from the CPU 62, the FD drive 2 writes the SMF into the set FD.

[1.2.2] Reproduction Operation Next, an operation for reproducing the SMF recorded by the above-described method and synchronizing the audio data of the music CD and the MIDI data of the SMF will be described. The music CD used in the following playback operation includes the same music as the music CD-A used in the recording operation described above, but the version is different and the performance speed of the music included in the music CD is slightly lower. Is different. Further, since the audio data included in the music CD is edited such as the acoustic effect on the master data, there is a slight difference in the content from the audio data included in the music CD-A. The level of the voice waveform shown is also different. Therefore, in order to distinguish the music CD used in the reproduction operation described below from the music CD-A, it is called a music CD-B.

[1.2.2.1] Playback start operation The user sets the music CD-B in the music CD drive 1 and the FD on which the SMF is recorded in the FD drive 2. Subsequently, the user presses the keypad of the operation display unit 5 corresponding to the start of reproduction of the performance data. The operation display unit 5 outputs a signal corresponding to the pressed keypad to the controller unit 6.

  When the CPU 62 receives a signal for instructing the start of reproduction of performance data from the operation display unit 5, it first transmits an SMF transmission command to the FD drive 2. The FD drive 2 reads the SMF from the FD in response to the SMF transmission command, and transmits the read SMF to the controller unit 6. The CPU 62 receives the SMF from the FD drive 2 and records the received SMF in the RAM 64.

  Subsequently, the CPU 62 transmits a music CD playback command to the music CD drive 1. In response to this playback command, the music CD drive 1 sequentially transmits audio data recorded on the music CD-B to the controller unit 6. The controller unit 6 receives one set of left and right data from the music CD drive 1 every 1/44100 seconds. Here, the value of data received by the CPU 62 from the music CD drive 1 is represented as (r (n), l (n)). The range of values of r (n) and l (n), and the definition of “sample value” used below are the same as those in R (n) and L (n).

[1.2.2.2] Transmission of Audio Data to Sound Generation Unit The CPU 62 receives sample values from the music CD drive 1, that is, (r (0), l (0)), (r (1), l (1). ), (R (2), l (2)),... Are transmitted to the sound generator 4. When the sound generation unit 4 receives the sample value from the controller unit 6, it converts it into a sound and outputs it. As a result, the user can listen to the music recorded on the music CD-B.

[1.2.2.3] Generation of Management Event The CPU 62 transmits the received sample values to the sound generator 4 and simultaneously records the received sample values in the RAM 64. When the CPU 62 records a predetermined number of sample values in the RAM 64, the CPU 62 transmits an execution instruction for management event generation processing to the DSP 63. When the DSP 63 receives this execution command, the DSP 63 performs the management event generation process described with reference to FIG. 4 on a series of sample values of the music CD-B sequentially recorded in the RAM 64. As a result, the DSP 63 sequentially transmits management events to the CPU 62 at the timing when the determination result of step S10 in FIG.

[1.2.2.4] Playback of MIDI Event On the other hand, when the CPU 62 receives the first sample value from the music CD drive 1, that is, (r (0), l (0)), the time is set to 0 seconds. To start. The elapsed time by this time measurement is called “time T”. At the same time, the CPU 62 reads the SMF from the RAM 64 and sequentially compares the time T at that time with the delta time included in the SMF. When the time T and the delta time match, if the event corresponding to the delta time is a MIDI event, the CPU 62 transmits the MIDI event to the automatic performance piano unit 3.

  In the automatic performance piano unit 3, when the MIDI event control circuit 34 receives a MIDI event from the CPU 62, it transmits the received MIDI event to the sound source 35 or the drive unit 36. When a MIDI event is transmitted to the sound source 35, the sound source 35 sequentially transmits sound data indicating the sound of the musical instrument to the sound generation unit 4 in accordance with the received MIDI event. The sound generation unit 4 outputs the performance of the musical instrument sound received from the sound source 35 from the speaker 44 together with the sound of the music CD-B that has already been reproduced. On the other hand, when a MIDI event is transmitted to the drive unit 36, the drive unit 36 moves the keys and pedals of the piano 31 according to the received MIDI event. In any case, the user can simultaneously listen to the music recorded on the music CD-B and the performance based on the musical instrument sound based on the performance information recorded on the SMF.

  However, as described above, the music recorded on the music CD-B is recorded at a slightly different speed from the music recorded on the music CD-A. The performance and music are going off. FIG. 9 shows the temporal relationship between the MIDI event and the management event recorded in the SMF, the management event generated when the music CD-B is played back, and the MIDI event after performing the timing adjustment process. FIG. In the example of this figure, the clock frequency used when the audio data is recorded on the music CD-B is slightly higher than the clock frequency used when the audio data is recorded on the music CD-A. As a result, when the music CD drive 1 plays the music, the music recorded on the music CD-B is played faster than the music recorded on the music CD-A. Therefore, if a MIDI event is transmitted to the automatic performance piano unit 3 according to the delta time recorded in the SMF, the automatic performance by the automatic performance piano unit 3 is delayed with respect to the music CD-B. In order to adjust the time lag between the performance by such a MIDI event and the reproduction of the music CD, the CPU 62 performs a timing adjustment process described below for the MIDI event recorded in the SMF.

  The timing adjustment process uses the timing of generating a management event generated based on the audio data of the music CD-B and the delta time of the management event recorded in the SMF to calculate the delta time of the MIDI event. It is a process to adjust. Hereinafter, the management event recorded in the SMF is referred to as “management event A”, and the management event generated based on the audio data of the music CD-B is referred to as “management event B”.

  When the CPU 62 receives the management event B from the DSP 63, it records the reception timing in the RAM 64. On the other hand, if the time T and the delta time match and the event corresponding to the delta time is a management event, the CPU 62 receives the management event B during a certain period before and after the time indicated by the delta time. Monitor whether or not. In the following description, as an example, it is assumed that the CPU 62 monitors the reception of the management event B for 0.2 seconds before and 0.2 seconds after the delta time of the management event A. That is, the CPU 62 confirms whether or not the management event B has been recorded in the RAM 64 within 0.2 seconds before the time indicated by the delta time of the management event A. Subsequently, the CPU 62 monitors whether or not the management event B is received until 0.2 seconds elapse from the time indicated by the delta time of the management event A. When the management event B is received before and after the time indicated by the delta time of the management event A, the CPU 62 determines that the management event A and the management event B are the audio data of the music CD-A and the music CD-B. It is determined that the event is a management event generated from the corresponding part. Then, the CPU 62 multiplies (the reception time of the management event B / the delta time of the management event A) by the delta time of the MIDI event and the management event that have not yet been executed by the SMF. This is a process for adjusting the delta time in the SMF with reference to the reception time of the management event B. The CPU 62 repeats this delta time adjustment process until the end of the music.

  FIG. 10 is an example of data showing the timing adjustment process. According to this data example, first, the delta time of the first management event A coincides with the time T at 0.13 seconds. Therefore, the CPU 62 confirms whether the management event B is received for 0.2 seconds before that time. In this case, since the first management event B has been received at the time of 0.11 seconds, the CPU 62 determines that these management events correspond to each other.

Subsequently, the CPU 62 uses the delta time of the management event A 0.13 seconds and the reception timing of the management event B 0.11 seconds, and the MIDI event and the management event A delta that have not been processed yet. Multiply the time by (0.11 / 0.13). As a result, the delta time after the first adjustment is the first MIDI event: 0.41 seconds, the second management event: 0.69 seconds, the second MIDI event: 1.04 seconds, and so on.
Thereafter, since the delta time of the first MIDI event coincides with the time T at 0.41 seconds, the CPU 62 transmits the first MIDI event to the automatic performance piano unit 3.

After that, since the delta time of the second management event A coincides with the time T at 0.69 seconds, the CPU 62 determines whether the management event B has been received in 0.2 seconds before that time. Check. In this case, since the second management event B is received at 0.75 seconds, the CPU 62 determines that these management events correspond to each other. As a result, as in the case of the first management event A, the CPU 62 multiplies (0.75 / 0.69) by the delta time of the MIDI event that has not yet been processed and the management event A. As a result, the delta time after the second adjustment is the second MIDI event: 1.14 seconds, the third management event: 1.34 seconds, the third MIDI event: 2.02 seconds, and so on. .
Thereafter, since the delta time of the second MIDI event coincides with the time T at 1.14 seconds, the CPU 62 transmits the second MIDI event to the automatic performance piano unit 3.

  Hereinafter, the same processing is repeated until the end of the music, but in the case of the data example of FIG. 10, the fourth management event A is between 0.2 seconds before and 0.2 seconds after the timing indicated by the delta time. Is ignored because there is no management event B. Similarly, the eighth management event B is ignored because there is no management event A between 0.2 seconds before and 0.2 seconds after the generation timing.

  As a result of transmitting the MIDI event to the automatic performance piano unit 3 in accordance with the delta time subjected to the timing adjustment processing as described above, as shown in the lower part of FIG. The MIDI event is played back at the correct timing.

  Contrary to the above-described example, when the music recorded on the music CD-B is played later than the music recorded on the music CD-A, the timing of reproduction of the MIDI event is similar to the above processing. Adjusted. Therefore, the description is omitted.

[2] Second Embodiment In the second embodiment of the present invention, the time recorded on the music CD is used for the synchronization adjustment of the reproduction of the audio data recorded on the music CD and the MIDI event recorded on the SMF. Code is used. Further, in performing the timing adjustment process, first, after completing the management event generation process for the audio data of the music CD to the end of the music piece, the timing adjustment process is performed based on values obtained by using regression analysis.

[2.1] Music CD Drive The overall configuration, functions of each component, and data format in MIDI data in the second embodiment are the same as those in the first embodiment except for the function of the music CD drive 1. Therefore, only the function of the music CD drive 1 will be described, and other description will be omitted.
In the second embodiment, the music CD drive 1 transmits the time code to the controller unit 6 together with the audio data recorded on the music CD. The other points are the same as those of the music CD drive 1 in the first embodiment.

[2.2] Operation Regarding the operation of the synchronous recording / reproducing apparatus SS in the second embodiment, the following three points are different from those in the first embodiment.
(1) In the performance data recording operation, the time code corresponding to the reception time of these events is recorded as the delta time of the management event and the MIDI event recorded in the SMF.
(2) In the performance data playback operation, the audio data of the music CD is first played to the end, and management event generation processing is performed on the audio data. A regression line is obtained for the management event obtained as a result and the time information of the management event recorded in the SMF, and the delta time corresponding to the MIDI event is adjusted based on the regression line. After this timing adjustment process is completed, the music CD is played again, and the MIDI event is played back synchronously with the playback.
(3) In the performance data reproduction operation, the MIDI event is transmitted to the automatic performance piano unit 3 based on the time clock transmitted from the music CD drive 1.

  Since other operations in the second embodiment are the same as those in the first embodiment, a detailed description thereof will be omitted. In the following description, it is assumed that the music CD-A is used in the recording operation and the music CD-B is used in the reproduction operation, as in the first embodiment. The time code expression format uses hours, minutes, seconds, and frames. Similar to the delta time recorded in the SMF, in the following description, time information indicated by the time code is used for simplification. Is expressed in seconds.

[2.2.1] Recording Operation In the synchronous recording / reproducing apparatus SS of the second embodiment, when the user gives an instruction to start recording performance data using the operation display unit 5, the audio data of the music CD-A is It is sequentially transmitted from the music CD drive 1 to the controller unit 6 together with the time code.

  In the controller unit 6, the CPU 62 sequentially transmits the sample values of the received audio data to the sound generation unit 4. As a result, the sound generator 4 outputs the music of the music CD-A as a sound. On the other hand, the CPU 62 sequentially records the received sample values in the RAM 64 together with the time code.

  When the CPU 62 records a predetermined number of sample values in the RAM 64, the CPU 62 transmits an execution instruction for management event generation processing to the DSP 63. Upon receiving this execution command, the DSP 63 performs management event generation processing on a series of sample values of the music CD-A recorded sequentially in the RAM 64. The management event generation process in the second embodiment is the same as the management event generation process in the first embodiment described with reference to FIG. As a result, the DSP 63 sequentially transmits management events to the CPU 62 at the timing when the determination result of step S10 in FIG. When the CPU 62 receives the management event, the CPU 62 generates a system exclusive event indicating the management event. Subsequently, the CPU 62 converts the time indicated by the time code received from the music CD drive 1 into a delta time format immediately before receiving the management event, and then converts the delta time into a system exclusive event indicating the management event. Correspondingly, it is recorded in the RAM 64.

  On the other hand, the user performs a performance using the piano 31 while listening to the music of the music CD-A output as a sound from the sound generator 4. The user performance information is transmitted as a MIDI event from the MIDI event control circuit 34 to the CPU 62. When receiving the MIDI event from the MIDI event control circuit 34, the CPU 62 converts the time indicated by the time code received from the music CD drive 1 immediately before to the delta time format, and associates the delta time with the MIDI event, Recorded in the RAM 64.

  Thereafter, when the user gives an instruction to end recording of performance data, the music CD drive 1 stops the reproduction of the music CD-A. Further, the CPU 62 reads the system exclusive event and the MIDI event indicating the management event recorded in the RAM 64 together with the corresponding delta time, and combines these data to generate an SMF similar to that shown in FIG. However, the SMF delta time in the second embodiment is based on the time code of the music CD-A and is not based on the time count of the CPU 62. The SMF generated in this way is transmitted to the FD drive 2 and written to the FD set by the FD drive 2.

[2.2.2] Reproduction Operation Next, an operation when reproducing the SMF recorded by the above-described method and synchronizing the audio data of the music CD-B and the MIDI data of the SMF will be described.
When the user gives an instruction to start playing performance data using the operation display unit 5, first, the SMF recorded in the FD is transmitted from the FD drive 2 to the CPU 62, and the CPU 62 records the received SMF in the RAM 64. Subsequently, the music CD drive 1 starts playback of the music CD-B, and the audio data and time code recorded on the music CD-B are sequentially transmitted to the controller unit 6. The CPU 62 records the audio data together with the time code in the RAM 64. However, the CPU 62 does not transmit the received audio data to the sound generation unit 4, so that the user cannot hear the music CD-B.

  When the CPU 62 records a predetermined number of sample values in the RAM 64, the CPU 62 transmits an execution instruction for management event generation processing to the DSP 63. Upon receiving this execution command, the DSP 63 performs management event generation processing on a series of sample values of the music CD-B recorded sequentially in the RAM 64. As a result, the DSP 63 sequentially transmits management events to the CPU 62 at the timing when the determination result of step S10 in FIG. When the CPU 62 receives the management event, the CPU 62 converts the time indicated by the time code received immediately before into the delta time format, and then records the delta time in the RAM 64. Hereinafter, the management event recorded in the SMF is “management event A”, the delta time recorded in the SMF is “delta time A”, and the management event generated by the audio data of the music CD-B is “ The delta time recorded for the reception of the management event B according to the management event B ”and the time code of the music CD-B is referred to as“ delta time B ”.

  When the reproduction of the music CD-B reaches the end of the music, the music CD drive 1 ends the reproduction of the music CD-B. Subsequently, the CPU 62 reads the delta time A and the delta time B from the RAM 64, and performs a delta time adjustment process similar to the timing adjustment process in the first embodiment. FIG. 11 is an example of data showing how the delta time is adjusted in the second embodiment. In FIG. 11, the reception timing of the delta time and the management event B is different from the first embodiment in that it is based on the time code of the music CD-B. Also, delta time adjustment for MIDI events is not performed at this stage. Since other points are the same as the timing adjustment process in the first embodiment, the description thereof is omitted. The delta time A and the delta time B are associated by this delta time adjustment process. Hereinafter, the value of the delta time A of the nth set is represented as A (n), and the value of the delta time B is represented as B (n). Moreover, those sets are represented as (A (n), B (n)). Therefore, (A (1), B (1)) = (0.13, 0.11), (A (2), B (2)) = (0.81, 0.75), (A (3 ), B (3)) = (1.45, 1.36).

  Subsequently, the CPU 62 obtains a regression line by the least square method for these (A (n), B (n)). A regression curve or other approximate curve may be obtained instead of the regression line. Actually, a regression line is obtained with respect to (A (n), B (n)) obtained for the audio data of the entire music piece, but for the sake of explanation, regression is performed using nine sets of data obtained from FIG. A straight line is to be obtained.

  FIG. 12 is a graph showing nine sets of (A (n), B (n)) and the regression line corresponding thereto. The equation indicating the regression line is B = 0.9414A−0.006. This formula is based on the time code of the music CD-B, and the start time of the SMF performance data is delayed by 0.006 seconds with respect to the start time of the audio data of the music CD-B, and the performance speed is 5.86% slower. Therefore, the CPU 62 adjusts the timing for each delta time by substituting the value of the delta time of the MIDI event recorded in the SMF for the calculated equation A of the regression line. FIG. 13 shows the delta time before the timing adjustment and the delta time after the adjustment. This timing adjustment of the CPU 62 means that the delta time of each event of the SMF is reduced by 5.86% as a whole, and further advanced by 0.006 seconds. The CPU 62 updates the delta time for the SMF MIDI event with the delta time after the timing adjustment.

  When the timing adjustment processing is completed, the CPU 62 transmits a music CD-B playback command to the music CD drive 1 again. In response to this playback command, the music CD drive 1 starts playback of the music CD-B, and the audio data and time code recorded on the music CD-B are sequentially transmitted to the controller unit 6. The CPU 62 sequentially transmits the received audio data to the sound generator 4, and the music of the music CD-B is output from the sound generator 4 as sound.

  The CPU 62 sequentially transmits the received audio data to the sound generator 4 and simultaneously compares the time information indicated by the delta time of the MIDI event with the time information indicated by the time code of the music CD-B transmitted from the music CD drive 1. If they match, a MIDI event corresponding to the matched delta time is transmitted to the automatic performance piano unit 3. When the automatic performance piano unit 3 receives the MIDI event, the musical performance sound data electronically synthesized from the sound source 35 is transmitted to the sound generation unit 4, or the drive unit 36 moves the keys and pedals of the piano 31. Perform an automatic performance according to the event. As a result, the user can simultaneously listen to the music recorded on the music CD-B and the performance based on the performance information recorded on the SMF.

[3] Modified Examples The first embodiment and the second embodiment described above are examples of the embodiments of the present invention, and various modifications can be made to the above embodiments without departing from the spirit of the present invention. Can be added. Hereinafter, a modification is shown.

[3.1] First Modification In the first modification, the components of the synchronous recording / reproducing apparatus SS are not arranged in the same apparatus, but are arranged separately for each group.
For example, it is possible to separately arrange in the following groups.
(1) Music CD drive 1
(2) FD drive 2
(3) Automatic performance piano part 3
(4) Mixer 41 and D / A converter 42
(5) Amplifier 43
(6) Speaker 44
(7) Operation display unit 5 and controller unit 6
Furthermore, the controller unit 6 may be configured separately for a device that performs only a recording operation and a device that performs only a reproduction operation.

The groups of these components are connected by an audio cable, a MIDI cable, an audio optical cable, a USB (Universal Serial Bus) cable, a dedicated control cable, and the like. Commercially available FD drive 2, amplifier 43, speaker 44, etc. may be used.
According to the first modified example, the flexibility of the arrangement of the synchronous recording / reproducing apparatus SS is increased, and at the same time, the user does not prepare all of the synchronous recording / reproducing apparatus SS and prepares only necessary components. Can reduce the necessary cost.

[3.2] Second Modification In the second modification, there is no music CD drive 1 and FD drive 2 in the synchronous recording / reproducing apparatus SS. On the other hand, the communication interface has a function connectable to a LAN (Local Area Network) and is connected to an external communication device via the LAN and WAN. Further, the controller unit 6 has an HD (Hard Disk).

  The controller unit 6 receives digital audio data including audio data and a time code from another communication device via the LAN, and records the received audio data on the HD. Similarly, the controller unit 6 receives the SMF created corresponding to the audio data from another communication device via the LAN, and records the received SMF on the HD.

The controller unit 6 reads the digital audio data from the HD instead of receiving the audio data and time code of the music CD from the music CD drive 1. The controller unit 6 performs the same operation on the HD instead of writing and reading the SMF to and from the FD drive 2.
According to the second modification, the user can transmit / receive digital audio data and SMF to / from a geographically distant communication device via the LAN. The LAN may be connected to a wide area communication network such as the Internet.

  Note that the recording level of music may differ between different versions of the same music, but in the present invention, the management event used for adjusting the playback timing of performance data is generated at the same timing regardless of the music recording level. Is done. This is because the two indices used to generate the management event change in proportion to the recording level, so the magnitude relationship does not change. Therefore, according to the technique of the present invention, synchronized reproduction of performance data can be realized at the correct timing even for audio data of different recording levels.

It is a figure which shows the structure of the synchronous recording / reproducing apparatus SS which concerns on 1st Embodiment and 2nd Embodiment of this invention. It is a figure which shows the structure of a MIDI event. It is a figure which shows the structure of SMF. It is a flowchart of the management event production | generation process which concerns on 1st Embodiment and 2nd Embodiment of this invention. It is a figure which shows the structure of the comb filter which concerns on 1st Embodiment and 2nd Embodiment of this invention. It is a figure which shows the temporal relationship of the medium-term parameter | index, long-term parameter | index, and management event which concern on 1st Embodiment and 2nd Embodiment of this invention. It is a schematic diagram which shows the temporal relationship of the production | generation of the management event which concerns on 1st Embodiment of this invention, and 2nd Embodiment, and the production | generation of a MIDI event. It is a figure which shows the outline of SMF which concerns on 1st Embodiment and 2nd Embodiment of this invention. It is a figure which shows the temporal relationship of the MIDI event before timing adjustment which concerns on 1st Embodiment of this invention, the MIDI event after timing adjustment, and the management event. It is an example of data which shows the mode of the timing adjustment process which concerns on 1st Embodiment of this invention. It is an example of data which shows the mode of the timing adjustment process which concerns on 2nd Embodiment of this invention. It is a figure which shows the regression line with respect to the delta time of the management event which concerns on 2nd Embodiment of this invention. It is a data example which shows the mode of the adjustment process of the delta time which concerns on 2nd Embodiment of this invention. It is a figure which shows the temporal relationship of the MIDI event before timing adjustment which concerns on 2nd Embodiment of this invention, the MIDI event after timing adjustment, and the management event.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Music CD drive, 2 ... FD drive, 3 ... Automatic performance piano part, 4 ... Sound generation part, 5 ... Operation display part, 6 ... Controller part, 31 ... Piano, 32 ... key sensor, 33 ... pedal sensor, 34 ... MIDI event control circuit, 35 ... sound source, 36 ... drive unit, 41 ... mixer, 42 ... D / A Converter, 43 ... Amplifier, 44 ... Speaker, 61 ... ROM, 62 ... CPU, 63 ... DSP, 64 ... RAM, 65 ... Communication interface.

Claims (9)

Performance data in which the control data instructing the sound generation control of the musical sound constituting the music and the first delta time data indicating the control timing based on the control data are paired, the management event data, and the generation timing of the management event data are shown. First receiving means for receiving first music data recorded as a pair of 2 delta time data is recorded along the progress of the music;
Output means for outputting the control data in accordance with the control timing indicated by the first delta time data paired with the control data received by the first receiving means;
Second receiving means for receiving audio data of music pieces to be sequentially played back as second music data;
First generation means for generating a first index indicating a medium-term fluctuation element of a voice waveform indicated by the second music data;
Second generation means for generating a second index indicating a long-term fluctuation factor that is longer than the middle period of the speech waveform;
A sudden change in the voice waveform is detected by comparing the first index and the second index, and a rapid change in the voice waveform is detected by the time measurement started with the reproduction of the second music data. Third generation means for generating time data indicating timing;
It is monitored whether or not the timing indicated by the time data generated by the third generation means exists within a certain period before and after the second delta time data received by the first reception means. And a adjusting unit that adjusts an output timing of the control data by the output unit according to a comparison result between the data and the second delta time data. Performance data in which the control data instructing the sound generation control of the musical sound constituting the music and the first delta time data indicating the control timing based on the control data are paired, the management event data, and the generation timing of the management event data are shown. First receiving means for receiving first music data recorded as a pair of 2 delta time data is recorded along the progress of the music;
Output means for outputting the control data in accordance with the control timing indicated by the first delta time data paired with the control data received by the first receiving means;
Second receiving means for receiving audio data of music pieces to be sequentially played back as second music data;
First generation means for generating a first index indicating a medium-term fluctuation element of a voice waveform indicated by the second music data;
Second generation means for generating a second index indicating a long-term fluctuation factor that is longer than the middle period of the speech waveform;
A sudden change in the voice waveform is detected by comparing the first index and the second index, and a rapid change in the voice waveform is detected by the time measurement started with the reproduction of the second music data. Third generation means for generating time data indicating timing;
The output timing of the control data by the output means based on the analysis result obtained by performing regression analysis on the time data generated by the third generation means and the second delta time data received by the first reception means A playback device comprising: an adjusting means for adjusting the sound. The first generation means generates, as the first index, data indicating a waveform including only a frequency component lower than one frequency among frequency components included in the waveform indicated by the audio data,
The second generation means generates, as the second index, data indicating a waveform including only a frequency component lower than another frequency that is lower than the one frequency among frequency components included in the waveform indicated by the audio data. The reproducing apparatus according to claim 1 or 2. Performance data in which the control data instructing the sound generation control of the musical sound constituting the music and the first delta time data indicating the control timing based on the control data are paired, the management event data, and the generation timing of the management event data are shown. A first reception process in which a pair of two delta time data receives first music data recorded along with the progress of the music;
An output process for outputting the control data according to the control timing indicated by the first delta time data paired with the control data received by the first reception process;
A second receiving process of receiving the audio data of the music that is sequentially played back as the second music data;
A first generation step of generating a first index indicating a medium-term fluctuation element of a voice waveform indicated by the second music data;
A second generation step of generating a second index indicating a long-term fluctuation factor that is longer than the middle period of the speech waveform;
A sudden change in the voice waveform is detected by comparing the first index and the second index, and a rapid change in the voice waveform is detected by the time measurement started with the reproduction of the second music data. A third generation process for generating time data indicating timing;
Whether or not there is a timing indicated by the time data generated by the third generation process within a certain period before and after the second delta time data received by the first reception process, and if there is, the time And adjusting the output timing of the control data according to the output process according to a comparison result between the data and the second delta time data. Performance data in which the control data instructing the sound generation control of the musical sound constituting the music and the first delta time data indicating the control timing based on the control data are paired, the management event data, and the generation timing of the management event data are shown. A first reception process in which a pair of two delta time data receives first music data recorded along with the progress of the music;
An output process for outputting the control data according to the control timing indicated by the first delta time data paired with the control data received by the first reception process;
A second receiving process of receiving the audio data of the music that is sequentially played back as the second music data;
A first generation step of generating a first index indicating a medium-term fluctuation element of a voice waveform indicated by the second music data;
A second generation step of generating a second index indicating a long-term fluctuation factor that is longer than the middle period of the speech waveform;
A sudden change in the voice waveform is detected by comparing the first index and the second index, and a rapid change in the voice waveform is detected by the time measurement started with the reproduction of the second music data. A third generation process for generating time data indicating timing;
Output timing of control data by the output process based on an analysis result obtained by performing regression analysis on the time data generated by the third generation process and the second delta time data received by the first reception process A reproduction method comprising: an adjustment process for adjusting the frequency. In the first generation process, data indicating a waveform including only a frequency component lower than one frequency among frequency components included in the waveform indicated by the audio data is generated as the first index,
In the second generation process, data indicating a waveform including only a frequency component lower than another frequency that is lower than the one frequency among frequency components included in the waveform indicated by the audio data is generated as the second index. The reproduction method according to claim 4 or 5. Performance data in which the control data for instructing the sound generation control of the musical sound constituting the music to the computer and the first delta time data indicating the control timing by the control data are paired, the management event data, and the generation timing of the management event data. A first receiving process of receiving first music data recorded along with the progress of the music, with a pair of second delta time data shown;
An output process for outputting the control data according to the control timing indicated by the first delta time data paired with the control data received by the first reception process;
A second receiving process for receiving the audio data of the music that is sequentially played back as the second music data;
A first generation process for generating a first index indicating a medium-term fluctuation element of a voice waveform indicated by the second music data;
A second generation process for generating a second index indicating a long-term variation factor that is longer than the middle period of the speech waveform;
A sudden change in the voice waveform is detected by comparing the first index and the second index, and a rapid change in the voice waveform is detected by the time measurement started with the reproduction of the second music data. A third generation process for generating time data indicating timing;
It is monitored whether there is a timing indicated by the time data generated by the third generation process within a certain period before and after the second delta time data received by the first reception process. A program for executing adjustment processing for adjusting output timing of control data by the output processing according to a comparison result between the data and the second delta time data. Performance data in which the control data for instructing the sound generation control of the musical sound constituting the music to the computer and the first delta time data indicating the control timing by the control data are paired, the management event data, and the generation timing of the management event data. A first receiving process of receiving first music data recorded along with the progress of the music, with a pair of second delta time data shown;
An output process for outputting the control data according to the control timing indicated by the first delta time data paired with the control data received by the first reception process;
A second receiving process for receiving the audio data of the music that is sequentially played back as the second music data;
A first generation process for generating a first index indicating a medium-term fluctuation element of a voice waveform indicated by the second music data;
A second generation process for generating a second index indicating a long-term variation factor that is longer than the middle period of the speech waveform;
A sudden change in the voice waveform is detected by comparing the first index and the second index, and a rapid change in the voice waveform is detected by the time measurement started with the reproduction of the second music data. A third generation process for generating time data indicating timing;
Based on an analysis result obtained by performing regression analysis on the time data generated by the third generation process and the second delta time data received by the first reception process, the output timing of the control data by the output process A program that executes the adjustment process to adjust the. In the first generation process, data indicating a waveform including only a frequency component lower than one frequency among frequency components included in the waveform indicated by the audio data is generated as the first index,
In the second generation process, data indicating a waveform including only a frequency component lower than another frequency that is lower than the one frequency among frequency components included in the waveform indicated by the audio data is generated as the second index. The program according to claim 7 or 8.

Images