JP5287617B2 - Sound processing apparatus and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To synchronize the display of lyrics and musical score with the reproduction of acoustic signals, with high accuracy. <P>SOLUTION: A storage device 14 stores musical sound data DA for designating a time series of musical sounds composing a piece of music, and display data DB for designating a time series of the lyrics of the piece of music. A synchronization analyzing section 48 executes the chroma vector &Psi;1 of the reference section &sigma;REF of an acoustic signal P1 and the chroma vector &Psi;2 of the candidate section &sigma;X of an acoustic sound P2 with respect to each of the plurality of candidate sections &sigma;X of different start points SX. Among the candidate sections &sigma;X, the synchronization analyzing section specifies a candidate section &sigma;X in which the chroma vector &Psi;2 is similar to the chroma vector &Psi;1 of the reference section &sigma;X as a synchronization section &sigma;S, corresponding to the reference section &sigma;REF. On the basis of the result of the process by the synchronization analyzing section 48, a display control section 60 makes a display device 72 to display the characters designated by the display data DB so that they are synchronized with the reproduction of the acoustic signals P1. <P>COPYRIGHT: (C)2011,JPO&amp;INPIT

Description

  The present invention relates to a technique for specifying correspondence between a plurality of acoustic signals representing a musical tone waveform of music.

  Display of lyrics and musical scores (for example, chord scores and tablatures) specified in time series by MIDI (Musical Instrument Digital Interface) data of the music, and an acoustic signal of the music acquired from, for example, a CD (hereinafter referred to as “reproduced acoustic signal”) If playback is performed in parallel, it is possible to practice singing and playing while confirming the lyrics and score under the high-quality accompaniment sound of the playback sound signal. However, a complicated operation of creating MIDI data and a reproduced sound signal so as to be synchronized with each other on the time axis is required.

  Thus, for example, Patent Document 1 discloses a technique for converting MIDI data so that the display of the lyrics of a song is synchronized with the reproduction of a reproduction acoustic signal. Specifically, a configuration in which the tempo of the MIDI data is adjusted according to the length of the total playback time of the playback sound signal and the playback time of the MIDI data, the starting point of the singing sound in the playback sound signal, and the MIDI data A configuration has been proposed in which the tempo of MIDI data is adjusted so that the start point of the lyrics matches.

JP 2004-212473 A

  However, it is actually difficult to synchronize the reproduction of the reproduced sound signal and the display of the lyrics and the score with the technique of Patent Document 1 with high accuracy. For example, since the time length from the start point of the playback sound signal to the actual start of the music does not necessarily match the time length from the start point of the MIDI data to the actual start of the music, the playback sound as in Patent Document 1 Even if the start and end points of the signal playback section and the MIDI data playback section are matched, there is a possibility that the playback of the accompaniment sound and the display of the lyrics and the score are not synchronized. In addition, when the playback point of the playback sound signal is changed, for example, by the user, there is a problem that it is difficult to synchronize the playback of the accompaniment sound and the display of the lyrics and the score from the playback point after the change. In the above description, the display of lyrics is exemplified, but the same problem may occur when the processing of control data instructing various operations other than the display of lyrics is synchronized with the playback of the playback sound signal. In view of the above circumstances, an object of the present invention is to synchronize control data processing and sound signal reproduction with high accuracy.

  In order to solve the above problems, the acoustic processing device of the present invention includes a storage unit that stores control data for designating information related to music in time series, and a feature amount of a first acoustic signal that represents a waveform of a musical tone of the music. First feature extraction means for sequentially generating, second feature extraction means for sequentially generating feature quantities of a second sound signal representing a waveform of a musical sound synchronized with a time series of information designated by control data, and first sound The comparison between the feature amount of the reference section of the signal and the feature amount of the candidate section of the second acoustic signal is performed for each of the plurality of candidate sections having different positions on the time axis (for example, the positions of the start point and the end point). A synchronization point detecting means for identifying a candidate section whose feature quantity is similar to the feature quantity of the reference section among a plurality of candidate sections as a synchronization section corresponding to the reference section; and reproduction of the reference section of the first acoustic signal; Sync point detection for the reference section As the processing of the control data corresponding to the synchronous interval stages identified are synchronized, and a control means for processing the control data.

  In the above embodiment, the feature quantity of the reference section of the first acoustic signal and the feature quantity of the candidate section of the second acoustic signal are compared for each of a plurality of candidate sections having different positions on the time axis, and a plurality of candidates are compared. A candidate section whose feature amount is similar to the feature amount of the reference section among the sections is specified as a synchronization section corresponding to the reference section. Therefore, even when the time point when the music actually starts (the time length from the start point) differs between the first acoustic signal and the second acoustic signal, the processing of the control data and the reproduction of the first acoustic signal are synchronized with high accuracy. It is possible to make it.

  In a preferred aspect of the present invention, the synchronization point detecting means compares the feature amount of each of the plurality of candidate sections having the same start point position and different time lengths in the second acoustic signal with the feature amount of the reference section. This is executed for a plurality of cases where the position of the starting point is changed, and a candidate section whose feature quantity is similar to the feature quantity of the reference section among the plurality of candidate sections compared with the reference section is specified as a synchronization section. In the above aspect, since the candidate section of the reference section of the first acoustic signal and the candidate section of the second acoustic signal are compared while changing the time length in addition to the position on the time axis for the candidate section, the first acoustic signal and Even when the tempo is different from that of the second sound signal, there is an advantage that the synchronization section corresponding to the reference section in the music can be detected with high accuracy.

  In the first aspect of the present invention, the synchronization point detecting means includes a first reference point whose intensity first exceeds a threshold value in the first acoustic signal, and a second reference whose intensity first exceeds the threshold value in the second acoustic signal. A point is specified, and the feature amount of the reference section starting from the first reference point is compared with the feature amount of each of the plurality of candidate sections whose start points are changed within a predetermined range including the second reference point. In the above aspect, while changing the starting point of the candidate section within a predetermined range including the second reference point, the feature amount of the reference section starting from the first reference point of the first acoustic signal and the second acoustic signal Since the feature amount of the candidate section is compared, for example, even when a sound that does not exist in the first sound signal (for example, a beat sound) is included in the head portion of the second sound signal, the first sound signal There is an advantage that it is possible to accurately detect the time when the first acoustic signal and the second acoustic signal first correspond to each other. In addition, the specific example of the above aspect is later mentioned, for example as 1st Embodiment.

  In the second aspect of the present invention, the synchronization point detecting means includes a feature amount of a reference section including a reproduction instruction point that is variably set in accordance with an instruction from the user in the first acoustic signal, and a second acoustic signal. Are compared with the feature amounts of the plurality of candidate sections whose start points are changed within a predetermined range including the reference point set according to the reproduction instruction point. In the above aspect, the feature amount of the reference section starting from the playback instruction point in the first sound signal while changing the starting point of the candidate section within a predetermined range including the reference point set according to the playback instruction point Are compared with the feature quantities of the candidate sections of the second acoustic signal, it is possible to synchronize the reproduction of the first acoustic signal and the processing of the control data from an arbitrary reproduction instruction point. In addition, the specific example of the above aspect is later mentioned, for example as 2nd Embodiment.

  In the specific example of each aspect described above, the storage means includes data conversion means for storing musical sound data specifying a time series of musical sounds constituting a musical piece and generating a second acoustic signal from the musical sound data. In the above aspect, since the 2nd acoustic signal applied to the process (synchronous analysis) by a synchronous analysis means is produced | generated from musical sound data, there exists an advantage that the music data for karaoke can be diverted, for example. Further, as compared with a configuration in which the second sound signal is prepared separately from the musical sound data, there is an advantage that the capacity required for the storage device is reduced.

  The feature amount in each of the above forms is, for example, a chroma vector. That is, the first feature extraction unit sequentially generates the chroma vector of the first acoustic signal for each unit section, and the second feature extraction unit sequentially generates the chroma vector of the second acoustic signal for each unit section. In the above configuration, since a chroma vector that is an index of harmony in the acoustic signal is generated as a feature quantity, for example, compared with a configuration in which the beat point of the first acoustic signal and the beat point of the second acoustic signal are synchronized. And there exists an advantage that the position corresponding to a 1st acoustic signal and a 2nd acoustic signal can be pinpointed with high precision. The chroma vector is a numerical value (for example, an added value or an average) in which each of a plurality of elements corresponding to different pitch names (chroma) adds the intensity of the component of the pitch name corresponding to the element in the acoustic signal over a plurality of octaves. This means a vector (pitch class profile) set according to (value).

  The sound processing apparatus according to each of the above aspects is realized by hardware (electronic circuit) such as a DSP (Digital Signal Processor) dedicated to sound processing, and general-purpose arithmetic processing such as a CPU (Central Processing Unit). This is also realized by cooperation between the apparatus and the program. A program according to the present invention is a first program for sequentially generating feature quantities of a first acoustic signal representing a waveform of musical tone of music on a computer having storage means for storing control data for designating information related to the music in time series. A feature extraction process; a second feature extraction process for sequentially generating feature quantities of a second sound signal representing a waveform of a musical sound synchronized with a time series of information specified by control data; and a reference section of the first sound signal. Is compared with the feature quantity of the candidate section of the second sound signal for each of a plurality of candidate sections having different positions on the time axis, and the feature quantity of the plurality of candidate sections is a feature of the reference section. Synchronization point detection processing that identifies candidate intervals similar to the amount as a synchronization interval corresponding to the reference interval, reproduction of the reference interval of the first acoustic signal, and synchronization interval specified by the synchronization point detection processing for the reference interval Compatible with As processing of the control data and to synchronize that, to execute a control process for processing the control data. According to the above program, the same operation and effect as the sound processing apparatus according to the present invention are exhibited. The program of the present invention is provided to a user in a form stored in a computer-readable recording medium and installed in the computer, or provided from a server device in a form of distribution via a communication network and installed in the computer. Is done.

1 is a block diagram of a sound processing apparatus according to a first embodiment. It is a schematic diagram of a correspondence table showing correspondence between a plurality of acoustic signals. It is a flowchart of operation | movement of a synchronous point detection part. It is a flowchart of the operation | movement which pinpoints a reference point. It is explanatory drawing of operation | movement of a synchronous point detection part. It is a flowchart of the detailed operation | movement of a synchronous point detection part. It is a schematic diagram for demonstrating operation | movement of the area control part. It is explanatory drawing of operation | movement of a display control part when the display of a lyric precedes reproduction | regeneration of a musical sound. It is explanatory drawing of operation | movement of a display control part when the display of a lyric is delayed to reproduction | regeneration of a musical sound. It is explanatory drawing of operation | movement of the sound processing apparatus which concerns on 2nd Embodiment. It is a flowchart of operation | movement of the sound processing apparatus which concerns on 2nd Embodiment.

<A: First Embodiment>
FIG. 1 is a block diagram of a sound processing apparatus 100 according to the first embodiment of the present invention. The sound processing apparatus 100 is suitably used as a karaoke apparatus that executes the reproduction of musical tone of music and the display of lyrics of the music in parallel. As shown in FIG. 1, a display device 72, a sound collection device 74, a sound emission device 76, an input device 78, and a signal supply device 80 are connected to the sound processing device 100.

  The display device (for example, a liquid crystal display device) 72 displays the lyrics of music under the control of the sound processing device 100. The sound collection device (microphone) 74 generates an acoustic signal QB corresponding to surrounding sounds (singing sound, performance sound of musical instrument, etc.). An A / D converter that converts an analog signal corresponding to the surrounding sound into a digital sound signal QB is not shown. The sound emitting device (for example, a speaker or headphones) 76 reproduces sound according to the sound signal QC output from the sound processing device 100. The input device 78 includes an operator that receives an operation from a user.

  The signal supply device 80 supplies the sound processing device 100 with a digital sound signal P1 representing the waveform of the musical tone (singing sound or accompaniment sound) of the music. For example, the acoustic signal P1 is acquired from a recording device such as a semiconductor recording medium or an optical disk, and the acoustic signal P1 is acquired from a communication device such as the Internet. A communication device that receives and outputs the signal to the sound processing device 100 is preferably employed as the signal supply device 80.

  The sound processing device 100 is a computer system that includes an arithmetic processing device 12 and a storage device 14. The storage device 14 stores a program executed by the arithmetic processing device 12 and data used by the arithmetic processing device 12. A known recording medium such as a semiconductor recording medium or a magnetic recording medium is arbitrarily employed as the storage device 14. A configuration in which the acoustic signal P1 is stored in the storage device 14 is also employed.

  As shown in FIG. 1, the storage device 14 stores a plurality of music data D0 corresponding to different music. For example, music data D 0 received via a communication network such as the Internet is stored in the storage device 14. The music data D0 includes music data DA for designating each musical tone constituting the music in time series, and display data DB for designating the lyrics of the music in time series. For example, MIDI in which event data that specifies the pitch / intensity of musical tone and text of lyrics and timing data that specifies the time point of processing of each event data (for example, the interval of processing of event data that follows each other) are arranged in time series A data string in a format (specifically, XF format) is suitably employed as the music data D0 (DA, DB).

  The musical sound data DA and the display data DB are created so that the time series of musical sounds specified by the musical sound data DA and the time series of lyrics specified by the display data DB correspond to (synchronize) with each other on the time axis. On the other hand, the music of the music data D0 and the music of the sound signal P1 may be common, but since the music data D0 and the sound signal P1 are created separately, the time series of the music specified by the music data DA (display data DB) Is not completely synchronized with the musical tone waveform represented by the sound signal P1.

  The arithmetic processing unit 12 executes a program stored in the storage device 14 to thereby synchronize the reproduction of the acoustic signal P1 (streaming reproduction of music) and the display of lyrics specified by the display data DB. (First holding unit 161, second holding unit 162, reproduction control unit 20, output processing unit 30, analysis processing unit 40, display control unit 60) are realized. Therefore, the user of the sound processing apparatus 100 practices singing while confirming the lyrics under the reproduced sound of the sound signal P1 (generally, a high-quality musical sound than the musical sound generated from the musical sound data DA). It is possible. A configuration in which each function of the arithmetic processing unit 12 is distributed over a plurality of integrated circuits, or a configuration in which a dedicated electronic circuit (DSP) realizes each function may be employed.

  The acoustic signal P1 of the music designated by the user by operating the input device 78 is supplied from the signal supply device 80 to the first holding unit 161. The first holding unit 161 is a buffer circuit that sequentially acquires and holds the acoustic signal P1 supplied from the signal supply device 80. On the other hand, the second holding unit 162 is a buffer circuit that sequentially acquires and holds the music data D0 (musical sound data DA and display data DB) of the music selected by the user by operating the input device 78 from the storage device 14.

  The reproduction control unit 20 generates an acoustic signal QA from the acoustic signal P1 held in the first holding unit 161. Specifically, the playback control unit 20 includes a component suppression unit 22, a pitch conversion unit 24, and a tempo adjustment unit 26. The component suppression unit 22 suppresses (ideally removes) a component of a specific part in the acoustic signal P1. For example, when the user of the acoustic processing device 100 sings along with the reproduced sound (accompaniment sound) of the acoustic signal P1, the component suppression unit 22 suppresses the component of the singing sound part of the acoustic signal P1. A known technique is arbitrarily employed for the processing by the component suppressing unit 22 (minus one processing). The pitch conversion unit 24 changes the pitch of the acoustic signal P1 in accordance with an operation (key designation) from the user with respect to the input device 78. The tempo adjustment unit 26 changes the tempo of the acoustic signal P1 according to an operation (tempo designation) by the user with respect to the input device 78.

  The acoustic signal QA processed by the reproduction control unit 20 (tempo adjustment unit 26) is supplied to the output processing unit 30. The output processing unit 30 generates an acoustic signal QC from the acoustic signal QA and the acoustic signal QB and outputs the acoustic signal QC to the sound emitting device 76. The output processing unit 30 mixes the acoustic signal QA from the reproduction control unit 20 and the acoustic signal QB from the sound collecting device 74, and various effects on the acoustic signal processed by the mixing unit 32. An effect imparting unit 34 that imparts (for example, reverb or echo) and a D / A conversion unit 36 that converts the acoustic signal processed by the effect imparting unit 34 into an analog acoustic signal QC are configured. Sound corresponding to the acoustic signal QC is emitted from the sound emitting device 76.

  The analysis processing unit 40 shown in FIG. 1 specifies (synchronized analysis) the corresponding positions in the music for the acoustic signal P1 held in the first holding unit 161 and the time series of each musical tone designated by the musical tone data DA. The analysis processing unit 40 includes a first feature extraction unit 42, a data conversion unit 44, a second feature extraction unit 46, and a synchronization analysis unit 48.

  The first feature extraction unit 42 performs a chroma vector (pitch class profile) of the acoustic signal P1 for each unit section (frame) F1 obtained by dividing the acoustic signal P1 held by the first holding unit 161 on the time axis (see FIG. 2). (PCP: pitch class profile)) Ψ1 is calculated sequentially. The chroma vector Ψ1 is a feature value that serves as an index of harmony (harmony) of the acoustic signal P1, and different pitch names (C, C #, D, D #, E, F, F #, It is expressed by a 12-dimensional vector corresponding to G, G #, A, A #, B). Each of the twelve elements composing the chroma vector Ψ1 has a frequency component of a pitch name corresponding to the element of the acoustic signal P1 (12 frequency bands obtained by dividing a frequency band corresponding to one octave for each pitch name). This corresponds to a numerical value obtained by adding or averaging the intensities of the frequency band component corresponding to the pitch name) for a plurality of octaves. That is, the first feature extraction unit 42 generates a frequency spectrum (power spectrum) for each unit section F1 of the acoustic signal P1 by a known technique such as fast Fourier transform, and a frequency band corresponding to one octave of the frequency spectrum is generated. A chroma vector Ψ1 is generated by adding a plurality of octaves.

  The data conversion unit 44 in FIG. 1 generates an acoustic signal P2 representing a waveform of a musical tone designated by the musical tone data DA among the music data D0 held by the second holding unit 162. Specifically, a known sound source (MIDI sound source) that sequentially generates a tone waveform specified by event data of the tone data DA is preferably employed as the data conversion unit 44. The waveform of the musical sound represented by the acoustic signal P2 generated by the data converter 44 is synchronized with the time series of the lyrics specified by the display data DB. That is, the point in time when each musical sound is generated when the sound signal P2 is reproduced and the point in time when the display data DB designates the lyrics to be sung at the time of reproduction of the musical sound correspond on the time axis (ideally match) )

  The second feature extraction unit 46 sequentially applies the chroma vector (pitch class profile) Ψ 2 of the acoustic signal P 2 for each unit section F 2 obtained by dividing the acoustic signal P 2 generated by the data conversion unit 44 on the time axis (see FIG. 2). To calculate. Similarly to the chroma vector Ψ1, the chroma vector Ψ2 has the twelve elements corresponding to different pitch names (chroma), and the intensity of the component of the pitch name corresponding to the element in the acoustic signal P2 is set to a plurality of octaves. It is a 12-dimensional vector set to a numerical value added or averaged over. The second feature extraction unit 46 generates a chroma vector Ψ 2 of the acoustic signal P 2 for each unit section F 2 in the same manner as the first feature extraction unit 42. The unit section F1 and the unit section F2 are sections having the same time length. However, a configuration in which the unit interval F1 and the unit interval F2 are set to different time lengths may be employed.

  The synchronization analysis unit 48 in FIG. 1 compares each chroma vector Ψ1 generated by the first feature extraction unit 42 for each unit section F1 with each chroma vector Ψ2 generated by the second feature extraction unit 46 for each unit section F2. The positions on the time axis corresponding to each other in the music (for example, the position corresponding to the same musical sound of the music) in the music are specified according to the comparison result. Specifically, the synchronization analysis unit 48 generates the correspondence table TBL1 illustrated in FIG. The correspondence table TBL1 is a chroma of the plurality of unit sections F1 (F1 [1] to F1 [M]) of the acoustic signal P1 and the plurality of unit sections F2 (F2 [1] to F2 [N]) of the acoustic signal P2. This is a data table that associates a unit interval F1 and a unit interval F2 in which the vector ψ1 and the chroma vector ψ2 are similar to each other.

  As shown in FIG. 1, the synchronization analysis unit 48 includes a synchronization point detection unit 52 and a section contrast unit 54. The synchronization point detector 52 detects the synchronization point S1 of the acoustic signal P1 and the synchronization point S2 of the acoustic signal P2. The synchronization point S1 and the synchronization point S2 are points in time when the singing and accompaniment of the music starts (at the time when the silent section before the music starts). That is, the synchronization point S1 and the synchronization point S2 correspond to the time when the acoustic signal P1 and the acoustic signal P2 are first synchronized. The synchronization point S1 is designated by the number of the unit section F1, and the synchronization point S2 is designated by the number of the unit section F2.

  FIG. 3 is a flowchart of the operation of the synchronization point detection unit 52. For example, the processing of FIG. 3 is executed before the start of the reproduction of the acoustic signal P1 triggered by a predetermined operation on the input device 78. The synchronization point detection unit 52 identifies the reference point α1 of the acoustic signal P1 and the reference point α2 of the acoustic signal P2 (SA1, SA2). The reference point αk (k = 1, 2) is the time when the intensity (volume) of the acoustic signal Pk first exceeds the threshold value.

  FIG. 4 is a flowchart of processing (SA1, SA2) in which the synchronization point detection unit 52 specifies the reference point αk. The synchronization point detector 52 initializes the reference point αk to zero (first unit interval Fk of the acoustic signal Pk) (SB1), and the intensity Lk at the reference point αk of the acoustic signal Pk is a predetermined threshold value LTH. It is determined whether or not (SB2). When the intensity Lk is lower than the threshold value LTH (SB2: NO), the synchronization point detection unit 52 updates the reference point αk by a predetermined value δ (for example, δ = 1) to a later time point (SB3), and the updated reference point The process of step SB2 is executed for αk. When the intensity Lk exceeds the threshold value LTH (SB2: YES), the synchronization point detection unit 52 ends the process of FIG. Therefore, the time point at which the intensity Lk first exceeds the threshold value LTH in the acoustic signal Pk is determined as the reference point αk.

  By the way, there is a possibility that the reference point α2 in which the intensity L2 first exceeds the threshold value LTH in the acoustic signal P2 does not correspond to the time when the music actually starts (the time when the music starts with the acoustic signal P1). For example, assuming karaoke music data D0 of a music whose singing sound starts prior to the accompaniment sound, the singer grasps the time signature in the acoustic signal P2 generated from the music data D0 (musical sound data DA). The time signature is generated before the start of the accompaniment sound. Therefore, there is a possibility that the time point of the time signature is detected as the reference point α2. Therefore, the synchronization point detection unit 52 compares each chroma vector Ψ1 of the acoustic signal P1 with each chroma vector Ψ2 of the acoustic signal P2 in step SA3 of FIG. 3 (that is, determines the musical similarity). Thus, an accurate synchronization point S2 in the acoustic signal P2 is specified. The synchronization point S1 of the acoustic signal P1 is set to the reference point α1 specified in step SA1.

  FIG. 5 is an explanatory diagram showing an overview of the process of step SA3 of FIG. The synchronization point detection unit 52 includes a plurality of candidates in which the chroma vector Ψ1 in the reference section σREF starting from the synchronization point S1 (reference point α1) in the acoustic signal P1 and the starting point SX and the time length WX in the acoustic signal P2 are different. Each chroma vector Ψ 2 in the interval σ X is compared. That is, as shown in FIG. 5, the synchronization point detection unit 52 has a plurality of time lengths WX from the start point SX of the acoustic signal P2 that are different for each of a plurality of cases where the start point SX on the time axis is changed. Is compared with the chroma vector ψ1 of the reference interval σREF of the acoustic signal P1. Then, the synchronization point detection unit 52 identifies a candidate section σX having the chroma vector Ψ 2 most similar to the chroma vector Ψ 1 of the reference section σ REF among the plurality of candidate sections σ X as the synchronization section σ S corresponding to the reference section σ REF. The starting point of the synchronization interval σS corresponds to the synchronization point S2.

  FIG. 6 is a flowchart of the operation of FIG. As shown in FIG. 6, the synchronization point detection unit 52 sets the variable β to which the musical similarity index (hereinafter referred to as “similarity index”) γ between the candidate section σX and the reference section σREF is substituted to the maximum value γMAX. And the synchronization point S1 of the reference interval σREF is set to the reference point α1 (SC1). In addition, the synchronization point detection unit 52 initializes the time length WX of the candidate section σX to the lower limit value LA, and the reference point α2 identified at step SA2 in FIG. 3 as the starting point SX of the candidate section σX as shown in FIG. To a time point (α2-T) that is a predetermined time length T before (SC2). The lower limit value LA is set to a time shorter than the reference interval σREF (for example, half of the time length WREF). The time length T is set to, for example, half (WREF / 2) of the time length WREF of the reference interval σREF.

  The synchronization point detector 52 includes the chroma vector Ψ1 of each unit section F1 in the reference section σREF of the acoustic signal P1 and the chroma vector of each unit section F2 in the candidate section σX extending from the start point SX to the time length WX of the acoustic signal P2. Ψ2 is compared (SC3, SC4). Specifically, the synchronization point detection unit 52 expands / contracts the candidate section σX to the time length WREF of the reference section σREF (that is, matches the start point and the end point with the reference section σREF), and references the candidate section σX after expansion / contraction. The chroma vector Ψ2 at the time corresponding to each unit section F1 in the section σREF is calculated by interpolation of the chroma vector Ψ2 of each unit section F2 in the candidate section σX before expansion / contraction (SC3). Then, the synchronization point detection unit 52 determines the distance D between the chroma vector Ψ1 in the reference section σREF and the interpolated chroma vector Ψ2 in the candidate section σX as a pair of the corresponding chroma vector Ψ1 and chroma vector Ψ2. It is calculated every time, and the sum or average of the distances D is calculated as the similarity index γ (SC4). The distance D is an index value that is a measure of the similarity between the chroma vector Ψ1 and the chroma vector Ψ2. For example, the inner product or the geometric distance between the chroma vector Ψ 1 and the chroma vector Ψ 2 is calculated as the distance D. Therefore, as the similarity index γ is smaller, the reference interval σREF of the acoustic signal P1 and the candidate interval σX of the acoustic signal P2 tend to be musically similar.

  The synchronization point detector 52 determines whether or not the similarity index γ calculated in step SC4 is less than the variable β (SC5). When the similarity index γ is less than the variable β (SC5: YES), the synchronization point detection unit 52 substitutes the similarity index γ calculated in the immediately preceding step SC4 for the variable β, and the current time length WX of the candidate section σX Is set as the provisional time length WS of the synchronization section σS, and the starting point SX of the current candidate section σX is set as the provisional synchronization point S2 (starting point of the synchronization section σS) (SC6). On the other hand, if the similarity index γ exceeds the variable β (SC5: NO), the synchronization point detection unit 52 proceeds to step SC7 without executing step SC6. That is, the time length WS and the synchronization point S2 are sequentially updated to the time length WX and the start point SX of the candidate section σX that is musically similar to the reference section σREF.

  Next, the synchronization point detection unit 52 adds a predetermined value B to the current time length WX of the candidate section σX (SC7), and determines whether or not the added time length WX exceeds the upper limit value LB (SC8). . The upper limit value LB is set to a time longer than the reference interval σREF (for example, 1.5 times the time length WREF). If the result of step SC8 is negative, the synchronization point detector 52 moves the process to step SC3. That is, as shown in FIG. 5, the similarity between the candidate section σX before the update and the reference point σREF is determined for the new candidate section σX having the same start point SX and a different time length WX (longer by the predetermined value B). (SC3 to SC8).

  On the other hand, if the result of step SC8 is positive (WX> LB), the synchronization point detector 52 moves the starting point SX of the candidate section σX backward by a predetermined value C as shown in FIG. Is initialized to a lower limit value LA (for example, half the time length WREF of the reference interval σREF) (SC9). Then, the synchronization point detection unit 52 starts from the time when the updated start point SX in step SC9 has passed a predetermined value T (for example, half the time length WREF) from the reference point α2 of the acoustic signal P2, as shown in FIG. Is determined to be backward (SC10). If the result of step SC10 is negative, the synchronization point detection unit 52 performs the processing subsequent to step SC3 for the updated starting point SX at step SC9 (the reference interval σREF and the starting point SX while changing the time length WX of the candidate interval σX). The process of comparing with the candidate section σX from the above is repeated. On the other hand, if the result of step SC10 is affirmative, the synchronization point detector 52 ends the process of FIG.

  As a result of the above processing, as shown in FIG. 5, each of a plurality of cases where the starting point SX is moved by a predetermined value C within a range extending from the reference point α2 of the acoustic signal P2 to the preceding and following time length T (total 2T). Then, the musical similarity between each of the plurality of candidate sections σX and the reference section σREF in which the time length WX is changed by a predetermined value B within the range from the lower limit value LA to the upper limit value LB is determined. 6 is completed, the time length WX and the starting point SX of the candidate section σX having the smallest similarity index γ (the candidate section σX most similar to the reference section σREF) among the plurality of candidate sections σX are obtained. The time length WS of the synchronization section σS and the synchronization point S2 are determined. The above is the operation of the synchronization point detection unit 52.

  The section contrast unit 54 in FIG. 1 generates a correspondence table TBL1 by analyzing the correspondence between each unit section F1 of the acoustic signal P1 and each unit section F2 of the acoustic signal P2. For comparison between the acoustic signal P1 and the acoustic signal P2 by the section contrast unit 54, DP (dinamic programming) matching exemplified below is preferably employed. As shown in FIG. 7, the section contrast unit 54 includes the time series (horizontal axis) from the unit section F1 [1] corresponding to the synchronization point S1 to the last unit section F1 [M] in the acoustic signal P1, and the sound. Of the signal P2, the synchronization point detection unit 52 compares the time series (vertical axis) from the unit interval F2 [1] to the last unit interval F2 [N] corresponding to the synchronization point S2 identified in the process of FIG. . In other words, the above-described processing by the synchronization point detection unit 52 is a process of matching the head (starting point of music) of the section to be analyzed by the section contrast unit 54 with the sound signal P1 and the sound signal P2 (head adjustment). Equivalent to.

  First, the section control unit 54 generates a unit from each of the time series of the unit sections F1 [1] to F1 [M] of the acoustic signal P1 and the time series of the unit sections F2 [1] to F2 [N] of the acoustic signal P2. For all combinations of selecting the section F1 [m] (m = 1 to M) and the unit section F2 [n] (n = 1 to N), the chroma vector Ψ1 of the unit section F1 [m] and the unit section F2 [ The distance D [m, n] (for example, inner product or geometric distance) between the n] and the chroma vector Ψ 2 is calculated. Therefore, a matrix (cost matrix) of M rows × N columns composed of distances D [1,1] to D [M, N] is generated.

  The section contrast unit 54 selects an arbitrary unit section F1 [m from the time point p [1,1] corresponding to the combination of the unit section F1 [1] (synchronization point S1) and the unit section F2 [1] (synchronization point S2). ] And a cumulative value g [m, n] of distances D [1,1] to D [m, n] among a plurality of routes to time point p [m, n] corresponding to the combination of unit interval F2 [n] ] Are sequentially identified for the route having the minimum value. Then, as shown in FIG. 7, the section contrast unit 54 performs a time point p [M, N corresponding to the last unit section F1 [M] of the acoustic signal P1 and the last unit section F2 [N] of the acoustic signal P2. ], The optimum route (matching path) MP for which the accumulated value g [M, N] is minimum is specified, and the unit interval F1 [m] and the unit interval F2 [n] corresponding to each time point on the optimum route MP The correspondence table TBL1 in FIG.

  The display control unit 60 in FIG. 1 causes the display device 72 to sequentially display each character of the lyrics designated by the display data DB held in the second holding unit 162. The display control unit 60 variably sets the tempo of the processing of the display data DB (display instruction to the display device 72) in synchronization with the reproduction of the acoustic signal P1 according to the correspondence table TBL1. Specifically, the lyrics characters of the unit section F2 associated with each unit section F1 in the correspondence table TBL1 are displayed on the display device 72 when the unit section F1 is reproduced in the sound signal P1. As described above, the display control unit 60 controls the tempo of processing of the display data DB. Further, the tempo of processing of the display data DB also depends on the tempo set by the tempo adjustment unit 26. That is, the display control unit 60 increases the processing tempo of the display data DB as the tempo set by the tempo adjustment unit 26 is faster.

  For example, as shown in FIG. 8, the unit section F1 [1] of the acoustic signal P1 corresponds to the unit section F2 [1] of the acoustic signal P2, and the unit sections F1 [2] and F1 [3] of the acoustic signal P1 Assume that the unit interval F2 [2] of the acoustic signal P2 corresponds. When the tempo of the processing of the display data DB is fixed under the relationship of FIG. 8, the display of the lyrics proceeds with respect to the reproduction of the acoustic signal P1, and the reproduction of the acoustic signal P1 and the display of the lyrics are not synchronized. It becomes. Therefore, as shown in FIG. 8, the display control unit 60 displays in the unit section F2 [2] at a tempo that is half the tempo (tempo × 1/2) of processing the display data DB of the unit section F2 [1]. Process data DB. Accordingly, in the section where the unit section F1 [2] and the unit section F1 [3] of the acoustic signal P1 are reproduced (from the start point of the unit section F1 [2] to the end point of the unit section F1 [3]), the display data DB Among them, the lyrics of the unit section F2 [2] are displayed, and when the reproduction of the unit section F1 [4] of the sound signal P1 starts, the display of the lyrics of the unit section F2 [3] starts.

  On the other hand, as shown in FIG. 9, the unit section F1 [1] of the acoustic signal P1 corresponds to the unit section F2 [1] of the acoustic signal P2, and the unit section F1 [2] of the acoustic signal P1 and the unit of the acoustic signal P2 Assume that the sections F2 [2] to F2 [4] correspond to each other. When the processing tempo of the display data DB is fixed under the relationship of FIG. 9, the display of the lyrics is delayed with respect to the reproduction of the acoustic signal P1, and the reproduction of the acoustic signal P1 and the display of the lyrics are not synchronized. Become. Therefore, as shown in FIG. 9, the display control unit 60 has unit intervals F2 [2] to F2 [4] at a tempo (tempo × 3) that is three times the tempo for processing the display data DB of the unit interval F2 [1]. ] Display data DB is processed. Therefore, in the section where the unit section F1 [2] of the acoustic signal P1 is reproduced (from the start point to the end point of the unit section F1 [2]), the unit sections F2 [2] to F2 [4] of the display data DB are displayed. The lyrics are sequentially displayed, and the display of the lyrics of the unit section F2 [5] is started when the reproduction of the unit section F1 [3] of the sound signal P1 is started.

  The processing tempo of the display data DB is controlled by the following method, for example. First, the display control unit 60 sets the playback speed of the display data DB (for example, the number of unit sections F2 to be processed within a predetermined time (frame / sec)) according to the preceding and following of the unit sections F1 and F2. calculate. For example, when the display of the lyrics precedes the reproduction of the acoustic signal P1 (in the case of FIG. 8), the reproduction speed is set to a low speed, and the display of the lyrics is delayed with respect to the reproduction of the acoustic signal P1 (in the case of FIG. 9). Set the playback speed to high. The display control unit 60 variably sets the tick interval that defines the unit (resolution) of the processing interval of the display data DB according to the reproduction speed (for example, the higher the reproduction speed, the greater the tick interval). The processing tempo of the display data DB is variably controlled.

  In the above embodiment, the reference section σREF of the acoustic signal P1 and the candidate section σX of the acoustic signal P2 are compared while moving the starting point SX of the candidate section σX, and a candidate similar to the reference section σREF among the plurality of candidate sections σX. The starting point SX of the section σX (synchronization section σS) is specified as the synchronization point S2 corresponding to the synchronization point S1 of the acoustic signal P1. Therefore, even when the synchronization point S1 of the acoustic signal P1 and the synchronization point S2 of the acoustic signal P2 have different positions on the time axis, it is possible to synchronize the reproduction of the acoustic signal P1 and the display of the lyrics with high accuracy. is there. Moreover, in the first embodiment, since the time length WX is changed in addition to the starting point SX for the candidate section σX, even when the tempo is different between the acoustic signal P1 and the acoustic signal P2, the reproduction of the acoustic signal P1 and the lyrics The display can be synchronized with high accuracy.

  Furthermore, since the starting point SX is changed within a predetermined range including the reference point α2 at which the intensity L2 of the acoustic signal P2 first exceeds the threshold value LTH, for example, sound that does not exist in the acoustic signal P1 (for example, beat sound) is acoustic signal P2. Even when included in the head portion of the signal, there is an advantage that the synchronization points (S1, S2) that are synchronized with each other between the acoustic signal P1 and the acoustic signal P2 can be accurately detected.

  Further, the correspondence between the unit section F1 and the unit section F2 in the music is analyzed according to the musical similarity between each unit section F1 of the acoustic signal P1 and each unit section F2 of the acoustic signal P2, and they correspond to each other. The tempo of processing of the display data DB is variably controlled so that the unit section F1 and the unit section F2 are synchronized. Therefore, for example, when compared with the configuration of Patent Document 1 in which the reproduction of the sound signal and the display of the lyrics are synchronized only at the start point or the end point of the entire song section or the singing section, for example, the sound signal P1 and the song data D0 It is possible to synchronize the reproduction of the sound signal P1 and the display of the lyrics with high accuracy even when the synchronization of the sound is lost.

  By the way, as a method of synchronizing the reproduction of the sound signal P1 and the display of the lyrics, for example, a method of associating the beat point of the sound signal P1 with the beat point of the sound signal P2 can be assumed. However, in the method using the beat points, for example, when the sound signal P1 is not reproduced and the lyrics are displayed due to omission of detection of the beat point of the sound signal P1 or the sound signal P2, for example, the sound signal P1 is reproduced. On the other hand, when the lyrics are displayed with a delay of one beat, it is difficult to restore the synchronization between the two. In the present embodiment, since the synchronization between the reproduction of the acoustic signal P1 and the display of the lyrics is ensured according to the comparison result of the chroma vectors (Ψ1, Ψ2) serving as an index of harmony, the two are out of synchronization. The possibility is low, and even if the two are out of synchronization for some reason, it is possible to quickly recover the synchronization.

<B: Second Embodiment>
Next, a second embodiment of the present invention will be described. In addition, about the element in which an effect | action and a function are equivalent to 1st Embodiment in each following form, the same code | symbol as the above is attached | subjected and each detailed description is abbreviate | omitted suitably.

  In the acoustic processing apparatus 100 of the second embodiment, the playback point of the acoustic signal P1 is changed in accordance with an operation from the user with respect to the input device 78. For example, the display control unit 60 displays the operation screen 720 of FIG. The operation screen 720 is a GUI including a linear reproduction area 721 that represents the entire section of the music (sound signal P1) and an indicator 722 arranged on the reproduction area 721. The indicator 722 is an image showing the time point of the music being played. Therefore, the indicator 722 moves over time from the left end to the right end of the reproduction area 721 as the reproduction of the music progresses. The user can arbitrarily change the playback point of the music piece by operating the input device 78 and moving the indicator 722.

  The synchronization point detection unit 52 of the second embodiment has a function similar to that of the first embodiment (or instead of the function of the first embodiment), and a reproduction point after the change by the user in the acoustic signal P1 (hereinafter referred to as “reproduction point”). A function of specifying the time point of the acoustic signal P2 corresponding to the music in the music. FIG. 11 is a flowchart of processing executed by the synchronization point detection unit 52 when a change of the reproduction point is instructed. The process of FIG. 11 is started when the playback point is changed by the user.

  When the processing of FIG. 11 is started, the synchronization point detector 52 sets a reference interval σREF including the reproduction instruction point t1 in the acoustic signal P1 (SE1). For example, as shown in FIG. 10, a section having a predetermined time length WREF starting from the reproduction instruction point t1 is specified as the reference section σREF. Next, the synchronization point detector 52 specifies a reference point t2 corresponding to the reproduction instruction point t1 in the acoustic signal P2 (SE2). For example, as shown in FIG. 10, the synchronization point detecting unit 52 is at the same time as the playback instruction point t1 in the acoustic signal P2, or the playback instruction point for all the sections of the acoustic signal P1 at the position relative to the entire section of the acoustic signal P2. A time point equivalent to the position of t1 (for example, a time point corresponding to 1/3 of the entire section when viewed from the start point of the acoustic signal P2) is specified as the reference point t2.

  Then, the synchronization point detection unit 52 uses the same method as the first embodiment to identify the synchronization point S1 and the synchronization point S2 from the reference point α1 and the reference point α2, and from the reproduction instruction point t1 and the reference point t2 to the synchronization point S1 and The synchronization point S2 is specified (SE3). That is, the synchronization point detection unit 52 specifies the synchronization point S1 and the synchronization point S2 by replacing the reference point α1 with the reproduction instruction point t1 and replacing the reference point α2 with the reference point t2 and executing the processing of FIG. To do.

  Specifically, as shown in FIG. 10, the synchronization point detection unit 52 is included in the candidate section σX defined by each chroma vector Ψ1 and the acoustic signal P2 in the reference section σREF specified in step SE1 in the acoustic signal P1. The similarity index γ with each of the chroma vectors ψ2 is calculated for each of a plurality of cases in which the starting point SX and the time length WX of the candidate section σX are changed. The starting point SX is sequentially changed within a range over a predetermined time length T (total 2T) before and after the reference point t2 specified in step SE2, and the time length WX is within the range from the lower limit value LA to the upper limit value LB ( For example, it is sequentially changed within a range of half to 1.5 times the time length WREF. The synchronization point detection unit 52 selects the candidate section σX having the smallest similarity index γ among the plurality of candidate sections σX (that is, the candidate section σX that is musically most similar to the reference section σREF specified in step SE1) as the synchronization section σS. And the start point of the synchronization interval σS is specified as the synchronization point S2.

  The section contrast unit 54 includes the unit sections F1 (unit sections F1 [1], F1 [2],...) After the synchronization point S1 (reproduction instruction point t1) in the acoustic signal P1 and the acoustic signal P2. 11 is the same method (DP matching) as in the first embodiment for the correspondence in the music with the unit sections F2 (unit sections F2 [1], F2 [2],...) After the synchronization point S2 identified in step SE3. ).

  In the second embodiment, the synchronization point S1 and the synchronization point S2 are specified from the reproduction instruction point t1 variably set according to the instruction from the user and the reference point t2 corresponding to the reproduction instruction point t1. It is possible to synchronize the reproduction of the acoustic signal P1 and the display of the lyrics from an arbitrary reproduction instruction point t1. Also, since the playback of the sound signal P1 and the display of the lyrics are synchronized from an arbitrary playback instruction point t1, it is necessary to specify the correspondence between the sound signal P1 and the sound signal P2 over the entire section before playing the music. There is no. That is, even when the processing for analyzing the correspondence between the acoustic signal P1 and the acoustic signal P2 in the music is executed in parallel with the reproduction of the acoustic signal P1, when the user designates the reproduction instruction point t1, the acoustic signal P1 And the display of lyrics can be synchronized from the reproduction instruction point t1. Therefore, a configuration for storing the sound signal P1 over the entire section of the music in the first holding unit 161 and a configuration for storing the musical sound data DA over the entire section of the music in the second holding unit 162 are unnecessary. That is, there is an advantage that the capacity of the first holding unit 161 and the second holding unit 162 is reduced.

<C: Modification>
Various modifications are added to the above embodiment. Specific modifications are exemplified below. Two or more aspects arbitrarily selected from the following examples may be merged.

(1) Modification 1
In the first embodiment, the start point SX and the time length WX of the candidate section σX are changed and sequentially compared with the reference section σREF. However, in the process of FIG. A configuration in which only the start point SX is changed by fixing the time length WX to a predetermined value (for example, the time length WREF) is also adopted. That is, step SC7 and step SC8 in FIG. 6 can be omitted. Similarly, in the process of the second embodiment for specifying the synchronization point S1 and the synchronization point S2 from the reproduction instruction point t1 and the reference point t2, a configuration in which only the start point SX is changed while the time length WX is fixed to a predetermined value can be adopted. .

(2) Modification 2
Each numerical value exemplified as a fixed value in each of the above embodiments can be appropriately changed to a variable value. For example, in each of the above embodiments, the time length WREF of the reference section σREF is fixed to a predetermined value, but a configuration in which the time length WREF is variably controlled is also suitable. Specifically, a configuration is adopted in which the synchronization analysis unit 48 variably controls the time length WREF in accordance with the tempo specified by the music data D0 and the number of sounds specified by the musical sound data DA. For example, the synchronization analysis unit 48 sets the time length WREF to a shorter time as the tempo is faster, and sets the time length WREF to a shorter time as the number of sounds per unit time increases. According to the above configuration, there is an advantage that the correspondence between the sound signal P1 and the sound signal P2 can be specified with high accuracy even for a music with a fast tempo or a music with a large number of sounds.

  Further, in each of the above embodiments, the time length T that defines the range (search range) in which the time length WX of the candidate section σX is changed is fixed to a predetermined value, but a configuration in which the time length T is variably controlled is also suitable. is there. For example, the synchronization analysis unit 48 sets the time length T to a shorter time as the tempo specified by the music data D0 is faster (sets the search range to a narrower range), and the number of sounds specified by the musical sound data DA per unit time. As the number increases, the time length T is set to a shorter time. According to the above configuration, there is an advantage that an appropriate synchronization section σS can be specified with high accuracy even when the tempo of music is fast or the number of sounds is large.

  Further, in each of the above embodiments, the variation amount B of the time length WX and the movement amount C of the start point SX of the candidate section σX are fixed to predetermined values, but a configuration in which the variation amount B and the movement amount C are variably controlled is also preferable. It is. For example, the synchronization analysis unit 48 sets the amount of change B or the amount of movement C to a smaller value as the tempo specified by the music data D0 is faster, and the amount of change as the number of sounds specified by the musical sound data DA per unit time increases. B and moving amount C are set to small numerical values. According to the above configuration, there is an advantage that an appropriate synchronization section σS can be specified with high accuracy even when the tempo of music is fast or the number of sounds is large.

  In addition, the deviation of the synchronization between the acoustic signal P1 and the acoustic signal P2 becomes more remarkable at the rear of the music. Therefore, in the second embodiment, it is preferable to variably control the time length WREF of the reference section σREF and the time length T defining the search range according to the movement amount Δt of the indicator 722 in FIG. Specifically, the synchronization analysis unit 48 sets the time length WREF or the time length T to a longer time as the movement amount Δt is larger (that is, as the reproduction instruction point t1 is located behind the music). According to the above configuration, even when the movement amount Δt is large, it is possible to accurately specify the correspondence between the acoustic signal P1 and the acoustic signal P2 after the reproduction instruction point t1.

(3) Modification 3
The information displayed on the display device 72 is not limited to lyrics. For example, a configuration in which the display data DB specifies a musical score (code score or tablature) in time series, or a configuration in which the display data DB specifies an image related to the music (for example, a karaoke background image) in time series is also preferable. . In other words, the display data DB is included as data for designating images (lyrics, music scores, background images) related to song singing and performance in time series. A configuration in which the display data DB is prepared separately from the musical sound data DA is not essential. For example, a configuration is also adopted in which the musical sound designated by the musical sound data DA in time series is used for displaying the musical score.

  Further, the operation to be synchronized with the reproduction of the acoustic signal P1 is not limited to the display of lyrics and musical scores. Specifically, a configuration is adopted in which the processing of instruction data for instructing operations related to music reproduction in time series (control of elements that execute the operations) is synchronized with the reproduction of the acoustic signal P1. As with the display data DB, the instruction data is created so that the time series of musical sounds designated by the musical sound data DA and the time series of instructions by the instruction data correspond (synchronize) with each other on the time axis. For example, the instruction data for designating the operation (blinking and illuminance) of the lighting device at the time of music reproduction in time series, and the mode (type and degree of effect) of the acoustic effect added by the effect imparting unit 34 are designated in time series. A configuration in which the instruction data is processed in synchronization with the reproduction of the acoustic signal P1 instead of the display data DB (or together with the display data DB) is also suitable.

  As can be understood from the above examples, the display data DB and the instruction data are included as control data for designating information related to music (lyrics, music score, or instruction of operation of elements related to music reproduction) in time series, The element that processes the control data (for example, the display control unit 60 in each of the above embodiments) controls the control data according to the analysis result by the synchronization analysis unit 48 so that the reproduction of the acoustic signal P1 and the processing of the control data are synchronized. Is included as an element (control means) for processing.

(4) Modification 4
In the above embodiment, the configuration in which the data conversion unit 44 generates the acoustic signal P2 from the musical sound data DA is exemplified, but the acoustic signal P2 for synchronization analysis is used instead of the musical sound data DA (or together with the musical sound data DA). The configuration stored in is also adopted. The analysis processing unit 40 performs synchronization analysis on the acoustic signal P2 stored in the storage device 14. In the configuration in which the acoustic signal P2 is stored in the storage device 14, the data conversion unit 44 can be omitted. However, according to the configuration provided with the data conversion unit 44, the musical sound data DA of the existing music data D0 created for karaoke can be used for the synchronization analysis by the analysis processing unit 40, and the acoustic signal P2 is stored in the storage device. There is an advantage that the capacity required for the storage device 14 is reduced compared to the configuration stored in the storage device 14.

(5) Modification 5
In the above embodiment, the configuration in which the musical sound data DA is used only for the synchronization analysis with the acoustic signal P1, but the configuration in which the acoustic signal P2 corresponding to the musical sound data DA is reproduced together with the acoustic signal P1 is also preferably employed. For example, if the sound signal P2 is generated from the musical sound data DA indicating the guide melody and reproduced together with the sound signal P1, it is possible to support the practice of singing and playing by the user. Further, the reproduction control unit 20 is omitted as appropriate. For example, a configuration in which the component suppression unit 22 is omitted and all the parts of the acoustic signal P1 are reproduced, or a configuration in which the pitch conversion unit 24 and the tempo adjustment unit 26 are omitted and the pitch and tempo are not changed may be employed.

(6) Modification 6
The method of analyzing the synchronization between the acoustic signal P1 and the acoustic signal P2 after the synchronization point S1 and the synchronization point S2 are specified is appropriately changed. For example, the method for analyzing the correspondence between each unit section F1 and each unit section F2 is not limited to the DP matching of the first embodiment. Further, in each of the above embodiments, the correspondence between each unit section F1 of the acoustic signal P1 and each unit section F2 of the acoustic signal P2 is specified, but the acoustic signal P1 and the plurality of unit sections (F1, F2) are used as units. A configuration for specifying the correspondence with the acoustic signal P2 is also employed. For example, the section contrast unit 54 includes the chroma vector Ψ1 of each unit section F1 in each reference section σREF in which the acoustic signal P1 after the synchronization point S1 is divided by the time length WREF corresponding to the plurality of unit sections F1, and the acoustic signal P2. The similarity index γ with the chroma vector ψ2 of each unit section F2 of the candidate section σX is calculated for each of a plurality of cases where the time length WX of the candidate section σX is changed. For the calculation of the similarity index γ, the same method as in step SC3 and step SC4 in FIG. 6 is applied. Then, the section contrast unit 54 identifies the candidate section σX having the smallest similarity index value γ among the plurality of candidate sections σX as the synchronization section σS corresponding to the reference section σREF. By repeating the above processing, the synchronization interval σS corresponding to each of the plurality of reference intervals σREF obtained by dividing the acoustic signal P1 is specified as the acoustic signal P2.

(7) Modification 7
The feature quantities extracted by the first feature extraction unit 42 and the second feature extraction unit 46 are not limited to chroma vectors (Ψ1, Ψ2). For example, a configuration in which the pitch of the acoustic signal P1 or the acoustic signal P2 is extracted and applied to the synchronization analysis may be employed. As understood from the above description, the feature amount is included as a scale for determining musical similarity between the acoustic signal P1 and the acoustic signal P2.

(8) Modification 8
In each of the above embodiments, the process of FIG. 4 that specifies the reference point αk according to the intensity (volume) of the acoustic signal Pk, and the process of FIG. 6 that specifies the synchronization point S2 by comparing the chroma vectors (Ψ1, Ψ2). The synchronization point detection unit 52 executed. However, since the chroma vector is a feature amount indicating a sense of harmony (chord sense), it is difficult to accurately specify the synchronization point S2 when the music starts from a percussion musical tone, for example. Therefore, a configuration in which the synchronization point detection unit 52 executes only the process of FIG. 4 (a configuration in which the process of FIG. 6 is omitted) is also employed. The reference point αk specified in the process of FIG. 4 is applied to the process by the section contrast unit 54 as the synchronization point Sk.

DESCRIPTION OF SYMBOLS 100 ... Sound processing device, 12 ... Arithmetic processing device, 14 ... Memory | storage device, 161 ... 1st holding | maintenance part, 162 ... 2nd holding | maintenance part, 20 ... Reproduction control part, 22 ... Component suppression part, 24 …… Pitch converter, 26 …… Tempo adjuster, 30 …… Output processor, 32 …… Mixer, 34 …… Effect imparter, 36 …… D / A converter, 40 …… Analysis processor, 42 …… First feature extraction unit, 44 …… Data conversion unit, 46 …… Second feature extraction unit, 48 …… Synchronization analysis unit, 52 …… Sync point detection unit, 54 …… Section control unit, 60 …… Display control unit, 72... Display device, 74... Sound collection device, 76.

Claims (5)

Storage means for storing control data for designating information related to music in time series;
First feature extraction means for sequentially generating feature quantities of a first acoustic signal representing a waveform of a musical tone of music;
Second feature extraction means for sequentially generating feature quantities of a second acoustic signal representing a waveform of a musical sound synchronized with a time series of information designated by the control data;
The comparison between the feature quantity of the reference section of the first acoustic signal and the feature quantity of the candidate section of the second acoustic signal is performed for each of the plurality of candidate sections having different positions on the time axis, Synchronization point detecting means for identifying a candidate section whose feature amount is similar to the feature amount of the reference section among the candidate sections as a synchronization section corresponding to the reference section;
Control processing the control data so that the reproduction of the reference section of the first sound signal and the processing of the control data corresponding to the synchronization section identified by the synchronization point detection means for the reference section are synchronized. A sound processing apparatus comprising: means. The synchronization point detection means compares the feature amount of each of a plurality of candidate sections having the same start position and different time lengths in the second acoustic signal with the feature quantity of the reference section. The sound processing according to claim 1, wherein a candidate section having a feature amount similar to the feature amount of the reference section is specified as the synchronization section among the plurality of candidate sections compared with the reference section. apparatus. The synchronization point detection means identifies a first reference point whose intensity first exceeds a threshold value among the first acoustic signals and a second reference point whose intensity first exceeds a threshold value among the second acoustic signals, The feature amount of the reference section starting from the first reference point is compared with the feature amount of each of a plurality of candidate sections whose start points are changed within a predetermined range including the second reference point. The sound processing apparatus according to claim 2. The synchronization point detecting means includes a feature amount of the reference section including a reproduction instruction point variably set in accordance with an instruction from a user in the first acoustic signal, and the reproduction instruction in the second acoustic signal. The acoustic processing device according to any one of claims 1 to 3, wherein the acoustic processing device compares a feature amount of each of a plurality of candidate sections whose start points are changed within a predetermined range including a reference point set according to the points. In a computer comprising storage means for storing control data for designating information related to music in time series,
A first feature extraction process for sequentially generating feature quantities of a first acoustic signal representing a musical tone waveform of the music;
A second feature extraction process for sequentially generating feature quantities of a second acoustic signal representing a waveform of a musical sound synchronized with a time series of information designated by the control data;
The comparison between the feature quantity of the reference section of the first acoustic signal and the feature quantity of the candidate section of the second acoustic signal is performed for each of the plurality of candidate sections having different positions on the time axis, A synchronization point detection process for identifying a candidate section whose feature quantity is similar to the feature quantity of the reference section among the candidate sections as a synchronization section corresponding to the reference section;
Control for processing the control data so that the reproduction of the reference section of the first sound signal and the processing of the control data corresponding to the synchronization section specified in the synchronization point detection process for the reference section are synchronized. A program that executes processing and.

Images