JP5143569B2 - Method and apparatus for synchronized modification of acoustic features - Google Patents

Description

  The present invention relates to signal modification. More specifically, but not exclusively, the invention is summarized in other cases where the corresponding features of the first and second audio signals do not occur at the same relative position in time within each signal. Related to the problems that occur in the modification of one digested speech signal, based on features in the recorded speech signal.

  It is difficult to speak or sing with an audio or audio / video clip if the new performance is an exact synchronized repeat of the words of the original actor or singer It is well known that there is. Thus, it is very rare for the start of a new performance recording and detailed acoustic features to synchronize with the start of the original audio track and detailed acoustic features. Similarly, features such as the new singer's pitch may not change as accurately or as complex as the features of the original singer. Professional audio recording industry and consumer computer-based games and activities, where audio recordings are made with newly recorded voice utterances and musical pitches, make the audio recording the original voice There are many cases that benefit from pitch adjustment (generally meaning correction) to synchronize with other recordings. In addition, even if synchronized, normal amateur song recordings may not have a professional singer's skilled vocal style and pitch inflection.

  FIG. 4 shows a professional singer pitch measurement (guide pitch 401) and several public (new pitch 402) songs with the same music track and the same words. The timing between the onsets and offsets of the corresponding section (pulse) of the uttered signal (non-zero Hz pitch value) and the position of the non-voiced or silent section (at zero Hz) Inconsistencies often occur and are significant. Applying pitch data from guide pitch 401 at the same relative time directly to new pitch 402 data is clearly wrong and inappropriate for the substantial amount of segments shown here. is there. This is a general result and represents a basic solution.

  Pitch adjustments for musical notes are generally commercially available, such as synchronizing input notes to a fixed grid with a specified pitch of acceptable notes. Can be automatically applied to recorded or live singing by various hardware and software devices. In such systems, each output tone can be corrected automatically, but this approach can often be unacceptable or have unpleasant consequences. Because it can remove natural and desirable "human" variations.

  The basic basis for target pitch identification in such known software and hardware devices is basically a list of those specific sound frequencies (as opposed to the device's first input). The scale of the music, which should be compared). Most devices come with a preset musical scale relative to the standard scale, for example to change the target pitch or leave a constant pitched sound intact. Allows customization.

  Known software devices can be set to automatic mode. This automatic mode generally determines how the hardware device operates (the device detects the input pitch, identifies the closest scale note in the user-specified preset scale, and the output pitch is specified. The input signal is changed so as to match the pitch of the sound of the selected scale. A ratio, at which the output pitch is slewed back to the target pitch, and the ratio (often referred to as “speed”) is a natural pitch counter (ie pitch as a function of time). ) Is more accurate and more natural, and is controlled to help enable a wider range of “styles”.

  However, amateur recorded singing should not be emphasized by such known automatic tuning techniques in order to achieve the complex and skilled pitch variation found in professional singer performances .

  Perform a pitch correction and / or other utterance modification to specify a desired modification using a target utterance or a sequence of other stored target utterance parameter data There are also voice processing methods and systems. These known methods have one or more major drawbacks. For example,

1. A karaoke track or other such accompaniment in which the target pitch (or other voicing features) provided to the user's input utterance signal is strictly followed by the user, generally in real time No attempt is made to match the characteristics of the corresponding uttered sound in accordance with the timing (Patent Document 1: US Pat. No. 5,966,867, Patent Document 2: Japanese Patent No. 2003044066). If the user's utterance begins too early compared to the timing of the target feature (eg pitch) data, the target feature will be incorrectly applied to later words or syllables. A similar problem occurs if the user voice is slow. During a phrase, any words or syllables that are out of time in relation to the music track will be assigned to an incorrect pitch or other feature for that word or syllable. Similarly, any voiced segments that occur when unvoiced segments are expected will receive no stored target pitch or other target feature information at all. do not do.

2. The target pitch (or other utterance feature) applied to the user's input utterance is the expected sequence of stored input phonemes, or similar uttered / Rely on and follow non-voicing patterns, or simply vowel detection (US Pat. No. 5,057,912). Such methods generally require user training or input of fixed characteristics of phoneme data and / or require close enough pronunciation of the same words for accurate identification to occur. To do. If there is no training and the user's phoneme set is sufficiently different from the stored set that it is not recognized, the system will not function properly. If the user's phonemes are not kept long enough, or if they are kept too short, the output notes can be truncated or deleted. If the phoneme arrives too early or too late, the pitch or feature can be applied to the correct phoneme, but out of time with the musical accompaniment. If the user asks for incorrect phonemes (one or more), the system will easily fail to maintain a match. Furthermore, in a song, a single phoneme is often given a range of multiple pitches and / or a series of pitches (for these, phoneme-based systems have the correct pitch or feature change). Hard to run). Accurate phoneme recognition also requires non-zero processing time, which can slow down the application of the correct features in a real time system. Non-voiced sounds (eg, flutes) cannot be used as a guide signal or input.

3. The target pitch model is generally based on a set of discrete notes described by a table (eg, as Midi data) that is typically quantized in both pitch and time. In this case, the correction to the input utterance sound is limited to the stored sound. This approach leads to a limited set of available vocal patterns that can be generated. Inter-note transitions, vibrato, and glissando control are generally limited to rough sound-based descriptions (ie Midi). Also, the processed pitch-corrected utterances can take on mechanical (monotone) sounds, and if the pitch is applied to the wrong part of the word due to miss timing If so, the song can be heard weirdly sung, and in some cases, out of tune.

4). The system is designed to operate in near real time (like a live karaoke system), and the output is output immediately after the input (to be corrected) is received (ie, a few minutes per second). Within one). Systems that use phonemes or similar features (eg, US Pat. No. 5,509,912) are limited to highly localized time slots. Such a system can be out of step, for example, leading to karaoke singer vowels matching the wrong part of the target song to guide.

US Patent No. 5966867 JP2003-044066 US Patent No. 5750912 US Patent No. 5750912

  Therefore, first, there is a need for a method and apparatus that establishes a detailed timing relationship between the time-varying characteristics of a new song and the corresponding characteristics of the guiding song. Second, this timing alignment path can be used as a time map to correctly determine and apply features (eg, pitch) adjustments to new vocal performance at the correct time. It must be. When performed correctly, this method can affect the new singing with the nuance and complexity found in singing vocalizations (eg, for pitch, vibrato, inflection curves, glide, jumps, etc.) Make it possible. In addition, if time alignment is applied, other features in addition to or as an alternative to pitch; for example, glottal characteristics (eg, breath sounds or snarling ( raspy), tract resonances, EQ, and others can be controlled.

Another object of the present invention is that, in the case of non-ideal input signal conditions, particularly new inputs (eg, user-generated sounds)
(A) Bandwidth limited and / or dynamic range limited (eg, input via telephone system)
(B) if it contains certain types of noise or distortion;
(C) Guiding (target) generated voice (voice) is from a person with a different accent, gender or age, or words and phonemes (they are guiding (target) signals) Either from the same or different, and even from different input languages) from people with very different timings of delivery,
To provide a method for correcting utterances.

  Further objectives require any prior information about the detailed set of possible signal states that can be applied to the signal to be stored or the output signal, eg regarding the phonemic nature of the signal It is to provide a method that does not. Accordingly, a further related object is to provide a method that can operate with a guiding audio signal and a new audio signal (either or both of which are not required to be speech or singing).

  Systems and methods for time mapping and audio signal alignment already exist. A method and system for determining the time difference between two audio signals and automatically aligning one of the audio signals to the other audio signal by automatic waveform editing is described in British Patent 2117168 and US Patent 4591928 (Bloom et al.). Other techniques for time alignment are described in J Holmes and W Holmes, (2001), "Speech synthesis and recognition, 2nd Edirtion", Taylor and Francis, London.

  Techniques for pitch changes and other vocal corrections are also well established. One example is K. Lent (1989), “An efficient method for pitch shifting digitally sampled sounds,” Computer Music Journal Vol. 13, No. 4, (pages 65 to 71).

  The invention is defined by the claims, to which reference will now be made.

  The preferred embodiment of the present invention automatically and correctly converts one or more signal characteristics of the second digitized audio signal to a specified feature in the first digitized audio signal. Methods and apparatus for modifying to be a function are provided. In these embodiments, the relative timing of specified features in both signals is first established. Based on these timing relationships, detailed and temporally critical corrections of signal features can be applied correctly. To accomplish this, a mapping between the first signal features and the second signal features is generated, and a function for optionally editing the second (user) signal is provided. A time alignment function is generated.

  For specific applications of the present invention, the selected audio characteristics of a professional singer's digitized vocalization song are accurately transferred to the digitized song of a less skilled person, Thus, enhancing it is included. One particular application of the present invention is that the pitch of a new audio signal ("new signal") generated by a general member of the public is different from another audio signal ("guide signal") generated by a professional singer. This is an application in automatically adjusting to follow the pitch. An example of this is a karaoke-style recording and playback system using digitized music video as the original source, where the user's voice is digitized during playback of the original audio and optional corresponding video. System that is then entered into the device (as a new record). This system can generate a modified user voice signal that is automatically time and pitch corrected. When the modified voice signal is played in sync with the original video, the user's voice will be the recorded voice of the original singer in terms of both pitch and time, including some lip synching. Can be exactly substituted. If the original, replaced speech signal is not audible during playback due to the recording of the user's modified voice, the impact of this replacement will be even more effective during playback of the music video. . As described in WO 2004/040576, the modified voice recording can be combined with the original background music.

  An additional application of the present invention resides in the generation of personalized audio files for use in a telephone system. In such an application, the user sings or even speaks and is recorded and then enhanced (eg, pitch and time to follow the characteristics of a professional singer's version). Corrected) and optionally provides an utterance signal that is mixed with the appropriate background track. The resulting improved user record can then be made available to the telephone user as a personalized ring tone or as an audio file for other purposes. The device embodying the invention can then take the form of a server computer coupled to a telecommunication network and / or a telecommunication system comprising the Internet, for example, as an interface between the device and the user. You can use the phone. Additionally or alternatively, a mobile phone can be applied to implement the present invention. In such a system, a modified utterance signal, or data representing such a signal, generated by an embodiment of the present invention may be used as a ring tone or other identifying voice signal for use in a ring tone ( ring tone) may be sent to the selected recipient via the delivery system.

  In the preferred embodiment of the present invention, the inclusion of the step of generating a mapping function between a time-dependent guide (Guide) and a new signal (NewSignal) means that the modification of the signal characteristics is between two signals. To ensure that it is done at the appropriate time within the new signal. The time alignment function is used to map the control feature function data to the desired signal modification process. The modification process accesses the new signal and modifies it as required. This operation generates a new, third audio signal from the new signal. Thus, the third signal then has the desired time-varying features determined by the features specified as the control features of the guide signal.

  In one embodiment, the second audio signal (new signal) is time corrected (non-linearly time compressed or expanded) using the mapping information from the time alignment function, so that its time variation The features are arranged in time with the first audio signal (guide signal). This time alignment may be performed before or after the desired correction described above is performed.

  In an alternative embodiment, the time alignment process is not performed on the new or modified waveform. Instead, a time warping path is used to modify the appropriate portion of the waveform of the second signal to maintain its original timing, and the first signal (guide signal voice control parameter). Are mapped to the second signal.

  By performing processing without the constraints of real-time processing, a detailed analysis of the stored version of the guide signal and the new signal can be made, statistically significant and substantial before the time alignment process begins. Both signal quantities (eg, up to 30 seconds, or even the entire signal) are processed, and important decisions are made regarding long-term signal characteristics.

  Thus, large size (eg, a few seconds) time excursions can be accommodated and corrected, and localized optimal alignment within words and phrases can be performed. In addition, functional features can also be performed “offline”, so that the highest quality processing is applied, and correction-related data interpolation and / or smoothing introduces some obvious terrible errors. , Allowing it to be removed before being applied to a new signal.

  The set of output feature values for the new signal need not be pre-defined. For example, if the pitch of a new signal provided by the user should be corrected to match the pitch of a guide signal in the form of a professional singer recording, an acceptable pitch value is defined or set. There is no need to Instead, the user's voice is present in the recording of the guide signal and is adjusted to the measured value.

  The new signal need not be limited to resemble the guide signal, or need to be generated by the same type of audio process as the guide signal. For example, monotonic speech can be time and pitch modified to follow a solo woodwind or bird song. As long as both signals have the same time-varying characteristics that can be treated as related, a method embodying the present invention can produce an output signal with appropriately modified properties. Furthermore, the new signal and guide signal features can be offset from each other in frequency. For example, the pitch of one signal can be one octave away from the other signal.

  It should also be understood that one or both audio signals can be in the ultrasonic or infra sound region.

  By operation of the preferred embodiment of the present invention, the complex, skilled pitch variations (and optionally other characteristics) found in the professional singer's demonstrations can be sung by the user (eg, amateur) singer. Can be accurately converted into a digitized voice. This improves many aspects of the user's demonstration to a professional level.

  Embodiments of the present invention can also be applied to the field of Automatic Dialogue Replacement (ADR) to improve actors' ADR studio recorded performances. Embodiments record studio-recorded vocal features, such as pitch, energy level, and pprosodic features, with images, either on set or location. It can be used to modify to match or follow that of the original guide signal. Further, the performer in the studio can be a different performer than the person who recorded the guide signal.

  Furthermore, the present invention is flexible in the range of processing that can be applied. For example, in the case of pitch adjustment, additional pitch change features such as time aligned harmony generation are introduced as a function of multiple pitch adjustment functions to generate an alternative output signal. Can be done. Furthermore, one measured feature in the guide signal can be mapped by any function to control other completely different features in the new signal.

  The method of implementing the present invention can be implemented in a computer program in a computer system, such as a PC or a computer-based game console, having means for voice input and output.

  There are many permutations of arrangements of processing sequences that can be performed (some have advantages over others in certain circumstances). The following example will illustrate how fluctuations affect processing complexity and / or reduce the potential for generating adjustable signal artifacts in the output signal: Shown with respect to processing pitch. Similar considerations and results can arise in consideration of processing features other than pitch, such as kuddness, tone, or format structure.

In general, in an embodiment, initially, new and guide signals are digitally sampled and stored. Next, a robust, speaker-specific, short-time feature analysis extracts feature change profiles in both signals. Spectrally every 10 ms over a continuous windowed signal “frame” given a noise and level compression algorithm (eg, as described in US Pat. No. 4,591,928). Energy measurement. This analysis is performed over the entire input signal to maximize processing accuracy and robustness. Other short-term feature measurements can alternatively be used. An example is L. R. Rabiner and RW Schfer (1978) “Digital Processing of Speech Signals,” shown in Prentice Hall.

  Taking the example of pitch determination, the remaining major signal processing steps to be performed in the computer system for the recorded signals and their measured signal function data are:

Method 1
(A) The time-dependent feature sequence of the guide signal and the new signal is processed with a pattern matching algorithm that determines and outputs an optimal time alignment path function as the data sequence. . This path optimally maps the frame of the new signal to the frame of the guide signal.
(B) Data from the time-aligned path is used to edit the new signal and generate a new signal that is time-aligned to the guide signal.
(C) The guide signal is segmented into discrete and continuous frames, and the pitch of each frame is measured. The pitch measurement sequence value is smoothed to provide a guide signal pitch contour.
(D) The processing step in step (c) is repeated for the aligned (edited) new signal to generate its pitch contour.
(E) Correction for each pitch contour value of the guide signal to be applied to each frame of the new aligned signal by dividing and octave shifting by the corresponding pitch contour value for the new aligned signal. Adjusted to produce a correction contour, which is a set of values giving the factor. This correction contour is facilitated to remove any gross errors.
(F) A pitch shift algorithm is used to shift the pitch of the new aligned signal to a value in time and pitch by shifting to a value according to the smoothed correction contour from step (e). Generate a new signal matching to the guide signal of
It is.

  Method 1 employs two editing algorithms in series and measures the pitch of the new signal after the new signal undergoes one editing step. Thus, the quality of the generated output in Method 1 depends on the output quality of the edited signal from step (b). As a result, imperfections in the signal introduced during editing can degrade the output quality of steps (d) and (f). This leads to occasional small errors in the corrected pitch, and in some cases produces a slight roughness in the generated output.

Method 2
In order to reduce the risk of such errors, other embodiments combine steps (b) and (f) described above to produce a single editing stage. Also, any characteristic of the new signal (pitch in this example) can be measured from the unmodified new signal, not from the time aligned (edited) version. This is accomplished by calculating the inverse of the time alignment path. The reverse path maps each frame of the new unedited signal to its corresponding frame of the guide signal. From this mapping, a pitch correction contour is calculated for the new signal, aligned with the guide signal in time. In effect, the guide signal is aligned with the new signal in time before the pitch correction contour is calculated.

The following steps summarize the method.
(A) a sequence of features dependent on the time of the guide signal and the new signal for determining and outputting the optimal time aligned path feature as a data sequence that optimally maps the new signal frame to the frame of the guide signal; Processed by a matching algorithm,
(B) data from the time aligned path is used to generate an inverse path function that maps a frame of the guide signal to a corresponding frame of the new signal;
(C) The guide signal is segmented into discrete frames and the pitch of each frame is measured. The pitch measurement sequence value is smoothed to give a guide signal pitch contour,
(D) the process in step (c) is repeated for a new signal (unedited) to generate its pitch contour;
(E) Using an inverse path function to align the guide signal pitch contour to the new signal pitch contour and each pitch contour value of the mapped guide signal is divided by the corresponding pitch contour value for the new signal And adjusted for octave shift to produce an aligned correction contour that is a set of values that gives a correction factor to each frame of the new signal. This aligned correction contour is smoothed to remove any gross errors.
(F) A new signal required to generate an output signal aligned with the guide signal in time and pitch using a time aligned path function and a smoothed aligned correction contour. The new signal is edited using a processing algorithm that shifts the pitch and also compresses the time or decompresses the new signal.
As an alternative to (g) or step (f), a smoothed and aligned correction contour can be applied without time alignment from the new signal to the guide signal. This can maintain the original timing of the new signal, but will apply pitch correction to the correct frame of the new signal, even if the new signal is not aligned with the guide signal in time. Let's go.

  Any form of Method 2 can accurately follow subtle nuances such as vibrato and other details and reproduce it, more reliable and natural voice pitch through all words and phrases Provide corrections.

Method 3
Method 2 edits the new signal only once, but it utilizes processing techniques that simultaneously correct the pitch and time alignment. By slightly changing the sequence of steps, it is possible to handle pitch shifting and time correction separately without using Method 1. This introduces two stages of editing, but the most appropriate specialized processing algorithm can be selected separately for each stage.

The following steps summarize this third method.
(A) A pattern matching in which a time-dependent function sequence of the guide signal and the new signal determines and outputs an optimal time alignment path function as a data sequence that optimally maps the new signal frame to the frame of the guide signal.・ Processed by algorithm,
(B) The guide signal is segmented into discrete frames and the pitch of each frame is measured. The pitch measurement sequence values are smoothed to provide a guide signal pitch contour.
(C) Processing step (b) is repeated for the new signal (unedited) to generate its pitch contour.
(D) Using a time aligned path function, the new signal pitch contour is effectively time aligned to the guide signal pitch contour.
(E) Each guide signal pitch contour value is divided by the corresponding time aligned new signal pitch contour value, and the result is adjusted for octave shift. This produces an aligned correction contour that includes a correction factor to apply to each frame of the new time aligned signal. This aligned correction contour is smoothed to remove any severe errors.
(F) Data from the time alignment path is used to edit the new signal and generate a new signal time aligned to the guide signal.
(G) Using a pitch shifting algorithm, the pitch of the new time aligned signal is shifted by the smoothed and aligned correction contour generated in step (e). This gives a new edited signal that is aligned with a given guide signal in time and in pitch.

  Method 3 uses the original time-aligned path function, not inverse. Furthermore, it has the advantage that, as in method 2, the pitch of the new unmodified signal is measured rather than the time aligned (edited) version of the pitch. However, it cannot correct the pitch of the new signal (step g) without first generating a time aligned version (step f).

In a further embodiment, other features of the audio signal other than the pitch can be modified to cause the features in the guide signal to follow once the time alignment function has been generated. Additional types of time-synchronizable modifiable features include instantaneous volume, equalization, speech format or resonant pattern, reverberation, and echo characteristics, and even analysis of specified features This includes the modification of audio signal features, such as the word itself, if an appropriate mechanism for modification is available.

  In the present invention, a video signal is not necessary. And the input audio signal may only be needed to accompany or replace other audio signals.

In a preferred embodiment of the present invention, the time between the time-varying feature of the first (guide) audio signal and the corresponding second (new) audio signal, which is optimally and fully detailed, Means are included for determining a time aligned function or time warping path that may provide a mapping. This mapping ensures that the time-varying alterations are based on specified features in the portion of the guide (control) signal that corresponds to the appropriate portion of the new signal being modified. The measurement of a particular time-varying feature used to determine the time alignment is a short portion or window of the sampled signal waveform (each window is a period of T ′, where T ′ is T For every T seconds. Measurements are usually made on a continuous frame-by-frame basis with overlapping sampling windows. This is a “short-time” signal analysis, as described in LRRabiner and RW Schfer (1978) “Digital Processing of Speech Signals” Prentice Hall.

  Note that the features measured for the time alignment process are likely to be different from both the modified features and the features used as controls. The basic relationship between the features to be changed and the control feature parameters must be defined. For example, one simple relationship that will be described in more detail hereinafter is that the pitch of the new signal is changed to the pitch of the guide signal, with adjustments to maintain the natural pitch range of the person generating the new signal. Modify to match. If necessary, this definition of the correction function and other definitions can be changed over time. The correction function is an output value in a speech processing computer system. versus. It can be programmed as a data array of input values, as a mathematical function, or as a set of processing rules. Note that the function does not necessarily depend on the signal itself, so the signal may not require any analysis. In a further step, both the feature designated to be modified with the second signal and the designated control feature in the first signal are measured as a function of time. These measurements are stored as data.

  Computer systems that can simultaneously reproduce background audio and / or video signals from digitized computer video and audio files while recording audio input are well known. A typical PC system component and environment capable of supporting these functions is shown in FIG. 1 of the accompanying drawings, which is the basis for providing a hardware and software environment for embodiments of the present invention. Can be used with the software of FIG.

In FIG. 1, user interface hardware including a CPU (Central Processing Unit) 112, a RAM (Random Access Memory) 118, a pointing device 120, typically a mouse, a keyboard 125, and a display screen 130. An internal storage device 140 such as a hard disk or further RAM, a device 160 for accessing data on a fixed or removable storage medium 165 such as a CD ROM or DVD ROM, and optional Illustrated is an environmental computer system 100 comprising a computer 110 having a modem or network interface 170 for providing access to the Internet 175. The pointing device 120 controls the position of the displayed screen cursor (not shown) and the function section displayed on the screen 130.

The computer 110 may be any conventional home or business computer, such as a PC or Apple Macintosh, or alternatively a Microsoft® Xbox ^™ with a pointing device 120 that is a game controller device. It can also be a dedicated “game device” such as Sony Playstation2 ^™ . Some components shown in FIG. 1 may not be present on a particular gaming device. FIG. 2 shows additional software that may be installed on the computer 110.

  A user may obtain a digital data file 115 containing audio and optional attached video clips (clips) from CD ROM, the Internet, or other means. The digital data file 115 can be a widely used format, such as avi or QuickTime® movie format, and the digital data file 115 is, for example, on the hard disk 140 or Copied and stored in RAM. Computer 110 is a known operating system 135, sound card 150, such as that provided by any available version of Microsoft® Windowa® or Mac® OS. Or a DAC (digital-to-analog) connected from one or more loudspeakers 156 for playing back audio, including an ADC (analog-to-digital converter) connected from a microphone 159 for recording. Audio software and hardware in the form of equivalent hardware on the computer's motherboard, including the converter).

  As shown in FIG. 2, such an operating system 135 is typically an audio recording and editing software 18 (Windows () that supports audio recording and editing functions via a sound card 150. (Such as a “Sound Recorder” application program) that is shipped with a registered trademark). A recording program and / or other program can convert an incoming analog audio signal into digital audio data and record the data to a computer file on the hard disk drive 140. 150 can be used. Audio / video player software 190, such as Windows® Media Player, shipped with Windows® and / or other software, sound card 150, additional built-in video hardware and software It can be used to play a composite digital video and audio file, or just an audio file, via the display screen 130 and the speaker 156. Composite video and audio files consist of video data and one or more tracks of parallel synchronized audio data. Alternatively, the audio data may be maintained as a separate file assigned to store multiple streams of audio data. The audio data can be utterance data such as conversation, singing, instrumental music, “sound effects”, or some combination thereof. Blocks 180 and 190 may also represent software and hardware that, in conjunction with 135 and 110, may implement the song processing system described herein.

  Alternatively, distributed embodiments of hardware and software systems at 100 and 110 may be employed. One example is such that the main elements of the computer system 100 are provided to the user by a remote server. In such a case, in a state where an analog or digital audio signal is transmitted between the user and 100, on the user side, a telephone system network, and a telephone or microphone and speaker connected to the user's PC system, and Alternatively, input and output transducers 159, 156 may be provided via the Internet. A user can remotely control system operations by a vast number of methods including a telephone touchtone keypad, computer keyboard, voice input, or other means.

  Embodiments of the present invention in the form of a non-real-time consumer karaoke system record the voice of some public singing pop songs along with music videos to a computer-based system Make it possible. When the user's recorded voice is modified and then played back, the modified voice is lip-synchronized with the mouth movements of the original singer and the voice of the replaced singer in the music video Have the same pitch variation. The system of FIG. 2 allows for audio playback of the original performer song singing with or without accompanying video. The user can play the song and the system digitizes the user's voice and records (stores) it on a computer hard disk or other memory device. It is preferred that the voice signal be separate from the background music track, as there is a need to accurately measure the original singer's voice features. This can be most effectively accomplished by requesting a voice-isolated recording from a record company or organization that provides media content.

  In this embodiment, a digitized recording of a singer performing the song in isolation (eg, a solo vocal track transferred from a multi-track recording from the original recording session). The first signal (guide signal), preferably without additional processing such as echo or reverberation, is used. Such a digitized guide signal, g (n), may be provided to the user's system on a CD or DVD / ROM 165 or via the Internet 175. Alternatively, in further embodiments, the required features of the guide signal (for both time alignment and feature modification control) can be pre-configured in the same or other systems to extract the required data. Can be analyzed. This data may be input to the system 100 for use as a data file via 165, 175 or via other data transfer methods. An example data storage and processing module is shown in FIG.

  A user running a voice recording and playback program can play a desired song while the original singer is audible or inaudible, and can sing at the same time. The user's song is digitized and recorded in a data file in the data storage unit 310. This digitized signal is the second signal, the new signal, s (n).

  The embodiment of FIG. 3 implements Method 1 described below. The purpose is to correct the pitch and timing of the user's new signal to mimic the pitch and timing of the guide signal. In this case, the feature in the guide signal is used as a control function, and the feature modified by the new signal is the same feature, ie the pitch contour of the respective signal. The process of tracking the difference between the new time-aligned signal pitch measurement and the guide signal pitch measurement is a calculation of the pitch adjustment function so that the corrected new signal pitch follows the guide signal pitch. Used in. Here, it is assumed that the new signal, s (n), is similar to the guide signal, g (n), in phrasing, content, and length. For non-real-time karaoke-type applications, this is a reasonable assumption. This is because the user usually tries to simulate the original singing performance in timing, pitch, and words.

  Here, method 1 is performed on digital audio data in non-real time as follows.

<Input signal description and measurement>
There is very little that the new and guide signals are properly time aligned without processing. U.S. Pat. No. 4,919,928 (Bloom et al.) Describes differences between energy patterns of similar speech signals that are not time aligned but similar to the energy of the filter bank output as an input to the time alignment process. Explain the use of related measurements.

  FIG. 4 shows a function of pitch measurement frame number M (where M = 0, 1, 2,... N), hereinafter referred to as pitch contour 401, obtained by measuring the pitch of a professional female singer's guide signal. Of the time series Pg (M) and the time series Ps (M) shown as pitch contour 402 of a new signal (male voice) of a general amateur before time alignment, along the same time scale Show. Differences in the pitch contour of both signals and their misalignment in time are obvious. In time, the second series, the first series that is not aligned with Ps (M), Pg (M), will produce a large and audible error without generating a large audible error. Cannot be used directly as a control or target pitch function.

  A data point indicated as zero HZ in pitch contour 401 or 402 indicates that the corresponding pitch measurement frame contains either silence or unvoiced speech. A non-zero measurement indicates a pitch measurement of each signal in that frame.

  In FIG. 4, the non-zero value segments (pulses) of the uttered sound in the new signal pitch contour 402 are generally delayed relative to the corresponding features in the guide signal pitch contour 401, and , With different durations. In addition, the voiced voices of the two pitch contours are in different octaves. Further, the pitch range variation in each pulse of the guide signal pitch contour 401 is wider than in the corresponding pulse in the pitch contour 402 of the new signal. This is expected because the guide signal pitch contour 401 is taken from a professional singer. Such details and timing of the guide signal pitch contour 401 are provided to the recorded song of the amateur user.

<Time alignment of new signals>
In FIG. 3, the new sampled signal waveform s (n) read from the data store 310 is first obtained from the data store 312 in time using techniques such as those described in US Pat. No. 4,591,928. Aligned with the read guide signal, g (n), produces an intermediate audio signal, ie a new time-aligned signal, s ′ (n) (which is stored, for example, on disk 330). This ensures that the detail of the energy pattern in s ′ (n) occurs at the same relative time as that in the guide signal. This also means that any lip-synching is valid, and any transfer of features from the guide signal to the new signal does not require further time mapping. Also guarantee. The sampling frequency used in generating the new signal, s (n) and guide signal g (n) in this example is 44.1 kHz.

  The time alignment process described in US Pat. No. 4,591,928 measures spectral energy features (eg, filter bank output) every 10 ms, has a path point every 10 ms, and is new A time alignment or “time warping” path is generated that associates similar spectral features in the signal with the closest corresponding feature in the guide signal.

  FIG. 5 shows that each feature frame of the new signal has a frame number j, each feature frame of the guide signal has a frame number k, and the frame sampling interval is T seconds (where An example of a time warping path, w (k), k = 0, 1, 2,. Such a warping path is generated within the time alignment processing module 320, which is a new signal in module 320 in generating a time aligned new signal s ′ (n) stored on disk 330. Used to control editing (ie, time compression / decompression) of s (n). As shown in US Pat. No. 4,591,928, a new time-aligned signal, s ′ (n), is generated by module 320 by building an edited version of s (n). Note that in this edited version of s (n), the s (n) portion has been iterated according to w (k) and additional timing error feedback from the editing system. Or it has been deleted and this edited version of s (n) is limited to making pitch-synchronized editing when utterances are present.

<Generation of new signal pitch contour>
The raw pitch contour, Ps ′ (M), of the aligned new signal, s ′ (n), is taken of s ′ (n) using the Haan window of motion analysis in successive discrete pitch measurement frames. Generated from measurements (where M is the frame number and M = 1, 2, 3,...). In order to obtain an accurate pitch measurement, it is recommended that the length of the analysis window be 2.5 to 3.0 times the lowest period length to be measured. Therefore, in this embodiment, an analysis window (or about 35 ms) of 1536 samples (at a 4401 kHz sampling frequency) is used to measure a pitch as low as 72 Hz with a period of about 0.0139 s. The sampling interval of the pitch measurement frame is 10 ms. The analysis window of the pitch estimator module 340 is centered within each pitch measurement frame of the sample. For each pitch measurement frame, pitch estimation is performed using one of the well-known methods for pitch prediction (eg, autocorrelation, comb filtering, etc.). A detailed description of these techniques can be found in Wolfgang Hess (1983) "Pitch Determination of Speech Signals, Algorithms and Devices", Springer-Verlag; RJMcAulay and TFQuatieri, (1990); "Pitch estimation and voicing detection based on a sinusoidal model, "Proc. Int Conf. On Acoustics, Speech and Signal Processing, Albuquerque, NM, pp. 249-252; and TFQuatieri (2002)" Discrete-Time Speech Signal Processing; Principles and Practice, "In references such as Prentice Hall Can be found.

  Measurements can be taken without analysis window overlap, but continuous windowed data overlap of between 25 and 50% is generally recommended. In this example, the measurement frame rate of M is 100 Hz (ie, a 10 ms interval), which provides sufficient overlap while at the same time conveniently measuring the time alignment function. Same as rate. In order to correctly measure the first and last few pitches (where the analysis window naturally exceeds the available data samples), before starting those pitch measurements, Both are padded with the length of one analysis window of a maximum, zero magnitude sample.

  To generate the final smoothed pitch contour (Ps ′ (M) for the new time aligned signal), the filter module 350 uses a three-point median filter and then an averaging filter. , The pitch measurement of the individual frames is facilitated. Furthermore, silence and unvoiced frames of the new time aligned signal s ′ (n) are marked as having a zero pitch in P ′s ′ (M).

<Guide pitch contour generation>
Similarly, the pitch estimator module 345 uses the same method and parameters as described for generating the pitch contour Ps ′ (M) and uses the pitch contour Pg ( M) is generated and smoothed in the filter module 355 to generate a smoothed pitch contour P′g (M) for the guide signal.

<Pitch adjustment calculation>
The next step is the calculation of the pitch adjustment or correction factor for each frame of the new time aligned signal. This is done by the pitch adjustment module 370 and takes into account the ratio of the new signal pitch time aligned with the guide signal pitch, and any desired octave shift. This calculation is done for each set of pitch measurement frames having the same frame number M. The low pass filter in module 370 then smoothes the correction factor. There are two steps: determining the octave and shifting the pitch of the new signal. There are two main options, taking into account the adjustment of the pitch: (a) adjusting the output pitch to be the same as the pitch of the guide signal, or (b) the adjusted voice. Is to maintain the pitch range of the new input signal so that it sounds the most natural. The octave adjustment for realizing the latter effect will now be described. The octave adjustment module 358 calculates an octave multiplier Q (which is kept constant during the period of the signal). This emphasizes that it is necessary to analyze all or at least a substantial amount of the new signal before this value can be set.

For each pitch analysis, a time aligned new signal frame M (unsmoothed pitch prediction for frame M from pitch predictor modules 350 and 355) is used to produce local pitch correction C _L ( M) (where M is the frame number) and the calculations on those frames are performed if both the time-aligned new signal and its corresponding guide signal frame are spoken, i.e. these Only when both frames have an effective pitch. In these frames, the local pitch correction factor C _L (M) (which makes the pitch of frame M of the new time aligned signal the same as the pitch of frame M of the guide signal) is

CL (M) = Pg (M) / Ps ′ (M) (1)

Represented by

Each ratio C _L (M) is then rounded to its nearest octave by selecting a powers of 2 according to the following table.

All the resulting octave values are entered into the histogram and then the most frequently occurring octave correction value Q is selected. In some embodiments, Q is not a function of time, but a function of time. If necessary, Q can be multiplied by other factors to achieve some desired offset in pitch frequency. The calculation of Q is performed in module 358. The octave correction value Q is provided to the pitch adjustment module 370 and used in equation (2) below to generate an octave corrected pitch correction factor C (M).

C (M) = P′g (M) / (Q * P ′s ′ (M)) (2)

Where C (M) is the pitch correction factor in the frame M of the signal,
P ′s ′ (M) and P′g (M) are the smoothed predicted pitch in frame M of the new time aligned signal and the guide signal, respectively.

  To generate the pitch correction signal, from Equation (2), for every frame of the new time aligned signal, the pitch correction factor C (M) is calculated to produce the modified time aligned new signal. The pitch register of the current signal is made to closely match the pitch register of the original new signal.

  If there is no corresponding guide signal pitch in frame M (i.e., the guide signal is unvoiced or the time-aligned new signal is slightly longer than the guide signal) The last correction factor value in M-1 is reused. In this case, it is also possible to obtain a better prediction using interpolation.

  In the example of the correction value obtained, a correction factor C (M) of 1.0 means that there is no change to s ′ (n) in frame M, and 0.5 means a lower pitch by one octave. 2.0 means that the pitch is increased by one octave.

<New signal shift pitch>
Each value C (M) in the pitch correction signal gives the correction multiplication value required for the corresponding frame M of the samples of the new time-aligned signal s ′ (n). In this example, the frame rate of C (M) is selected to be the same as that used by the time alignment algorithm (which is 100 frames per second (ie, 100 fps)). In other words, C (M) will have 100 samples every s ′ (n) every second.

  In order to function correctly, some pitch shifting algorithms must have a frame rate that is much smaller than the frame rate of the time-aligned algorithm (ie, the sampling interval (analysis frame) is , Very long). For example, time-domain pitch shifting techniques typically have a frame rate of about 25-30 fps (if they are to operate at frequencies of 50-60 Hz). However, their frame rate need not be constant throughout the signal. The rate can then vary with the basic pitch of the signal s ′ (n), for example. However, in this embodiment, a fixed frame rate is used for pitch shifting.

  In this embodiment, the operation of the respective frame rate and pitch shifting algorithm for the calculation of the pitch correction factor C (M) is different, so linear interpolation is used, and the pitch in time A prediction of the pitch correction required at the center of each analysis frame of the pitch shifting algorithm is derived from the C (M) sample closest to the center of each analysis frame of the shifting algorithm. This interpolated correction factor is derived as follows:

The frame M of the pitch correction signal has a length equal to the Lc sample of the new signal s (n). Where Lc is given by:

Lc = sampling rate of new signal s (n) / frame rate of C (M)
(3)

  The sample number along s ′ (n) at the center of each analysis frame of the pitch shifting algorithm (where prediction of pitch correction is required) is determined as follows.

If Nc (Fps-1) is the sample number along s' (n) at the center of the pitch shifting analysis frame Fps-1, the sample number Nc (Fps) at the center of the next frame Fps is ,

Nc (Fps) = Nc (Fps−1) + Ls (Fps, T ₀ (Fps−1)) (4)

It is. here,
Fps is the pitch shifting analysis frame number, Fps = 0,1,2, ...
Ls (Fps, T ₀ (Fps−1)) = (New signal sampling rate) / (Pitch shifting algorithm frame rate)
It is.

In this general case, Ls is a function of the frame numbers Fps and T ₀ (Fps-1) (which is the pitch period duration in Fps-1), allowing a time-varying frame rate. . In this embodiment, Ls is held constant and is set to 1536 samples, ie 34.83 ms.

Sample number along s ′ (n) in the middle of both the pitch shifting analysis frame (Nc (−1)) before the first calculated frame and the first calculated frame Nc (0) The initial value for depends on the pitch shifting algorithm. In this example, Nc (−1) = 0.5 * T ₀ (−1) and Nc (0) = 0.

Nc (Fps) and Lc are used to bind or contain the sample at the center of a particular analysis frame Fps in the pitch shifting algorithm C (M) pitch correlation frame number Fc (M) is

Fc (Fps) = Nc (Fps) / Lc (5)

It is. here,
/ Means integer division,
Fc (Fps) is a C (M) frame that occurs just before or in the middle of the pitch shifting algorithm frame Fps.
Lc is as defined above.

  If Fc (Fps) is a pitch correction frame occurring just before or in the middle of the pitch shifting algorithm frame, (F (Fps) +1) occurs after that center. This is the next pitch correction frame.

Linear interpolation between pitch corrections C (Fc (Fps)) and C (Fc (Fps) +1) gives an interpolated correction factor Cs (Fps) at the center of the pitch shifter analysis frame, and pitch pitch Control the shifter.

Cs (Fps) = C (Fc (Fps)) * (1−α) + α * C (Fc (Fps) +1) (6)

here,
α = (Nc (Fps) −Lc * Fc (Fps)) / Lc
And where
/ Means integer division,
Other symbols are those described above.

  The interpolated correction factor value Cs (Fps) is smoothed by simple low-pass filtering to become C's (Fps) as the output of module 370 (given to the pitch changer module 380 (pitch changer moduke)). Expressed. For pitch correction, the new time aligned signal s ′ (n) is processed in a frame Fps corresponding to the pitch shifting algorithm frame. Each such frame Fps of the new time-aligned signal s ′ (n) is dynamically shifted in pitch by the smoothed correction factor in module 380 and the resulting pitch corrected time The aligned new signal s ″ (n) is written to disk 390 for later playback with background music and optionally a corresponding music video when available. This output signal s '' (n) will have both the required time alignment and pitch correction to be reproduced as a replacement for or in synchronization with the guide signal g (n). The time-aligned signal that will be observed at s ″ (n) as a result of multiplying the pitch value of the new time-aligned signal s ′ (n) shown in FIG. 6 by the corresponding correction factor value. An example of the corrected pitch contour 701 is shown in FIG. Most of the details of the guide signal pitch contour 401 now appear in this example of the calculated modified pitch contour 701.

  Pitch shifting performed by module 380 to generate pitch corrected time aligned output signal waveform s '' (n) in storage 390 is described by K. Lent (1989), “An efficient method for pitch shifting digitally sampled sounds, "Computer Music Journal Vol. 13, No.4, 65-71; N. Schnell, G. Peeters, S. Lemouton, P. Manoury, and X. Rodet (2000)," Synthesizing a choir in real-time using Pitch Synchronous Overlap Add (PSOLA), "International Computer Music Conference, pp. 102-108; J. Laroche and M. Dolson (1999)," New Phase-Vocoder Techniques for Pitch-Shifting, Harmonizing and other Exotic Effects. "Proc. 1999 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, pages 91-94; G. Peeters (1998)," Analyse-Synthese des sons musicaux par Ia methode PSOLA, "Proceedings of the Journees d'lnformatique Musicale, Agelonde, France; and V. Goncharoff and P. Gries (1998), "An algorithm for accuratel Standards such as TDHS, PS-OLA, FFT explained in references such as "y marking pitch pulses in speech signals", Proceedings of the IASTED International Conference Signal and Image Processing (SIP'98), October 28-31 It can be implemented using any of the pitch shifting methods.

  In this example, D. Malah (1979) "Time Domain Algorithms for Harmonic Bandwidth Reduction and Time Scaling of Speech Signals", IEEE Transactions Acoustics, Speech and Signal Processing, Volume 27, No. 2, 121-133 A time domain algorithm as described on the page is used in module 380 to shift the pitch of signal s ′ (n).

At every frame Fps of s ′ (n), a pitch period defined here as T ₀ (Fps) is measured. Henceforth, for simplicity, variables based on calculations involving T ₀ (Fps) are also variables of Fps, but the parameter Fps is not made explicit in their representation.

In this example, a new time-aligned signal s ′ (n) is transformed into s ′ (n) h (p) (analysis window function 801 (shown in FIG. 10 (a))) (which is By being decomposed into a sequence of windowed samples s ′ (n) of the signal by multiplying by (periodically shifted)

s ′ (u, n) = h (n) * s ′ (n−ta (u)) (7)

Is obtained. here,
h (p) is the pitch shifting analysis window of the length of the P sample, and the length of the P sample in time is twice the measured pitch period of the frame Fps, ie 2 * (Fps) Is equal to In this example, h (p) is a Hann window of P samples.

Also, ta (u) is the u th analysis instance set at the pitch synchronization rate for the uttered frame, ta (u) −ta (u−1) = T ₀ (Fps), where , U = 0,1,2,... For unvoiced frames, ta (u) is set to a constant rate of 10 ms. It can also be set to the last valid value of T ₀ to form a spoken frame.

From the smoothed pitch correction C ′s (Fps), a new output period T ₀ ′ (Fps) of the corrected signal is calculated. For non-speech signals in the frame Fps, T ₀ ′ (Fps) = T ₀ (Fps). For voiced signals in frame Fps,

T ₀ '(Fps) = T ₀ (Fps) / C's (Fps) (8)
It is.

From this process, a sequence 802 of the short-term synthesis window ts (v) is generated. This sequence 802 is synchronized with a new output period T ₀ ′ (Fps),

ts (v) −ts (v−1) = T0 ′ (Fps) (9)

become that way. Here, ts (v) is the vth composite instance in the output frame.

  As illustrated in FIGS. 10 (a) and 10 (b), for each ts (v), that window ta (u) of s ′ (n) data closest in time is selected. The selected window ta (u) of s ′ (n) data is then added to an output stream buffer (not shown) to convert the output signal stream s ″ (n) to all of one frame Fps. One frame at a time is generated by a known method of overlap and addition combining short-term composite windows ts (v). Effectively, the windowed sample s ′ (u, n) is recombined with the pitch period of T0 ′ (Fps) rather than being recombined with the period of T0 (Fps). .

Further embodiments are described.
In addition to the pitch including vibrato and inflection curves, many other features of the speech signal are measurable and can be modified. Examples of these are instantaneous loudness, glottal sound characteristics, speech format or resonance pattern, equalization, reverberation, and echo characteristics. Furthermore, the new and guide signals are not necessarily limited to having prosodic, rhythmic or acoustic similarities.

  In FIG. 8, a feature analysis operation is shown that operates on new and guide signals in modules 840 and 850, respectively, to generate fs (N) and fg (M). These are shown bold as feature vectors specifying the selected features, measured in frames N and M, respectively. The vectors need not have the same characteristics. While fg (M) must contain at least one feature, in a further embodiment, fs (N) can be a null vector with no features.

  A feature adjustment function, A (fs (N), fg (M), M) must be provided, here the input from the source 865 to the system as a specification. This function is the desired relationship between the feature vectors of the two signals in frames N and M, where these two can be the same frame or not the same frame, It defines the elapsed time as expressed by the frame parameter M and the time-varying signal correction process implemented in software and applied in module 870. This function and variation is generally defined and entered by the system programmer and can be subsequently presented as a set of presets and / or provide user defined variations that can be selected by the system user Will be able to.

  An example of using two different features in A (fs (N), fg (M), M) is the guide signal under the condition that the new signal contains energy in the band of the moving bandpass filter. Loudness is included to control the central frequency of the moving bandpass filter process in the new signal. Making A a function of M also generalizes that the process includes a possible time-based modification to the function.

  Another embodiment that employs previously described Method 2 is shown in FIG. 9A. Here, a new time aligned signal waveform is not generated as the first step. Instead, the time alignment data obtained in module 920 as in the embodiment of FIGS. 3 and 8 is used, and in module 960, the measured characteristics of the guide signal are applied to the appropriate times in the new signal. ) Is time distorted. Module 970 makes time aligned corrections to the new signal. Selective time alignment can be performed simultaneously (combining the processing of modules 970 and 975 into one algorithm) in the feature modification step module 970, or in a new signal that has been modified, or in a subsequent process module 975 It can be performed on the feature-modified signal. Further details of this approach are given below.

The inverse of the time alignment function in FIG. 5 maps the matching frame of the guide signal at frame k to each frame of the new signal at frame j. If Fs is the frame number of the new signal and W (Fs) is the (inverse) time warping function (or mapping function) generated by the time alignment process module 920,

Fag (Fs) = W (Fs) (10)

Where Fag is the corresponding frame number of the time-aligned guide.

From this mapping, a time aligned or warped version of the feature adjustment function is generated and used in the adjustment module 960 of 9A (not shown).
As an example, returning to the application in pitch correction, based on equation (1), the warped version of the pitch correction function is

C (Fs) = Pg (Fag (Fs)) / Ps (Fs) (11)

Is calculated as
From (10) and (11),

C (Fs) = Pg (W (Fs)) / Ps (Fs) (12)

Here, C (Fs) is a correction factor of the frame Fs of the new signal.
Ps (Fs) is the predicted pitch of the new signal frame Fs. W (Fs) is the corresponding frame in the guide from the warping function. Further processing of C (Fs), previously described, including octave correction (if necessary) is performed in the adjustment module 960. This adjustment module 960 is then based on equation (2)

C (Fs) = P'g (W (Fs)) / (Q * P's (Fs)) (13)

Provides the correction function given by.

  This modification function is applied to s (n) on a frame by frame basis in the modification module 970 to produce a modified output s * (n).

  To allow any signal feature to be specified for analysis and modification, the process shown in FIG. 9A is generalized as shown in FIG. The difference is that the output s * (n) has the timing of the original new signal s (n) instead of being time aligned with the guide signal. The time alignment of the modified output s * (n) to the guide signal g (n) is relative to the pitch correction in a single step such that the feature correction in module 970 and the time alignment in module 975 are performed simultaneously. Can be realized. For example, a description of a method for performing pitch and time correction at the same time (which can reduce potential processing artifacts and improve computational efficiency) can be found in J. McAulay and T. Quatieri (1992), " Shape Invariant Time-Scale and Pitch Modification of Speech ", IEEE Trans. Sig. Processing, IEEE Trans. Sig. Processing, March, Vol. 40 No. 3, pages 497-510 and D. O'Brien and A. Monaghan (1999), “Shape Invariant Pitch Modification of Speech Using a Harmonic Model”, EuroSpeech 1999, pages 1059-1062. These references assume either an arbitrary constant pitch shift or a constant pitch shift based on measurements of the original signal to determine the amount of shift to apply. For example, if a non-voicing frame is detected in the original voicing waveform, it is normal to switch off, or at least reduce, any time or pitch correction applied during that frame. It is the way.

  Optionally, a normal time alignment function can also be applied to the non-linear editing process in module 975 to generate the signal s ′ * (n). This s ′ * (n) is a time aligned version of the new signal s * (n) with modified features.

Another embodiment for performing method 3 is illustrated in FIG. 9B. In this figure, the time aligned signal s ′ (n) in the storage module 982 is generated by module 975 using the original time aligned path generated in module 920. In this arrangement, a new signal feature contour is generated by module 840 from an unmodified new signal s (n) and a guide signal feature contour is generated by module 850. In module 960, the formula

C (M) = P'g (M) / Q * P's (w (M)) (14)

(Where w (M) is the time warping path generated by module 920) is executed to generate a feature modified contour C (M). This modified contour is applied at module 972 to the new time aligned signal to generate a new signal s * (n) that is time aligned and feature corrected at the output storage module 987.

  In a further embodiment, instead of one continuous signal, the guide signal may consist of a series of different individual signals. Alternatively, multiple guide signals (eg, harmony vocals) can be used to generate multiple vocal parts from a single new signal.

  In a further embodiment, the features in the new signal need not be measured or need to be input into the new signal feature adjustment calculation, simply based on the measurement of the feature of the guide signal or features. Can be modified. An example of this could be the reverberation or application of EQ to a new signal as a function of their characteristics in the guide signal.

  The processing modules used in the above embodiments are software modules when implemented in a system such as the system 100 of FIGS. 1 and 2, but in alternative implementations are hardware modules, or It will be understood that it can be a mixture of hardware and software modules.

  One application of the present invention is to generate a personalized voice file with a user voice that can provide, for example, a telephone ring tone in a cell phone or computer-based telephone system. is there. Other examples include replacing some ringing tone or other voice that may be presented to a caller or call recipient during a telephone call or other data exchange. Such an exchange may be done via a telephone network, a VOIP (Voice over Internet Protocol) system, or other message delivery system. Further examples include the generation of personalized audio files for any device or system that may use personalized prerecorded messages.

  FIG. 11 illustrates an embodiment of the present invention that allows a user to generate, ship and receive such audio files. In operation, a user initiates a telephone call via telecommunications network 1140 from a landline handset 1110 or a mobile telephone handset 1120. A suitable converter 1150 receives the signal from the telecommunications network 1140 and converts it into a digital voice signal and operational command tones (which are processed by the server computer 1160). Server computer 1160 operatively provides an interactive voice response (IVR) from module 1165 to provide the user with feedback on selection and operation.

  Server computer 1160 may be implemented in one or more computers and may incorporate a voice processing module 1170 for performing the steps described in FIG. 3 or 8 or 9A or 9B. The computer 1160 accesses the storage module 1180 to store song audio files and accesses a database to reference the song files. The computer 1160 also stores in the storage module 1185 the original and processed user voice recordings and a database for referencing those recordings.

Server computer 1160 interprets the touchtone or other signal and initiates operation. For example, with the telephone keypad in this implementation, the user can access the computer 1160
(a) Choose to select “track” (eg, part of a song (stored in module 1180)),
(b) The user transmits the selected track via the converter 1150 and the network 1140 to the telephone handset 1110 or 1120 in order to hear it (selected track) and rehearse against it.
(c) the selected track is replaying through the telephone handset 1110 or 1120 and the user's voice is recorded while the user sings into the handset microphone;
(d) play a processed recording of the user's voice mixed with an appropriate background track (eg, a version of the original singer's voiceless track);
Can be ordered.

  In step (c), the user's voice is recorded in the storage module 1185 and processed through a processing module 1170 that performs processing such as that shown in FIG. 3 or 8 or 9A or 9B, and the result is the module 1185. Is remembered.

  Finally, the user then enters the recipient's mobile phone number on his / her handset 1110 or 1120 keypad. The computer 1160 then sends the data message to the recipient's number using a ring tone delivery system 1190 such as a “WAP push” system. This data message gives the recipient the information needed to download the processed voice to his mobile phone or other device.

  In an alternative implementation, the user's computer 100 having a microphone 159 and speaker 156 is used to access the server computer 1160 directly over the Internet 175 or by telephone call using VOIP software 1135. The user may then perform the same procedure as previously described, but using the computer 100 to listen, record and send commands entered on the keyboard 125 (not shown) of the computer 100 to the server computer 1160. send. The user can finally designate a mobile phone by its number to receive the audio file generated through the delivery system 1190. The audio file may also be used on the user's computer 100 or other designated computer (such as a friend's computer) as a ring tone or other identification sound file in the designated computer's VOIP system.

  In other alternative implementations in which a user accesses server computer 1160 over the Internet, some or all of the processing modules of FIG. 3 or 8 or 9A or 9B are represented by module 1130, It can be downloaded to the user's computer 100. Audio files obtained as a result of the use of the module 1130 with or without the assistance of the voice processing module in the server computer 1160 and stored in either the user's computer 100 or the storage module 1185 are Internet 175 or telecommunications network 1140 may be sent to the requested destination phone or other personal computer.

  In a further embodiment, the processor is implemented in whole or in part in a telephone or in some other device including a computer system and memory and means for inputting and outputting the requested audio signal. Can be done.

  In a further embodiment, the server computer 1160 may provide a video signal (such as a music video) along with an audio file of a song received by the user. The user can reproduce these audio and video signals, and can perform audio recording as described above. The processed file, mixed with the background track and synchronized video, is delivered to a designated phone, personal computer, or other device capable of playing audio / video files.

  An audio file of a song is not limited to a song, and can be any audio recording including speech, sound effects, music, or a combination thereof.

FIG. 2 is a block diagram of a computer system suitable for use in the practice of the present invention. FIG. 2 is a block diagram illustrating additional software components that may be added to the computer of FIG. 1 to implement the present invention. FIG. 4 is a block diagram of one embodiment of the present invention showing a signal and processing module used to generate an output audio signal with pitch adjustment based on input signals having different pitch and timing characteristics. Pitch measurement as a function of time for a professional singer's recorded guide utterance, and the same measurement in a new recorded signal from an untrained user who sings the same song for the same musical accompaniment It is a graph which shows a general example. Fig. 6 is a graph representing a time warping function or algorithm path. FIG. 4 is a graph showing the pitch of the guide signal and the aligned new signal from FIG. 4 (before pitch correction) for the left frequency axis and the calculated smoothed pitch correction factor for the right vertical axis. is there. FIG. 7 is a graph of the pitch of the guide signal and the corrected new signal pitch (shown in the uncorrected state in FIG. 6). FIG. 6 is a block diagram of another embodiment of the present invention showing a signal and processing module used to generate an output audio signal with some general signal feature modification based on time-aligned features of any input signal. FIG. 6 is a block diagram of a further embodiment of the process according to the present invention, wherein the process is characterized in that new signal characteristics are modified with or without simultaneous time alignment to the guide signal. Processing according to the invention, in which a time aligned path generates a new time aligned signal and accurately determines the modification to be made to the new time aligned signal FIG. 6 is a block diagram of a further embodiment having the process used for both providing a mapping function to: FIG. 10 (a) is a graphical representation of an example of the relative position and shape of the analysis window used to generate the signal s ″ (n) using overlap and additive synthesis. FIG. 10 (b) is a graphical representation of an example of the relative position and shape of the synthesis window used to generate the signal s ″ (n) using overlap and additive synthesis. FIG. 6 is a block diagram of a further embodiment of the present invention utilizing a communication system.

Claims (27)

A method for modifying at least one acoustic feature of an audio signal, comprising:
The first sampled audio signal that is the guide signal and the second sampled audio signal that is the new input signal are compared to generate a time-dependent feature in the second sampled audio signal. Determining time alignment data configured to indicate a timing difference between time and the time of occurrence of a time dependent feature in the first sampled audio signal;
Measuring at least one acoustic feature of the first sampled audio signal at a selected location along the first sampled audio signal to generate a sequence of first signal feature measurements; And
Measuring at least one acoustic feature of the second sampled audio signal at a selected location along the second sampled audio signal to generate a second sequence of signal feature measurements; And
Processing the first signal feature measurement value sequence and the second signal feature measurement value sequence according to the time alignment data, and the first signal feature measurement value corresponding in timing to the second signal feature measurement value; Generating a sequence of feature correction data for correction, wherein the processing of the sequence compares the first signal feature measurement value with the second signal feature measurement value, and the feature correction data is derived from the comparison. Including deciding and
Applying the sequence of feature modification data to the second sampled audio signal to modify at least one acoustic feature of a selected portion of the second sampled audio signal. Method. The step of applying the feature correction data uses the time alignment data to generate a time aligned second signal from the second sampled audio signal, and the feature correction data is applied to the time correction data. The method of claim 1, comprising applying to the aligned second signal.   2. The process of claim 1, wherein the processing step includes generating the feature correction data for time aligning the second signal feature measurement using the first signal feature measurement and the time alignment data. Method. Applying the feature modification data to modify the at least one acoustic feature of the selected portion of the second signal by using the mathematical factor or function with the feature modification data. 4. A method according to any of claims 1 to 3, comprising modulating the feature correction data according to a mathematical factor or function. 5. A method according to any preceding claim, wherein the at least one acoustic feature of the first sampled audio signal is pitch. 6. A method according to any preceding claim, wherein the at least one acoustic feature of the second sampled audio signal is pitch. 7. A method according to any preceding claim, wherein the time-dependent characteristic of the first and second signals is a sampled spectral energy measurement. The at least one acoustic feature of the first sampled audio signal is a pitch and the at least one acoustic feature of the second sampled audio signal is a pitch;
The processing step determines a multiplication factor from the ratio of the pitch measurement value of the first signal and the time-aligned pitch measurement value of the second signal, and determines the factor as the feature correction data. Shifting the frequency range of the pitch change in the second sampled audio signal in the modified selected signal portion.
The method of claim 4. Further comprising scaling the multiplication factor by a power of two to change the pitch in the modified selected signal portion according to the power-of-two selection.
The method of claim 8. Measuring at a selected location along the second sampled audio signal;
Using the time alignment data from said second sampled speech signal, a second signal which is time-aligned, in the the second signal, said second sampled speech signal Generating a second signal in which the time of occurrence of the time dependent feature is aligned with the time of occurrence of the time dependent feature in the first sampled audio signal;
The time-aligned second at a position along the time-aligned second signal that is selected such that timing is associated with the selected position along the first sampled audio signal. Measuring the at least one acoustic feature in the signal;
The method of claim 1. The at least one acoustic feature of the first sampled audio signal is a pitch;
The at least one acoustic feature of the second sampled audio signal is a pitch;
Applying the feature correction data comprises:
The time-aligned data is used to generate a time-aligned second signal from the second sampled audio signal, and the feature correction data is converted to the time-aligned second signal. Applying to generate a pitch corrected and time aligned second signal;
The method of claim 1. To correct the pitch in the selected portion of the second signal by using the mathematical factor or function with the feature correction data,
Applying the feature correction data comprises:
The method of claim 11, comprising modulating the feature correction data according to a mathematical factor or function. The function is, the pitch measurement in the first sampled speech signal, the value of the ratio of the corresponding pitch measurement in the second sampled the second sampled audio signals along the sound signal The method of claim 12, wherein the method is a function. The first sampled audio signal that is the guide signal and the second sampled audio signal that is the new input signal are compared to generate a time- dependent feature in the second sampled audio signal. Means for determining time alignment data configured to indicate a timing difference between time and the time of occurrence of a time dependent feature in the first sampled audio signal;
Measuring at least one acoustic feature of the first sampled audio signal at a selected location along the first sampled audio signal to generate a sequence of first signal feature measurements; Means to
Measuring at least one acoustic feature of the second sampled audio signal at a selected location along the second sampled audio signal to generate a second sequence of signal feature measurements; Means to
Processing the first signal feature measurement value sequence and the second signal feature measurement value sequence according to the time alignment data, and a timing corresponding to the second signal feature measurement value. Means for generating a sequence of feature modification data for modification to a value, the processing of the sequence comparing the first signal feature measurement with the second signal feature measurement; Means comprising determining the feature correction data from a comparison; and
Means for applying the sequence of feature modification data to the second sampled audio signal to modify at least one acoustic feature of a selected location of the second sampled audio signal;
An apparatus for modifying at least one acoustic feature of an audio signal. The means for applying the feature correction data uses the time alignment data to generate a time aligned second signal from the second sampled audio signal, and the feature correction data Means for applying to the time aligned second signal;
The apparatus according to claim 14. The means for processing includes : means for generating the feature correction data time aligned with the second signal feature measurement using the time alignment data for the first signal feature measurement. The device described. Applying the feature modification data to modify the at least one acoustic feature of the selected portion of the second signal by using the mathematical factor or function with the feature modification data . 15. The apparatus of claim 14, wherein the means includes means for modulating the feature correction data according to a mathematical factor or function. The apparatus of claim 14, wherein the at least one acoustic feature of the first sampled audio signal is pitch. The apparatus of claim 14, wherein the at least one acoustic feature of the second sampled audio signal is pitch. The apparatus of claim 14, wherein the time dependent feature of the first and second signals is a sampled spectral energy measurement. The at least one acoustic feature of the first sampled audio signal is a pitch;
The at least one acoustic feature of the second sampled audio signal is a pitch;
Said processing means, and the pitch measurement of the first sampled voice signal, from the value of the ratio of time-aligned pitch measurement of the second sampled speech signal, to determine a multiplication factor, the feature Means for including the factor in the application of correction data to shift the frequency range of pitch changes in the second sampled audio signal of the corrected selected signal portion;
The apparatus according to claim 14.   22. The means of claim 21, further comprising means for scaling the multiplication factor by a power of two and changing the pitch in the second modified and selected signal portion according to the power of two selection. apparatus. The means for measuring at a selected portion along the second sampled audio signal;
Using said time aligned data from said second sampled speech signal, a second signal which is time-aligned, in the second signal, the second sampled speech signal time of occurrence of features that depend on the time aligned with the time of occurrence of the feature that depends on the time in the first sampled speech signal comprises means for generating the second signal,
The time-aligned second at a position along the time-aligned second signal that is selected such that timing is associated with the selected position along the first sampled audio signal. The apparatus of claim 14, wherein the at least one acoustic feature in a signal is measured. The position timing has been selected such that is associate is, the timing is aligned with the selected location along said first sampled voice signal, apparatus according to claim 23. The at least one acoustic feature of the first sampled audio signal is a pitch;
The at least one acoustic feature of the second sampled audio signal is a pitch;
The means for applying the feature correction data comprises:
The time aligned data is used to generate a time aligned second signal from the second sampled audio signal and the feature correction data is applied to the time aligned second signal. Means for generating a second signal that is pitch corrected and time aligned,
The apparatus according to claim 14. Means for applying the feature correction data;
For modulating the feature modification data according to the mathematical factor or function to modify the pitch in the selected portion of the second signal by using the mathematical factor or function with the feature modification data. 26. The apparatus of claim 25, comprising: The mathematical factor or function, the pitch measurement in the first sampled voice signal, corresponding pitch measurement in the second sampled the second sampled audio signals along the sound signal 27. The apparatus of claim 26, wherein the apparatus is a function of the ratio value.

Images