KR20150067139A - Social music system and method with continuous, real-time pitch correction of vocal performance and dry vocal capture for subsequent re-rendering based on selectively applicable vocal effect(s) schedule(s) - Google Patents

Abstract

Despite the many practical limitations imposed by mobile device platforms and application execution environments, vocal music performances can be captured and, in some instances or embodiments, And / or processed according to a user selectable vocal effects schedule for mixing and rendering. In some cases, vocal performances of individual users are captured on mobile devices in a situation where the lyrics are displayed in a karaoke-style in response to an audible rendering of the accompaniment. These performances can be pitch-corrected in real time in the mobile device according to the pitch correction settings. Vocal effect schedules may also be selectively applied to these performances. In this way, even amateur users / performers with imperfect pitches can be challenged to challenge "stardom " and / or participate in gameplay, social network or vocal accomplishment application architecture that facilitates musical collaboration on a global scale, And / or in some cases or embodiments, obtain a container to do so to initiate a transaction in an application that generates revenue.

Description

TECHNICAL FIELD [0001] The present invention relates to a social music system and method by continuous real-time pitch correction of vocal performance and dry vocal capture for subsequent re-rendering based on selectively applicable vocal effect schedules PERFORMANCE AND DRY VOCAL CAPTURE FOR SUBSEQUENT RE-RENDERING BASED ON SELECTIVELY APPLICABLE VOCAL EFFECT (S) SCHEDULE (S)}

The present invention generally relates to the capture and / or processing of vocal performances, and more particularly to techniques suitable for selectively applying vocal effect schedules to captured vocals.

The installed base of mobile phones and other portable computing devices is growing rapidly in terms of daily numbers and computing power. These mobile phones and other portable computing devices, which are very common and deeply embedded in the lives of people around the world, transcend almost all cultural and economic barriers. Operationally, today's mobile phones offer unparalleled speed and storage power in desktop computers in less than a decade, making them astonishingly fit for real-time sound synthesis and other musical applications. As a result, in part, some of the modern mobile phones, such as iPhone ^® handheld digital devices, available from Apple Inc. are very well supported audio and video playback.

Similar to traditional acoustic instruments, mobile phones can be familiar sound generating devices. However, compared to most traditional instruments, mobile phones are somewhat limited in acoustic bandwidth and power. Despite these shortcomings, mobile phones have advantages of being able to use them everywhere (Ubiquity), numerical strengths and extreme mobility, so that almost anywhere, anytime, jam sessions, rehearsals, It is feasible (at least theoretically) to bring the artists together for the performance. The mobile music field was developed by several research and development organizations. In general, see G. Wang and Designing Smule's iPhone Ocarina presented at 2009 New Interfaces for Musical Expression, Pittsburgh (June 2009). In ^{addition, iPhone ®, iPad ®, iPod} Touch ® and possible Ocarina ™ available from Smule, Inc. for other iOS ^® devices, Leaf Trombone: by applications such as the World Stage ™, and I Am T-Pain ™ applications Experience has shown that advanced digital sound technologies can be delivered in a way that provides a strong user experience. iPhone, iPad, iPod Touch are trademarks of Apple, Inc. iOS is a trademark of Cisco Technology, Inc., used by Apple under license.

Digital acoustic researchers have found that within the constraints of the real world that are imposed by the processor, the memory and other limited computing resources thereof, which are considerable real-world challenges, and / or within the communication bandwidth and transmission delay constraints common to wireless networks before their innovation operable iPhone ^® handheld and possible commercial applications used in handheld devices such as Hyundai and other platforms and parties. Improved techniques, features and user experiences are required.

Despite the many practical limitations imposed by mobile device platforms and application execution environments, vocal music performances can be captured and, in some instances or embodiments, And / or processed according to a user-selectable vocal effects schedule for mixing and rendering. In some cases, vocal performances of individual users are captured on mobile devices in a situation where they present karaoke-style lyrics in response to the audible renderings of the accompaniment. These performances may be adjusted in real time by the pitch correction settings in the mobile device (or more generally in a mobile computing device such as a mobile phone, personal digital assistant, laptop computer, notebook computer, pad-type computer or netbook) . Vocal effect schedules may also be selectively applied to these performances. In this way, even amateur users / performers with incomplete pitches may participate in game play, social network or vocal accomplishment application architectures that challenge "stardom " and / or facilitate world-class musical collaboration, In some cases or embodiments, it is encouraged to initiate a transaction within the application that generates revenue.

In some cases or embodiments, such transactions may include purchasing or licensing a computer-readable encoding of artist-, song- and / or performance-specific vocal effects schedules that may optionally be applied to the captured vocals have. In some cases or embodiments, the vocal effects schedule is specific to a music genre. In some cases or embodiments, transactions may include purchasing or licensing of computer-readable encoding of lyric, timing and / or pitch correction settings or plug-ins. In some instances or embodiments, transactions may include purchasing a "replay" or retry of all or a portion of the vocal performance. In some cases or embodiments, in addition to (or instead of) purchase-type transactions in the application, the computer-readable encodings of vocal effects schedules, lyrics, timing, pitch correction settings and / Access may be obtained based on the vocal achievement (e.g., based on the pitch, timing, or other corresponding to the target score or other vocal performance) or successful traversal of the game play logic.

As in the vocal effect scheduling transactions, social interactions performed by the application or social network infrastructure, such as group formation, group participation, performance sharing, open call initiation, etc., "Or an applicable currency or credit for transactions involving retry privileges. In some cases, the viewing of the advertisement content by the user may generate an applicable currency or credit for these transactions.

In some cases or embodiments, the pitch correction settings code a particular key or scale for the vocal performance or a portion thereof. In some cases or embodiments, the pitch correction settings include a score-coded melody and / or Mars sequence for receiving or associating with lyrics and accompaniment. Mars notes or chords can be coded for explicit pitches or for scored coded melodies, or even, if desired, for actual pitches produced by the vocalist. In some cases or embodiments, vocal effects schedules and / or pitch correction settings for feeding or associating with lyrics and accompaniment may be used only in some (e.g., lead vocal, backup An artist's vocals, a chorus or chorus, a duet part, or a part of a three-part Mars).

In these various ways, user performances (typically performances of amateur vocalists) can be significantly improved in tone or performance quality, and the user can be provided with immediate and courageous feedback, In the examples, the user may emulate or assume the style of a preferred artist, representative performance or musical genre, or a persona. Typically, the feedback may include both the pitch-corrected vocals themselves and the visual enhancement when the user / vocalist fits the correct note (during vocal capture). Typically, "correct" notes are those notes that match the key and correspond to a score-coded melody or harmony expected according to a particular point in the performance. In that regard, in a cappella modes in which there is no working score and facilitates an immediate off-score or in which specific pitch correction settings are disabled, the pitches heard in a given vocal performance may optionally include notes that are closest to a particular key or scale For example, C major, C minor, E flat major, etc.). In each case, the vocal sound of the "correct" notes may acquire user-vocal list points (e.g., in a game play sequence) and / or credits (e.g. In general, these points or credits may include vocal effects schedules specific to additional artist-, song-, performance-, or musical genres (or partially using transaction handling logic implemented in a handheld device) Can be applied to additional vocal scores of the vocal capture "repeat" for the user-selectable portion of the previously captured vocal performance and to the purchase or licensing of the lyrics.

Based on the powerful and translational nature of vocal effects specific to artist-, song-, performance- or musical genres and of pitch-calibrated vocals, user / vocalists can use other natural Shyness or anxiety can be overcome. Instead, even just amateurs are encouraged to share and contribute to their vocal performances as part of a virtual "choir" or "open recruitment" to share with friends and family. In some implementations, these interactions are facilitated through invitations to participate in social network- and / or eMail-mediated sharing and group performance of performances. Using the uploaded vocals captured at clients such as the portable computing devices described above, the content server (or service) arbitrates these virtual choruses or public offerings by manipulating and mixing uploaded vocal performances of a number of contributed vocalists can do. Depending on the purposes and implementation of a particular system, the uploads may include (i) dry vocal versions of the user's captured vocal performance suitable for application (re-application) of the vocal effects schedule and / or pitch-correction, (ii) (Iii) control tracks or other indications of the user key, pitch correction and / or vocal effects schedule selection, and the like. By including dry vocals in the upload, post-processing (in a content server or service) by a selectable vocal effects schedule, and mixing, crossfading and mixing of each vocal contribution into the appropriate score or performance template slots or positions, Considerable flexibility is imparted to the pitch shifting.

The virtual chorus or open recruitment audition can be done in any of a variety of ways. For example, in some cases or embodiments, a portable computing device may be capable of capturing (e.g., pitch-compensated) cues that have been captured for an accompaniment (and which are scored according to score-coded melody and / or cosmic cues The vocal performance of the first user is provided to the other potential vocal players through the content server or service. Typically, the captured vocal performance is supplied as a dry vocal with or in an encodable form associated with pitch-correction and / or vocal effects schedule settings or selections. The vocal effect schedule may be selectively applied to the supplied vocal performance (or a portion thereof) (in a content server or service or, optionally, a portable computing device), the result being a 2 are mixed with background musical instruments / vocals to form a production accompaniment.

In some cases, consecutive vocal contributors may be geographically separated and may not know each other (at least a priori), but the intimacy of the vocals with the cooperative experience itself tends to minimize this physical separation. In other cases, the open recruitment audition may be posted to a group of potential contributors selected by the initiating user-vocalist or otherwise associable therewith. As successive vocal performances are captured (e.g., in each portable computing device), added in response to an open recruitment audition, or as part of a virtual choir, the accompaniment with which each vocal is captured evolves, Quot; members "or public recruitment audition entrants. In some cases, storing or maintaining dry vocal versions of the captured vocal performances may facilitate application of changeable (or later selectable) vocal effects schedules.

Depending on the purpose and implementation of a particular system, the vocal effects (EFX) schedule may be used for spectral equalization, audio compression, pitch correction, stereo delay and reverberation effects for application to one or more respective portions of a user's vocal performance And / or parameters (for computer-readable medium encoding) for one or more of < RTI ID = 0.0 > In some cases or embodiments, the vocal effects schedule may feature an artist, song, or performance and may be applied to audio encoding of the user's captured vocal performance, so that a derivative audio encoding or The audible rendering may cause the selected artist, song or performance features to take on.

In the context of the present description, the term vocal effect schedule is used to denote, at least in some cases or embodiments, the listed and operative (e.g., vocal) effects of a vocal EFX to be applied to some or all of the captured vocal performance Quot; and " a " Thus, different vocal effects schedules can be acquired or transcribed and applied to captured dry vocals to provide a "Katy Perry effect" or "T-Pain effect ". In some cases, social interactions, such as group formation, group participation, performance sharing, public recruitment audition initiation, etc., made by an application or social network infrastructure, generate applicable currencies or credits for such transactions do. In some cases, the viewing of the advertisement content by the user may generate an applicable currency or credit for these transactions.

In some cases, different vocal effects schedules may be applied to the user's captured dry vocals to fill the derivative audio encoding of the audible rendering by the studio or "live " playing feature of a particular artist or song. In at least some instances or embodiments, the term vocal effect schedule further includes additional vocals (e.g., separate vocal EFX sets for pre-chorus and chorus portions of a song, and / or duets or other multi- And a set of vocal EFXs that vary in time or in a template corresponding to portions of the vocal score (with a distinct vocal effect set for each portion of the vocalist performance). Similarly, each part of a single vocal effect schedule (or a pair of separate vocal effect schedules for that situation) may be used for each vocal performance capture, so that the duet performed by the first user For each vocal performance capture of the first part of the duet performed by the second user and for separate vocal performance captures of the second part of the duet performed by the second user.

In some cases or embodiments capturing visual animation and / or facilities for listener comments and ranking, as well as public recruitment audition management or vocal performance add logic, may be performed by mixing background instruments and / or vocals For example, in an audible rendering of a vocal performance (captured in a similarly configured mobile device, for example). The synthesized Mars and / or additional vocals (e.g., vocals captured from other vocalists at other positions and optionally pitch-shifted to be harmonized with other vocals) can also be included in the mix have. Geocoding of captured vocal performances (or individual contributions to combined performance) and / or listener feedback may facilitate animations in a manner that represents performance or assurance originating from a particular geographic location on a user-operable globe, Lt; / RTI > In this manner, implementations of the described functionality can otherwise transform everyday mobile devices into social instruments that create a global connection, collaboration, and community of unique meanings.

In some embodiments of the invention, the method includes using a portable computing device for vocal performance capture, wherein the portable computing device has a touch screen, a microphone interface, and a communication interface. The method includes, in response to a user selection on the touch screen, retrieving vocal scores synchronized in time with a corresponding accompaniment and lyrics through a communication interface, wherein the vocal scores are synchronized in time with corresponding accompaniments and lyrics. In a handheld computing device, the accompaniment is rendered audibly, and at the same time, the corresponding portions of the lyrics corresponding to the accompaniment and time are presented on the display. Corresponding to the accompaniment in time, the user's vocal performance is captured via the microphone interface, and the dry vocal version of the user's captured vocal performance is stored on the portable computing device. Depending on the vocal score, the portable computing device performs continuous real-time pitch shifting of at least some portions of the user's captured vocal performance and mixes the user's pitch-shifted resultant vocal performance into an audible rendering of the accompaniment. The method further includes applying at least one vocal effect schedule to the user's captured vocal performance. The vocal effects schedules may be stored in a computer readable encoding of settings and / or parameters for one or more of spectral equalization, audio compression, stereo delay, and reverberation effects for application to one or more respective portions of a user's vocal performance .

In some cases, the vocal effects schedule encodes different effects for application to the respective portions of the user's vocal performance in time correspondence to the accompaniment or lyrics. In some cases, the vocal effect schedule is characterized by a particular music genre. In some cases, the vocal effects schedule is characterized by a particular artist, song or performance.

In some embodiments, the method further comprises transactionally purchasing or licensing at least a portion of the vocal effects schedule from the portable computing device. In some embodiments, the method includes retrieving or unlocking an existing stored instance of a computer-readable encoding of a vocal effect schedule via a communication interface to facilitate a transactional step. In some embodiments, the method includes calculating a mathematical evaluation of the correspondence of the vocal score and at least a portion of the captured vocal performance of the user, and calculating, based at least in part on the threshold of merit, And granting a license or access to the user.

In some cases, the vocal effects schedule is subsequently applied to the dry vocal version of the user's captured vocal performance. In some cases, the subsequent application to the dry vocals is done in a portable device, and the method further includes the step of audibly re-rendering the user's captured vocal performance to the portable device using the applied vocal effects and pitch shifting . In some embodiments, the method further comprises transmitting the audio signal encoding of the dry vocal version of the user's captured vocal performance to the remote service or server via the communications interface, for subsequent application of the vocal effects schedule at the remote service or server .

In some embodiments, the method further comprises, for the sake of or relating to the transmitted audio signal encoding of the dry vocal, the user's captured vocal performer constitutes only one of a plurality of vocal performances to be combined at the remote service or server And transmitting an open call indication. In some cases, the open recall audition display is directed to a remote service or server to request additional one or more vocal performances to be mixed for audible rendering with the user's vocal performance from one or more other vocalists. In some cases, the request may include (i) a listed set of potential other vocalists identified by the user; (ii) Members of the Intimacy Group defined or recognized by the remote service or server; Or (iii) the user's set of social network entities. In some instances, the public recall audition indication specifies a second vocal score and a second utterance for at least one additional vocalist position for provision to a responding additional vocalist. In some cases, the open recall audition indication further specifies a second vocal effect schedule for at least one additional vocal position for application to the vocal performance of the responding additional vocalist.

In some embodiments, the method includes receiving from a remote service or server a version of a captured vocal performance of a user that is processed according to a vocal effects schedule; And audibly re-rendering the captured vocal performance of the user to which the vocal effect has been applied in the portable device.

In some cases, the vocal effect schedule is applied to a portable computing device in a rendering pipeline that includes continuous real-time pitch shifting such that the audible rendering includes scheduled vocal effects.

In some embodiments, the method further comprises transaction from a portable computing device with authority to initiate vocal re-capture of a previously selected user selection of the vocal performance. In some embodiments, the method includes the steps of: calculating a threshold performance index and, using a vocal score, computationally evaluating a correspondence of at least a portion of a captured vocal performance of a user to determine a vocal performance of a user selected portion of a previously captured vocal performance And authorizing the user to start re-capturing.

In some cases, pitch shifting is based on a continuous time-domain estimate of the pitch for the user's captured vocal performance. In some cases, the continuous time-domain pitch estimation includes an operation of a lag-domain periodogram for the current block of the sampled signal corresponding to the user's captured vocal performance , The operation of the lag-domain pyramidogram involves an evaluation of the autocorrelation function for the mean size difference function (AMDF) or the range of lags, for the analysis window of the sampled signal.

In some embodiments, the method further comprises the step of retrieving the accompaniment through the data communication interface, also in response to the user selection. In some instances, the accompaniment is stored in storage local to the portable computing device, and the retrieving step uses an identifiable identifier from the locally stored accompaniment to identify a vocal score that is synchronizable in time with the corresponding accompaniment and lyrics do. In some instances, the accompaniment includes a musical instrument and background vocals and is rendered with multiple versions, and the version of the accompaniment that is rendered audibly in response to the lyrics is a monophonic scratch version, The pitch-corrected vocal versions of the vocal performance and the version of the accompaniment mixed are polyphonic versions of higher quality or sound quality than the scratch version.

In some embodiments, the portable computing device is a mobile phone; A personal digital assistant; Media player or gaming device; And a group of laptop computers, notebook computers, tablet computers, or netbooks. In some embodiments, the display includes a touch screen. In some embodiments, the display is wirelessly coupled to a portable computing device.

In some embodiments, the method further comprises geocoding the transmitted audio signal encoding of the dry vocals. In some embodiments, the method includes receiving an audio signal encoding comprising a second vocal performance captured at a remote device from a remote service or server via a communication interface; And displaying the geographic origin for the second vocal performance in response to an audible rendering comprising a second vocal performance. In some cases, the display of geographic origin is by a display animation that represents a performance originating from a specific location on earth.

In some embodiments according to the present invention (method), the method comprises the steps of: (i) using a portable computing device for vocal performance capture, the portable computing device having a touch screen, a microphone interface and a communication interface; (ii) in response to a user selection on the touch screen, retrieving a vocal score synchronized in time with a corresponding accompaniment and lyrics through a communication interface, the vocal score comprising a sequence of target notes for at least a portion of the vocal performance Gt; (iii) in a portable computing device, rendering the accompaniment audibly and simultaneously presenting corresponding portions of the lyrics on the display in time to the accompaniment; (iv) capturing the vocal performance of the user through the microphone interface and corresponding in time to the accompaniment; And (v) transmitting the audio signal encoding of the dry vocal version of the user's captured vocal performance to the remote service or server via the communication interface, with the selection of at least one vocal effect schedule to be applied to the user's captured vocal performance .

In some embodiments, the method further comprises applying the selected vocal effect schedule to a remote service or server. In some embodiments, the method may include performing successive real-time pitch shifting of at least some portions of a user's captured vocal performance to match the vocal score at the portable computing device and generating a user ' s pitch- Mixing the performance into an audible rendering of the accompaniment.

In some cases, the selected vocal effect schedule may include settings for one or more of spectral equalization, audio compression, pitch correction, stereo delay, and reverb effects for application to one or more respective portions of a user's vocal performance And / or computer-readable encoding of the parameters. In some cases, the vocal effect schedule is specific to the music genre. In some cases, the vocal effects schedule is characterized by a particular artist, song or performance.

In some embodiments, the method further comprises transactionally purchasing or licensing at least a portion of the vocal effects schedule from the portable computing device. In some embodiments, the method includes the steps of: arithmetically evaluating a correspondence of a vocal score and at least a portion of a user's captured vocal performance, and determining a license or access to at least a portion of the vocal effect schedule based on the threshold performance index And further includes a step of granting to the user. In some embodiments, the method further comprises the step of transactionally authorizing to recaptivate a selected portion of the vocal performance from the portable computing device. In some embodiments, the method includes calculating a threshold value based on a threshold performance index and using a vocal score to computationally evaluate a correspondence of at least a portion of a captured vocal performance of a user to determine a right to recapture a selected portion of the vocal performance To the user.

In some embodiments according to the present invention, the portable computing device includes a microphone interface, an audio transducer interface, a data communication interface, a user interface code, a pitch correction code, and a rendering pipeline. Wherein the user interface code is executable on the portable computing device to capture user interface gestures selected for the accompaniment and to initiate a search for at least a corresponding vocal score thereof and the vocal score comprises a sequence of note goals for at least a portion of the vocal performance Is encoded for the accompaniment. The user interface code may be used to (i) render the accompaniment audibly, (ii) simultaneously present the lyrics on the display, (iii) capture the user's vocal performance using the microphone interface, and (iv) To initiate storage of the user interface gestures in the computer readable storage. The pitch correction code is executable on the portable computing device to continuously and real-time pitch correction of the vocal performance captured to match the vocal score, simultaneously with audible rendering. The rendering pipeline is operable to mix the user's pitch-corrected vocal performance with the audible rendering of the user's vocal performance to the captured accompaniment. The rendering pipeline is further executable to apply vocal effect schedules to a user's captured vocal performance, wherein the vocal effect schedules are selectable by the user and are adapted to apply to one or more respective portions of the user's vocal performance , Computer-readable encoding of settings and / or parameters for one or more of spectral equalization, audio compression, stereo delay and reverberation effects.

In some embodiments, the portable computing device includes a display. In some embodiments, the data communication interface provides a wireless interface to the display.

In some embodiments, the user interface code is configured to capture user interface gestures indicative of a user selection of a vocal effects schedule, and in response thereto, for subsequent application of a vocal effects schedule selected at the remote service or server, And to transmit the audio signal encoding of the dry vocal version of the recorded vocal performance to the remote service or server via the data communication interface. In some cases, the transmission may include a public recall audition that the user's captured vocal performer constitutes only one of a plurality of vocal performances to be combined at the remote service or server, either in connection with or related to the audio signal encoding of the dry vocal Display.

In some embodiments, the portable computing device evaluates the correspondence of the vocal score and at least a portion of the user's captured vocal performance and, based on the threshold performance index, provides a license or access to at least a portion of the vocal effect schedule to the user Lt; RTI ID = 0.0 > executable on a portable computing device. ≪ / RTI > In some embodiments, the portable computing device may be adapted to present the user with authority to recaptivate a selected portion of a vocal performance, based on a threshold performance index, and using a vocal score to provide at least a portion of the user's captured vocal performance The portable computing device further comprising code executable on the portable computing device.

In some embodiments, the portable computing device further includes local storage, wherein the retrieved searches include, if there are instances of vocal score information in the local storage, checking instances for available instances from a remote server, And retrieving from the remote server whether instances in local storage are unavailable or outdated.

In some embodiments according to the present invention, a computer program product encoded on one or more non-volatile media, the computer program product comprising: a portable computing device operable to cause a portable computing device to perform one of the above- And instructions executable on the processor of the device.

These and other embodiments in accordance with the present invention will be understood with reference to the following description and appended claims.

The present invention is illustrated by way of example and not limitation with reference to the accompanying drawings, in which like references generally denote like elements or features.
Figure 1 illustrates information flows between exemplary mobile phone-type portable computing devices and a content server in accordance with some embodiments of the present invention.
2 is a block diagram illustrating, in accordance with some embodiments of the present invention, for a captured vocal performance, real-time continuous pitch-correction and Mars generation based on a score-coded pitch or Mars cue, And / or < / RTI > storing and / or uploading a dry vocal version of a captured vocal performance for a remote application.
3 is a flow diagram illustrating an exemplary mobile phone-type portable computing device < RTI ID = 0.0 > (e. G., ≪ / RTI >Lt; / RTI > is a functional block diagram of hardware and software components executable in a computer system.
Figure 4 illustrates features of a mobile device that may function as a platform for execution of software implementations in accordance with some embodiments of the present invention.
Figure 5 is a network diagram illustrating the cooperation of exemplary devices in accordance with some embodiments of the invention.
Figures 6A and 6B present complementary (and in some cases, collaborative) uses of the signal processing architecture for application of vocal effects schedules in flow diagram form, in accordance with each and illustrative embodiments of the present invention . Specifically, FIG. 6A illustrates content server-centric utilization of a signal processing architecture that includes interactions with a client application (e.g., hosted by a handheld computing device) vocal capture platform. Figure 6b similarly illustrates client application-centric utilization (e.g., hosted by a handheld computing device) of a signal processing architecture that includes interactions with a content server.
Those skilled in the art will recognize that the elements or features of the figures are illustrated for simplicity and clarity and are not necessarily drawn to scale. For example, in an effort to help improve understanding of embodiments of the present invention, the dimensions or significance of some of the illustrated elements or features may be exaggerated relative to other elements or features.

Techniques have been developed to facilitate capture, pitch correction, harmonization, vocal effect (EFX) processing, encoding and audible rendering of vocal performances on handheld or other portable computing devices. On the basis of these techniques, mixes including these vocal performances can be used to perform audible rendering on products including desktops, workstations, gaming stations and even target products for calls, as well as handheld or portable computing devices Can be prepared for. Implementations of the described techniques are well suited in view of the generally limited capabilities of such handheld or portable computing devices and are suitable for wireless and / or wireline bandwidth-limited networks for rendering on portable computing devices or other products Utilizes the allocation of system functions and signal processing techniques to facilitate efficient encoding and communication of pitch-corrected vocal performances (or their precursors or variations).

The pitch detection and correction of the user's vocal performance is performed continuously and in real time for the audible rendering of the backing track in a handheld or portable computing device. In this way, the pitch-corrected vocals can be mixed with audible rendering to superimpose the user's vocal performance (in real time) with the corresponding instruments and / or vocals of the captured accompaniment. In some implementations, pitch detection may be performed using an average magnitude difference function (AMDF) with zero-crossing and / or peak picking techniques to identify differences between the pitch of the captured vocal signal and the score- Domain pitch correction techniques that use autocorrelation-based techniques. Based on the detected differences, pitch correction based on pitch-synchronized superposition additive (PSOLA) and / or linear predictive coding (LPC) techniques can be used to determine if the captured vocals are pitch-coded melody targets and pitch Allowing pitch shifting in real time to "correct" notes in accordance with the correction settings. Frequency domain techniques, such as FFT peak picking for pitch detection and phase vocoding for pitch shifting, may be used in some implementations, especially when computation facilities substantially exceed current conventional generation mobile devices or when offline processing is used . Pitch detection and shifting (for example, pitch correction, Martians of a synthetic multi-vocal list and / or preparations, virtual chorus mixes) may also be performed in the post-processing mode.

In general, "correct" notes are those notes that correspond to a specified key or scale, or, in some embodiments, corresponding to a score-coded melody (or Mars) predicted according to a particular point in the performance. Thus, in a cappella modes that do not have an operant score (or allow the user to dynamically change the pitch correction settings of an existing score during vocal capture), in some implementations, Can be provided. For example, user interface gestures captured on a mobile phone (or other portable computing device) can be used to (i) switch off (and turn on) score-coded note goals for a particular piece of music, (ii) (E.g., C major, C minor, E flat major, etc.); and / or (iii) dynamically switching back and forth between melody and harmonic note sets as the pitch correction settings to be operated, (At the gesture selected points in the vocal capture) to the settings that cause only the notes closest to the vocals to be corrected. That is, the user interface gesture capture and dynamic variable pitch correction settings can provide a free style mode for advanced users.

In some cases, the pitch correction settings may be selected to distort the captured vocal performance according to the desired effect, e.g., according to the particular musical performance or pitch correction effects popularized by a particular artist. In some embodiments, the pitch correction is performed using an autocorrelation applied to the variable window of samples from the captured vocal signal, such as plug-in implementations of the Auto-Tune 占 technology, available from and popularized by Antares Audio Technologies. May be based on techniques that simplify calculations computationally.

Depending on the purpose and implementation of a particular system, a user selectable vocal effect (EFX) schedule may be used for spectral equalization, audio compression, pitch correction, stereo delay and so on for application to one or more respective portions of a user & And / or parameters (for computer-readable medium encoding) for one or more of the reverberation effects. In some instances or embodiments, the vocal effects schedule may feature an artist, song, or performance, and the derived audio encoding or audible rendering may take the characteristics of the selected artist, song, It can be applied to audio encoding of vocal performance.

Thus, one vocal effect schedule may feature, for example, studio recordings of lead vocals by artist Michael Jackson singing "PYT (Pretty Young Thing)", while other vocal effects schedules may be featured by other artist T You can feature a cover version of the same song by -Pain. In such a case, the first vocal effects schedule (corresponding to the original performance by Michael Jackson) would be bass roll-off (using terms often used by studio engineers), intermediate compression moderate compression, and digital plate reverberation. < RTI ID = 0.0 > EFX < / RTI > More specifically, the first vocal effect schedule is a tube compressor with a 12 dB / octave high pass filter at 120 Hz, a 4: 1 ratio and a threshold of -10 dB, and a 30 ms pre-delay and 15% wet / dry mix And may encode parameters or settings of a digital remainder having a warm plate setting. Conversely, the second vocal effects schedule (which corresponds to a cover version by T-Pain) is a high-pass equalization, pop-compression, fast (using terms often used by studio engineers) Pitch correction, vocal doubling for some words, and light reverberation for "airiness ". More specifically, the second vocal effects schedule includes digital compression with a 24 dB / octave high pass filter at 200 Hz, a 4: 1 ratio and a threshold of -15 dB, a pitch correction with 0 ms attack, a rate of 0.3 Hz, Stereo chorus with 100% intensity and 100% mix, and 300 Hz high pass filtering, 2.5 second length and 10 < RTI ID = 0.0 >Gt; impulse-response-based < / RTI > reverberation parameters or settings for a concert hall with% wet / dry mix.

Similarly, in some instances or embodiments, a vocal effects schedule may feature a particular musical genre. For example, one vocal effect schedule may feature a dance genre (for example, a digital compressor with a 24 dB / octave high pass filter at 250 Hz, a 6: 1 ratio and a threshold of -15 dB, Stereo delay with channel [200 ms delay, 15% wet / dry mix, 40% feedback factor] and right channel [260 ms delay, 15% wet / dry mix, 40% feedback factor] (E.g., encode parameters or settings of a digital reflector having a dry mix), other vocal effects schedules may feature a ballad genre (e.g., 12 dB / octave high pass filter at 120 Hz, 4: And a digital compressor with a threshold of -8 dB and a digital reflector with large concert hall settings of 30 ms pre-delay and 20% wet / dry mix). Although specific parameterization of musical genre-by-vocal effects schedules generally depends on implementation, based on the description herein, those skilled in the art will appreciate that appropriate variations of the vocal effects schedule for different musical genres and other parameterization Will recognize. Dance and ballad genres are merely illustrative.

In the context of this disclosure, the term vocal effects schedule refers to a set of vocal EFXs, in at least some instances or embodiments, that are arranged and operated sets of vocal EFXs to be applied to some or all of the captured vocal performance (typically a dry vocal version) As used herein. Thus, different vocal effect schedules can be transacted and applied to the captured dry vocals to provide a "Katy Perry effect" or "T-Pain effect. Similarly, different vocal effects schedules can be transacted and applied to the captured dry vocals, in order to introduce musical genre-star effects into the derivative audio encoding or audible rendering. In some cases, different vocal effect schedules may be transacted and alternatively applied to the user's captured dry vocals, in order to inject studio or "live " performance features into the derivative audio encoding or audible rendering. On the other hand, while the artist-, song-, and vocal-vocal EFX schedules are described separately from the musical genre-specific vocal EFX schedules, in some instances or embodiments, certain vocal EFX schedules may be artist-, And / or may combine musical genre-like aspects.

In at least some instances or embodiments, the term vocal effect schedule further includes additional vocals, such as for the portions of the vocal score (e.g., distinct vocal EFX sets for the pre-chorus and chorus portions of the song, Or a set of vocal EFXs that vary in temporal or template correspondence (for a separate set of vocal effects for respective portions of a duet or other multi-vocalist performance). Thus, in the vocal effects schedule for the symbolic performance of Cher's "Believe ", the specific score-aligned portions corresponding to the pre-chorus section of the performance are (in terms of the terms often used by studio engineers) Can be encoded in a computer-readable form of EFX, including equalization, intermediate compression, strong pitch correction, and light stereo delay, while portions corresponding to chorus sections of the performance can be encoded with a base roll-off, pop compression, Pass-through stereo delay and rich / worm echo can be encoded. In more technical terms, the pre-chorus section EFX of the vocal effects schedule has a 24 dB / octave high pass filter at 400 Hz and a 12 dB / octave low pass filter at 2.2 kHz, a 3: 1 ratio and a threshold of -10 dB A soft-knee compressor, a pitch correction with 0 ms attack, and a delayed quarter note on the left channel, offset by an eighth note on the right channel, both channels having a 15% wet / dry mix It is possible to encode parameters and settings with 33% feedback. Conversely, in the vocal effects schedule, the chorus section EFX has a 12-dB / octave high-pass filter at 120 Hz, a tube compressor with a 4: 1 ratio and a threshold of -15 dB, a delayed second-note sink on the left channel, Response-based reverberation feature of the concert hall offset by 20 ms, both channels having 45% feedback at 25% wet / dry mix, high frequency pass filtering at 200 Hz for 4.5 seconds length and 18% wet / dry mix Parameters or settings.

Similarly, each part of a single vocal effect schedule (or a pair of separate vocal effect schedules for that situation) may be used for each vocal performance capture, so that the duet performed by the first user For each vocal performance capture of the first part of the duet performed by the second user and for separate vocal performance captures of the second part of the duet performed by the second user.

Based on the powerful and transformative nature of pitch-corrected vocals and selectable vocal effects (EFX), user / vocalists typically overcome other natural shyness or anxiety associated with sharing their vocal performances . Even amateurs are encouraged to share and contribute to vocal performances as part of an intimate group or to share with friends and family. In some implementations, these interactions are facilitated through social network- and / or eMail-mediated sharing of performances and invitations to participate in group play or virtual chorus. Using the uploaded vocals captured at clients such as the portable computing devices described above, the content server (or service) can arbitrate these intimate groups by manipulating and mixing uploaded vocal performances of a number of contributed vocalists . Depending on the purposes and implementations of the particular system, the upload may include pitch-corrected vocal performances, dry (i.e., uncorrected) vocals, and / or control tracks of user keys and / can do.

Often, first and second encodings of the same basic audio source material (often of different quality or sound quality) may be used. For example, the use of first and second encodings of an accompaniment (e.g., one in a handheld or other portable computing device where vocals are captured, and one in a content server) Constraints or the requirements in the particular device / platform in which they are used. In some embodiments, the first encoding of an audible rendered accompaniment in a handheld or other portable computing device with an audio backdrop for vocal capture may be accomplished by using a first encoding of the accompaniment, May be of lower quality or sound quality than the second encoding of the same accompaniment. In this way, high quality mixed audio content can be provided, while limiting data bandwidth requirements for handheld devices used for capture and pitch correction of vocal performance.

Nevertheless, the accompaniment encodings used in the portable computing device may, in some cases, be equivalent or even better quality / sound quality to the accompaniment encodings at the content server. For example, embodiments or situations in which the appropriate encoding of the accompaniment already exists in the mobile phone (or other portable computing device), such as based on a previous download from or a music library residing on the mobile phone The download data bandwidth requirements may be very low. The lyrics, timing information, and applicable pitch correction settings may be made from fingerprinting techniques that are applied to audio metadata, track titles, associated thumbnails, or even audio if desired, using any of a variety of identifiable identifiers, Can be searched in association with the accompaniment.

Laoke - Style vocal performance capture

Mobile-phone-hosted, pitch-corrected karaoke-style vocal captures provide useful illustrative situations, although embodiments of the present invention are not necessarily limited thereto. For example, in some embodiments as illustrated in FIG. 1, an iPhone ™ handheld (or more generally a handheld 101), available from Apple, Inc., may be used for vocal capture and for a series of captured vocals Lt; RTI ID = 0.0 > real-time < / RTI > score-coded pitch correction and harmonization. Karaoke-style applications (e.g., "I am T-Pain" application for the iPhone originally released in September 2009, both available from Smule, Inc., or "Glee & As is conventional, the accompaniment and / or vocals of instruments may be rendered audibly for a vocalist / user singing about it. In these cases, the lyrics can be displayed 102 in response to an audible rendering to facilitate karaoke-style vocal performance by the user. In some cases or situations, the background audio may be rendered from a local store such as, for example, the contents of the iTunes ™ library residing on the handheld.

The user vocals 103 are captured in the handheld 101 and are pitch-corrected continuously and in real time (also in the handheld) to provide the user with an improved tone quality rendition of his own vocal performance (See note 104, mixed with the accompaniment). Pitch correction is typically based on score-coded set of notes or signals (e.g., pitch and harmonics cues 105), which are successively generated with performance synchronized sequences of target notes in the current key or scale And provides a pitch-correction algorithm. In addition to the performance-synchronized melody goals, the score-coded melodious note sequences (or sets) are used for pitch-shifting to Mars versions of the user's own captured vocals (typically to the lead melody note track Lt; / RTI > are coded as offsets and are typically scored only for selected portions of goals). In some cases, the pitch correction settings may be a feature of a particular artist, such as an artist performing vocals associated with a particular accompaniment.

In the illustrated embodiment, both the background audio (here, one or more musical instruments and / or vocal tracks), the lyrics and timing information, and the pitch / fade cues are stored in one or more content servers or hosted service platforms 110) < / RTI > In the case of a given song and play, for example, "Hot N Cold ", some versions of the background track may be stored on the content server, for example. For example, in some implementations or applications, the versions include:

· Uncompressed stereo wav format accompaniment,

· Uncompressed mono wav format accompaniment, and

· Compressed mono m4a format accompaniment

. ≪ / RTI >

It is also contemplated that sets of lyrics, melodies and Martian track notes and associated timing and control information may be stored in a suitable container or object (e.g., a MIDI (Musical Instrument Digital Interface) or JavaScript object Notation < / RTI > json type format). With this information, the handheld 101 can respond to audible performance of visual cues, harmonics, and accompaniment (s) associated with the lyrics and even the target notes, to facilitate karaoke-style vocal performance by the user So that the currently detected vocal pitch can be displayed.

Thus, if the people who want to become vocalists choose "Hot N Cold" on the handheld device as it was originally famed by artist Katy Perry, HotNCold.json and HotNCold.m4a (cached or already available based on previous downloads) (If not) can be downloaded from the content server, and then the background music, synchronized lyrics, and, in some situations or embodiments, the score-coded for continuous real-time pitch- Can be used to provide note track (s). Alternatively, for at least certain embodiments or genres, Mars note track can be score-coded for Mars shifting for captured vocals. Typically, the captured pitch-corrected (possibly harmonized) vocal performance is stored locally on the handheld device as one or more wav files, and subsequently (for example, ALE (Apple Lossless Encoder ) Or lost AAC (Advanced Audio Coding) or Vorbis codec) and encoded for upload 106 to the content server 110 as an MPEG-4 audio, m4a, or ogg container file. MPEG-4 is an international standard for the transmission of coded representations of digital multimedia content and the Internet, mobile networks and advanced broadcast applications. OGG is an open standard container format often used in connection with Vorbis audio format specifications and codecs for lossy audio compression. Other suitable codecs, compression techniques, coding formats and / or containers can be used, if desired.

Depending on the implementation, the encodings of dry vocal and / or pitch-corrected vocals may be uploaded to the content server 110 (106). These vocals, which are then generally encoded (e.g., as wav, m4a, ogg / vobis content or otherwise), whether already pitch-corrected or pitch-corrected in content server 110, Background audio and other captured (and possibly pitch-shifted) vocal performances 111 may be mixed with the capabilities or limitations of a particular product (e.g., handheld 120) And may generate files or streams of quality or coding characteristics to be selected accordingly. For example, to produce different quality streams, the pitch-corrected vocals may be mixed with both stereo and mono wav files. In some cases, a high quality stereo version may be created for web playback and a lower quality mono version may be created for streaming to devices such as the handheld device itself.

As described elsewhere herein, performances of multiple vocalists may be attached in response to an open recruitment. In some embodiments, excellence may be granted (e.g., as lead vocals) to one of the vocals (e.g., the main vocals captured in the handheld 101 in the example of FIG. 1) have. In general, a user-selectable vocal effect schedule may be applied 112 to each captured and uploaded encoding of the vocal performance. For example, initially captured dry vocals can be processed (e.g., 112) in the content server 100 according to Katy Perry's studio performance vocal effects schedule characteristics of "Hot N Cold". In some cases or embodiments, processing may include pitch correction (at server 100) in accordance with pitch cues 105 previously described. In some embodiments, the captured pitch-corrected main vocals that are mixed with the resulting mix (e. G., EFX applied, compressed mono m4a format accompaniment and one or more additional vocals, additional vocals themselves are EFX applied Shifted to the respective Mars positions above or below the main vocals) may be used to create a visual rendering 121 as a second generation accompaniment for capturing additional vocal performances and / The user may be supplied to another user in the personal computer (e.g.

Score-coded pitch Shifting  And vocal effect schedules

2 is a flow chart illustrating real-time continuous score-coded pitch-correction and / or Mars generation for a captured vocal performance in accordance with some embodiments of the present invention. The user / vocalist sings with the accompaniment karaoke style as well as in the illustrated arrangement, as previously described. The vocals captured 251 from the microphone input 201 may be mixed with the main vocal pitch cues or in some cases with an accompaniment that is audibly rendered in one or more acoustic transducers 202 (252) in succession to a corresponding Mars Cue in real time. In some cases or embodiments, the audible rendering of the captured vocal pitch corrected to the "main" melody may optionally include synchronizing harmonics (HARMONY1, HARMONY2) from the vocals captured according to the scored coded offsets, (254).

As will be apparent to those skilled in the art, it is generally desirable to limit feedback loops from the transducer (s) 202 to the microphone 201 (e.g., through the use of headphones or earphones). Indeed, much of the exemplary description of the present disclosure is based on mobile phone situations, and more particularly on portable computing devices that are established on features and capabilities familiar to the Apple iPhone handheld, but that do not even have built- Provided with headphone / microphone jacks, it can act as a platform for continuous vocal capture by real-time pitch correction and harmonization. Apple iPod Touch handhelds and Apple iPad tablets are two examples.

Pitch correction and optionally added harmonics (to the main or Mars pitches) are applied to the device in which the vocal capture and pitch-correction is to be performed, together with the audio encoding of the lyrics 208 and the accompaniment 209 1, it is selected to correspond to score 207 (261) that is wirelessly communicated (from content server 110 to iPhone handheld 101 or other portable computing device). One difficulty encountered in some designs and implementations is that the user may tend to hear Mars well only if he or she chooses to sing the anticipated melody of the song. If a user wants to decorate a song or call it their own version, Mars can be heard in the lane. In order to deal with this difficulty, relative Mars (e.g., for selected songs and selected portions of a song) is pre-scored and coded for specific content. Target pitches are selected at runtime for Mars based on both the score and what the user is singing. This approach has led to a strong user experience.

In some embodiments of the techniques described herein, we have determined from our scores the notes (at the current scale or key) closest to what is heard by the user / vocal list. This closest note may, but need not necessarily, be the main pitch, which typically corresponds to a score-coded vocal melody. Indeed, in some cases, the user / vocalist may intend to sing Mars, and the heard notes may come closer to the Mars track. In either case, the pitch corrector 252 and / or the Mars generator 255 may be implemented by generating appropriate pitch-shifted versions of the captured vocals (even if the user / vocalist intentionally sings Mars) Other portions of the score-coded chord can be synthesized. One or more of the resultant pitch-shifted versions that are combined 254 or aggregated for a dry vocal version of the user's captured vocal performance and, optionally, a mix 253 with the audibly rendered accompaniment, Or wirelessly to a remote device (e. G., Handheld 120).

Although user-selectable vocal effects (EFX) schedules are similarly applied in the signal processing flows 250 implemented in portable computing devices (e.g., 101 and 120), although the content server 100 side application of vocal effects has been described As will be appreciated by those skilled in the art. As before, in this case, the selected vocal effects (EFX) schedule, which may be encoded and included in the wireless transmission 261, may include spectral equalization, audio Settings, and / or parameters for one or more of compression, pitch correction, stereo delay, and reverberation effects. In the illustrated arrangement, an optional signal processing flow is provided for audio signal encodings of dry vocals stored in local storage and mixed (253) with the previously described accompaniment for audible rendering using acoustic transducer 202 do. Typically, depending on the nature of the selected EFX and the computational complexity, a real-time continuous session (including score-coded pitch correction) may be provided in some embodiments, but a user- Application is a post-processing application.

Those skilled in the art will recognize that any of a variety of score-coding frameworks may be used, but the exemplary implementations described herein may be applied to extensions to widely used and standardized musical instrument digital interface data formats . Once established for this framework, the scores may be coded as a set of tracks represented as a MIDI file, data structure, or container, and in some implementations or applications the container may be:

Control track: key changes, gain changes, pitch correction controls, Mars controls, etc.

One or more lyrics tracks: Lyrics events by display customization

· Pitch track: Main melody (usually coded)

According to the control track events, the notes specified in a given Mars track are recorded as absolute scored pitches or relative to the current pitch of the user (according to the current settings It may be interpreted as uncorrected or uncorrected).

Chord tracks: Desired Mars are set on Mars tracks, but if the user's pitch is different from the scored pitch, relative offsets can be kept close to the set of notes in the current chord

.

Once established for this, it is possible to set the runtime actions of the pitch corrector 252 and / or the Mars generator 255, and thus, to overcome what is achievable by conventional static Mars (a wide range of vocal skill levels Considerable score-coded specializations can be defined to provide pitch-corrected vocals and user experience (e.g.

Referring specifically to control track features, in some embodiments, the following text markers may be supported:

Key: <string>: Represents the keys whose audible notes are to be corrected (for example, G? Major g # M, E minor Em, B flat major BbM, etc.). C is the default.

PitchCorrection: {ON, OFF}: Coding to correct the pitch of the user / vocal list. The default is ON. It can be turned ON and OFF at points synchronized in time in vocal performance.

SwapHarmony: {ON, OFF}: If the pitch heard by the user / vocalist is closest to Mars, code to pitch correction to Mars rather than melody. The default is ON.

Relative: {ON, OFF}: When ON, Mars tracks are interpreted as offsets relative to the user's current pitch (calibrated according to different pitch correction settings). Offsets from Mars tracks are offsets of Mars tracks for the scored pitch track. If OFF, Mars tracks are interpreted as absolute pitch targets for Mars shifting.

Relative: {OFF, <+/- N> ... <+/- N>}: If not OFF, the Mars offsets (as many as you want) are scored according to any active key or set of notes It is relative to the pitch track.

RealTimeHarmonyMix: {value}: Codes the changes in the mix ratio at the time-synchronized points in the vocal performance of the main voices and harmonies in an audibly rendered Mars / main vocal mix. 1.0 is all the Mars voices. 0.0 is all main audio.

RecordedHarmonyMix: {value}: Codes the changes in the mix ratio at the time-synchronized points in the vocal performance of the main voice and Mars in the uploaded Mars / Main vocal mix. 1.0 is all the Mars voices. 0.0 is all main audio.

The chord track events, in some embodiments, represent root and quality (e.g., C min7 or Ab maj) and include the following text markers that allow the set of notes to be defined. Desired Mars are set in the Mars track (s), but if the user's pitch is different from the scored pitch, relative offsets can be kept close to the notes in the current chord. As used for the chord tracks of a score, the term "chord" will be understood to mean a set of available pitches, since chord track events do not need to encode standard chords in the usual sense . These and other score-coded pitch correction settings may be used to facilitate the inventive techniques described herein.

Pitch detection, correction and Shifting  Computational techniques for

As will be appreciated by those of ordinary skill in the art having the benefit of the disclosure of this disclosure, pitch-detection and correction techniques include correction of the captured vocal signal to a target pitch or note, and pitch- The creation of Mars can be used for both. Figures 2 and 3 illustrate an example of a mobile device 101 that may be used to create pitch-corrected and optionally harmonized vocals for audible rendering (locally and / or at a remote target device) illustrating basic signal processing flows 250 and 350 according to particular implementations suitable for an iPhone ™ handheld.

Based on the description herein, those skilled in the art will appreciate that the functional blocks of software (e.g., decoder (s) 352, digital-to- (Such as sampling, filtering, decimation, and the like) and data representations to the analog to digital (D / A) converter 351, capture 253 and encoder 355). Similarly, for signal processing flows 250 and exemplary score-coded musical note goals (including musical note goals), those skilled in the art will appreciate that, as implemented as at least a portion of software executable on a handheld or other portable computing device (E.g., decoder (s) 258, capture 251, digital-to-analog (D / A) converter 256, mixers 253, and 254, and encoder 257) to properly allocate signal processing techniques and data representations.

Once established for any of the various suitable implementations of the signal processing arrangements, it will be appreciated that the pitch correction, harmonic generation, and combined pitch correction / smoothing blocks 252, 255, and 354 shown in FIGS. Reference is made to pitch detection and correction / shifting techniques that may be used in the various embodiments described herein, including acceleration.

As will be appreciated by those skilled in the art, pitch-detection and pitch-correction have a rich technical history in the field of music and speech coding. Indeed, a wide range of feature peaking, time domain, and even frequency domain techniques have been used in this field and can be used in some embodiments in accordance with the present invention. The present description is not intended to be a complete listing of a wide variety of signal processing techniques that may be suitable for a variety of designs or implementations in accordance with the present description, Lt; RTI ID = 0.0 &gt; (e. G., &Lt; / RTI &gt; like) implementations.

Thus, in view of the above, and without limitation, certain exemplary embodiments operate as follows:

1) Obtain a buffer of audio data containing sampled user vocals.

2) down-sampling from a 44.1 kHz sampling rate by low-pass filtering, and 22 k (sampled (for use as a score-coded melody note target, for use in pitch detection and correction of vocals sampled as main audio) For pitch detection of vocals and shifting of their Mars transformations).

3) Call a pitch detector (PitchDetector:: CalculatePitch ()) that first checks to see if the sampled audio signal has sufficient amplitude and the sampled audio is not too noisy (excessive zero crossings) to proceed . If the sampled audio is acceptable, the CalculatePitch () method computes an average magnitude difference function (AMDF) and executes logic to select a peak corresponding to an estimate of the pitch period. Additional processing improves that estimate. For example, in some embodiments, parabolic interpolation of peaks and adjacent samples may be used. In some embodiments and given appropriate computation bandwidth, the additional AMDF may be performed at a higher sampling rate around the peak sample to obtain better frequency resolution.

4) By shifting the main voice to a score-coded target pitch by using a Pitch-Synchronous Overlap Addition (PSOLA) technique (for higher quality and overlapping accuracy) at a 22 kHz sampling rate. The PSOLA implementation (Smola:: PitchshiftVoice ()) is invoked with class variables and data structures that contain the information required to specify the desired correction (detected pitch, pitch target, etc.). Generally, the target pitch is selected based on the score-coded targets (which frequently change in response to the melody note track) and according to the current scale / mode settings. Scale / mode settings may be updated during a particular vocal performance, but may not be updated too frequently, typically based on score-coded information, and may be updated in an a capella or freestyle mode based on user selections.

PSOLA techniques are well known in the art to facilitate the resampling of waveforms to produce pitch-shifted distortions while reducing the aperiodic effects of the splice. PSOLA techniques, if cross-faded between two periodic waveforms during overlapping segments, are similar points in the periodic oscillation of the waveforms (e.g., at zero crossings, which ideally have approximately the same slope and proceed positive) It is based on the observation that it is much smoother to splice two periodic waveforms. For example, the samples {a, b, c, ...} and the indices 0, 1, 2,

(Where .1 symbols represent deviations from the period), and if you want to jump backward or go somewhere forward, you can choose positive progressive cd transitions in indices 2 and 10, and instead of simply jumping Until (0 * c + 1 * c.1) is reached at index 10/18

(8 indexes) but make the acyclicity less obvious at the edit point. This is pitch synchronous because it is performed on 8 samples which is the cycle closest to the detectable period. Note that this crossfade is a linear / triangular overlap-add, but (more generally) complementary cosine, 1-cosine or other functions as desired.

5) Generate Mars voices using methods that use both PSOLA and LPC techniques. Mars notes are selected based on current settings, which often change according to score-coded Mars goals, or can be changed by the user in Freestyle. These are the target pitches as described above, but different techniques can be used if generally greater pitch shifting is given for Mars. The main voice (now 22k or alternatively 44k) is pitch-corrected to the target using PSOLA techniques as described above. Similarly, pitch shifting to each Mars is performed using PSOLA techniques. Linear Predictive Coding (LPC) is then applied to each to generate the residual signal for each Mars. The LPC is applied to the main pitch-uncompensated voice at 11k (or alternatively 22k) to derive a spectral template to apply to the pitch-shifted remainders. This tends to avoid head-size modulation problems (making people's voices like Darth Vader for chipmunk or munchkinification or downward shifting for upward shifting) .

6) Finally, the remainder are mixed together and used to reconstruct each pitch-shifted Mars using a filter defined by LPC coefficients derived for the main pitch-uncompensated speech signal. The resulting mix of pitch-shifted harmonics is then mixed with the pitch-corrected main audio.

7) The resulting mix is re-sampled up to 44.1k, mixed with accompaniment (mixed in freestyle mode), or mixed with its enhanced sound quality variations buffered for handoff to the audio subsystem for playback. As will be appreciated by those skilled in the art, AMDF computations are only one time domain computation technique suitable for measuring the periodicity of a signal. More generally, the term rag-domain pyrimogram takes as input a series or time domain function of the discrete-time samples x (n) of the signal and determines the function or signal as a series of delays, , And lag-domain) to determine the periodicity of the original function x. This is done in interesting lugs. Thus, for the techniques described herein, examples of suitable lag-domain pyramidogram operations for pitch detection include a lagged version of the vocal input signal x (n) captured for the current block, (Difference function), or taking the absolute value of the subtraction (AMDF), or multiplying the signal by its delayed version and summing the values (autocorrelation).

AMDF will represent a valley in cycles corresponding to the frequency components of the input signal, and autocorrelation will indicate a peak. If the signal is aperiodic (e.g., noise), the pyrograms will not exhibit any distinct peaks or valleys except at the zero lag position. Mathematically,

to be.

In the implementations described herein, AMDF-based lag-domain pyramid calculations can be performed efficiently even using computing facilities of current generation mobile devices. Nonetheless, those skilled in the art, based on the description herein, will recognize implementations that presently or in the future establish any of a variety of pitch detection techniques that may be easier to operate on a given target device or platform.

Attachment of vocal performance in response to "public recruitment audition"

Once the vocalist is captured on a handheld device, the captured vocal performance audio (typically dry vocals, but optionally pitch corrected) is converted to an audio codec (e.g., Advanced Audio Coding (AAC) or ogg / And uploaded to the content server. Figures 1, 2 and 3 each illustrate such an upload. Generally, the content server (e.g., content server 110, 310) then processes the uploaded dry vocals according to the selected vocal effects (EFX) schedule and applicable score-coded pitch correction sets (112, 312). The content server then remixes these captured, pitch-corrected, and EFX applied vocal performance encodings with other content (111, 311). For example, the content server may mix these vocals with high quality or quality musical instrument (and / or background vocal) tracks to produce high quality master audio of the mixed performance. As shown in FIG. 1 and described herein, other captured vocal performances may also be mixed.

Generally, the resulting master is then encoded with the selected vocals at various bit rates and / or with significance using an appropriate codec (e.g., an AAC codec), and then transmitted to the capture handheld device (and / Devices) and generates compressed audio files suitable for streaming / playback over the web. In general, it may be desirable in terms of audio data bandwidth to limit uploaded data to the data needed to represent the vocal performance while mixing at the required point-in-time and place, compared to the capabilities of wireless networks that are typically utilized. In some cases, the data streamed for playback or use as a second (or Nth) generation accompaniment may independently encode the vocal tracks for mixing with the first production accompaniment in an audible rendering target. Generally, vocal and / or accompaniment audio exchanges between the handheld device and the content server may be adapted to the quality and capabilities of the available data communication channels.

In some embodiments of the invention, additional or alternative mixes may be desirable for certain social network configurations that facilitate open recruitment audition handling. For example, in some embodiments, the attachment of the pitch-corrected and EFX-applied vocals captured from the initial or previous contributor may result in a subsequent vocal capture (e.g., from another user's handheld device) Can form the basis of the accompaniment to be used in. Thus, for the supply and use of the accompaniments shown and described herein, the captured, pitch-corrected, EFX applied (and, though not usually, but possibly harmonized) vocals are themselves mixed so that subsequent vocal captures &Quot; background track "that is used for motion, guide, or framing.

In general, additional vocalists may be invited to sing or simply sign a particular part (e.g., tenor, part B, etc. in a duet), at which time the content server 110 sends their captured vocals to a pitch- And may be placed in one or more positions within a public recruitment audition or virtual choir. Typically, a user-vocalist who initiates an open recruitment audition will be able to play the vocals that are subsequently attached to the vocabulary, such that the vocal performances that are subsequently attached are slotted or arranged (by applicable pitch cues and / or by applied EFX, ) &Lt; / RTI &gt; Mixed vocals may be included in this accompaniment, but since the illustrated and described systems separately capture individual vocal performances and apply vocal effect schedules and pitch-correction, a content server (e.g., content server 110) ) Will understand that it is in a position to manipulate (112) the mixes in such a way as to accommodate the emotion of the user vocalist who initiated the public offer audition or add the objects of the virtual choir.

For example, in some embodiments of the invention, alternative mixes of three different contributing vocalists may be presented in various ways. The mixes provided to (or for) the first contributor may include a vocal of the first contributor (e.g., as a lead vocal with appropriate pitch correction for the main melody, and an artist-, song-, performance- or musical genre - can be featured more important than the other two vocals (with the application of certain vocal effects (EFX) schedules). In general, the content server 110 may modify the mixes so that one vocal performance becomes more important than the others, by manipulating the pitch corrections and EFX applied to the various captured vocals here.

World Stage

Most of the description in this specification focuses on the use of each of the first and second encodings of the accompaniment for vocal performance capture, pitch correction, and mixing and capturing of the user's own vocal performance, It will be appreciated that facilities for the audible rendering of performances of the present invention may be provided in some situations or embodiments. In these situations or embodiments, the vocal performance capture occurs at another device, and after the corresponding encoded (and typically pitch-corrected) vocal performance encoding is received at the current device, And is rendered audibly in conjunction with a visual display animation that represents a vocal performance originating from a particular location. Figure 1 illustrates a snapshot of this visual display animation in handheld 120 which shows the handheld 120 in the capture and pitch-correction mode (as shown in the snapshot) (Or other portable computing device), such as those described and illustrated with reference to handheld device instances 101 and 301 (see FIG. 3), except that the handheld device instances 101 and 301 are operating in a play (or listener) Lt; / RTI &gt; device).

When a user runs a handheld application and accesses this play (or listener) mode, a world stage is presented. More specifically, a network connection is made to the content server 110 that reports the current network connection status and playback preferences of the handheld (e.g., random global, most popular, my performances, etc.). Based on these parameters, the content server 110 selects a performance (e.g., a pitch-corrected and EFX applied vocal performance that may have been captured in the handheld device instance 101 or 301 initially) And transmits metadata associated therewith. In some implementations, the metadata may include additional information, such as a uniform resource locator (URL) that allows the handheld 120 to search for an actual audio stream (which is of high or low quality depending on the size of the pipe) For example, a geocoded location (including geocodes for additional vocal performances included as Mars or backup vocals) of vocal performance capture (using GPS) and a favorite, tagged, or commented And attributes of other listeners left behind. In some embodiments, the listener feedback itself is geocoded. During playback, the user can tag the performance and leave his own feedback or comments for subsequent listeners and / or original vocalists. Once the performance is tagged, a relationship can be established between the player and the listener. In some cases, the listener may be allowed to filter for additional performances by the same performer, and the server may also be able to more intelligently provide new "random" performances that the user will listen to based on an evaluation of user preferences .

Although not specifically shown in the snapshot, geocoded listener feedback indications may be displayed on the globe (e.g., at locations to present respective geographic locations that have transmitted corresponding listener feedback, in accordance with the geocoded metadata) Quot ;, asterisk, or "thumbs up"), or may be presented selectively. In some embodiments, the visual display animation is interactive and will further recognize that it is subject to a viewpoint operation corresponding to user interface gestures captured in the touch screen display of the handheld 120. [ For example, in some embodiments, moving a finger or stylus over a displayed image of a globe of a visual display animation causes the globe to rotate about an axis that is generally orthogonal to the direction of finger or stylus movement. Both the visual display animation representing the vocal performance originating from a particular location on the globe and the listener feedback display are thus presented in locations that correspond to their respective geotags within the interacting and rotating globe user interface.

Exemplary mobile device

Figure 4 illustrates features of a mobile device that may serve as a platform for execution of software implementations in accordance with some embodiments of the present invention. More specifically, FIG. 4 is a block diagram of a mobile device 400, generally according to commercially available versions of an iPhone ™ mobile digital device. While embodiments of the present invention are not limited to iPhone applications or applications (or even to iPhone-type devices), the iPhone device may include its rich sensor complement, multimedia facilities, application programmer interfaces, and wireless application delivery model Along with a high-capacity platform for leveraging specific implementations. Based on the description herein, one of ordinary skill in the art will recognize a wide variety of additional mobile device platforms that may be suitable for a given implementation or use of the inventive techniques described herein (presently or in the future).

Briefly summarized, the mobile device 400 includes a display 402 that is sensitive to haptic and / or haptic contact with a user. The touch-sensitive display 402 may support multi-touch features for processing multiple simultaneous touch points, including processing of data associated with the pressure, degree and / or position of each touch point. This processing facilitates interactions and gestures with multiple fingers, chords, and other interactions. Of course, other touch-sensitive display technologies, such as, for example, a display in which a contact is made using a stylus or other pointing device, may also be utilized.

Typically, the mobile device 400 presents a graphical user interface on the touch-sensitive display 402 to provide user access and convey information to the various system objects. In some implementations, the graphical user interface may include one or more display objects 404,406. In the illustrated example, display objects 404 and 406 are graphical representations of system objects. Examples of system objects include device functions, applications, windows, files, alerts, events, or other identifiable system objects. In some embodiments of the invention, the applications, when executed, provide at least some of the digital acoustic functions described herein.

The mobile device 400 typically includes a network connection that includes both mobile radio and wireless internetworking capabilities so that the user can move with the mobile device 400 and its associated network- Support. In some cases, the mobile device 400 may interact with other nearby devices (e.g., via Wi-Fi, Bluetooth, etc.). For example, the mobile device 400 may be configured to interact with peers or base stations for one or more devices. Thus, the mobile device 400 may grant or deny network access to other wireless devices.

The mobile device 400 includes various input / output (I / O) devices, sensors and transducers. For example, speakers 460 and microphone 462 for facilitating audio, such as audio rendering of pitch-corrected vocal performances mixed with captures and accompaniment of vocal performances as described elsewhere herein, Are usually included. In some embodiments of the invention, the speaker 460 and the microphone 662 may provide appropriate transducers for the techniques described herein. An external speaker port 464 may be included to facilitate a hands-free voice function, such as speaker phone functions. An audio jack 466 may also be included for use with headphones and / or a microphone. In some embodiments, external speakers and / or microphones may be used as transducers for the techniques described herein.

Other sensors may also be used or provided. Proximity sensor 468 may be included to facilitate detection of the user location of mobile device 400. [ In some implementations, an ambient light sensor 470 may be utilized to facilitate adjusting the brightness of the touch-sensitive display 402. Accelerometer 472 may be utilized to detect movement of mobile device 400, as indicated by directional arrow 474. Thus, display objects and / or media may be presented in accordance with the detected orientation, e.g., portrait or landscape. In some implementations, the mobile device 400 may include a Global Positioning System (GPS) or other positioning systems (e.g., Wi-Fi access points, television signals, etc.) to facilitate the geocodes described herein , Cellular grids, systems using Uniform Resource Locators (URLs), etc.). The mobile device 400 may also include a camera lens and a sensor 480. In some implementations, the camera lens and sensor 480 may be located on the back side of the mobile device 400. [ The camera may capture still images and / or video associated with the captured pitch-corrected vocals.

The mobile device 400 may also include one or more wireless communication subsystems, such as an 802.11b / g communication device and / or a Bluetooth communication device 488. Other communications, including other 802.x communication protocols (e.g., WiMAX, Wi-Fi, 3G), Code Division Multiple Access (CDMA), Universal System for Mobile Communications (GSM), Enhanced Data GSM Environment Protocols can also be supported. To establish a wired connection to other computing devices, e.g., other communication devices 400, network access devices, personal computers, printers, or other processing devices capable of receiving and / or transmitting data , A port device 490, such as a universal serial bus (USB) port, or a docking port or some other wired port connection, may be included and utilized. The port device 490 may also allow the mobile device 400 to synchronize with the host device using one or more protocols such as, for example, TCP / IP, HTTP, UDP, and any other known protocol .

5 illustrates an example of each instance 501 and 520 of a portable computing device, such as mobile device 400 programmed with user interface code, pitch correction code, audio rendering pipeline, and playback code, in accordance with the functional descriptions herein. Lt; / RTI &gt; Device instance 501 operates in vocal capture and continuous pitch correction mode while device instance 520 operates in listener mode. Both communicate with the server 512 or the service platform hosting the functions and / or storage described herein with respect to the content servers 110, 210 via the wireless data transfer and arbitration networks 504. The captured and pitch-corrected vocal performances may (optionally) be streamed from the laptop computer 511 and rendered audibly on the laptop computer 511.

Other Examples

While the invention (s) have been described with reference to various embodiments, it is to be understood that such embodiments are illustrative and that the scope of the invention (s) is not limited thereto. Many variations, changes, additions, and improvements are possible. For example, pitch correction vocal performances captured in accordance with a karaoke-style interface have been described, but other variations will be appreciated. In addition, while certain exemplary signal processing techniques have been described in the context of particular illustrative applications, those skilled in the art will recognize that it is straightforward to modify the described techniques to accommodate other suitable signal processing techniques and effects.

Embodiments in accordance with the present invention can take the form of a computer program product encoded on a machine-readable medium as instruction sequences and other functional configurations of software, and / or can be provided as a computer program product, The software may be executed in an operating system (such as an iPhone handheld, mobile or portable computing device, or a content server platform) to perform the methods described herein. In general, a machine-readable medium may be in a form readable by a type of storage upon transmission of information, such as a machine (e.g., computational facilities of a computer, mobile device or a portable computing device, , Source or object code, functionally descriptive information, and the like). The machine-readable medium can be a magnetic storage medium (e.g., disks and / or tape storage); Optical storage media (e.g., CD-ROM, DVD, etc.); A magneto-optical storage medium; A read only memory (ROM); A random access memory (RAM); Erasable programmable memory (e. G., EPROM and EEPROM); Flash memory; Or other types of media suitable for storing electronic instructions, operational sequences, functionally descriptive informational encodings, and the like.

In general, a plurality of instances may be provided for the components, acts, or structures described herein as a single instance. The boundaries between the various components, operations and data stores are somewhat arbitrary, and the specific operations are illustrated in the context of certain exemplary configurations. Other assignments of functionality are devised and may fall within the scope of the present invention (s). In general, the structures and functions presented in separate components in the exemplary arrangements may be implemented as a combined structure or component. Similarly, the structures and functions presented as a single component may be implemented as separate components. These and other variations, changes, additions, and improvements may fall within the scope of the invention (s).

Claims (48)

Using a portable computing device for vocal performance capture, the portable computing device having a touch screen, a microphone interface and a communication interface,
In response to a user selection on the touch screen, searching through the communication interface for a vocal score synchronized in time with a corresponding accompaniment and lyrics, the vocal score comprising a sequence of target notes for at least a portion of a vocal performance Encoding for accompaniment - and,
At the portable computing device, audibly rendering the accompaniment, at the same time presenting a corresponding portion of the lyrics on the display in time corresponding to the accompaniment,
Capturing a vocal performance of the user over the microphone interface and corresponding to the accompaniment in time;
Storing a dry vocal version of the user's captured vocal performance in the portable computing device, wherein, in accordance with the vocal score, the portable computing device is configured to: And mixing the user's pitch-shifted resultant vocal performance into an audible rendering of the accompaniment;
Applying at least one vocal effect schedule to the user's captured vocal performance, wherein the vocal effect schedule comprises at least one of spectral equalization, audio compression, stereo delay And a computer-readable encoding of parameters and / or parameters for one or more of a reverberation effect
Way.
The method according to claim 1,
The vocal effects schedule may be generated by coding a different effect for application to each portion of the vocal performance of the user corresponding to the accompaniment or lyrics in time
Way.
3. The method according to claim 1 or 2,
The vocal effect schedule may include at least one of
Way.
3. The method according to claim 1 or 2,
The vocal effect schedule may be a vocal effect schedule, which is characterized by a particular artist, song or performance
Way.
5. The method according to any one of claims 1 to 4,
&Lt; / RTI &gt; further comprising the step of: &lt; RTI ID = 0.0 &gt; transacting &lt; / RTI &gt; purchasing or licensing at least a portion of said vocal effects schedule from said portable computing device
Way.
6. The method of claim 5,
Further comprising retrieving or unlocking an existing stored instance of the computer readable encoding of the vocal effects schedule via the communication interface to facilitate the transaction step
Way.
7. The method according to any one of claims 1 to 6,
Arithmetically evaluating a correspondence between the vocal score and at least a portion of the captured vocal performance of the user;
Further comprising awarding to the user a license or access to at least a portion of the vocal effects schedule based on a threshold figure of merit
Way.
8. The method according to any one of claims 1 to 7,
The vocal effect schedule is subsequently applied to the dry vocal version of the user &apos; s captured vocal performance
Way.
9. The method of claim 8,
Subsequent application to the dry vocals is done in the portable computing device,
The method comprises:
Further comprising audibly re-rendering the user's captured vocal performance at the portable computing device using the applied vocal effect and pitch shifting,
Way.
9. The method of claim 8,
Transmitting the audio signal encoding of the dry vocal version of the user's captured vocal performance to the remote service or server via the communication interface for the subsequent application of the vocal effects schedule at the remote service or server, More included
Way.
11. The method of claim 10,
For the purpose of relating or relating to the transmitted audio signal encoding of the dry vocal, the user's captured vocal performer constitutes only one of a plurality of vocal performances to be combined at the remote service or server. ) &Lt; / RTI &gt;
Way.
12. The method of claim 11,
The open recruitment audition indication instructs the remote service or server to request one or more additional vocal performances to be mixed for one or more other vocalists to be audibly rendered with the user's vocal performance
Way.
12. The method of claim 11,
The request includes:
A listed set of potential other vocalists identified by the user,
A member of an affinity group defined or recognized by the remote service or server,
To one or more of the user's set of social network relations
Way.
12. The method of claim 11,
The open recruitment audition indication may include identifying a second vocal score and a second utterance for at least one additional vocalist position for provision to a responding additional vocalist
Way.
15. The method of claim 14,
Wherein the open recruitment audition indication further identifies a second vocal effect schedule for the at least one additional vocalist position for application to the vocal performance of the responding additional vocalist
Way.
11. The method of claim 10,
Receiving from the remote service or server a version of the user's captured vocal performance to be processed according to the vocal effects schedule;
Further comprising the steps of audibly re-rendering the captured vocal performance of the user to which the vocal effect has been applied in the portable computing device
Way.
17. The method according to any one of claims 1 to 16,
Wherein the vocal effect schedule is applied to the portable computing device in a rendering pipeline including the continuous real time pitch shifting such that the audio rendering includes a scheduled vocal effect
Way.
18. The method according to any one of claims 1 to 17,
Further comprising transaction from the portable computing device authorization to initiate vocal re-capture of a previously selected user selection of vocal performance
Way.
19. The method according to any one of claims 1 to 18,
The method comprising: computing the vocal score and at least a portion of the captured vocal performance of the user in an arithmetical manner; and determining, based on the threshold performance index, an authority to initiate vocal re- To &lt; RTI ID = 0.0 &gt;
Way.
20. The method according to any one of claims 1 to 19,
Wherein the continuous real-time pitch shifting is based on a continuous time-domain estimate of the pitch for the user's captured vocal performance
Way.
21. The method of claim 20,
Wherein the continuous time-domain pitch estimation comprises an operation of a lag-domain periodogram for a current block of sampled signals corresponding to the user's captured vocal performance, The computation of the domain pyramidogram comprises an evaluation of an autocorrelation function for an analysis window of the sampled signal, an average magnitude difference function (AMDF) or a range of lag
Way.
22. The method according to any one of claims 1 to 21,
Further comprising, in response to the user selection, retrieving the accompaniment also via a data communication interface
Way.
23. The method according to any one of claims 1 to 22,
Wherein the accompaniment is stored in storage local to the portable computing device,
Wherein the searching step uses an identifier identifiable from the locally stored accompaniment to identify a vocal score that is synchronizable in time with the corresponding accompaniment and lyrics
Way.
24. The method according to any one of claims 1 to 23,
The accompaniment includes one or both of a musical instrument and a background vocal, rendered in multiple versions,
Wherein the version of the accompaniment that is audibly rendered corresponding to the lyrics is a monophonic scratch version and the version of the accompaniment mixed with the pitch-corrected vocal version of the user's vocal performance is greater than the monophonic scratch version A polyphonic version of high quality or fidelity &lt; RTI ID = 0.0 &gt;
Way.
25. The method according to any one of claims 1 to 24,
The portable computing device
Mobile phones,
A personal digital assistant,
A media player or a gaming device,
A laptop computer, a laptop computer, a tablet computer, or a netbook
Selected from the group of
Way.
26. The method according to any one of claims 1 to 25,
Wherein the display comprises a touch screen
Way.
27. The method according to any one of claims 1 to 26,
Wherein the display is wirelessly coupled to the portable computing device
Way.
12. The method of claim 11,
Further comprising geocoding the transmitted audio signal encoding of the dry vocals
Way.
29. The method of claim 28,
Receiving an audio signal encoding including a second vocal performance captured at a remote device from the remote service or server via the communication interface;
And displaying the geographic origin for the second vocal performance in response to an audible rendering comprising the second vocal performance
Way.
30. The method of claim 29,
Wherein the display of the geographic origin is by a display animation representing a performance originating from a specific location of the earth,
Way.
Using a portable computing device for vocal performance capture, the portable computing device having a touch screen, a microphone interface and a communication interface,
In response to a user selection on the touch screen, searching through the communication interface for a vocal score synchronized in time with a corresponding accompaniment and lyrics, the vocal score comprising a sequence of target notes for at least a portion of a vocal performance Encoding for accompaniment - and,
At the portable computing device, rendering the accompaniment audibly and simultaneously presenting a corresponding portion of the lyrics on the display in time corresponding to the accompaniment,
Capturing a vocal performance of the user over the microphone interface and corresponding to the accompaniment in time;
Transmitting the audio signal encoding of the dry vocal version of the user's captured vocal performance to the remote service or server via the communication interface with the selection of at least one vocal effect schedule to be applied to the user's captured vocal performance Included
Way.
32. The method of claim 31,
And applying the selected vocal effect schedule to the remote service or server
Way.
33. The method according to claim 31 or 32,
Performing successive real-time pitch shifting of at least a portion of the captured vocal performance of the user to match the vocal score at the portable computing device and to provide the user's pitch- Further comprising mixing to an audible rendering
Way.
34. The method according to any one of claims 31 to 33,
Wherein the selected vocal effect schedule includes at least one of settings and / or parameters for one or more of spectral equalization, audio compression, pitch correction, stereo delay, and reverberation effects for application to one or more respective portions of the vocal performance of the user. Including readable encoding
Way.
35. The method according to any one of claims 31 to 34,
The vocal effect schedule may be a feature of a particular artist, song or performance
Way.
36. The method according to any one of claims 31 to 35,
The vocal effect schedule may include at least one of
Way.
37. The method according to any one of claims 31 to 36,
&Lt; / RTI &gt; further comprising the step of: &lt; RTI ID = 0.0 &gt; transacting &lt; / RTI &gt; purchasing or licensing at least a portion of said vocal effects schedule from said portable computing device
Way.
37. The method according to any one of claims 31 to 37,
Further comprising computing the vocal score and at least a portion of the captured vocal performance of the user in a computational manner and awarding a license or access to at least a portion of the vocal effect schedule to the user based on a threshold performance index doing
Way.
39. The method according to any one of claims 31 to 38,
&Lt; / RTI &gt; further comprising the step of &lt; RTI ID = 0.0 &gt; transacting &lt; / RTI &gt; authority from the portable computing device to recapture a selected portion of the vocal performance
Way.
39. The method according to any one of claims 31 to 38,
The method further comprises computing the vocal score and at least a portion of the captured vocal performance of the user in an arithmetical manner and granting the user the right to recapture a selected portion of the vocal performance based on the threshold performance index doing
Way.
As a portable computing device,
A microphone interface, an audio transducer interface, a data communication interface,
A user interface code executable on the portable computing device for capturing a user interface gesture selected for the accompaniment and initiating a search for at least one vocal score corresponding to the selected user interface gesture, (Ii) display of the lyrics on the display at the same time; (iii) capture of the user's vocal performance using the microphone interface; And (iv) further executable to capture a user interface gesture to initiate storage of the dry vocal version of the captured vocal performance in a computer readable storage,
A pitch correction code executable on the portable computing device for continuously and real-time pitch correction of the captured vocal performance to coincide with the vocal score,
A rendering pipeline executable to mix the user's pitch-corrected vocal performance with the audible rendering of the accompaniment of which the user's vocal performance has been captured, the rendering pipeline comprising a vocal effect schedule for the user's captured vocal performance Wherein the vocal effect schedule is selectable by the user and comprises at least one of spectral equalization, audio compression, stereo delay, and reverberation effects for application to one or more portions of the user's vocal performance And / or &lt; / RTI &gt; parameters for one or more of &lt; RTI ID = 0.0 &gt;
Portable computing device.
42. The method of claim 41,
Further comprising the display
Portable computing device.
43. The method of claim 41 or 42,
Wherein the data communication interface provides a wireless interface to the display
Portable computing device.
44. The method according to any one of claims 41 to 43,
Wherein the user interface code is adapted to capture a user interface gesture representing a user selection of a vocal effect schedule and in response to the selection of a vocal performance schedule of the user for a subsequent application of the selected vocal effect schedule at a remote service or server, Further operable to transmit the dry vocal version of the audio signal encoding to the remote service or server via the data communication interface
Portable computing device.
45. The method of claim 44,
The transmission may include a public recall audition indication that the user's captured vocal performer constitutes only one of a plurality of vocal performances to be combined at the remote service or server, in association with or relating to the audio signal encoding of the dry vocal Included
Portable computing device.
46. The method according to any one of claims 41 to 45,
For evaluating a correspondence of the vocal score and at least a portion of the user's captured vocal performance and awarding to the user a license or access to at least a portion of the vocal effects schedule based on a threshold performance index, Lt; RTI ID = 0.0 &gt;
Portable computing device.
46. The method according to any one of claims 41 to 45,
To evaluate a corresponding correspondence of the vocal score and at least a portion of the captured vocal performance of the user and to award to the user authority to recapture a selected portion of the vocal performance based on a threshold performance index on the portable computing device More executable code
Portable computing device.
49. The method according to any one of claims 41 to 47,
Further including local storage,
Wherein the searching comprises: if there are instances of vocal score information in the local storage, checking the instance for available instances from a remote server; and if the instance in the local storage is unavailable or outdated And retrieving from the remote server
Portable computing device.

Images