DE112018004717T5 - Audiovisual effects system for expanding a recorded performance based on its content - Google Patents

Abstract

Visual effect schemes are applied to audiovisual performances, with different visual effects applied according to different elements of a musical structure. Segmentation techniques applied to one or more audio tracks (e.g., vocal or backing tracks) are used to compute some of the components of the musical structure. In some cases, visual effects schemes applied are mood-indicative and can be selected by a performer as a component of his visual expression or determined from an audiovisual performance using machine learning techniques.

Description

TECHNICAL AREA

The invention relates generally to the recording and / or editing of vocal audio performances and, more particularly, to techniques suitable for use in applying selected visual effects to performance-synchronized video in a manner consistent with or underlying the musical structure of the performance.

STATE OF THE ART

The installed inventory of mobile phones and other portable computing devices increases significantly in number and computing power every day. Omnipresent and deeply rooted in the lifestyles of people all over the world, they overcome almost every cultural and economic barrier. Mathematically, today's mobile phones offer speed and storage capabilities comparable to desktop computers less than ten years ago, which surprisingly makes them suitable for real-time sound synthesis and other musical applications. In part, modern cell phones, such as iPhone® handheld digital devices available from Apple Inc., therefore support audio and video playback quite well.

Like traditional acoustic instruments, cell phones can be personal sound generating and recording devices. Compared to most traditional instruments, however, they are somewhat limited in acoustic bandwidth and performance. Nevertheless, despite these disadvantages, mobile phones have the advantages of omnipresence, their large number and ultra-mobility, which (at least in theory) makes it possible to bring artists together for a performance almost anywhere and at any time. The field of mobile music has been researched in several development research projects. Indeed, recent experience has included applications such as Smule Ocarina ™, Smule Magic Piano and Smule Sing! Karaoke ™ (all available from Smule, Inc.) demonstrated that advanced digital acoustic techniques can be delivered in ways that provide an engaging user experience.

While researchers in digital acoustics are trying to transform their innovations into commercial applications that can be used with modern handheld devices such as the iPhone® handheld and other platforms that are within the real world constraints imposed by the processor, memory, and other restrictive computational resources are imposed, and / or operate within wireless bandwidth and transmission latency limitations typical of wireless networks, pose significant practical challenges. Improved techniques and capabilities are desired, especially with regard to video.

DISCLOSURE OF THE INVENTION (DE)

It has been discovered that, despite many practical restrictions imposed by mobile device platforms and application execution environments, audiovisual performances, including vocal music, are recorded or manipulated and (in some cases) can be coordinated with other users in ways that create engaging user experiences. In some cases, singing performances by individual users (along with performance-synchronized video) are recorded on mobile devices or using a set-top box-like device in connection with a karaoke-like presentation of lyrics to audible renditions of a backing track. In some cases, pitch notes can be presented to singers in connection with the karaoke-like presentation of lyrics, and optionally, continuous automatic pitch correction (or pitch shifting to harmony) can be provided.

A user's vocal audio along with performance-synchronized video is in some cases or embodiments recorded and coordinated with other users' audiovisual contributions to form composite duet-like or glee-club-like or window-based music video-like audiovisual performances. In some cases, the singing performances of individual users (along with performance-synchronized video) are recorded on mobile devices, a television-like display and / or a set-top box device in the context of karaoke-like presentations of a song text corresponding to audible reproductions of a backing track. Multiple singer contributions can be coordinated and mixed in a manner that, at any given time along a given performance timeline, selects a performance-synchronized video of one or more of the contributors for presentation. Selections provide a sequence of visual layouts corresponding to other encoded aspects of a performance score, such as pitch tracks, backing audio, lyrics, sections, and / or vocal parts.

Visual effect schemes are applied to audiovisual performances, with different visual effects applied according to different elements of the musical structure will. In some cases, segmentation techniques applied to one or more audio tracks (e.g. vocal tracks or backing tracks) are used to determine elements of the musical structure. In some cases, visual effects schemes applied are mood-indicative and can be selected by a performer as a component of their visual expression or can be determined from an audiovisual presentation using machine learning techniques.

In some embodiments of the present invention, a method includes accessing machine readable encoding of a first audiovisual presentation and applying a first visual effects scheme to at least a portion of the first audiovisual presentation encoding. The first audiovisual performance is recorded as a vocal audio with performance-synchronized video and has an associated musical structure coding, which at least contains music segment delimitations that are coded for temporal comparison with the first audiovisual performance coding. The applied visual effects scheme encodes different visual effects for different musical structural elements of the first audiovisual performance coding and provides transitions for visual effects in chronological comparison with at least some of the encoded music section boundaries.

In some embodiments, the method further includes segmenting at least one audio track of the first audiovisual performance encoding to provide the associated musical structure encoding. In some cases or embodiments, the associated musical structure encoding includes group part or musical section metadata. In some cases or embodiments, the different visual effects differ either in grade or type, or both in grade and type.

In some embodiments, the method further includes selecting the first visual effects scheme from a plurality of mood-designing visual effects schemes. In some cases or embodiments, the selection is based on a computed mood for at least the recorded vocal audio. In some cases or embodiments, the selection is based on user interface selection by the performer of the vocal audio prior to or concurrently with the recording of the vocal audio. In some embodiments, the method further includes (i) selecting (after the audiovisual rendering of the first audiovisual performance) a second visual effects scheme from the plurality of mood-designating visual effects schemes, the second visual effects scheme being different from the first visual effects scheme, and (ii) applying of the second visual effect scheme on at least part of the first audiovisual performance coding. In some embodiments, the method further includes selecting a second visual effects scheme from the plurality of mood-designating visual effects schemes, the second visual effects scheme being different from the first visual effects scheme; and applying the second visual effects scheme to at least a portion of the first audiovisual performance encoding.

In some embodiments, the method further includes streaming the first audiovisual performance to an audience on one or more remote client devices. In some cases or embodiments, the streamed first audiovisual performance is mixed with encoding of a backing track against which the vocal audio was recorded. In some cases or embodiments, the streamed first audiovisual presentation is streamed with the first visual effects scheme applied. In some embodiments, the method further includes providing identification of the applied visual effects scheme for video effects rendering on one or more of the remote client devices.

In some embodiments, the method further includes transferring (to, from or via a content server or service platform) the first audiovisual presentation along with at least one identifier for the one or more visual effects schemes applied. In some cases or embodiments, the selection is based on user interface selection during or before the audiovisual playback of the first audiovisual performance.

In some cases or embodiments, for a particular mood-designing visual effects scheme, mood values are parameterized as a two-dimensional quantity, a first dimension of the mood parameterization encoding an emotion and a second dimension of the mood parameterization encoding an intensity. In some embodiments, the method further includes determining an intensity dimension of mood parameterization based on one or more of the following: (i) a time-varying audio signal strength or a measure of the vocal energy density that is computed from the vocal audio, and (ii) beats, Tempo, signal strength or energy density of a backing audio track.

In some embodiments, the method further includes segmenting the first audiovisual performance encoding to identify the different musical structural elements. In some cases or embodiments, segmentation is based at least in part on a mathematical determination of a vocal intensity, at least some segmentation limits being limited in order to be compared in time with beats or a tempo that are mathematically extracted from a corresponding audio backing track. In some cases or embodiments, segmentation is based, at least in part, on a similarity analysis that is computationally performed on a timed song text track to classify specific portions of the first audiovisual performance encoding as a verse or refrain.

In some cases or embodiments, the different visual effects encoded by the applied visual effects scheme for a given element thereof include one or more of the following: (i) a particle-based effect or lens reflection, (ii) transitions between different source videos, (iii) animations or Movement of a single image within a source video, (iv) vector graphics or images of patterns or textures; and (v) color, saturation or contrast. In some cases or embodiments, the associated musical structure encodes sections of music of different types and the visual effects scheme applied defines different visual effects for different of the encoded sections of music. In some cases or embodiments, the associated musical structure encodes events or transitions, and the visual effects scheme applied defines different visual effects for different decoded events or transitions.

In some cases or embodiments, the machine-readable encoding further encodes at least a portion of a second audiovisual performance that is recorded as a second vocal audio with performance-synchronized video, the first and second audiovisual performance forming a group performance. In some cases or embodiments, the associated musical structure encodes group parts and the visual effects scheme applied is selectable in time for a special performance-synchronized video according to the encoded musical structure.

In some cases or embodiments, the first and second audiovisual performances are presented as a duet. In some cases or embodiments, the visual effects scheme applied encodes color matching of the performance-synchronized video for respective performers in the group performance for at least some musical structural elements. In some cases or embodiments, the visual effects scheme applied encodes visual blurring or blending at least for some musical structural elements at a transition point between performance-synchronized video for respective performers in the group performance. In some cases or embodiments, the first and second audiovisual performances are recorded against a common backing track.

In some embodiments, the method further includes recording the first audiovisual performance on a network-connected vocal recording device that is communicatively coupled to a content server or service platform from which the musical structure coding is provided. In some cases or embodiments, the recording of the visual performance in the network-connected vocal recording device is performed according to a karaoke-like operating mechanism in which a song text is visually presented according to an audible reproduction of a backing track.

In some embodiments, the method is performed at least in part on a content server or a service platform to which geographically distributed, network-connected vocal recording devices are communicatively coupled. In some embodiments, the method is performed, at least in part, on a network-connected vocal recording device that is communicatively coupled to a content server or service platform. In some embodiments, the method is implemented, at least in part, as a computer program product encoding of instructions executable on a content server or service platform to which a plurality of geographically distributed, network-connected vocal recorders are communicatively coupled.

In some embodiments in accordance with the present invention (s), a system includes a geographically distributed set of network-connected devices configured to record audiovisual performances, including vocal audio, with performance synchronized video, and a service platform. The service platform is designed to (i) receive encodings of the recorded audiovisual performances and, in accordance with associated musical structural encodings, which at least include musical segment boundaries which are encoded for a time comparison with the audiovisual presentation encodings, (ii) special visual effect schemes on at least one part the audiovisual Perform encoding. The visual effect schemes used encode different visual effects for different musical structural elements of the audiovisual performance encoding and provide transitions for visual effects in a temporal comparison with at least some of the encoded music section boundaries.

In some cases or embodiments, the service platform is designed (in response to user or performer selections) to change the visual effects schemes applied and deliver resulting recorded audiovisual performances to respective ones of the geographically distributed, network-connected devices for audiovisual reproduction thereon. In some cases or embodiments, the visual effect schemes applied are selected from a plurality of mood-designing visual effect schemes.

In some embodiments according to the present invention (s), a system includes at least one guest-and-host pairing of network-connected devices that are configured to record at least vocal audio. The host device is designed to (i) receive a coding of a respective coding of at least vocal audio from the guest device and, in accordance with an associated musical structure coding, which includes at least music segment boundaries which are coded for a time comparison with an audiovisual performance coding, (ii) apply selected visual effect schemes to the audiovisual performance coding. The visual effect schemes used encode different visual effects for different musical structural elements of the audiovisual performance coding and provide transitions for visual effects in chronological comparison with at least some of the encoded musical section delimitations.

In some cases or embodiments, the host and guest devices are coupled as local and remote peers over a communication network with non-negligible peer-to-peer latency for broadcasts of audiovisual content, with the host device communicatively coupled as the local peer for media encoding a mixed audio performance that forms vocal audio recorded on the guest device, and the guest device is communicatively coupled as the remote peer to provide the media encoding that is recorded by a first one of the performers and mixed with a backing audio track. In some cases or embodiments, the associated musical structure coding is computed at the host device based on segmentation of at least one audio track received from the guest device. In some cases or embodiments, the host device is configured to render the audiovisual performance encoding as a mixed audiovisual performance including vocal audio and performance-synchronized video from the first and second of the performers, and to apply the audiovisual performance encoding as an apparent live broadcast with the selected visual effects scheme transfer.

These and other embodiments according to the present invention (s) will be understood with reference to the description and the appended claims that follow.

Figure list

• 1 bildet Informationsflüsse zwischen illustrativen mobiltelefonartigen tragbaren Rechenvorrichtungen, fernseherartigen Anzeigen, Set-Top-Box-artigen Medienanwendungsplattformen und einem beispielhaften Inhaltsserver gemäß manchen Ausführungsformen der vorliegenden Erfindung(en) ab, bei denen ein visuelles Effektschema an einer audiovisuellen Darbietung angewendet wird.
• 2A, 2B und 2C sind aufeinanderfolgende Momentaufnahmen von gesangsdarbietungssynchronisiertem Video entlang einer Zeitleiste koordinierter visueller Darbietungen, wobei, gemäß manchen Ausführungsformen der vorliegenden Erfindung, an einem Video für einen, den anderen oder beide von zwei beitragenden Performern Gesangseffekte basierend auf einer Stimmung und basierend auf einem rechnerisch definierten Audiomerkmal wie etwa Gesangsintensität, die über den aufgenommenen Gesang berechnet wird, angewendet werden.
• 3A, 3B und 3C veranschaulichen eine beispielhafte Implementierung einer Segmentierung-und-Videoeffekt(VFX)-Engine gemäß manchen Ausführungsformen der vorliegenden Erfindung(en). 3A bildet Informationsflüsse ab, die eine beispielhafte musikalische Strukturcodierung beinhalten, während 3B eine alternative Ansicht abbildet, die den Schwerpunkt auf eine beispielhafte VFX-Wiedergabepipeline legt. Schließlich bildet 3C eine beispielhafte Abbildung von Gesangsparts und Segmenten zu visuellen Layouts, Übergängen, nachbearbeiteten Videoeffekten und partikelbasierten Effekten grafisch ab.
• 4 bildet Informationsflüsse zwischen illustrativen mobiltelefonartigen tragbaren Rechenvorrichtungen in einer Gastgeber-und-Gast-Konfiguration gemäß manchen Ausführungsformen der vorliegenden Erfindung(en) ab, bei denen ein visuelles Effektschema an einer duettartigen audiovisuellen Livestream-Gruppendarbietung angewendet wird.
• 5 ist ein Flussdiagramm, das Informationstransfers veranschaulicht, die bei einer zusammengesetzten audiovisuellen Darbietung beitragen oder diese beinhalten, die segmentiert ist, um eine musikalische Struktur für eine Abbildung visueller Effekte gemäß manchen Ausführungsformen der vorliegenden Erfindung(en) bereitzustellen.
• 6 ist ein Funktionsblockdiagramm von Hardware- und Softwarekomponenten, die an einer illustrativen mobiltelefonartigen tragbaren Rechenvorrichtung ausführbar sind, um die Bearbeitung einer aufgenommenen audiovisuellen Darbietung gemäß manchen Ausführungsformen der vorliegenden Erfindung(en) zu ermöglichen.
• 7 veranschaulicht Prozessschritte und Ergebnisse der Bearbeitung gemäß manchen Ausführungsformen der vorliegenden Erfindung(en), um Farbkorrektur und stimmungsbezeichnende Videoeffekte an Video für jeweilige Performer einer Gruppendarbietung, die getrennt unter Verwendung von Kameras jeweiliger Aufnahmevorrichtungen aufgenommen werden, anzuwenden.
• 8A und 8B veranschaulichen Bildmaterial für eine Gruppendarbietung mit und ohne Verwendung einer visuellen Unschärfetechnik, die gemäß manchen Ausführungsformen der vorliegenden Erfindung(en) angewendet wird.
• 9 veranschaulicht Merkmale einer Mobilvorrichtung, die als eine Plattform zur Ausführung von Softwareimplementierungen einschließlich audiovisueller Aufnahme gemäß manchen Ausführungsformen der vorliegenden Erfindung(en) dienen kann.
• 10 ist ein Netzwerkdiagramm, das eine Zusammenarbeit beispielhafter Vorrichtungen gemäß manchen Ausführungsformen der vorliegenden Erfindung(en) veranschaulicht.

The present invention is illustrated by way of example and without limitation with reference to the accompanying figures, in which like reference numerals generally indicate like elements or features.

• 1 depicts information flows between illustrative mobile phone type portable computing devices, television type displays, set-top box type media application platforms, and an example content server in accordance with some embodiments of the present invention (s) that apply a visual effects scheme to an audiovisual performance.
• 2A , 2 B and 2C 10 are sequential snapshots of vocal performance-synchronized video along a timeline of coordinated visual performances, wherein, in accordance with some embodiments of the present invention, vocal effects on a video for one, the other, or both of two contributing performers based on a mood and based on a computed audio feature such as Singing intensity, which is calculated from the recorded singing, can be used.
• 3A , 3B and 3C 13 illustrate an exemplary implementation of a Segmentation and Video Effect (VFX) engine in accordance with some embodiments of the present invention (s). 3A maps information flows that include exemplary musical structure coding, while 3B depicts an alternative view that focuses on an exemplary VFX rendering pipeline. Finally forms 3C an exemplary illustration of vocal parts and segments to visual layouts, transitions, post-processed video effects and particle-based effects.
• 4th depicts flows of information between illustrative cellular phone type portable computing devices in a host-and-guest configuration, in accordance with some embodiments of the present invention (s), where a visual effects scheme is applied to a duet type audiovisual live stream group performance.
• 5 FIG. 10 is a flowchart illustrating information transfers that contribute to or include a composite audiovisual performance that is segmented to provide a musical structure for visual effects mapping in accordance with some embodiments of the present invention (s).
• 6 FIG. 10 is a functional block diagram of hardware and software components executable on an illustrative cellular phone type portable computing device to enable editing of a recorded audiovisual performance in accordance with some embodiments of the present invention (s).
• 7 illustrates process steps and results of processing in accordance with some embodiments of the present invention (s) to apply color correction and mood-designing video effects to video for respective performers of a group performance, which are recorded separately using cameras of respective recording devices.
• 8A and 8B Figure 13 illustrates visuals for a group performance with and without the use of a visual blurring technique used in accordance with some embodiments of the present invention (s).
• 9 illustrates features of a mobile device that can serve as a platform for executing software implementations, including audiovisual recording, in accordance with some embodiments of the present invention (s).
• 10th 10 is a network diagram illustrating collaboration of exemplary devices according to some embodiments of the present invention (s).

Those skilled in the art will recognize that elements or features are illustrated in the figures for simplicity and clarification and have not necessarily been drawn to scale. For example, the dimensions or highlighting of some of the illustrated elements or features may be exaggerated relative to other elements or features to improve understanding of embodiments of the present invention.

MODES FOR CARRYING OUT THE INVENTION (DE)

Techniques have been developed to enable the recording, pitch correction, harmonization, coding and / or reproduction of audiovisual performances on portable computing devices and entertainment devices in the living room. Vocal audio along with performance-synced video can be recorded and coordinated with other users' audiovisual contributions to form duet-like or glee-club-like or window-based music video-like audiovisual performances. In some cases, the singing performances of individual users (along with performance-synchronized video) are recorded on mobile devices, a television-like display and / or a set-top box device in the context of karaoke-like presentations of a song text corresponding to audible reproductions of a backing track. In some cases, pitch notes can be presented to singers in connection with the karaoke-like presentation of lyrics, and optionally, continuous automatic pitch correction (or pitch shifting to harmony) can be provided.

Often, multiple singers' contributions are coordinated and mixed in a manner that selects for presentation and at times applies mood-designing visual effects to a performance-synchronized video of one or more of the contributors along a given performance timeline. In some cases or embodiments, techniques of the present invention (s) may be applied even to a single performer's audiovisual content. Generally, selections are made according to segmentation of a certain audio track to determine the musical structure of the audiovisual performance. Based on the musical structure, particle-based effects, transitions between video sources, animations or movement of single images, vector graphics or images of patterns / textures, color / saturation / contrast and / or other visual effects, which are coded in a visual effect scheme, are applied to the respective parts audiovisual performance. In this way, visual effects are applied according to coded aspects of a performance or features such as vocal tracks, backing audio, lyrics, sections and / or vocal parts. The particular visual effects applied vary during the course of a given audiovisual performance based on a segmentation based on a vocal intensity that is computational for one or more vocal tracks is determined, carried out and / or based on it.

In general, for a given song, aspects of the song's musical structure are selectable for the particular visual effects used by a mood-designing visual effects scheme, and intensity measures (typically vocal intensity, but in some cases, power density of non-vocal audio) are used to order the magnitude or modulate or otherwise control prominence of the visual effects applied. In some cases, situations, or embodiments, for example, song form such as {verse, refrain, verse, refrain, bridging ...} is used to limit the illustration. In some cases, such as in a duet, singing parts sequencing (e.g. you sing one line, I sing one line, you sing two words, I sing three, we sing together ...) provides structural information that is used to generate a series of visual layouts. In some cases, situations, or embodiments, a song's intensity of formation (e.g., as measured by acoustic performance, pace, or other measure) may be selectable for the particular visual effects applied by a particular vocal effects scheme.

Optionally, and in some cases or embodiments, vocal audio can be pitch corrected in real time on the vocal recording device (e.g., a portable computing device such as a cellular phone, personal digital assistant, laptop computer, notebook computer, pad-type computer, or netbook) according to pitch correction settings will. In some cases, pitch correction settings encode a specific key or scale for the vocal performance or for parts thereof. In some cases, pitch correction settings include a score-encoded melody and / or harmony sequence that are provided with or for connection to the lyrics and backing tracks. Harmony notes or chords can be encoded as explicit targets or relative to the score-encoded melody or even actual pitches sung by a singer if desired. Machine-usable MIDI-like encodings (MIDI: Musical Instrument Digital Interface) can be used for lyrics, backing tracks, note targets, vocal parts (e.g. vocal part 1, vocal part 2, ... together), music section information (e.g. intro / outro , Verse, pre-chorus, refrain, bridging, transition and / or other section encodings) etc. are used. In some cases or embodiments, conventional MIDI-like encodings can be expanded to encode a score-matched history of visual effects to be applied.

Based on the captivating and transformative nature of pitch-coded vocals, performance-synced video, and score-coded harmony mixes, users / singers can overcome an otherwise natural shyness or fear associated with sharing their vocal performances. Instead, even geographically dispersed singers are encouraged to share or collaborate with friends or family, and to contribute vocal performances as part of social music networks. In some implementations, these interactions are enabled through social networking and / or email mediated sharing of performances and invitations to attend a group performance. By using uploaded vocals recorded on clients such as the aforementioned portable computing devices, a content server (or service) can deliver such coordinated performances by manipulating and mixing the uploaded audiovisual content of multiple contributing singers. Depending on the goals and implementation of a special system, uploads may include pitch-corrected vocal performances (with or without harmonies), dry (i.e. uncorrected) vocals and / or user-type control tracks and / or pitch correction selections, in addition to video content, in addition to video content.

Social music can be conveyed in any of a variety of ways. For example, in some implementations, a first user's vocal performance recorded against a backing track on a portable computing device and typically pitch corrected according to score-encoded melody and / or harmony cues is provided as a seed performance to other potential vocal performers. Performance-synchronized video is also recorded and can be delivered with the pitch-corrected recorded vocals. The delivered vocals are typically mixed with backing instrumentals / vocals and form the backing track for recording vocals from a second (and potentially subsequent) user. Often the subsequent singers are geographically separated and may (at least a priori) be unknown to each other, but the intimacy of the singing, together with the collaborative experience itself, tends to minimize this separation. While subsequent vocal performances and video are recorded (e.g. on respective portable computing devices) and accumulated as part of the social music experience, the backing track versus which each vocal is recorded may develop to include previously recorded vocals by other performers contains.

In some cases, vocals (and typically dubbed video) are recorded as part of a live performance or improvised performance with vocal interactions (e.g., a duet or dialogue) between collaborators. It is envisioned that non-negligible network communication latencies will exist between at least some of the collaborating contributors, especially if these contributors are geographically separate. As a result, a technical challenge is to manage latencies and the recorded audiovisual content in such a way that a combined audiovisual performance can nonetheless be broadcast (e.g., broadcast) in a manner that is useful to recipients, listeners, and / or viewers is presented as an interactive live collaboration.

For example, in one technique for achieving a reproduction of an interactive live performance collaboration, an actual and non-negligible network communication latency (in effect) is masked in one direction between a guest and host performer and tolerated in the other direction. For example, a recorded audiovisual performance by a guest performer on a host show's “live show” internet broadcast could include a guest + host duet that is sung in apparent real-time synchrony. In some cases, the guest could be a performer who popularized a particular music performance. In some cases, the guest could be an amateur singer who was given the opportunity to sing "live" (though remotely) with the popular artist or group "in the studio" as the host (or host) of the show. Notwithstanding a non-negligible guest-to-host network communication latency involved in delivering the guest's audiovisual stream of contributions (perhaps 200-500 ms or more), the host performs in apparent synchronicity (although in an absolute sense, temporally shifted from it) with the Guest through and the apparently synchronized singing is recorded and recorded and mixed with the guest's contribution to the broadcast or broadcast.

The result is an interactive, seemingly live performance (at least from the perspective of the host and the recipients, listeners and / or viewers of the broadcast or broadcast performance). Although the non-negligible network communication latency is masked from the guest to the host, it is understood that the latency is present and is tolerated in the host-to-guest direction. The host-to-guest latency, although visible to the guest (and perhaps quite recognizable), need not be apparent in the apparent live broadcast or other broadcast. It has been discovered that delayed audible playback of host singing (or more generally, the host's recorded audiovisual performance) need not psychoacoustically disrupt the guest's performance.

Performance-synchronized video can be recorded and included in a combined audiovisual performance that forms the apparent live broadcast, with imagery based at least in part on time-varying, computationally-defined audio features extracted from (or calculated from) recorded vocal audio. In some cases or embodiments, these arithmetically defined audio features can be selected over the course of a coordinated audio-visual mix for a special synchronized video of one or more of the participating singers (or their celebrities).

In some cases, engaging visual animations and / or facilities for listener commenting and grading, as well as duet, glee club, or chorus grouping or gathering logic are combined with audible playback of a vocal performance (e.g., recorded on a similarly configured mobile device and pitch corrected) mixed with backing instrumentals and / or vocals. Synthesized harmonies and / or additional vocals (e.g. vocals recorded by another singer at other locations and optionally corrected for pitch to harmonize with other vocals) can also be included in the mix. The geocoding of recorded vocal performances (or individual contributions to a combined performance) and / or listener feedback can enable animations or display artifacts in ways that suggest a performance or confirmation that emanate from a specific geographic location on a user-manipulable globe. In this way, implementations of the functionality described can transform otherwise sophisticated mobile devices into social instruments that promote a sense of global connectivity, collaboration, and community.

Karaoke-like vocal performance recording

Although embodiments of the present invention (s) are not limited to this, pitch corrected, karaoke-like vocal recording using a cellular phone and / or television type audiovisual device provides a useful descriptive context. Likewise, although embodiments of the present invention (s) are not limited to multi-performer content coordinated, multi-performer audio-visual content, including multi-chant content that is recorded or edited asynchronously or recorded and streamed live using latency management techniques described herein, a useful descriptive context.

In some embodiments, such as in 1 an iPhone® handheld, available from Apple Inc., (or more generally, a handheld 101 ) Software that is in coordination with a content server 110 is performed to provide vocal recording and continuous score-coded real-time pitch correction and harmonization of the recorded vocal. Performance synchronized video can be recorded using a camera through or in conjunction with a television or other audiovisual media device 101A or a connected set-top box device ( 101B ) such as an Apple TV ™ device. Performance-synchronized video can also be recorded using an in-camera device that is held by the handheld 101 provided.

As is typical of karaoke-like applications (such as the Sing! Karaoke ™ app available from Smule, Inc.), a backing track of instrumental and / or vocals can be audibly played back by a user / singer to sing against. In such cases, lyrics ( 102 , 102A ) according to the audible reproduction ( 104 , 104A ) are displayed, so that a karaoke-like vocal performance by a user is made possible. In the illustrated configuration of 1 can include lyrics, timing information, pitch and harmony notes ( 105 ), Backing tracks (e.g. instrumental / vocals), performance-coordinated video, video effects schemes ( 107 ) etc. from a network connected content server 110 can be obtained. In some cases or situations, backing audio and / or video may be played back from a media store, such as an iTunes ™ library or other audiovisual content store, on the handheld, a set-top box, or a media streaming device etc. is available or accessible from them.

For the sake of simplicity, it can be assumed that a wireless local area network 180 Communications between the handheld 101 , any audio-visual device and / or set-top box device and a gateway of a large area network to hosted service platforms such as the content server 110 provides. 10th depicts an exemplary network configuration. However, based on the description herein, those skilled in the art will recognize that any of a variety of data communication systems including 802.11 WiFi, Bluetooth ™, 4G-LTE wireless, wired data networks, wired or wireless audio-visual interconnects such as HDMI, AVI, Wi -Di standards or devices, individually or in combination, to enable communications and / or the audiovisual reproduction described herein.

Again referring to the example of 1 becomes user vocals 103 on the handheld 101 recorded and optionally continuously and in real time either on the handheld or using computing devices of the audiovisual display and / or the set-top box device ( 101B ) pitch corrected and audibly mixed with the backing track (see 104, 104A) to provide the user with a reproduction of their own vocal performance with improved sound quality. It should be noted that while recorded vocals 103 and the audible playback 104 , 104A using a conventional visual symbology illustrated on handheld microphone and speaker systems 101 or the television / audiovisual media device 101A , those skilled in the art having the benefit of the present disclosure will recognize that in many cases, microphone and speaker functionality can be provided using attached or wirelessly connected ear plugs, headphones, speakers, feedback isolated microphones, etc. Accordingly, insofar as not specifically limited, vocal recording and audible reproduction should be understood broadly and without limitation to a specific audio converter configuration.

Pitch correction, if provided, is typically based on score-encoded scores or clues (e.g., pitch and harmony clues) 105 ) that provide continuous pitch correction algorithms with performance-synchronized sequences of target notes in a current key or scale. In addition to performance-synchronized melody targets, score-coded harmony note sequences (or sentences) can have pitch shift algorithms with additional targets (typically coded as offsets relative to a lead melodic note track and typically create a score only for selected parts thereof) for pitch shifting to harmony versions of the user's own recorded vocals provide. In some cases, pitch correction settings can be characteristic of a particular artist, such as the artist who originally performed (or popularized) the vocals associated with the particular backing track.

In addition, lyrics, melody and harmony track sheet music, and related timing and control information can be summarized as a score, stored in a suitable container or object (e.g. in a musical instrument digital interface). or MIDI or Java Script Object Notation or json-like format) for delivery together with the backing track (s). Using such information, the handheld 101 who have favourited Audiovisual Display 101A and / or the set-top box device or both display lyrics and even visual cues regarding target notes, harmony and currently detected vocal pitch according to an audible performance of the backing track (s) to enable karaoke-style vocal performance by a user. So if an aspiring singer chooses "When I was your Man", as made popular by Bruno Mars, your_man.json and your_man.m4a from the content server 110 downloaded (if not already available or cached based on a previous download) and used in return to provide background music, synchronized lyrics and, in some situations or embodiments, score-encoded note tracks for continuous real-time pitch correction as the user sings .

Optionally, at least for certain embodiments or genres, harmony note tracks for harmony shifts can be coded for recorded vocals. Typically, a recorded pitch-corrected (possibly harmonized) vocal performance along with performance-synchronized video is stored locally on the handheld device or set-top box as one or more audiovisual files and is then uploaded ( 106 ) to the content server 110 compressed and encoded as an MPEG-4 container file. MPEG-4 is an international standard for the encoded representation and transmission of digital multimedia content for the Internet, mobile networks and advanced broadcasting applications. Other suitable codecs, compression techniques, coding methods and / or containers can be used if desired.

Depending on the implementation, encodings of dry vocals and / or pitch corrected vocals can be sent to the content server 110 uploaded (106). In general, such vocals (e.g. in an MPEG-4 container or otherwise encoded), regardless of whether they are already pitch corrected or on the content server 110 pitch corrected, then z. B. mixed with backing audio or other recorded (and possibly pitch-shifted) vocal performances to produce files or streams with quality or coding characteristics that are compatible with the capabilities or limitations of a particular target or network (e.g., handheld 120 , audiovisual display and / or set-top box device, a social media platform, etc.) can be selected.

As further described in detail herein, performances of multiple singers (including performance-synchronized video) can be accumulated and combined to be presented as a duet-style performance, glee club, window-based music video composition, or vocal jam session. In some embodiments, a presentation-synchronized video post (for example, in the illustration of FIG 1 , the performance synced video 122 including a handheld performance 101 or using an audiovisual device and / or set-top box device 101A , 101B is recorded) in the resulting mixed audiovisual performance 123 with video effects that appear in the course of mixed audiovisual performance 123 applied and dynamically varied, presented. Video effects applied to it are based, at least in part, on an application of a video effect (VFX) scheme, which is selected based on either a user selection or a calculated mood ( 113 ) becomes. In some cases or embodiments, one or more VFX schemes may be a mood-defining set of formulas and / or filters that can be used to present a particular mood. A segmentation-and-VFX engine 112 determines the musical structure and applies special visual effects according to the selected video effects. In general, the special visual effects used are based on the segmentation of vocal and / or backing track audio, a specific or coded musical structure, a selected or detected mood or a selected or detected style and a computationally determined vocal or audio intensity.

The VFX scheme selection can be done by a user on the handheld 101 or using the audiovisual device and / or set-top box device 101A , 101B occur. For example, a user may select a mood-designing VFX scheme that includes video effects that are selected to provide a range of “sad” or “cloudy” video processing effects. Such a palette, in conjunction with the particular or coded musical structure, can provide and apply filters that provide colors, saturations, and contrast that tend to create a “sad” or “cloudy” mood to provide transitions between source videos with little visual energy and / or contain particle-based effects that represent rain, fog or other effects consistent with the selected mood. In turn, other palettes, in conjunction with the particular or encoded musical structure, can provide and apply filters that provide colors, saturations, and contrasts that tend to create a "swingy" or "energetic" mood, transitions between source videos with significant visual energy or movement Provide lens reflections or particle-based effects that augment a visual scene with bubbles, balloons, fireworks, or other visual features that are consistent with the mood selected.

In some embodiments, formulas and / or filters of a given VFX scheme may e.g. B. be parameterized based on computational features, such as average vocal energy, which are extracted from audio performances or based on tempo, beat or audio energy of backing tracks. In some cases or embodiments, lyrics or music selection metadata can be used for VFX scheme selection. In general, in connection with the description and the following claims, it is understood that visual effects schemes in some cases or embodiments can be iteratively selected and applied to a given performance or sub-performance, e.g. B. while a user or contributing singer or post-processing video editor tries to create a special mood, whether "sad", "thoughtful", "peppy" or "romantic".

To simplify the initial illustration forms 1 a recording of performance-synchronized audio ( 103 ) and video ( 105 ) a performance 106 from that to the content server 110 (or the service platform) and uploaded to one or more potential contributing singers or performers e.g. B. is distributed as a seed performance, while the other contributing singers or performers (No. 2, No. 3 No. N) record additional audiovisual (AV) performances. 1 forms the delivery of other recorded AV performances No. 2, No. 3 ... No. N for the audio mix and for the visual arrangement 111 on the content server 110 down to presentation synced video 122 to create. Generally, visual effects can be applied in the course of mixed audiovisual performance playback 123 can be varied according to a special visual effects scheme and segmentation of one or more of the constituent AV performances. In some cases, the segmentation may be based on vocal audio signal processing and / or may be based on a pre-encoded musical structure including vocal part or section notations, phrase or repetitive structure of a song text, etc.

The 2A , 2 B and 2C are consecutive snapshots 191 , 192 and 193 of vocal performance-synchronized video along a coordinated audiovisual performance timeline 151 A video in accordance with some embodiments of the present invention 123 has vocal effects for one, the other, or both of two contributing singers that are applied based on a mood and based on a computationally defined audio feature such as vocal intensity calculated from the recorded vocal. Although the pictures of the 2A , 2 B and 2C not attempting to faithfully reproduce special video effects (which tend to be dynamic and can be visually subtle), those of ordinary skill in the art having the benefit of the present disclosure will understand that for a first part (that of the snapshot 191 a coordinated audiovisual performance VFX is applied to the performance-synchronized video for individual performers based on the selected or detected mood for that performer and based on the vocal intensity of the particular performance. For a second part (that by the snapshot 192 of the coordinated audiovisual performance, VFX is applied to the performance-synchronized video for a single performer based on a selected or detected mood for that performer and a current vocal intensity. Finally, for a third part (such as a chorus caused by the snapshot 193 is represented) of the coordinated audiovisual performance VFX applied to the performance-synchronized video of both performers based on a common or composite mood (whether detected or selected) for the performers and a current measure of the common singing intensity.

As will be understood by those skilled in the art having the benefit of the present disclosure, the performance timeline guides 151 the presentation-synchronized video across different audio segmentation boundaries, across section and / or group part transitions and through different moments, so that the snapshots are expected 191 , 192 and 193 apply different aspects of a special VFX scheme to different parts of the performance timeline and based on the musical structure of the audio, e.g. B. different VFX formulas and VFX filters thereof.

The 3A , 3B and 3C illustrate an exemplary implementation of a Segmentation and Video Effects (VFX) engine 112, (see 1 ) according to some embodiments of the present invention (s). In particular forms 3A Information flows from an exemplary coding of a musical structure 115 involve extracting audio features from performance-synchronized vocal tracks (e.g., vocal # 1 and vocal # 2) and a backing track to segmentation and annotation for musical structure coding 115 to provide.

• • Gesang: Segmentierung „Singen“ gegenüber „Nichtsingen“, augenblickliche Lautstärke, relative Lautstärke jedes Segments.
• • Backing-Tracks: Tempo, augenblickliche Lautstärke, Beat-Detektion.
• • MIDI-Dateien: Tonhöhe, Harmonie, Liedtext, „Part“-Anordnungsmarkierungen (wann jeder Sänger singen sollte).

Feature extraction and segmentation 117 provide the annotations and transition marks of the musical structure coding 115 to formulas and filters from a selected visual effects scheme before playing video 119 to apply. In the exemplary implementation illustrated, the feature extraction and segmentation affects, for example:

• • Singing: segmentation "singing" versus "not singing", instantaneous volume, relative volume of each segment.
• • Backing tracks: tempo, instantaneous volume, beat detection.
• • MIDI files: pitch, harmony, lyrics, “part” arrangement marks (when each singer should sing).

• • Intro: Segment(e) vor dem Start der ersten Liedtextzeile.
• • Vers: Intensität des Segments ist geringer als die durchschnittliche Gesangsintensität des Sängers.
• • Überbrückung: Wie Vers, aber in der zweiten Hälfte eines Liedes vorhanden.
• • Pre-Chorus: Ein Segment vor dem Refrainsegment.
• • Inter: Stille Segmente, aber keine Intro- oder Outro-Segmente.
• • Outro: Segment(e) nach dem Ende der letzten Liedtextzeile.

In an example implementation, a vocal track is treated as consistent with singing and silent segments. Feature extraction tries to classify part of a solo vocal track as silence and singing segments. For duet vocal tracks with parts 1 and 2, the feature extraction tries to classify them as silence, part 1 singing, part 2 singing and singing together segments. Next, segment typing is performed. For example, in some implementations, a global average vocal intensity and average vocal intensities per segment are authorized to determine the "musical intensity" of each segment with respect to a particular performance of a singer of a song. In other words, segmentation algorithms attempt to determine whether a given section is a "louder" section or a "quiet" section. The start time and end time of each lyrics line are also retrieved from the lyrics metadata in some implementations to enable segment typing. Valid segment types and classification criteria include:

• • Intro: segment (s) before the start of the first line of lyrics.
• • Verse: Intensity of the segment is lower than the average singing intensity of the singer.
• • Bridging: Like verse, but present in the second half of a song.
• • Pre-chorus: one segment before the chorus segment.
• • Inter: Silent segments, but no intro or outro segments.
• • Outro: segment (s) after the end of the last line of lyrics.

• • Höhepunkt: Ein Segment ist auch als ein Höhepunktsegment markiert, falls es die höchste Gesangsintensität aufweist.
• • Halten: Falls eine Note eine Tonhöhenlänge länger als eine vorbestimmte Schwelle aufweist.
• • Crescendo: Eine Folge von Noten mit zunehmender Tonhöhe.

Feature extraction and segmentation 117 may also further include audio signal processing to extract the timing of beats and down beats in the backing track and to match the particular segments with down beats. In some implementations, a beats per minute (BPM) measure is used to determine the tempo of the song and moments such as climax, hold and crescendo, which are identified using vocal intensities and pitch information. Moment types and classification criteria can include, for example:

• • High point: A segment is also marked as a high point segment if it has the highest singing intensity.
• • Hold: If a note has a pitch length longer than a predetermined threshold.
• • Crescendo: A sequence of notes with increasing pitch.

In general, these and other annotations and segmentations with styles, formulas, and filters can be used to provide display-driven visual effects.

3B depicts additional details for an embodiment that translates its visual effects schemes into video style formulas (116B) used for VFX planning and special video filters ( 116A ) used in an exemplary VFX playback pipeline. A video style can be user-selected or, in some embodiments, can be selected based on a computed mood. In general, several formulas are defined for a given video style and specialized for special song tempo, recording type (solo, duet or partner artist) etc. A formula typically defines the visual effects, such as layouts, transitions, post-processing, color filters, watermarks and logos, for each segment type or moment. Based on the particular pace and Record type of a song is a suitable formula from the sentence ( 116B ) selected from it.

The VFX planner 118 forms the extracted features (segments and moments in the musical structure coding 115 annotated or marked as described above) for special visual effects based on the selected video style formula ( 116B ). The VFX planner 118 creates a video playback job that contains a number of visual effects configurations. A set of configuration parameters is generated for each visual effect configuration. Parameters such as the name of a previously created video effect, input video, start and end time, backing track intensities and vocal intensities during the effect, beat timing information during the effect, specific control parameters of the video effect and etc. Video effects that are configured can be created beforehand and used directly by the VFX renderer 119 be encoded to reproduce the encoded video effect. Vocal intensities and backing track intensities are used to control the visual effects. Beats timing information is used to match applied video effects with the audio.

Finally forms 3C an exemplary illustration of vocal parts and segments to visual layouts, transitions, post-processed video effects and particle-based effects graphically, such as as the musical structure coding 115 (please refer 3A) or, in some embodiments, by video style-defining formulas (116B) used for VFX planning and special video filters ( 116A ) can be represented for VFX playback. Calculated segments (intro, verse, inter, pre-chorus, bridging and outro) are mapped, for example, to special visual layouts, post-processed effects and particle-based effects with coded visual transitions between segments.

4th depicts a variation of the information flows described above. More specifically, forms 4th Flows between illustrative cellular phone-style portable computing devices in a host-and-guest configuration, in accordance with some embodiments of the present invention (s), where a visual effects scheme is applied to a duet-style audiovisual live stream group performance.

In the illustration of 4th controls a current host user of the current host device 101B at least partially the content of a live stream 122 who is for an audience on devices 120A , 120B ... 120N is buffered and streamed to them. In the illustrated configuration, a current guest user is wearing the current guest device 101A to the audiovisual group performance mix 111 at by the current host device 101B than the livestream 122 (finally through the content server 110 ) is delivered. Although the devices 120A , 120B ... 120N and indeed the current host and host device 101A , 101B For convenience, as illustrated as handheld devices, such as cellular phones, those skilled in the art having the benefit of the present disclosure will appreciate that any member of the accessory can watch the live stream 122 on any suitable computer, smart TV, tablet, set-top box or other streaming media enabled client.

A content that is blended to form the audiovisual group performance mix 111 to form is recorded in the illustrated configuration in connection with a karaoke-like performance recording in which a song text 102 , optional pitch notes 105 and typically a backing track 107 from the content server 110 to the current guest device 101A and / or the current host device 101B to be delivered. A current host (on the current host device 101B ) typically exercises final control over the livestream, e.g. B. by selecting a special user (or special user) from the audience to act as the one or more current guests, selecting a special song from a request queue (and / or vocal parts thereof for special users) and / or by starting, stopping or passing the group AV presentation. Once the current host selects or approves a guest and / or song, the guest user can (in some embodiments) play the backing track 107A start / stop / pass for local audible playback and otherwise the content of the guest mix 106 (Backing track playback mixed with recorded audiovisual guest content) control that of the current host device 101B is delivered. Playing the lyrics 102A and optional pitch notices 105A on the current guest device 101A corresponds in time to the backing track 107A and is also subject to control by the current guest to start / stop / pause. In some cases or situations, backing audio and / or video can be played from a media storage such as an iTunes ™ library located on or accessible from a handheld, set-top box, etc.

Those skilled in the art having the benefit of the present disclosure will understand that instances of the segmentation and VFX engine functionality are as previously described (see 1 , Segmentation-and-VFX engine 112 ) in the guest-host livestream configuration of 4th to the host 101B , the guest 101A and / or the content server 110 can be distributed. Descriptions of the segmentation and VFX engine 112 regarding the 3A , 3B and 3C are thus understood to be analog implementations of similar functionality 112A , 112B and or 112C regarding the devices or components of 4th describe.

In embodiments according to the guest-host livestream configuration of 4th are typically song requests 132 determined from the audience and through signaling paths to content selection and guest queue control logic 112 the content server 110 transmitted. The host controls 131 and the guest controls 133 are illustrated as bidirectional signaling paths. Other queuing and control logic configurations consistent with the described operation, including host or guest queuing and / or song selection, will be apparent based on the present disclosure.

Despite a non-negligible time delay (typically 100-250 ms, but possibly more), the current host device is receiving 101B the guest mix 106 and audibly reproduces this as a backing track, in contrast to the audiovisual presentation of the current host on the current host device 101B is recorded. Playing the lyrics 102B and optional pitch notices 105B on the current host device 101B corresponds in time to the backing track, here the guest mix 106 . For synchronization with the guest mix 106 given the time delay in the peer-to-peer communication channel between the current guest device 101A and the current host device 101B and to allow guest-side control of start / stop / pause, marker beacons can be encoded in the guest mix to provide appropriate phase control of the lyrics 102B and optional pitch notices 105B to provide on the screen. Alternatively, a phase analysis of any backing track can be done 107A who is in the guest mix 106 is included (or any bleed-through if the backing track is encoded or transmitted separately) can be used to control the appropriate phase of the lyrics 102B and optional pitch notices 105B on the screen on the current host device 101B to provide.

It is understood that the time delay in the peer-to-peer communication channel between the current guest device 101A and the current host device 101B both the guest mix 106 as well as communications in the opposite direction (e.g. signal coding of the host microphone 103C ) impaired. Any of a variety of communication channels can be used to convey audiovisual signals and controls between the current guest device 101A and the current host device 101B as well as between the host and host devices 101A , 101B and the content server 110 and between audience devices 120A , 120B ... 120N and the content server 110 be used. For example, respective wireless telecommunication carrier devices and / or wireless local area networks and respective gateways from large area networks (not specifically shown) can communicate to and from the devices 101A , 101B , 120A , 120B ... 120N provide. Based on the description herein, those skilled in the art will recognize that any of a variety of data communication devices including 802.11 WiFi, Bluetooth ™, 4G LTE wireless, wired data networks, wired or wireless audio-visual interconnects such as HDMI, AVI, Wi-Di -Standards or devices may be used individually or in combination to enable communications and / or audiovisual reproduction described herein.

User chants 103A and 103B are on respective handhelds 101A , 101B and can optionally be continuously and in real-time corrected for pitch and mixed with the locally suitable backing track (e.g. the backing track 107A on the current guest device 101A and the guest mix 106 on the current host device 101B ) are reproduced audibly in order to provide the user with a reproduction with improved sound quality of his own vocal performance. Pitch correction is typically based on score-coded musical notations or notes (e.g. the notes on pitch and harmony 105A , 105B working on the current guest device 101A or on the current host device 101B are displayed visually), which provide continuous pitch correction algorithms that are executed on the respective device with presentation-synchronized sequences of target notes in a current key or scale. In addition to performance-synchronized melody targets, score-coded harmony note sequences (or sets) pitch shift algorithms with additional targets (which are typically coded as offsets to a lead melody note track and are typically only provided with a score for selected parts thereof) include a pitch shift to harmony versions of one's own Singing of the user. In some cases, pitch correction settings can be made for a specific artist such as the Artist who performed the vocals associated with the special backing track.

Generally, lyrics, melody, and harmony track notations and related timing and control information can be stored in a suitable container or object (e.g., in a musical instrument digital interface, or MIDI or Java script) -Object-Notation- or json-like format) for delivery together with the backing track (s). Using such information, the devices 101A and 101B (as well as associated audiovisual displays and / or a set-top box device, not specifically shown) to display lyrics and even visual cues regarding the target notes, harmony and currently detected vocal pitch according to an audible performance of the backing track (s) to enable karaoke-like vocal performance by a user. Thus, if an aspiring singer chooses "When I Was Your Man" as popularized by Bruno Mars, your_man.json and your_man.m4a can be downloaded from the content server (if not already available or cached based on a previous download) and in return to provide background music, synchronized lyrics, and, in some situations or embodiments, score-encoded note tracks for continuous real-time pitch correction while the user is singing. At least for certain embodiments or genres, optional harmony note tracks for harmony shifts to recorded vocals can be coded. Typically, a recorded pitch-corrected (possibly harmonized) vocal performance along with performance-synchronized video is stored locally on the handheld device or set-top box as one or more audiovisual files and is then used for communication (e.g., as the guest mix 106 or the audiovisual group performance mix 111 or constituent codes thereof) to the content server 110 compressed and encoded as an MPEG-4 container file. MPEG-4 is a suitable standard for the encoded representation and transmission of digital multimedia content for the Internet, mobile networks and advanced broadcasting applications. Other suitable codecs, compression techniques, encoding format and / or containers can be used if desired.

As those skilled in the art having the benefit of the present disclosure will understand, performances of multiple singers (including performance-synchronized video) can be accumulated and combined to form a duet-style performance, glee club, or vocal jam session. In some embodiments of the present invention, social network entities may at least partially replace or report host control over the pairing of geographically distributed singers and / or the formation of geographically distributed virtual glee clubs. In terms of 4th can, for example, present individual singers as current host and guest users in a way that is recorded (with vocal audio and performance-synchronized video) and ultimately as a livestream 122 is streamed to an audience. Such recorded audiovisual content can be distributed in return for social media contacts of the singer, members of the audience, etc. via an open call which is mediated by the content server. In this way, the singers themselves, members of the audience (and / or content server or the service platform on their behalf) can invite others to participate in the coordinated audiovisual performance or as members of an audience or guest queue.

5 Fig. 3 is a flow diagram illustrating information transfers leading to a composite audiovisual performance 211 contribute or include that is segmented to provide a musical structure for video effects imaging according to some embodiments of the present invention (s). The video effects scheme 210 specifies special visual layouts and mood-designing visual effects such as particle-based effects, transitions between video sources, animations of single image movement, vector graphics / images of patterns / textures and / or color / saturation / contrast for the respective segmented elements of the musical structure. Generally, the intensity of applied video effects is determined based on an intensity measure from the recorded audiovisual performance (typically vocal intensity), although energy density may be included from one or more audio tracks, including a backing track, in some cases or embodiments.

In the illustrated configuration of signal processing pipelines operating on a user device such as the handheld 101 , 101A or 101B can be implemented, a user / singer sings karaoke-like together with a backing track. From a microphone input 201 recorded vocals become a mix ( 253 ) with the backing track connected to one or more acoustic transducers 202 is audibly reproduced, continuously corrected for pitch ( 252 ) and harmonized in real time ( 255 ).

Both pitch correction and added harmonies are chosen so that they track pitches 207 correspond to a musical score, those in the illustrated configuration wireless to the device (s) (e.g. from the content server 110 to the handheld 101 or set-top box device, see 1 ) on which the vocal recording and pitch correction is to be performed, along with the lyrics 208 and audio coding of the backing track 209 communicates ( 261 ) becomes.

In the computing flow of 5 can pitch-corrected and shifted vocals to the mix ( 253 ) combined with an audible backing track ( 254 ) or aggregated and / or to the content server 110 or a remote device (e.g. handheld 120 or 520 , Television and / or set-top box device or another media-compatible computing system 511 ) communicates ( 262 ) will. In some embodiments, the pitch correction or shifting of vocals and / or the segmentation of audiovisual performances on the content server 110 be performed.

As before, those skilled in the art having the benefit of the present disclosure will recognize instances of the segmentation and VFX engine functionality as described previously (see 1 , Segmentation-and-VFX engine 112 ), in other embodiments on a handheld 101 , an audiovisual device and / or set-top box device or another user device can be used. Accordingly, descriptions of the segmentation-and-VFX engine 112 regarding the 3A , 3B and 3C understood to be analog implementations of similar functionality 112D regarding the signal processing pipelines of 5 describe.

6 FIG. 4 is a functional block diagram of hardware and software components executable on an illustrative cellular phone type portable computing device that enable processing of a recorded audiovisual performance in accordance with some embodiments of the present invention (s). In some embodiments (see 1 ), the recording of vocal audio and performance-synchronized video can be performed using facilities of a television-like display and / or a set-top box device. In other embodiments, the handheld device (e.g., the handheld device 101 ), however, even support the recording of both vocal audio and performance-synchronized video.

Thus illustrated 6 basic signal processing flows according to certain implementations that apply to the cellular phone type handheld device 101 Suitable for recording vocal audio and performance-synchronized video, to produce pitch-corrected and optionally harmonized vocal for audible playback (locally and / or on a remote target device) and with a content server or service platform 110 to communicate the the segmentation-and-visual-effects engine 112 includes segmenting recorded audiovisual performances to reveal the musical structure and applying special visual effects from a video effects scheme based on the disclosed musical structure. As before, vocal intensity is measured and used (in some embodiments) to vary or modulate the intensity of mood-designing visual effects.

Exemplary visual effects for the cohesion of image material with several performers

7 illustrates process steps and results of processing in accordance with some embodiments of the present invention (s) to add color correction effects and mood-designing video effects (see 701B, 702B) to video for respective performers ( 701A and 702A) to apply a group performance that is recorded separately using cameras of respective recording devices. The 8A and 8B Figure 13 illustrate group performance images with (802) and without (801) using a visual blurring technique used in accordance with some embodiments of the present invention (s).

An exemplary mobile device and network

9 illustrates features of a mobile device that can serve as a platform for performing software implementations, including audio-visual recording, in accordance with some embodiments of the present invention (s). Illustrated in particular 9 Features of a mobile device that can serve as a platform for executing software implementations in accordance with some embodiments of the present invention. More specifically is 9 a block diagram of a mobile device 900 that is generally consistent with commercially available versions of a digital device of an iPhone ™ mobile phone. Although embodiments of the present invention are certainly not limited to iPhone uses or applications (or even iPhone-like devices), the iPhone device platform, along with its rich accessories of sensors, multimedia devices, application programmer interfaces, and wireless application delivery model, provides an extremely capable one Platform on which certain implementations are to be used. Based on the description herein, those of ordinary skill in the art will recognize a wide range of additional mobile device platforms that (now or in the future) may be suitable for a given implementation or use of the inventive techniques described herein.

In brief, the mobile device includes 900 an ad 902 that is sensitive to haptic and / or tactile contact with a user. The touch-sensitive display 902 can support multi-touch features that process multiple simultaneous touch points, including processing data regarding the pressure, degree, and / or position of each touch point. Such processing enables multi-finger gestures and interactions and other interactions. Of course, other touch-sensitive display technologies can also be used, e.g. B. A display where contact is made using a stylus or other pointing device.

Typically, the mobile device presents 900 a graphical user interface on the touch-sensitive display 902 , which provides the user with access to various system objects and for the transmission of information. In some implementations, the graphical user interface can have one or more display objects 904 , 906 include. In the example shown, the display objects are 904 , 906 graphic representations of system objects. Examples of system objects include device functions, applications, windows, files, warnings, events, or other identifiable system objects. In some embodiments of the present invention, applications when executed provide at least some of the digital acoustic functionality described herein.

Typically, the mobile device supports 900 Network connectivity including, for example, both cellular and wireless internetworking functionality to enable the user with the mobile device 900 and their associated network-based functions. In some cases, the mobile device 900 interact with other nearby devices (e.g. via WiFi, Bluetooth, etc.). The mobile device 900 For example, may be configured to interact with peers or a base station for one or more devices. Thus, the mobile device 900 Allow or deny network access for other wireless devices.

The mobile device 900 includes a variety of input / output (I / O) devices, sensors, and transducers. A loudspeaker 960 and a microphone 962 For example, are typically included to enable audio such as recording vocal performances and audible playback of backing tracks and mixed pitch corrected vocal performances, as described elsewhere herein. In some embodiments of the present invention, the speaker 960 and the microphone 962 provide suitable transducers for the techniques described herein. An external speaker connector 964 may be included to enable hands-free speech functionalities, such as hands-free device functions. An audio jack 966 may also be included for use with headphones and / or a microphone. In some embodiments, an external speaker and / or an external microphone can be used as a transducer for the techniques described herein.

Other sensors can also be used or provided. A proximity sensor 968 may be included to detect the user positioning of the mobile device 900 to enable. In some implementations, an ambient light sensor can be used 970 be used to adjust the brightness of the touch-sensitive display 902 to enable. An accelerometer 972 can be used to control the movement of the mobile device 900 to detect as indicated by the directional arrow 974 specified. Accordingly, display objects and / or media according to a detected orientation, e.g. B. portrait or landscape. In some implementations, the mobile device can 900 include circuitry and sensors to support a location device, such as those used by the global positioning system (GPS) or other positioning systems (e.g., systems that use WiFi access points, television signals, cellular networks, URLs (Uniform Resource Locators)) may be provided to enable the geocoding described herein. The mobile device 900 also includes a camera lens and an imaging sensor 980 . Some implementations have instances of a camera lens and a sensor 980 on the front and back surfaces of the mobile device 900 . The cameras enable the recording of still images and / or video for association with recorded pitch-corrected vocals.

The mobile device 900 may also include one or more wireless communication subsystems, such as an 802.11b / g / n / ac communication device and / or a Bluetooth ™ communication device 988 . Other communication protocols can also be supported, including other 802.x communication protocols (e.g. WiMax, WiFi, 3G), fourth generation protocols and - Modulations (4G-LTE) and beyond (e.g. 5G), CDMA (Code Division Multiple Access - code division multiple access), GSM (Global System for Mobile Communications - global system for mobile communications), EDGE (Enhanced Data GSM Environment - improved data GSM environment) etc. A connection device 990 , e.g. A universal serial bus (USB) port, or a docking port or other wired port connection may be included and for establishing a wired connection with other computing devices, such as other communication devices 900 , Network access devices, a personal computer, a printer or other processing devices that are capable of receiving and / or transmitting data. The connection device 990 can also the mobile device 900 enable synchronization with a host device using one or more protocols such as TCP / IP, HTTP, UDP and any other known protocol.

10th FIG. 10 is a network diagram illustrating collaboration of exemplary devices according to some embodiments of the present invention (s). Illustrated in particular 10th respective instances of handheld devices or portable computing devices such as the mobile device 1001 used in audiovisual recording and programmed with vocal audio and video recording code, user interface code, pitch correction code, an audio playback pipeline and playback code according to the functional descriptions herein. For example, a first device instance is mapped such that it is used when recording vocal audio and performance-synchronized video while the device instance 1020A works in a presentation or playback mode for mixed audiovisual performance with dynamic visual prominence for performance-synchronized video. An additional television display and / or a set-top box device 1020B is likewise depicted to operate in a presentation or playback mode, although, as described elsewhere herein, such a device may also operate as part of a vocal audio and performance sync video recorder. Each of the aforementioned devices communicates via wireless data transport and / or an intermediate network 1004 with a server 1012 or a service platform that provides storage and / or functionality described herein regarding the content server 110 (please refer 1 , 4th , 5 and 6 ) hosts. Recorded pitch-corrected vocal performances with mixed presentation-synchronized video, which are mixed to present a mixed AV presentation reproduction with applied visual effects, as described herein, can (optionally) on the laptop computer 1011 streamed and audiovisually played.

OTHER EMBODIMENTS

Although the invention (s) have been described with reference to various embodiments, it should be understood that these embodiments are illustrative and that the scope of the invention (s) is not limited thereto. Many variations, modifications, additions and improvements are possible. For example, although specific video effects, transitions, and audio-visual mixing techniques are illustrated and described, those skilled in the art having the benefit of the present disclosure will recognize a number of variations and adaptations suitable for a given development, implementation, music genre, or user demography. Similarly, although pitch corrected vocal performances have been described that are recorded on a karaoke-like surface, other variations and adaptations are apparent. Furthermore, although certain illustrative signal processing techniques have been described in the context of certain illustrative applications and device / system configurations, those of ordinary skill in the art will recognize that it is easy to modify the described techniques to accommodate other suitable signal processing techniques and effects.

Embodiments in accordance with the present invention may take the form of and / or be provided as a computer program product encoded in a machine-readable medium as instruction strings and other functional software images that are in return in a computing system such as an iPhone handheld, mobile, or portable Computing device or a content server platform can be executed to perform methods described herein. In general, a machine-readable medium may include tangible articles that provide information in a form that is readable by a machine (e.g., a computer, computing devices of a mobile device or portable computing device, etc.) (e.g., as applications, source or object code, functionally descriptive information etc.) as well as tangible memory that is suitable for the transmission of the information. A machine-readable medium can include a magnetic storage medium (e.g. disks and / or tape storage); an optical storage medium (e.g. CD-ROM, DVD, etc.); a magneto-optical storage medium; Read only memory (ROM); Random access memory (RAM); erasable programmable memory (e.g. EPROM and EEPROM); Flash memory; or other types of media suitable for storing electronic instructions, sequences of operations, functional descriptive information encodings, etc.

Generally, a variety of instances for components, operations, or structures may be provided, which are described herein as a single instance. Limits between various components, operations, and data stores are arbitrary, and specific operations are illustrated in the context of specific illustrative configurations. Other assignments of functionality are contemplated and may fall within the scope of the invention (s). In general, structures and functionality that are represented as separate components in the exemplary configurations can be implemented as a combined structure or component. Similarly, structures and functionality depicted as a single component can be implemented as separate components. These and other variations, modifications, additions and improvements may fall within the scope of the invention (s).

Claims (43)

A method comprising: Accessing a machine-readable coding of a first audiovisual performance that is recorded as vocal audio with performance-synchronized video and has an associated musical structure coding that includes at least music segment boundaries that are coded for a time comparison with the first audiovisual performance coding; Applying a first visual effects scheme to at least a portion of the first audiovisual performance encoding, the applied visual effects scheme encoding different visual effects for different musical structural elements of the first audiovisual performance encoding and providing transitions for visual effects in time alignment with at least some of the encoded musical section boundaries. Procedure according to Claim 1 , further comprising: segmenting at least one audio track of the first audiovisual performance encoding to provide the associated musical structure encoding. Procedure according to Claim 1 , wherein the associated musical structure coding includes group part or section music metadata. Procedure according to Claim 1 , with the different visual effects differing in either grade or type or both in grade and type. Procedure according to Claim 1 , further comprising: selecting the first visual effects scheme from a plurality of visual effects schemes. Procedure according to Claim 5 , whereby the selection is based on a calculated mood for at least the recorded vocal audio. Procedure according to Claim 5 , wherein the selection is based on a user interface selection by the vocal audio operator before or simultaneously with the recording of the vocal audio. Procedure according to Claim 5 further comprising: selecting a second visual effect scheme from the plurality of mood-designating visual effect schemes, the second visual effect scheme different from the first visual effect scheme; and applying the second visual effects scheme to at least a portion of the first audiovisual performance encoding. Procedure according to Claim 5 further comprising: streaming the first audiovisual show to an audience on one or more remote client devices. Procedure according to Claim 9 , wherein the streamed first audiovisual performance is mixed with a coding of a backing track, in contrast to which the vocal audio was recorded. Procedure according to Claim 9 , wherein the streamed first audiovisual presentation is streamed with the first visual effects scheme applied. Procedure according to Claim 8 further comprising: providing an identification of the visual effects scheme applied to video effects rendering on one or more of the remote client devices. Procedure according to Claim 5 , further comprising: transferring the first audiovisual presentation along with at least one identifier for the one or more visuals applied Effect schemes to, from, or through a content server or service platform. Procedure according to Claim 5 wherein the selection is based on a user interface selection during or before the audiovisual playback of the first audiovisual performance. Procedure according to Claim 14 further comprising: after the audiovisual rendering of the first audiovisual performance, selecting a second visual effects scheme from the plurality of mood-designating visual effects schemes, the second visual effects scheme being different from the first visual effects scheme; and applying the second visual effects scheme to at least a portion of the first audiovisual performance encoding. Procedure according to Claim 5 , wherein mood values are parameterized as a two-dimensional quantity for a special mood-designing visual effect scheme, a first dimension of the mood parameterization encoding an emotion and a second dimension of the mood parameterization encoding intensity. Procedure according to Claim 16 , further comprising: determining an intensity dimension of mood parameterization based on one or more of the following: (i) a time-varying audio signal strength or a measure of the vocal energy density that is computed from the vocal audio, and (ii) beats, tempo , Signal strength or energy density of a backing audio track. Procedure according to Claim 1 , further comprising: segmenting the first audiovisual performance encoding to identify the different musical structural elements. Procedure according to Claim 18 , wherein the segmentation is based at least in part on a mathematical determination of vocal intensity with at least some segmentation limits, which are limited in order to be temporally compared with beats or tempo, which are mathematically extracted from a corresponding audio backing track. Procedure according to Claim 18 , wherein the segmentation is based at least in part on a similarity analysis, which is carried out arithmetically on a chronologically synchronized song text track in order to classify specific parts of the first audiovisual presentation coding as a verse or refrain. Procedure according to Claim 1 , further comprising: segmenting an audio backing track associated with the first audiovisual performance encoding to identify the different musical structural elements. Procedure according to Claim 1 wherein the different visual effects encoded by the applied visual effects scheme includes, for a given element thereof, one or more of the following: a particle-based effect or a lens reflection; Transitions between or layouts of different source videos; Animations or movement of a frame within a source video; Vector graphics or images of patterns or textures; and color, saturation or contrast. Procedure according to Claim 1 wherein the different visual effects encoded by the applied visual effects scheme include, for a given element thereof, one or more of the following: visually expanding or embellishing a video image of a face, head, or body of a vocal performer; and background subtraction. Procedure according to Claim 1 , the associated musical structure encoding musical sections of different types; and wherein the applied visual effects scheme defines different visual effects for different of the encoded music sections. Procedure according to Claim 1 , the associated musical structure encoding events or transitions; and wherein the applied visual effects scheme defines different visual effects for different ones of the encoded events or transitions. Procedure according to Claim 1 , wherein the machine-readable encoding further encodes at least a portion of a second audiovisual performance that is recorded as second vocal audio with performance-synchronized video, the first and second audiovisual performance forming a group performance. Procedure according to Claim 26 , with the associated musical structure encoding group parts, and with the visual effects scheme applied for a special presentation-synchronized video can be selected according to the coded musical structure. Procedure according to Claim 26 , with the first and second audiovisual performances presented as a duet. Procedure according to Claim 26 , wherein the applied visual effect scheme encodes a color matching of performance-synchronized video for respective performers in the group performance for at least some musical structural elements. Procedure according to Claim 26 , wherein the applied visual effects scheme encodes for at least some musical structural elements visual blurring or mixing at an interface between performance-synchronized video for respective performers in the group performance. Procedure according to Claim 26 , with the first and second audiovisual performances recorded against a common backing track. Procedure according to one of the Claims 1 - 27th , further comprising: recording the first audiovisual performance on a network-connected vocal recording device that is communicatively coupled to a content server or service platform from which the musical structure coding is provided. Procedure according to Claim 32 wherein the recording of the audiovisual performance on the network-connected vocal recording device is carried out according to a karaoke-like operative mechanism in which a song text is visually presented in accordance with the audible reproduction of a backing track. Procedure according to one of the Claims 1 - 27th , which is carried out at least partially on a content server or a service platform with which geographically distributed, network-connected vocal recording devices are communicatively coupled. Procedure according to one of the Claims 1 - 27th , which is performed at least partially on a network-connected vocal recording device that is communicatively coupled to a content server or a service platform. Procedure according to one of the Claims 1 - 27th , which is implemented at least in part as a computer program product coding of instructions that can be executed on a content server or a service platform to which a plurality of geographically distributed, network-connected vocal recording devices are communicatively coupled. System that includes: a geographically distributed set of networked devices designed to record audiovisual performances, including vocal audio, with performance synchronized video; and a service platform that is designed to (i) receive encodings of the recorded audiovisual performances and, according to associated musical structural encodings, which at least include musical segment boundaries which are encoded for a time comparison with the audiovisual performance encodings, (ii) special visual effect schemes on at least apply some of the audiovisual performance encoding, wherein the visual effect schemes applied encode different visual effects for different musical structural elements of the audiovisual performance encoding and provide transitions for visual effects in time alignment with at least some of the encoded musical section boundaries. System according to Claim 37 wherein the service platform is configured in response to user or performer selections to change the visual effects schemes applied and deliver resulting recorded audiovisual performances to respective ones of the geographically distributed, network-connected audiovisual playback devices thereon. System according to Claim 38 , where the applied visual effect schemes are selected from several visual effect schemes. A system comprising: at least one guest-and-host pairing of networked devices configured to record at least vocal audio; and wherein the host device is configured to (i) receive a coding of a respective coding of at least vocal audio from the guest device and, correspondingly with an associated musical structure coding, which includes at least music segment boundaries which are coded for a time comparison with an audiovisual performance coding, (ii) Apply selected visual effects schemes to the audiovisual performance encoding, the applied visual effects schemes using different visual effects for different musical structural elements of the encode audiovisual performance encoding and provide transitions for visual effects in time alignment with at least some of the encoded musical section boundaries. System according to Claim 40 wherein the host and guest devices are coupled as local and remote peers over a communication network with non-negligible peer-to-peer latency for broadcasts of audiovisual content, the host device is communicatively coupled as the local peer for media encoding of a mixed audio presentation that forms vocal audio recorded on the guest device, and the guest device is communicatively coupled as the remote peer to provide the media encoding recorded by a first one of the performers and mixed with a backing audio track. System according to Claim 40 , wherein the associated musical structure coding is computationally determined on the host device based on a segmentation of at least one audio track received by the guest device. System according to Claim 40 , wherein the host device is configured to reproduce the audiovisual performance encoding as a mixed audiovisual performance including vocal audio and performance-synchronized video from the first and second of the performers, and to transmit the audiovisual performance encoding as an apparent live broadcast with the selected visual effects scheme applied.

Images