KR20200105700A - Voice effects based on facial expressions - Google Patents

Abstract

Embodiments of the present disclosure may provide techniques for adjusting audio and/or video information of a video clip based at least in part on facial feature features and/or voice feature features extracted from hardware components. have. For example, in response to detecting a request to create an avatar video clip of a virtual avatar, a video signal and an audio signal associated with a face within the camera's field of view may be captured. Speech feature features and facial feature features may be extracted from the audio signal and the video signal, respectively. In some examples, in response to detecting a request to preview the avatar video clip, an adjusted audio signal may be generated based at least in part on the facial feature characteristics and the voice feature characteristics, and the adjusted audio A preview of the video clip of the virtual avatar using the signal may be displayed.

Description

Voice effects based on facial expressions

Cross-reference to related applications

This application is filed on February 28, 2018, and is entitled "Voice Effects Based on Facial Expressions" in US Provisional Patent Application Nos. 15/908,603 and July 11, 2018. Claiming the benefit of priority from US Part Continuing Patent Application No. 16/033,111, filed dated and titled "Techniques for Providing Audio and Video Effects", these disclosures Are incorporated herein by reference in their entirety for all purposes.

Multimedia content such as emojis can be transmitted as part of messaging communications. Emojis can represent a variety of predefined people, objects, actions, and/or others. Some messaging applications allow users to select from a predefined library of emojis that can be sent as part of a message that can include other content (eg, other multimedia and/or text content). Animojis are one type of this different multimedia content, and users can select an avatar (eg, doll) to represent themselves. Animoji can move and speak as if it were a user's video. Animojis allow users to create personalized versions of emojis in a fun and creative way.

Embodiments of the present disclosure may provide systems, methods, and computer-readable media for implementing avatar video clip revision and playback techniques. In some examples, the computing device may present a user interface (UI) for tracking the user's face and presenting a virtual avatar representation (eg, a doll or a video character version of the user's face). Upon identifying the request to record, the computing device captures the audio and video information, extracts and detects facial feature features and voice feature features as well as context, and is based at least in part on the extracted/identified features. Accordingly, the audio and/or video information may be revised, and a video clip of the avatar may be presented using the revised audio and/or video information.

In some embodiments, a computer implemented method may be provided for implementing various audio and video effect techniques. The method may include displaying a virtual avatar creation interface. The method may also include displaying the first preview content of the virtual avatar in the virtual avatar creation interface, wherein the first preview content of the virtual avatar comprises real-time preview video frames of the user's headshot within the field of view of the camera and It can respond to associated headshot changes in appearance. The method may also include detecting an input in the virtual avatar creation interface while displaying the first preview content of the virtual avatar. In some examples, in response to detecting an input at the virtual avatar generation interface, the method further comprises: capturing, via a camera, a video signal associated with a user headshot during a recording session, via a microphone, a user audio signal during the recording session. Capturing, extracting audio feature features from the captured user audio signal, and extracting facial feature features associated with the face from the captured video signal. Additionally, in response to detecting the expiration of the recording session, the method further comprises generating an adjusted audio signal from the captured audio signal based at least in part on the facial feature characteristics and the audio feature characteristics, the facial feature The virtual avatar generation interface may include generating second preview content of the virtual avatar according to the sub-characteristics and the adjusted audio signal, and presenting the second preview content in the virtual avatar generation interface.

In some embodiments, the method further comprises storing facial feature metadata associated with facial feature features extracted from the video signal, and based at least in part on the facial feature features and audio feature features. And generating the adjusted facial feature metadata from the facial feature metadata. Additionally, the second preview of the virtual avatar may be additionally displayed according to the adjusted face metadata. In some examples, the first preview of the virtual avatar may be displayed according to preview facial feature characteristics identified according to changes in the appearance of the face during the preview session.

In some embodiments, an electronic device may be provided for implementing various audio and video effect techniques. The system may include a camera, a microphone, a library of prerecorded/predetermined audio, and one or more processors in communication with the camera and microphone. In some examples, processors may be configured to execute computer-executable instructions to perform operations. The operations may include detecting an input in the virtual avatar generation interface while displaying the first preview of the virtual avatar. The actions may also include in response to detecting input at the virtual avatar creation interface, initiating an containing capture session. The capture session includes, via the camera, capturing a video signal associated with a face within the camera's field of view, via a microphone, capturing an audio signal associated with the captured video signal, and extracting audio feature characteristics from the captured audio signal. And extracting facial feature features associated with the face from the captured video signal. In some examples, the actions also include, at least in response to detecting the expiration of the capture session, generating an adjusted audio signal based at least in part on the audio feature features and facial feature features, and creating a virtual avatar. It may include presenting the second preview content in the interface.

In some cases, the audio signal may be further adjusted based at least in part on the type of virtual avatar. Additionally, the type of the virtual avatar may be received based at least in part on the avatar type selection affordance presented in the virtual avatar generation interface. In some cases, the type of the virtual avatar may include an animal type, and the adjusted audio signal may be generated based at least in part on a predetermined sound associated with the animal type. The use and timing of the predetermined sounds may be based on audio features from captured audio and/or facial features from captured video. This predetermined sound can also be modified itself based on audio features from captured audio and facial features from captured video. In some examples, one or more processors may be further configured to determine whether a portion of the audio signal corresponds to a face in the field of view. Additionally, depending on the determination that a portion of the audio signal corresponds to a face, a portion of the audio signal may be stored for use in generating the adjusted audio signal and/or upon the determination that the portion of the audio signal does not correspond to a face. Accordingly, at least a portion of the audio signal may be discarded and not considered for modification and/or reproduction. Additionally, audio feature features may include features of speech associated with a face within the field of view. In some examples, one or more processors may be further configured to store facial feature metadata associated with facial feature features extracted from the video signal. In some examples, the one or more processors may be further configured to store audio feature metadata associated with audio feature features extracted from the audio signal. Further, the one or more processors may be further configured to generate the adjusted facial metadata based at least in part on the facial feature characteristics and the audio feature characteristics, and the second preview of the virtual avatar is It can be generated according to the data and the adjusted audio signal.

In some embodiments, a computer-readable medium may be provided. A computer-readable medium may contain computer-executable instructions that, when executed by one or more processors, cause one or more processors to perform operations. The actions are the following actions in response to detecting a request to create an avatar video clip of a virtual avatar: capturing a video signal associated with a face in the field of view of the camera, via the camera of the electronic device, the microphone of the electronic device Via, capturing an audio signal, extracting voice feature features from the captured audio signal, and extracting facial feature features associated with the face from the captured video signal. Also in response to detecting a request to preview the avatar video clip, generating an adjusted audio signal based at least in part on the following actions: facial feature features and voice feature features, and adjusting And displaying a preview of the video clip of the virtual avatar using the generated audio signal.

In some embodiments, the audio signal may be adjusted based at least in part on a facial expression identified in facial feature characteristics associated with the face. In some cases, the audio signal may be adjusted based, at least in part, on a level, pitch, duration, format, or change of a voice characteristic associated with the face. Further, in some embodiments, the one or more processors may be further configured to perform operations including transmitting a video clip of the virtual avatar to another electronic device.

The following detailed description in conjunction with the accompanying drawings will provide a better understanding of the nature and advantages of the present disclosure.

1 is a simplified block diagram illustrating an example flow for providing audio and/or video effects techniques as described herein, according to at least one example.
2 is another simplified block diagram illustrating an exemplary flow for providing audio and/or video effects techniques as described herein, according to at least one example.
3 is another simplified block diagram illustrating hardware and software components for providing audio and/or video effects techniques as described herein, according to at least one example.
4 is a flow chart to illustrate providing audio and/or video effect techniques as described herein, according to at least one example.
5 is another flow chart to illustrate providing audio and/or video effect techniques as described herein, according to at least one example.
6 is a simplified block diagram illustrating a user interface for providing audio and/or video effects techniques as described herein, according to at least one example.
7 is another flow chart to illustrate providing audio and/or video effect techniques as described herein, according to at least one example.
8 is another flow chart to illustrate providing audio and/or video effect techniques as described herein, according to at least one example.
9 is a simplified block diagram illustrating a computer architecture for providing audio and/or video effects techniques as described herein, in accordance with at least one example.

Certain embodiments of the present disclosure provide devices, computer-readable media, and methods for implementing various techniques for providing voice effects (e.g., revised audio) based at least in part on facial expressions. It is about. Additionally, in some cases, various techniques may also provide video effects based at least in part on the audio characteristics of the recording. Furthermore, various techniques may also provide (eg, together) voice effects and video effects based at least in part on one or both of the facial expressions and audio characteristics of the recording. In some examples, voice effects and/or video effects may be presented in a user interface (UI) configured to display a cartoon representation of the user (eg, an avatar or digital puppet). Such an avatar representing the user can be considered an animoji because it may appear to be an emoji character familiar to most smart phone users, but it can be animated to mimic the actual movements of the user.

For example, a user of the computing device may be presented with a UI for creating an Animoji video (eg, a video clip). The video clip may be limited to a predetermined amount of time (eg, 10 seconds, 30 seconds, etc.), or the video clip may not be limited. In the UI, the preview area may present a real-time expression of the user's face to the user using an avatar character. Various avatar characters can be provided, and the user can even create or import his own avatars. The preview area may be configured to provide an initial preview of the avatar and a preview of the recorded video clip. Additionally, the recorded video clip may be previewed in its original form (eg, without any video or audio effects), or may be previewed with audio and/or video effects. In some cases, the user may select an avatar after the initial video clip is recorded. The video clip preview can then be appropriately changed from one avatar to another avatar to which the same or different video effects have been applied accordingly. For example, if a user switches avatar characters while a raw preview (e.g., the original form with no effects) is being shown, the UI will display a rendering of the same video clip but will be updated to have the newly selected avatar. I can. In other words, facial features and audio (eg, the user's voice) captured during recording may be presented from any of the avatars (eg, without any effects). In the preview, it will appear as if the avatar character moves in the same direction the user moved during recording, and says what the user said during recording.

For example, the user may select a first avatar (eg, a unicorn head) through the UI, or a default avatar may be initially provided. The UI will present an avatar (in this example the head of a cartoon unicorn if selected by the user or any other available doll by default) in the preview area, and the device (e.g., one or more microphones and/or one or more Using cameras) will begin to capture audio and/or video information. In some cases, only video information is needed for the initial preview screen. Video information can be analyzed, and facial features can be extracted. These extracted facial features can then be mapped to the unicorn face in real time so that the initial preview of the unicorn head appears to reflect the user's head. In some cases, the term "real-time" is used to indicate that the results of extraction, mapping, rendering, and presentation can be performed in response to each movement of the user and presented substantially immediately. To the user, this will appear to be looking in a mirror except that the image of the user's face has been replaced with an avatar.

While the user's face is within the line of sight (eg, view) of the device's camera, the UI will continue to present the initial preview. Upon selection of the recording affordance (eg, a virtual button) on the UI, the device may start capturing video with an audio component. In some examples, this includes the camera capturing frames and the microphone capturing audio information. Special cameras capable of capturing three-dimensional (3D) information can also be utilized. Additionally, in some examples, any camera capable of capturing video may be utilized. The video may be stored in its original form and/or metadata associated with the video may be stored. As such, capturing video and/or audio information may differ from storing information. For example, capturing information may include sensing the information and at least caching the information to be available for processing. The processed data can also be cached until a decision is made whether to store or simply utilize the data. For example, during the initial preview, while the user's face is being presented as a doll in real time, video data (eg, metadata associated with the data) may be cached while it is mapped to the doll and presented. However, such data is not stored permanently at all, so the initial preview may not be reusable or recoverable.

Alternatively, in some examples, once the user selects the recording affordance of the UI, the video data and audio data may be more permanently stored. In this way, the audio and video (A/V) data can be analyzed, processed, etc. to provide the audio and video effects described herein. In some examples, video data may be processed to extract facial features (eg, facial feature features), and these facial features may be stored as metadata for an Animoji video clip. The set of metadata may be stored as an identifier (ID) representing the time, date and user associated with the video clip. Additionally, audio data may be stored with the same or different IDs. Once stored, or in some examples, prior to storage, the system (eg, the processors of the device) may extract audio feature features from the audio data and facial feature features from the video file. This information can be used to identify the user's context, key words, intent, and/or emotions, and video and audio effects can be introduced into the audio and video data before rendering the puppets. In some examples, the audio signal may be adjusted to include different words, sounds, tones, tones, timing, etc. based at least in part on the extracted features. Additionally, in some examples, video data (eg, metadata) can also be adjusted. In some examples, audio features are extracted in real time during the preview itself. These audio features are avatar specific and can only be created if the associated avatar is being previewed. Audio features are avatar agnostic and can be created for all avatars. The audio signal can also be adjusted based in part on these real-time audio feature extractions, and during or after the recording process, but using pre-stored extracted video features created before previewing.

Once the video and audio data is adjusted based at least in part on the extracted features, a second preview of the doll may be rendered. This rendering can be performed for each possible doll, for example as the user scrolls and selects different dolls, the adjusted data is already rendered. Alternatively, rendering may be performed after selection of each doll. In any case, once the user selects the doll, a second preview may be presented. The second preview will play a video clip recorded by the user but with audio and/or video adjusted. Using the example from above, if the user recorded himself in an angry tone (e.g., a rough voice and a wrinkled forehead), the context or intent of the anger could be detected, and the audio file would contain a growling sound. Can be adjusted. Thus, the second preview can be adjusted to make the unicorn appear to speak words spoken by the user, but the user's voice to sound as if growling, or to make the tone more baritone (eg, lower). The user can then save the second preview or select it for transmission to another user (eg, via a messaging application, etc.). In some examples, the below and above Animoji video clips can be shared as .mov files. However, in other examples, the described techniques may be used in real time (eg, using video messaging or the like).

1 is a simplified block diagram illustrating an exemplary flow 100 for providing audio and/or video effects based at least in part on audio and/or video features detected in a user's recording. In the exemplary flow 100, there are two separate sessions, a recording session 102 and a playback session 104. In the recording session 102, the device 106 can capture a video with the audio component of the user 108 at block 110. In some examples, video and audio may be captured (eg, collected) separately using two different devices (eg, microphone and camera). The capture of video and audio may be triggered based at least in part on the selection of recording affordance by the user 108. In some examples, user 108 can say the word “hello” at block 112. Additionally, at block 112, device 106 may continue to capture video and/or audio components of the user's actions. At block 114, device 106 may continue to capture video and audio components, and in this example, user 108 may say the word “bark”. At block 114, device 106 may also extract spoken words from the audio information. However, in other examples, speech word extraction (or any audio feature extraction) may actually occur after recording session 102 is complete. In other examples, speech word extraction (or any audio feature extraction) may actually occur in real time during preview block 124. Further, extraction (eg, analysis of audio) may be performed in real time while recording session 102 is still in progress. In either case, the avatar process executed by device 106 can identify through extraction that the user has spoken the word "barking" and employ some logic to determine which audio effects to implement.

As an example, the recording session 102 may be performed when the user 108 reselects the recording affordance (e.g., indicating a request to end recording), or selects the termination recording affordance (e.g., recording affordance During, it may act as an end recording affordance), or may end based at least in part on the expiration of a period (eg, 10 seconds, 30 seconds, etc.). In some cases, this period can be automatically predetermined, while in other cases it can be user selection (e.g., it can be selected from a list of options or entered in a free form via a text input interface. ). When recording is complete, the user 108 may select a preview affordance indicating that the user 108 wants to view a preview of the recording. One option may be to play the original recording without any visual or audio effects. However, another option could be to play a revised version of the video clip. Based at least in part on detection of the spoken word “barking”, the avatar process may have revised the audio and/or video of the video clip.

At block 116, device 106 may present an avatar (also referred to as a doll and/or animoji) 118 on the screen. Device 106 may also be configured with a speaker 120 capable of playing audio associated with a video clip. In this example, block 116 corresponds to the same time point as block 110, and user 108 may have opened his mouth, but not yet spoken. As such, the avatar 118 may be presented with its mouth open, but the audio is not yet presented from the speaker 120. At block 122 corresponding to block 112 where user 108 said "hello", the avatar process may present the avatar 118 with an avatar-specific voice. In other words, at block 122, a predefined dog voice may be used to say the word “hello”. The dog spoken word "Hello" may be presented by speaker 120. As described in more detail below, there are a variety of different animal (and other character) avatars available for selection by the user 108. In some examples, each avatar may be associated with a specific predefined voice that best fits that avatar. For example, a dog may have a dog voice, a cat may have a cat voice, a pig may have a pig voice, and a robot may have a robot voice. These avatar-specific voices can be pre-recorded or be associated with specific frequency or audio conversions that can occur by executing mathematical operations on the original sound so that any user's voice can be converted to sound like a dog's voice. I can. However, each user's dog voice may sound different based at least in part on the particular audio conversion being performed.

At block 124, the avatar process may replace the spoken word (eg, “barking”) with an avatar-specific word. In this example, the sound of a dog barking sound (eg, recorded or simulated dog barking) may be inserted into the audio data (eg, instead of the word “barking”) to be played during presentation of the video clip. In this case, a “woof” may be presented by the speaker 120. In some examples, at 124, different avatar-specific words will be presented based at least in part on different avatar selections, and in other examples, the same avatar-specific word may be presented regardless of the avatar selections. For example, if user 108 says "barking", then "bruising" may be presented when a dog avatar is selected. However, in this same case, if the user 108 later selects a cat avatar for the same flow, there are several options to revise the audio. In one example, the process can convert "barking" to "bruising" even if it is not suitable for a cat to "bruise." In a different example, the process may convert "barking" into recorded or simulated "meow" based at least in part on the selection of a cat avatar. And in another example, the process may ignore "barking" for avatars other than dog avatars. As such, there may be a second level of audio feature analysis performed even after extraction at 114. Video and audio features can also affect processing for avatar specific utterances. For example, the level and tonality and intonation that the user says "barking" may be detected as part of the audio feature extraction, which allows the system to select a specific "bruise" sample before and/or during the preview process, or You can instruct the sample to be converted.

2 is another simplified block diagram illustrating an exemplary flow 200 for providing audio and/or video effects based at least in part on audio and/or video features detected in a user's recording. In the exemplary flow 200, very similar to the exemplary flow 100 of FIG. 1, there are two separate sessions, a recording session 202 and a playback session 204. In the recording session 202, the device 206 can capture a video with the audio component of the user 208 at block 210. The capture of video and audio may be triggered based, at least in part, on the selection of recording affordance by the user 208. In some examples, user 208 can say the word “hello” at block 212. Additionally, at block 212, device 206 may continue to capture video and/or audio components of the user's actions. At block 214, device 206 may continue to capture video and audio components, and in this example, user 208 may keep his mouth open, but not say anything. At block 214, device 206 may also extract facial expressions from the video. However, in other examples, facial feature extraction (or any video feature extraction) may actually occur after the recording session 202 is complete. Still, extraction (eg, analysis of the video) may be performed in real time while recording session 202 is still in progress. In either case, the avatar process executed by device 206 can identify through extraction that the user has briefly opened his or her mouth (e.g., without saying anything), and implement some audio and/or video effects. Some logic can be employed to determine whether or not. In some examples, the determination that a user has said nothing and kept their mouths open may require extraction and analysis of both audio and video. For example, extraction of facial feature features (e.g., open mouth) may not be sufficient, and the process may also need to detect that the user 208 has not said anything during the same recording period. have. Video and audio features can also affect processing for avatar specific utterances. For example, the duration of mouth open, eye open, etc. may instruct the system to select a particular “bruise” sample, or to convert such a sample before and/or during the preview process. One such transformation is to change the level and/or duration of the bruise to match the detected opening and closing of the user's mouth.

As an example, the recording session 202 may be performed when the user 208 reselects the recording affordance (e.g., indicating a request to end recording), or selects the termination recording affordance (e.g., recording affordance May be terminated based at least in part upon expiration of an end recording affordance), or an expiration of a period (eg, 20 seconds, 30 seconds, etc.). When recording is finished, the user 208 may select a preview affordance indicating that the user 208 wishes to view a preview of the recording. One option may be to play the original recording without any visual or audio effects. However, another option could be to play the revised version of the recording. Based at least in part on detection of a facial expression (eg, an open mouth), the avatar process may have revised the audio and/or video of the video clip.

At block 216, device 206 may present an avatar (also referred to as a doll and/or animoji) 218 on the screen of device 206. Device 206 may also be configured with a speaker 220 capable of playing audio associated with a video clip. In this example, block 216 corresponds to the same time point as block 210 and user 208 may not have spoken yet. As such, the avatar 218 may be presented with its mouth open, but the audio has not yet been presented from the speaker 220. At block 222 corresponding to block 212 where user 208 says "hello", the avatar process may present the avatar 218 with an avatar-specific voice (as described above).

At block 224, the avatar process may replace the silence identified at block 214 with an avatar-specific word. In this example, the sound of a dog barking sound (eg, recorded or simulated dog barking) can be inserted into the audio data (eg, instead of silence), so that when played during presentation of a video clip, " A bruise may be presented by speaker 220. In some examples, at 224, different avatar-specific words will be presented based at least in part on different avatar selections, and in other examples, the same avatar-specific word may be presented regardless of the avatar selections. For example, if the user 208 keeps his mouth open, a “bruise” may be presented when a dog avatar is selected, a “meow” sound may be presented for a cat avatar, and the like. In some cases, each avatar is a predefined to be played when it is detected that the user 208 has not spoken and has kept his mouth open for a certain amount of time (e.g., 0.5 seconds, 1 second, etc.) You can have a good sound. However, in some examples, the process may ignore detection of an open mouth for avatars that do not have a predefined effect on such facial features. Additionally, there may be a second level of audio feature analysis performed even after extraction at 214. For example, if the process determines that a "bruise" should be inserted for a dog avatar (e.g., based on detection of an open mouth), the process may also (for example, allow the user to indicate "barking"). It is possible to detect how many "bruise" sounds will be inserted (when keeping the mouth open at twice the time used), or (for example, in the scenario of Figure 1, the user says "barking" In other words, it is possible to detect whether it is not possible to insert as many as the requested number of "barks"), indicating that the "bruising" sound should be inserted. Thus, based on the above two examples, it will be apparent that the user 208 can control the effects of playback (eg, recorded avatar messages) with their facial expressions and voice expressions. Also, although not explicitly shown in Figure 1 or Figure 2, the user device can process avatars (e.g., capture A/V information, extract features, analyze data, implement logic, And/or software for executing revising video files and rendering previews, as well as an application that allows a user to build avatar messages and subsequently transfer them to other user devices (e.g. , An avatar application having its own UI).

3 is a simplified block diagram 300 illustrating components (eg, software modules) utilized by the avatar process described above and below. In some examples, more or fewer modules may be utilized to implement the provision of audio and/or video effects based at least in part on audio and/or video features detected in the user's recording. In some examples, the device 302 is a display to present camera 304, microphone 306, and UI and avatar previews (e.g., a preview of the recording prior to transmission as well as an initial preview before recording). It can be composed of a screen. In some examples, the avatar process consists of an avatar engine 308 and a speech engine 310. The avatar engine 308 manages the list of avatars, processes video features (e.g., facial feature features), revise video information, communicates with the voice engine 301 when appropriate, and processes all When this is complete and the effects have been implemented (or discarded), the video of the avatar 312 can be rendered. Revision of video information may include adjusting or otherwise editing metadata associated with the video file. In this way, when video metadata (adjusted or not adjusted) is used to render dolls, facial features can be mapped to the doll. In some examples, the speech engine 310 can store audio information, perform logic to determine which effects to implement, revise the audio information, and when all processing is complete and effects have been implemented (or discarded). When) the revised audio 314 can be provided.

In some examples, if the user chooses to record a new avatar video clip, the video features 316 may be captured by the camera 304 and the audio features 318 may be captured by the microphone 306. have. In some cases, up to 50 (or more) facial features may be detected within video features 316. Exemplary video features include, but are not limited to, duration of facial expressions, open mouth, frowns, laughter, raised eyebrows or frowns, and the like. Additionally, the video features 316 may only include metadata that identifies each of the facial features (eg, data points indicating which locations on the user's face have moved or where they are). In addition, video features 316 may be passed to avatar engine 308 and voice engine 310. In the avatar engine 308, metadata associated with the video features 316 may be stored and analyzed. In some examples, avatar engine 308 may perform feature extraction from the video file prior to storing the metadata. However, in other examples, feature extraction may be performed before the video features 316 are sent to the avatar engine (in this case, the video features 316 will be the metadata itself). In speech engine 310, video features 316 may be helpful to match which audio features correspond to which video features (e.g., to see if certain audio and video features occur simultaneously). When compared to the audio features 318.

In some cases, audio features are also passed to speech engine 310 for storage. Exemplary audio features include, but are not limited to, level, tone, dynamics (eg, changes in level, tone, voice, formats, duration, etc.). Raw audio 320 contains the unprocessed audio file as it was captured. The raw audio 320 can be passed to the speech engine 310 for further processing and potential (e.g., ultimate) revision, which can also be stored separately so that the original audio can be used if desired. have. Raw audio 320 may also be passed to the speech recognition module 322. The speech recognition module 322 can be used to spot words and identify their intent from the user's speech. For example, the voice recognition module 322 may determine when the user is angry, sad, or happy. Additionally, if the user speaks a key word (eg, “barking” as described above), the speech recognition module 322 will detect it. The information detected and/or collected by the speech recognition module 322 may then be passed to the speech engine 310 for further logic and/or processing. As mentioned, in some examples, audio features are extracted in real time during the preview itself. These audio features are avatar specific and can only be created if the associated avatar is being previewed. The audio features are avatar agnostic and can be created for all avatars. The audio signal can also be adjusted based in part on these real-time audio feature extractions, and during or after the recording process, but using pre-stored extracted video features created before previewing. Additionally, some feature extraction may be performed by speech engine 310 during rendering at 336. Some of the pre-stored sounds 338 may be suitably used by the speech engine 310 to fill in the blanks or replace other extracted sounds.

In some examples, the speech engine 310 will make a decision as to how to process the information extracted from the speech recognition module 322. In some examples, the speech engine 310 can pass information from the speech recognition module 322 to the feature module 324 to determine which features correspond to data extracted by the speech recognition module 322. . For example, the feature module 324 indicates that the sad voice detected by the speech recognition module 322 corresponds to an increase in pitch of the voice or a deceleration of the speed or cadence of the voice (e.g., On a set and/or logic basis). In other words, the feature module 322 may map the extracted audio features to specific voice features. The effect type module 326 may then map the specific voice features to the desired effect. Speech engine 310 may also be responsible for storing each specific voice for each possible avatar. For example, there may be standard or hardcoded voices for each avatar. If no other changes are made and the user selects a particular avatar, the speech engine 310 may select an appropriate standard voice for use with playback. In this case, the modified audio 314 may only be the original audio 320 converted to an appropriate avatar voice based on the selected avatar. As the user scrolls through the avatars and selects different ones, the speech engine 310 can modify the raw audio 320 on the fly to sound like a newly selected avatar. Thus, the avatar type 328 needs to be provided to the speech engine 310 to make this change. However, if an effect is to be provided (e.g., tones, tones, or actual words have to be changed within the audio file), the speech engine 310 revises the original audio file 320 and modifies the audio 314 ) Can be provided. In some examples, the user will be provided an option to use the original audio file on/off 330. When the user selects “off” (e.g., effect off), the raw audio 320 is combined with the video of the avatar 312 (e.g., corresponding to the unchanged video) to output A/V ( 332) can be created. The A/V output 332 may be provided to the avatar application presented on the UI of the device 302.

The avatar engine 308 may be responsible for providing the initial avatar image based at least in part on the selection of the avatar type 328. Additionally, the avatar engine 308 is responsible for mapping the video features 316 to the appropriate facial markers of each avatar. For example, when the video features 316 indicate that the user is smiling, metadata representing the smile may be mapped to the mouth area of the selected avatar so that the avatar appears to be smiling in the video of the avatar 312. Additionally, the avatar engine 308 may properly receive timing changes 334 from the speech engine. For example, the voice engine 310 determines that the voice effect makes the audio somewhat whispering voice (e.g., based on the feature module 324 and/or effect type 326 and/or avatar type) and In the case of modifying the voice to a whispering voice, this effect change may include a slowdown of the voice itself, in addition to reduced level and other format and tone changes. Thus, the speech engine can produce modified audio with a slower playback speed compared to the original audio file for the audio clip. In this scenario, the speech engine 310 will need to instruct the avatar engine 308 via timing changes 334 to allow the video file to be properly slowed down, otherwise the video and audio will not be synchronized. will be.

As mentioned, the user can use the avatar application of the device 302 to select different avatars. In some examples, the voice effect can be changed based at least in part on this selection. However, in other examples, the user may be given the opportunity to select a different voice for a given avatar (eg, cat voice for a dog avatar, etc.). This type of free-form voice effect change can be executed by the user through selection on the UI, or in some cases by voice activation or facial movement. For example, a predetermined facial expression may trigger the voice engine 310 to change a voice effect for a given avatar. Also, in some examples, the voice engine 310 may be configured to make the child's voices sound higher in pitch, or alternatively, the raw audio 320 for the child's voice may already be in high pitch. Considering that it may sound inappropriate, it can be configured to decide not to make a child's voice higher pitched. Making such user specific decisions about the effect may be driven in part by the extracted audio features, in which case those features may include tonal values and ranges throughout the recording.

In some examples, the speech recognition module 322 may include a recognition engine, a word recognizer, a tone analyzer, and/or a format analyzer. The analysis performed by the speech recognition module 322 may identify whether the user is upset, angry, happy, or the like. Additionally, the speech recognition module 322 can not only identify the context and/or intonation of the user's speech, but also change the intention of wording and/or determine the user's profile (eg, virtual identity). I can.

In some examples, the avatar process 300 may be configured to package/render a video clip by combining the video of the avatar 312 and the modified audio 314 or raw audio 320 into an A/V output 332. have. To package the two, the speech engine 310 only needs to know the ID for the metadata associated with the video of the avatar 312 (e.g., it does not actually need the video of the avatar 312, It just needs the ID of the metadata). Messages within a messaging application (eg, an avatar application) can be sent to other computing devices, and the message includes an A/V output 332. When the user selects the "send" affordance in the UI, the latest video clip to be previewed may be transmitted. For example, if the user previews his video clip with a dog avatar and then switches to a cat avatar for preview, the cat avatar video will be sent when the user selects "Send". Additionally, the state of the recent preview is saved and can be used later. For example, if a recently transmitted message (eg, an avatar video clip) uses a specific effect, the first preview of the next message generated may utilize the specific effect.

The logic implemented by the speech engine 310 and/or the avatar engine 308 may identify certain cues and/or features and then revise the audio and/or video files to achieve the desired effect. Some exemplary feature/effect pairs include detecting that the user has opened his mouth and paused. In this example, both facial feature features (eg, mouth open) and audio feature features (eg, silence) need to occur simultaneously in order to achieve the desired effect. For these feature/effect pairs, the desired effect revises the audio and video so that the avatar appears to emit an avatar/animal-specific sound. For example, a dog will make a barking sound, a cat will make a meow sound, and a monkey, horse, unicorn, etc. will make sounds suitable for that character/animal. Other exemplary feature/effect pairs include lowering the audio pitch and/or tone when distortion is detected. In this example, only video feature characteristics need to be detected. However, in some examples, this effect may be implemented based at least in part on the speech recognition module 322 that detects the sadness of the user's voice. In this case, the video features 316 would not be needed at all. Other exemplary feature/effect pairs include slowing down audio and video, lowering tone and/or whipping to reduce changes. In some cases, video changes can lead to modifications of the audio, while in other cases, audio changes can lead to modifications of the video.

As mentioned above, in some examples, the avatar engine 308 can act as a feature extractor, in which case the video features 316 and audio features 318 are prior to being sent to the avatar engine 308. It may not exist. Instead, the raw audio 320 and metadata associated with the raw video may be passed to the avatar engine 308, where the avatar engine 308 may include audio feature features and video (eg, face) feature features. Can be extracted. In other words, although not shown in this manner in FIG. 3, portions of the avatar engine 308 may actually exist within the camera 304. Additionally, in some examples, metadata associated with video features 316 can be stored within a secure container, and when speech engine 310 is running, it can read metadata from the container.

In some cases, since the avatar's preview video clip is not displayed in real time (e.g., it is rendered and displayed only after the video has been recorded, and sometimes only in response to a selection of playback affordance), audio and video information. Can be processed offline (eg, not in real time). As such, the avatar engine 308 and speech engine 310 can read audio and video information in advance and make context decisions in advance. The speech engine 310 can then revise the audio file accordingly. This ability to read ahead and make decisions offline will greatly increase the efficiency of the system, especially for longer recordings. Additionally, this allows for a second analysis step in which additional logic can be processed. Thus, the entire audio file can be analyzed before making any final decisions. For example, if a user says "barking" twice in a row, but the words "barking" are spoken too closely together, the actual prerecorded "bruising" sound will be in the amount of time it takes the user to say "barking, barking". It may not be possible to fit. In this case, the speech engine 310 may take the information from the speech recognition 322 and decide to ignore the second “barking”, because it would not be possible to include both “bruise” sounds in the audio file. Because it is.

As mentioned above, when the audio file and video are packaged together for A/V output 332, the voice engine does not need to actually access the video of the avatar 312. Instead, a video file (eg, a .mov format file, etc.) is created when the video is playing by accessing an array of features (eg, floating point values) recorded in the metadata file. However, all permutations/adjustments to audio and video files can be done in advance, and even some can be done in real time when audio and video are extracted. Additionally, in some examples, each modified video clip can be temporarily stored (e.g., cached) so that if the user reselects an avatar that has already been previewed, generating/rendering that particular preview. There is no need to duplicate the processing for this. As opposed to re-rendering the revised video clip each time the same avatar is selected during the preview section, the caching of the previously mentioned rendered video clips is particularly useful for longer recordings and/or recordings with a large number of effects. On the other hand, it will enable the realization of significant savings in processor power and instructions per second (IPS).

Additionally, in some examples, noise suppression algorithms may be employed to handle the case where the sound captured by the microphone 306 includes sounds other than the user's voice. For example, when the user is in a windy area or in a noisy room (eg, a restaurant or bar). In these examples, the noise suppression algorithm can lower the decibel output of certain portions of the audio recording. Alternatively or additionally, different voices may be separated and/or only audio coming from certain viewing angles (eg, the angle of the user's face) may be collected, and other voices may be ignored or suppressed. In other cases, if the avatar process 300 determines that the noise levels are too large or difficult to process, the process 300 may disable the write option.

4 illustrates an example flow diagram illustrating a process 400 for implementing various audio and/or video effects based at least in part on audio and/or video features, in accordance with at least some embodiments. In some examples, computing device 106 of FIG. 1 or other similar user device (eg, utilizing at least the avatar process 300 of FIG. 3) may perform process 400 of FIG. 4.

At block 402, computing device 106 can capture video with an audio component. In some examples, video and audio can be captured by two different hardware components (eg, a camera can capture video information, while a microphone can capture audio information). However, in some cases, a single hardware component may be configured to capture both audio and video. In any event, video and audio information may be associated with each other (eg, by sharing an ID, timestamp, etc.). As such, the video can have an audio component (eg, they are part of the same file), or the video can be linked with an audio component (eg, are two files associated together).

At block 404, computing device 106 may extract facial features and audio features, respectively, from the captured video and audio information. In some cases, facial feature information may be extracted via avatar engine 308 and stored as metadata. The metadata can be used to map each facial feature to a specific doll or to any animated or virtual face. Therefore, since there is no need to store an actual video file, memory storage efficiency and significant savings can be obtained. With regard to audio feature extraction, a speech recognition algorithm can be utilized to extract different speech features, eg words, phrases, tone, speed, and the like.

At block 406, computing device 106 can detect the context from the extracted features. For example, the context may include the user's intentions, moods, settings, location, background items, ideas, and the like. Context can be important when employing logic to determine which effects to apply. In some cases, the context may be combined with the detected spoken words to determine whether and/or how to adjust the audio file and/or video file. In one example, the user can frown his eyebrows and speak slowly. The brow frown is a video feature that may have been extracted at block 404 and slow speech is an audio feature that may have been extracted at block 404. Separately, these two features may mean different things, but when combined together, the avatar process may determine that the user is worried about something. In this case, the context of the message may be that the parent is speaking to the child, or the friend is speaking to another friend about a serious or worrying problem.

At block 408, computing device 106 may determine effects for rendering audio and/or video files based at least in part on the context. As mentioned above, one context can be a concern. As such, certain video and/or audio features may be employed for this effect. For example, a voice file can be adjusted to sound more depressing or slowing down. In other examples, the avatar-specific voice may be replaced with a version of the original (eg, raw) audio to convey the severity of the message. A variety of different effects can be employed for a variety of different contexts. In other examples, the context may be animal sounds (eg, based on the user saying “barking” or “meow”, etc. In this case, the determined effect is to replace the spoken word “barking” with the sound of a dog barking. Will be.

At block 410, computing device 106 may perform additional logic for additional effects. For example, if the user attempts to achieve the barking effect by saying the barking twice in a row, additional logic may need to be utilized to determine whether the additional barking is technically feasible. As an example, if the audio clip of the bark used to replace the spoken word in the raw audio information is 0.5 seconds long, but the user says "barking" twice over a 0.7 second period, the additional logic is that the two barking sounds are available. It can be determined that it cannot be set within 0.7 seconds. Thus, the audio and video file may need to be expanded to fit the two barking sounds, the barking sound may need to be shortened (e.g., by processing the stored barking sound), or the second voice Word barking may need to be ignored.

At block 412, computing device 106 may revise the audio and/or video information based at least in part on the determined effects and/or additional effects. In some examples, only one set of effects can be used. However, in either case, the original audio file can be adjusted (eg, revised) to form a new audio file as additional sounds are added and/or removed. For example, in the case of using "barking", the spoken word "barking" will be removed from the audio file and a new sound representing the actual dog barking will be inserted. The new file can be saved with a different ID, or with an attached ID (original audio ID with a .v2 identifier to indicate that it is not the original). Additionally, the raw audio file will be stored separately so that it can be reused for additional avatars and/or if the user decides not to use the determined effects.

At block 414, computing device 106 may receive a selection of an avatar from a user. The user can select one of a plurality of different avatars through the UI of the avatar application being executed by the computing device 106. Avatars can be selected via a scroll wheel, a drop-down menu, or an icon menu (eg, where each avatar can be viewed on the screen at its own location).

At block 416, computing device 106 can present the revised video with revised audio based at least in part on the selected avatar. In this example, each adjusted video clip (e.g., a final clip for an avatar that adjusted audio and/or adjusted video) may be created for each respective avatar prior to selection of the avatar by the user. . In this way, the processing has already been completed and the adjusted video clip is ready to be presented upon selection of the avatar. This may require an additional IPS prior to avatar selection, but this will accelerate the presentation. Additionally, processing of each adjusted video clip may be performed while the user is reviewing the first preview (eg, the preview corresponding to the first/default avatar presented in the UI).

5 illustrates an example flow diagram illustrating a process 500 for implementing various audio and/or video effects based at least in part on audio and/or video features, in accordance with at least some embodiments. In some examples, computing device 106 of FIG. 1 or other similar user device (eg, utilizing at least the avatar process 300 of FIG. 3) may perform process 500 of FIG. 5.

At block 502, computing device 106 can capture video with an audio component. As in block 402 of FIG. 4, video and audio can be captured by two different hardware components (e.g., a camera can capture video information, while a microphone can capture audio information. Can). As mentioned, the video can have an audio component (eg, they are part of the same file), or the video can be linked with an audio component (eg, are two files associated together).

At block 504, computing device 106 may extract facial features and audio features, respectively, from the captured video and audio information. As described above, the facial feature information may be extracted through the avatar engine 308 and stored as metadata. The metadata can be used to map each facial feature to a specific doll or to any animated or virtual face. Therefore, since there is no need to store an actual video file, memory storage efficiency and significant savings can be obtained. With regard to audio feature extraction, a speech recognition algorithm can be utilized to extract different speech features, eg words, phrases, tone, speed, and the like. Additionally, in some examples, the avatar engine 308 and/or the speech engine 310 may perform audio feature extraction.

At block 506, computing device 106 can detect the context from the extracted features. For example, the context may include the user's intentions, moods, settings, location, ideas, identity, and the like. Context can be important when employing logic to determine which effects to apply. In some cases, the context may be combined with spoken words to determine whether and/or how to adjust the audio file and/or video file. In one example, the user's age may be detected as a context (eg, child, adult, etc.) based at least in part on facial and/or voice features. For example, a child's face may have certain features that can be identified (eg, large eyes, a small nose, and a relatively small head, etc.). In this way, a child context can be detected.

At block 508, computing device 106 may receive a selection of an avatar from a user. The user can select one of a plurality of different avatars through the UI of the avatar application being executed by the computing device 106. Avatars can be selected via a scroll wheel, a drop-down menu, or an icon menu (eg, where each avatar can be viewed on the screen at its own location).

At block 510, computing device 106 may determine effects for rendering audio and/or video files based at least in part on the context and the selected avatar. In this example, effects for each avatar may be created upon selection of each avatar, as opposed to being unified. In some cases, this will enable the realization of significant processor and memory savings because only one set of effects and avatar rendering will be performed at a time. These savings can be realized especially when the user does not select multiple avatars to preview.

At block 512, computing device 106 may perform additional logic for additional effects, similar to that described above for block 410 of FIG. 4. At block 514, computing device 106 may perform audio and/or audio and/or additional effects based at least in part on the determined effects and/or additional effects for the selected avatar, similar to that described above for block 412 of FIG. 4. Or you can revise the video information. At block 516, computing device 106 may present the revised video with revised audio based at least in part on the selected avatar, similar to that described above with respect to block 416 of FIG. 4.

In some examples, the avatar process 300 may determine whether to perform the flow 400 or the flow 500 based at least in part on the history information. For example, if the user generally uses the same avatar every time, flow 500 would be more efficient. However, if the user regularly switches between avatars and previews a number of different avatars per video clip, flow 400 may be more efficient.

6 illustrates an exemplary UI 600 for enabling a user to utilize an avatar application (eg, corresponding to the avatar application affordance 602 ). In some examples, UI 600 may look different until avatar application affordance 602 is selected (e.g., it may be viewed as a standard text (e.g., Short Message Service (SMS)) messaging application. has exist). As mentioned, the avatar application communicates with the avatar process (e.g., avatar process 300 in FIG. 3) to capture and process audio and/video (e.g., extracting features, executing logic, etc.) ) And requests for mediation. For example, when the user selects a recording affordance (e.g., recording/transfer video clip affordance 604), the avatar application will enter the avatar process to begin capturing video and audio information using the appropriate hardware components. Application programming interface (API) calls can be made. In some examples, the record/transfer video clip affordance 604 may be represented as a red circle (or a plain without lines shown in FIG. 6) prior to the start of the recording session. In this way, affordance will look more like a standard record button. While recording a session, the appearance of the record/transfer video clip affordance 604 can be changed to look like a clock countdown or other representation of a timer (eg, if the length of the video clip records is limited). However, in other examples, the recording/transfer video clip affordance 604 may only change the colors to indicate that the avatar application is recording. If there is no timer or no limit on the length of the recording, the user may need to select the record/transfer video clip affordance 604 again to end recording.

In some examples, the user may use the avatar selection affordance 606 to select an avatar. This may be done prior to recording of the avatar video clip and/or after recording of the avatar video clip. When selected before recording, an initial preview of the user's movements and facial features will be presented as the selected avatar. Additionally, the recording will be performed by presenting a live (eg, real-time) preview of the recording, where the user's face is represented by the selected avatar. When the recording is complete, a second preview (eg, playback of the actual recording) will be presented using the selected avatar again. However, at this stage, the user can scroll through the avatar selection affordance 606 and select a new avatar to view the recording preview. In some cases, upon selection of a new avatar, the UI will begin to preview the recording using the selected avatar. The new preview can be presented with audio/video effects or as originally recorded. As mentioned, the decision as to whether to present the version with the effect applied or the original may be based at least in part on the recent playback method used. For example, when effects are used in recent playback, effects may be used in a first playback after selecting a new avatar. However, if effects are not used in the recent playback, effects may not be used in the first playback after selecting a new avatar. In some examples, use may play the video clip with effects by selecting the effect preview affordance 608 or play without effects by selecting the original preview affordance 610. Once satisfied with the video clip (eg, the message), the user can use the record/transmit video clip affordance 604 to send the avatar video in the message to another computing device. The video clip will be transmitted using a format corresponding to the recent preview (eg, with or without effects). At any time, if the user wishes, the delete video clip affordance 612 may be selected to delete the avatar video and to start or end the avatar and/or messaging applications.

7 shows a process (e.g., a computer implemented method) 700 for implementing various audio and/or video effects based at least in part on audio and/or video features, according to at least some embodiments. Illustrates an exemplary flow chart. In some examples, the computing device 106 of FIG. 1 or another similar user device (e.g., utilizing at least an avatar application similar to that shown in FIG. 6 and the avatar process 300 of FIG. 3) is the process of FIG. 700 can be performed.

At block 702, computing device 106 can display a virtual avatar creation interface. The virtual avatar creation interface may look similar to the UI illustrated in FIG. 6. However, any UI configured to enable the same features described herein may be used.

At block 704, computing device 106 may display the first preview content of the virtual avatar. In some examples, the first preview content may be a real-time representation of the user's face including movements and facial expressions. However, the first preview will provide an avatar (eg, cartoon character, digital/virtual doll) to represent the user's face instead of an image of the user's face. This first preview may be video only, or at least may be a rendering of an avatar without sound. In some examples, this first preview is not recorded and can be utilized as long as the user desires without limitation other than battery power or memory space of computing device 106.

At block 706, computing device 106 may detect a selection of an input (eg, record/transmit video clip affordance 604 of FIG. 6) in a virtual avatar generation interface. This selection may be made while the UI is displaying the first preview content.

At block 708, computing device 106 can begin capturing video and audio signals based at least in part on the input detected at block 706. As described, video and audio signals can be captured by suitable hardware components, and can be captured by one or a combination of those components.

At block 710, computing device 106 may extract audio feature features and facial feature features as detailed above. As mentioned, the extraction may be performed by specific modules of the avatar process 300 of FIG. 3 or by other extraction and/or analysis components of the avatar application and/or computing device 106.

At block 712, computing device 106 may generate an adjusted audio signal based at least in part on facial feature characteristics and audio feature characteristics. For example, the audio file captured at block 708 may be permanently (or temporarily) revised (or temporarily) to have the original tone, tone, volume, etc. adjusted, and/or to include new sounds, new words, etc. For example, it can be adjusted). These adjustments may be made based at least in part on the context detected through analysis of facial feature features and audio feature features. Additionally, adjustments may include certain movements, facial expressions, words, phrases performed by the user (e.g., expressed by the user's face) based on the type of avatar selected and/or during the recording session. , Or based on actions.

In block 714, the computing device 106 may generate the second preview content of the virtual avatar in the UI according to the adjusted audio signal. The generated second preview content may be based at least in part on the currently selected avatar or some default avatar. When the second preview content is generated, the computing device 106 may present the second preview content to the UI at block 716.

8 is a process for implementing various audio and/or video effects based at least in part on audio and/or video features, according to at least some embodiments (e.g., on an executable computer-readable memory Stored instructions) 800 illustrates an example flow diagram. In some examples, the computing device 106 of FIG. 1 or another similar user device (e.g., utilizing at least an avatar application similar to that shown in FIG. 6 and the avatar process 300 of FIG. 3) is the process of FIG. (800) can be performed.

At block 802, computing device 106 may detect a request to create an avatar video clip of the virtual avatar. In some examples, the request may be based at least in part on a user's selection of the transmit/record video clip affordance 604 of FIG. 6.

At block 804, computing device 106 can capture a video signal associated with the face in the camera's field of view. At block 806, computing device 106 can capture an audio signal corresponding to a video signal (eg, coming from a face being captured by a camera).

At block 808, computing device 106 can extract voice feature features from the audio signal, and at block 810, computing device 106 can extract facial feature features from the video signal.

At block 812, computing device 106 can detect a request to preview the avatar video clip. Such a request may be based at least in part on the user's selection of a new avatar through the avatar selection affordance 606 of FIG. 6, or may be based at least in part on the user's selection of the effect preview affordance 608 of FIG. I can.

At block 814, computing device 106 may generate an adjusted audio signal based at least in part on facial feature characteristics and voice feature characteristics. For example, the audio file captured at block 806 will be revised (e.g., adjusted) to contain new sounds, new words, etc., and/or to have the original pitch, tone, volume, etc. I can. These adjustments may be made based at least in part on the context detected through analysis of facial feature features and speech feature features. Additionally, adjustments may include certain movements, facial expressions, words, phrases performed by the user (e.g., expressed by the user's face) based on the type of avatar selected and/or during the recording session. , Or based on actions.

At block 816, computing device 106 may generate a preview of the virtual avatar of the UI according to the adjusted audio signal. The generated preview may be based at least in part on the currently selected avatar or some default avatar. Once the preview is generated, computing device 106 can also present the second preview content to the UI at block 816.

9 is a simplified block diagram illustrating an exemplary architecture 900 for implementing features described herein, in accordance with at least one embodiment. In some examples, computing device 902 (e.g., computing device 106 of FIG. 1) with exemplary architecture 900 presents relevant UIs, captures audio and video information, and extracts relevant data. And perform logic, revise audio and video information, and present Animoji videos.

Computing device 902 provides a user interface for recording, previewing, and/or transmitting virtual avatar video clips (e.g., user interface 600 of FIG. 6), such as, but not limited to, It may be configured to execute or otherwise manage applications or instructions for performing the described techniques. Computing device 602 receives inputs from a user (e.g., utilizing I/O device(s) 904 such as a touch screen) in a user interface, captures information, processes the information, and then I/O device(s) 904 (eg, a speaker of computing device 902) may also be utilized to present video clips as previews. Computing device 902 may be configured to revise audio and/or video files based at least in part on facial features extracted from the captured video and/or speech features extracted from the captured audio.

Computing device 902 is a mobile phone (e.g., a smart phone), a tablet computer, a personal digital assistant (PDA), a laptop computer, a desktop computer, a thin-client device, a smart watch, a wireless headset, etc. However, it may be any type of computing device, but not limited thereto.

In one exemplary configuration, computing device 902 may include at least one memory 914 and one or more processing units (or processor(s)) 916. The processor(s) 916 may be implemented as appropriate in hardware, computer-executable instructions, or combinations thereof. Computer-executable instructions or firmware implementations of processor(s) 916 may include computer-executable or machine-executable instructions written in any suitable program language to perform the various functions described.

The memory 914 may store program instructions that are loadable and executable on the processor(s) 916 and data generated during execution of the programs. Depending on the configuration and type of computing device 902, memory 914 may be volatile (e.g., random access memory (RAM)) and/or non-volatile (e.g., read-only memory (ROM), flash memory, etc.). have. Computing device 902 may also include additional removable storage and/or non-removable storage 926, including but not limited to magnetic storage, optical disk, and/or tape storage. Disk drives and their associated non-transitory computer readable media may provide non-volatile storage of computer readable instructions, data structures, program modules, and other data to a computing device. In some implementations, the memory 914 may include a number of different types of memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), or ROM. Although the volatile memory described herein may be referred to as RAM, any volatile memory that is not capable of holding stored data once unplugged from the host and/or power source would be suitable.

Both removable and non-removable memory 914 and additional storage 926 are examples of non-transitory computer-readable storage media. For example, a non-transitory computer-readable storage medium may be a volatile or nonvolatile, removable, volatile or nonvolatile, technology implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. It may include removable or non-removable media. Both memory 914 and additional storage 926 are examples of non-transitory computer storage media. Additional types of computer storage media that may exist within computing device 902 include phase-change RAM (PRAM), SRAM, DRAM, RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM (compact disc read-only memory), DVD (digital video disc) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or can be used to store desired information and computing It may include, but is not limited to, any other medium that can be accessed by device 902. Combinations of any of the above should also be included within the scope of non-transitory computer-readable storage media.

Alternatively, a computer-readable communication medium may include computer-readable instructions, program modules, or other data transmitted within a data signal such as a carrier wave, or other transmission. However, as used herein, computer-readable storage media does not include computer-readable communication media.

Computing device 902 also includes communication connection(s) 928 that enable computing device 902 to communicate with a data store, other computing device or server, user terminals, and/or other devices via one or more networks. ) Can be included. Such networks may be any one or both of many different types of networks, such as cable networks, the Internet, wireless networks, cellular networks, satellite networks, other private and/or public networks, or any combination thereof. It may include a combination of. Computing device 902 may also include I/O device(s) 904, such as touch input devices, keyboards, mice, pens, voice input devices, displays, speakers, printers, and the like.

Referring to the contents of the memory 914 in more detail, the memory 914 includes an operating system 932 and/or a user interface module 934, an avatar control module 936, an avatar application module 938, and a messaging module ( 940) and one or more application programs or services for implementing the features disclosed herein. Memory 914 may also be configured to store one or more audio and video files to be used to generate audio and video output. In this manner, computing device 902 may perform all of the operations described herein.

In some examples, the user interface module 934 can be configured to manage a user interface of the computing device 902. For example, the user interface module 934 can present any number of various UIs requested by the computing device 902. In particular, the user interface module 934 captures video and audio information as described above, extracts appropriate facial feature and voice feature information, and revises the video and audio information prior to presentation of the generated avatar video clips. It may be configured to present the UI 600 of FIG. 6 that enables implementation of the features described herein, including communication with the avatar process 300 of FIG. 3 responsible for doing so.

In some examples, the avatar control module 936 is configured to implement the avatar process 300 (eg, execute instructions to implement), while the avatar application module 938 is configured to implement a user-facing application. . As mentioned above, the avatar application module 938 may utilize one or more APIs to request and/or provide information to the avatar control module 936.

In some embodiments, the messaging module 940 may implement any standalone or add-on messaging application capable of communicating with the avatar control module 936 and/or avatar application module 938. In some examples, the messaging module 940 may be fully integrated with the avatar application module 938 (eg, as shown in the UI 600 of FIG. 6 ), and the avatar application appears to be part of the messaging application. . However, in other examples, the messaging application 940 may call the avatar application module 938 when the user requests to create an avatar video clip, and the avatar application module 938 may be connected to the messaging module 940 New applications that are integrated can be opened together.

Computing device 902 can also include a camera and a microphone, as shown at least in FIG. 3, and processors 916 can be configured to execute instructions for displaying a first preview of the virtual avatar. In some examples, while displaying the first preview of the virtual avatar, input may be detected through the virtual avatar creation interface presented by the user interface module 934. In some cases, in response to detecting an input in the virtual avatar generation interface, the avatar control module 936 captures, via a camera, a video signal associated with a face within the camera's field of view, via a microphone, captured Initiating a capture session comprising capturing an audio signal associated with the video signal, extracting audio feature features from the captured audio signal, and extracting facial feature features associated with the face from the captured video signal. have. Additionally, in response to detecting the expiration of the capture session, the avatar control module 936 generates an adjusted audio signal based at least in part on the audio feature features and facial feature features, and And display a second preview of the virtual avatar on the virtual avatar generation interface according to the adjusted audio signal.

Exemplary methods, computer readable media, and systems for providing various techniques for adjusting audio and/or video content based at least in part on voice and/or facial feature characteristics have been described above. Some or all of these systems and methods may, but need not, be implemented at least in part by architectures such as those shown at least in FIGS. 1-9 above. While many embodiments have been described above with reference to messaging applications, it should be understood that any of the above techniques may be used within any type of application, including real-time video playback or real-time video messaging applications. For illustrative purposes, specific configurations and details are set forth to provide a thorough understanding of examples. However, it will also be apparent to those skilled in the art that some examples may be practiced without specific details. Additionally, well-known features have sometimes been omitted or simplified so as not to obscure the illustrated examples.

Various embodiments are further implemented in a wide variety of operating environments, which in some cases may include one or more user computers, computing devices or processing devices that may be used to operate any of a number of applications. Can be. User or client devices are among a number of general purpose personal computers, such as desktop or laptop computers running standard operating systems, as well as cellular, wireless and handheld devices capable of running mobile software and supporting a number of networking and messaging protocols. It can include anything. Such a system may also include a number of workstations running any of a variety of commercially available operating systems and other known applications for purposes such as development and database management. These devices may also include other electronic devices, such as dummy terminals, thin clients, gaming systems and other devices capable of communicating over the network.

Most embodiments are those of ordinary skill in the art to support communications using any of a variety of commercially available protocols, such as TCP/IP, OSI, FTP, UPnP, NFS, CIFS, and AppleTalk. Use at least one network that you are familiar with. The network may be, for example, a local area network, a wide area network, a virtual private network, the Internet, an intranet, an extranet, a public switched telephone network, an infrared network, a wireless network, and any combination thereof.

In embodiments utilizing a network server, the network server is a variety of servers or mid-tiers, including HTTP servers, FTP servers, CGI servers, data servers, Java servers, and business application servers. tier) can run any of the applications. The server(s) may also be one or more scripts written in any programming language, such as Java ^® , C, C# or C++, or any scripting language, such as Perl, Python or TCL, as well as combinations thereof. Alternatively, programs or scripts may be executed in response to requests from user devices, such as by executing one or more applications that may be implemented as a program. Server (s), also, Oracle ^®, Microsoft ^®, can include database servers, including without limitation those commercially available from using a Sybase ^®, and IBM ^®.

The environment may include various data stores and other memory and storage media as discussed above. They may reside in a variety of locations, such as on a storage medium that is local to (and/or resides in) one or more of the computers or remote from some or all of the computers over a network. In a particular set of embodiments, the information may reside in a storage-area network (SAN) that is familiar to those of skill in the art. Similarly, any necessary files to perform functions belonging to computers, servers or other network devices may be stored locally and/or remotely, where appropriate. Where the system comprises computerized devices, each such device may comprise hardware elements that can be electrically coupled via a bus, which elements are, for example, at least one central processing unit (CPU). ), at least one input device (eg, mouse, keyboard, controller, touch screen or keypad), and at least one output device (eg, display device, printer or speaker). Such a system may also include one or more storage devices, such as disk drives, optical storage devices, and solid state storage devices such as RAM or ROM, as well as removable media devices, memory cards, flash cards, and the like. have.

Such devices may also include a computer readable storage medium reader, a communication device (eg, a modem, a network card (wireless or wired), an infrared communication device, etc.), and a working memory as described above. A computer-readable storage medium reader may temporarily and/or more permanently contain, store, or transmit computer-readable information as well as non-transitory computer-readable storage media representing remote, local, fixed, and/or removable storage devices. , And a storage medium for retrieval or may be configured to receive the same. The system and various devices also typically include an operating system and a number of software applications, modules, services or other elements located within at least one working memory device, including application programs, such as a client application or browser. something to do. It should be appreciated that alternative embodiments may have numerous variations from those described above. For example, customized hardware may also be used and/or certain elements may be implemented in hardware, software (including portable software, such as applets), or both. In addition, connections to other computing devices, such as network input/output devices, may be used.

Non-transitory storage media and computer-readable storage media for containing code or portions of code include RAM, ROM, Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other memory technology, CD-ROM, DVD or other Any method for storage of information such as computer readable instructions, data structures, program modules, or other data, including optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or Any suitable medium known or used in the art, such as, but not limited to, volatile and nonvolatile, removable and non-removable media implemented with technology (except temporary media such as carrier waves, etc.), or storing desired information And may include any other media that can be used to and accessed by a system device. Based on the disclosure and teaching provided herein, one of ordinary skill in the art will recognize other ways and/or methods for implementing various embodiments. However, as mentioned above, computer-readable storage media do not include transitory media such as carriers or the like.

Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the present disclosure as set forth in the claims.

Other variations are within the spirit of this disclosure. Thus, while various modifications and alternative configurations are possible to the disclosed techniques, certain illustrated embodiments thereof are illustrated in the drawings and described in detail above. However, there is no intention to limit the present disclosure to the specific form or form disclosed, and to the contrary, the intention is to make all modifications falling within the spirit and scope of the present disclosure, as defined by the appended claims, It should be understood that it covers alternative structures and equivalents.

Use of the terms “a”, “an” and “the” and similar referents in connection with describing the disclosed embodiments (especially in connection with the claims below) Is to be construed as covering both the singular and the plural unless otherwise indicated herein or clearly contradicted by context. The terms "comprising", "having", "including", and "containing" are open-ended terms, unless otherwise stated. (Ie, meaning “including but not limited to”). The term “connected” is to be interpreted as partially or wholly contained within, attached to, or joined to each other, even if there is something intervening. The phrase “based on” is to be understood as being open and in no way limiting, and is intended to be interpreted or otherwise read as “based at least in part on” where appropriate. The enumeration of ranges of values herein is intended to serve only as a shorthand method, individually referring to each distinct value falling within that range, unless otherwise indicated herein, and each distinct The values of are incorporated herein as if individually recited herein. All methods described herein may be performed in any suitable order, unless otherwise indicated herein or otherwise clearly contradicted by context. The use of any and all examples, or illustrative language (eg, “such as”) provided herein is only intended to clarify embodiments of the present disclosure, and unless otherwise claimed. It does not impose restrictions on the scope of the disclosure. No expression in the specification is to be construed as representing any non-claimed element as essential to the practice of the present disclosure.

A disjunctive language such as the phrase "at least one of X, Y, or Z", unless specifically stated otherwise, means that an item, term, etc. is any one of X, Y, or Z, or any of these. It is understood differently within the context as commonly used to suggest that it can be a combination of (eg, X, Y, and/or Z). As such, such an OR expression is generally not intended and should not be implied that certain embodiments require at least one of X, at least one of Y, or at least one of Z, respectively. Additionally, a conjunctive language such as the phrase “at least one of X, Y, and Z” refers to “X, Y, Z, or any combination thereof, including “X, Y, and/or Z”. It should also be understood as meaning.

Preferred embodiments of the present disclosure are described herein, including the best mode known to the inventors for carrying out the present disclosure. Variations of the preferred embodiments may become apparent to those skilled in the art upon reading the foregoing description. The inventors expect those skilled in the art to use such variations where appropriate, and the inventors intend for the present disclosure to be practiced in a manner different from that specifically described herein. Accordingly, the present disclosure includes all variations and equivalents of the subject matter recited in the claims appended hereto, as permitted by applicable law. Moreover, any combination of the elements in all possible variations of the elements described above is encompassed by the present disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

All references cited herein, including publications, patent applications, and patents, are hereby indicated that each reference is individually and specifically incorporated by reference and is the same as if herein incorporated in its entirety. It is included as a reference.

Claims (20)

As a method,
At least in an electronic device with a camera and a microphone:
Displaying a virtual avatar creation interface;
Displaying the first preview content of the virtual avatar in the virtual avatar generation interface-The first preview content of the virtual avatar is associated with real-time preview video frames of the user's headshot within the field of view of the camera and the appearance Corresponds to headshot changes -;
Detecting an input in the virtual avatar creation interface while displaying the first preview content of the virtual avatar;
In response to detecting the input in the virtual avatar creation interface:
Capturing, via the camera, a video signal associated with the user headshot during a recording session;
Capturing, via the microphone, a user audio signal during the recording session;
Extracting audio feature characteristics from the captured user audio signal; And
Extracting facial feature features associated with the face from the captured video signal; And
In response to detecting the expiration of the recording session:
Generating an adjusted audio signal from the captured audio signal based at least in part on the facial feature characteristics and the audio feature characteristics;
Generating a second preview content of the virtual avatar in the virtual avatar generation interface according to the facial feature characteristics and the adjusted audio signal; And
And presenting the second preview content in the virtual avatar generation interface. The method of claim 1, further comprising storing facial feature metadata associated with the facial feature features extracted from the video signal, and storing audio metadata associated with the audio feature features extracted from the audio signal. The method further comprising the step of. 3. The method of claim 2, further comprising generating adjusted facial feature metadata from the facial feature metadata based at least in part on the facial feature features and the audio feature features. The method of claim 3, wherein the second preview of the virtual avatar is further displayed according to the adjusted face metadata. As an electronic device,
camera;
microphone; And
One or more processors in communication with the camera and the microphone, the one or more processors:
While displaying the first preview of the virtual avatar, detecting an input in the virtual avatar generation interface;
In response to detecting the input in the virtual avatar creation interface:
Capturing, via the camera, a video signal associated with a face within the camera's field of view;
Capturing, via the microphone, an audio signal associated with the captured video signal;
Extracting audio feature characteristics from the captured audio signal; And
Initiating a capture session comprising extracting facial feature features associated with the face from the captured video signal;
In response to detecting the expiration of the capture session:
Generate an adjusted audio signal based at least in part on the audio feature characteristics and the facial feature characteristics;
The electronic device, configured to display a second preview of the virtual avatar in the virtual avatar generation interface according to the facial feature characteristics and the adjusted audio signal. 6. The electronic device of claim 5, wherein the audio signal is further adjusted based at least in part on a type of the virtual avatar. The electronic device of claim 6, wherein the type of the virtual avatar is received based at least in part on an avatar type selection affordance presented in the virtual avatar generation interface. 7. The electronic device of claim 6, wherein the type of the virtual avatar comprises an animal type, and the adjusted audio signal is generated based at least in part on a predetermined sound associated with the animal type. 6. The electronic device of claim 5, wherein the one or more processors are further configured to determine whether a portion of the audio signal corresponds to the face in the field of view. 10. The method of claim 9, wherein the one or more processors are further configured to store the portion of the audio signal for use in generating the adjusted audio signal, upon determining that the portion of the audio signal corresponds to the face. Consisting of, an electronic device. 10. The electronic device of claim 9, wherein the one or more processors are further configured to discard at least the portion of the audio signal upon determining that the portion of the audio signal does not correspond to the face. 6. The electronic device of claim 5, wherein the audio feature characteristics include speech features associated with the face within the field of view. 6. The electronic device of claim 5, wherein the one or more processors are further configured to store facial feature metadata associated with the facial feature features extracted from the video signal. 14. The electronic device of claim 13, wherein the one or more processors are further configured to generate adjusted facial metadata based at least in part on the facial feature characteristics and the audio feature characteristics. The electronic device of claim 14, wherein the second preview of the virtual avatar is generated according to the adjusted face metadata and the adjusted audio signal. A computer-readable storage medium storing computer-executable instructions, the computer-executable instructions, when executed by one or more processors, the one or more processors:
In response to detecting a request to create an avatar video clip of the virtual avatar:
Capturing, via a camera of the electronic device, a video signal associated with a face within the camera's field of view;
Capturing an audio signal via a microphone of the electronic device;
Extracting speech feature characteristics from the captured audio signal; And
Extracting facial feature features associated with the face from the captured video signal; And
In response to detecting a request to preview the avatar video clip:
Generating an adjusted audio signal based at least in part on the facial feature characteristics and the speech feature characteristics; And
The computer-readable storage medium configured to perform operations comprising displaying a preview of the video clip of the virtual avatar using the adjusted audio signal. 17. The computer readable storage medium of claim 16, wherein the audio signal is adjusted based at least in part on facial expressions identified in the facial feature characteristics associated with the face. 17. The computer readable storage medium of claim 16, wherein the adjusted audio signal is further adjusted by inserting one or more pre-stored audio samples. The method of claim 16, wherein the audio signal is a level, pitch, duration, variable playback rate, speech spectral-format positions, speech spectral-format levels, instantaneous playback rate, or the face and A computer-readable storage medium that is adjusted based at least in part on changes in associated speech. 17. The computer readable storage medium of claim 16, wherein the one or more processors are further configured to perform the operations comprising transmitting the video clip of the virtual avatar to another electronic device.

Images