CN117635414A

CN117635414A - Real-time tracking of compensated image effects

Info

Publication number: CN117635414A
Application number: CN202311642293.9A
Authority: CN
Inventors: S·E·黑尔; F·波利亚科夫; 王国晖; X·熊; 杨建朝; 杨林杰; S·T·阿尼尔库马尔
Original assignee: Snap Inc
Current assignee: Snap Inc
Priority date: 2017-09-15
Filing date: 2018-09-14
Publication date: 2024-03-01
Also published as: US20200050866A1; KR102467236B1; KR20200042019A; EP3682421A4; EP4270329A3; WO2019055822A1; CN111095353A; US10474900B2; US20210216776A1; CN111095353B; US20190087660A1; US20230274543A1; EP4270329A2; US10929673B2; EP3682421A1; EP3682421B1; US11676381B2

Abstract

Methods, systems, and computer readable media for tracking compensation image effects in real time are provided. The mobile device may use an asynchronous processing pipeline to generate real-time complex visual image effects. The first pipeline applies complex image processing such as neural networks to key frames of a sequence of live images. The second pipeline generates a flow graph that describes feature transformations in the image sequence. The flowsheet may be used to process non-key frames on the fly. The processed key frames and non-key frames may be used to display complex visual effects on the mobile device in real time or near real time.

Description

Real-time tracking of compensated image effects

The present application is a divisional application of patent application with application number 201880058843.3 and the name of real-time tracking compensation image effect filed on 2018, 9 and 14.

Priority application

The present application claims the benefit of priority from U.S. patent application Ser. No. 15/706,096 filed on 2017, 9 and 15, wherein the benefit of each priority is hereby claimed and the contents of the above applications are incorporated by reference in their entirety.

Technical Field

The present disclosure relates generally to machines configured to the technical field of special purpose machines that manage electronic image processing and to improved machines for such variants, and to techniques for improving such special purpose machines as compared to other special purpose machines for managing asynchronous real-time image sequence processing.

Background

The computer may perform image processing using a complex Computer Vision (CV) scheme, such as a convolutional neural network. CV schemes may require intensive computation and typically require a powerful processor and a large amount of memory.

Executing CV schemes on mobile devices is often impractical because of the limited computing resources of the mobile device, which can result in long processing times. Furthermore, it is currently very difficult, if not impossible, to apply dense CV schemes in real time (e.g., 30 frames per second) using mobile devices.

Drawings

To facilitate identification of a discussion of any particular element or act, the most significant digit(s) in a reference number refer to the figure ("figure") number in which that element or act is first introduced.

Fig. 1 is a block diagram illustrating an example messaging system for exchanging data (e.g., messages and related content) over a network.

FIG. 2 is a block diagram illustrating further details regarding a messaging system with an integrated virtual object machine learning system, according to an example embodiment.

Fig. 3 is a schematic diagram illustrating data that may be stored in a database of a messaging server system, according to some example embodiments.

Fig. 4 is a diagram illustrating the structure of a message generated by a messaging client application for communication in accordance with some embodiments.

Fig. 5 is a schematic diagram illustrating an example access restriction process according to which access to content (e.g., ephemeral messages and associated multimedia data payloads) or a collection of content (e.g., ephemeral message stories) may be time-restricted (e.g., made ephemeral).

FIG. 6 illustrates example internal functional components of a tracking-based imaging system, according to some example embodiments.

FIG. 7 illustrates a flowchart of an example method for implementing tracking compensated image effects, according to some example embodiments.

FIG. 8 illustrates a flowchart of an example method for implementing asynchronous tracking compensation for image processing, according to some example embodiments.

Fig. 9 illustrates an example of an image and corresponding image mask according to some example embodiments.

FIG. 10 illustrates an example data flow for implementing a tracking-based imaging system, according to some example embodiments.

FIG. 11 illustrates an example data flow for generating a modified image using a tracking scheme, according to some example embodiments.

FIG. 12 illustrates a data flow for refining an image mask according to some example embodiments.

Fig. 13 is a block diagram illustrating a representative software architecture that may be used in connection with the various hardware architectures described herein.

Fig. 14 is a block diagram illustrating components of a machine capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing a method of any one or more of the methods discussed herein, according to some example embodiments.

Detailed Description

The following description includes systems, methods, techniques, sequences of instructions, and computer program products embodying illustrative embodiments of the present disclosure. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide an understanding of various embodiments of the inventive subject matter. It will be apparent, however, to one skilled in the art that embodiments of the inventive subject matter may be practiced without these specific details. Generally, well-known instruction instances, protocols, structures, and techniques have not necessarily been shown in detail.

The computer may perform image processing using a complex Computer Vision (CV) scheme, such as using a Convolutional Neural Network (CNN) to generate an image mask (mask). An image mask is a collection of numerical values that mark a given image region. In some example embodiments, an image mask for a live video may be generated by generating a key frame image mask using a CNN and then creating an image mask for a non-key frame using a light flow tracking algorithm. Fig. 9 shows an example of an image 900 and a corresponding image mask 905. Image 900 is an image of a girl holding a wine glass. CNNs may be trained to detect different areas of a given image, such as skin areas, clothing areas, hat areas, and so forth. The trained CNN may then receive the image as input and determine which areas of the image correspond to, for example, a hat section, skin section, or clothing section, and create a new image mask with pixel values for the detected portion.

In the example of fig. 9, the CNN has received the image 900 as input and has generated an image mask 905 with marked-up segments, including a hat segment 910, one or more skin segments 915 (girl's face and hands), and a clothing segment 920. Each segment may have a value that has been pre-assigned as a label for the given segment. For example, a pixel of cap segment 910 may have a value of 1, one or more skin segments 915 may have a value of 2, and garment segment 920 may have a value of 3. As discussed in further detail below, according to some example embodiments, the mask values are stored in separate images having the same dimensional dimensions as the original image.

In some example embodiments, the image mask 905 may be used to apply effects to regions of the image 900 by specifying fragment values. For example, the color of a girl's cap may be changed from red (its original color) to blue by replacing the pixel values corresponding to cap segment 910 in image 900. Further, if image 900 is part of a video sequence, an image mask may be calculated for each image in the video sequence, and the girl's hat may be recoloured to blue in each image, such that the hat appears blue throughout the video sequence.

While complex CV schemes (e.g., CNN-based image segmentation) can achieve impressive visual effects, these schemes typically require intensive computation and require powerful processors and a large amount of memory. Thus, running the solution from a low power computer (e.g., a smartphone) is often impractical. Although some client devices may be able to perform complex CV schemes, the processing time may be longer than the user expects (e.g., four seconds per image). In view of the long mobile device processing time, most client devices cannot implement real-time display of CNN modified images, as real-time processing may require 30 or more frames per second.

To this end, tracking-based imaging systems may generate real-time complex image effects by processing key frames of an image sequence using a complex image processing scheme (e.g., CNN) and asynchronously processing non-key frames using a tracking scheme (e.g., optical flow map). FIG. 10 illustrates an example data flow 1000 for implementing a tracking-based imaging system, according to some example embodiments. Image sequence 1010 is an image sequence (e.g., video) that includes IMGs 1-IMG6, which are captured from left to right in the direction of time 1005. The modified image sequence 1015 is a sequence of modified images, such as an image mask. The modified image sequence 1015 may be used to generate an output image sequence 1020 including output images O1-O6, which are displayed on a display screen of the client device.

As an illustrative example, assume that IMG1 in image sequence 1010 is IMG1 of a girl depicted in image 900 of fig. 9, and the remaining images in image sequence 1010 are photographs of the girl moving around with a red cap. In addition, M1 (of the modified imaging sequence 1015) is the image mask 905 of FIG. 9. Further, M2 to M6 are masks that track a girl's segments (e.g., hair, clothing) as the girl moves around. The M1-M6 mask may be used to apply an image effect to generate an output image sequence 1020, which may depict a girl moving on a blue hat.

According to some example embodiments, some modified images M1-M6 in the modified image sequence may be processed with a slower algorithm (such as CNN) and some masks may be generated with a faster algorithm (such as optical flow). In particular, slow or more time consuming image processing is applied to key frames of the image sequence 1010, wherein the key frames are alternating images of the image sequence 1010.

For example, assume that a CNN-based image segment is applied to each other image; thus, IMG1 is a Key Frame (KF) and is used to generate M1 using CNN, IMG3 is a key frame and is used to generate M3, and so on. To process non-key frames (e.g., IMG2, IMG4, etc.), a dataflow graph is generated and applied to the modified image, as discussed in further detail below. By not using a slow CNN to process some image masks, the overall process can be greatly accelerated; the sequence of output images 1020 may be displayed in real-time or near real-time, which the user may record and post as a short message, as discussed in further detail below.

FIG. 11 illustrates an example data flow 1100 for generating a modified image using a tracking scheme, according to some example embodiments. Similar to fig. 10, fig. 11 shows a sequence of images 1105 captured by a client device, and a modified sequence of images 1110, which may be a mask or final image output for display. In some example embodiments, the CNN receives IMG1 as an input and generates M1 as an output. To generate M2, a dataflow graph algorithm is used. In particular, an optical flow map 1115 is generated that describes the transformation from IMG1 to IMG2, which are the original images in image sequence 1105. Flowchart 1115 is then applied to M1 to create M2, thereby avoiding the application of CNN to IMG2 to create M2. In some example embodiments, the two images used to generate the flowsheet are separated by additional sequential images 1120, as discussed in more detail below with reference to fig. 8, 11, and 12.

FIG. 1 illustrates a block diagram of an example messaging system 100 for exchanging data (e.g., social media postings created using a modified image sequence 1110) over a network 106. Messaging system 100 includes a plurality of client devices 102, each hosting a plurality of applications including messaging client application 104. Each messaging client application 104 is communicatively coupled to other instances of messaging client application 104 and messaging server system 108 via network 106 (e.g., the internet).

Thus, each messaging client application 104 is capable of communicating and exchanging data with another messaging client application 104 and messaging server system 108 via network 106. The data exchanged between messaging client applications 104 and messaging server system 108 includes functions (e.g., commands to invoke functions) as well as payload data (e.g., text, audio, video, or other multimedia data).

The messaging server system 108 provides server-side functionality to particular messaging client applications 104 via the network 106. Although certain functions of messaging system 100 are described herein as being performed by messaging client application 104 or by messaging server system 108, it should be understood that the location of certain functions within messaging client application 104 or messaging server system 108 is a design choice. For example, it is technically preferable to initially deploy certain techniques and functions in the messaging server system 108 and later migrate the techniques and functions to the messaging client application 104 if the client device 102 has sufficient processing power.

The messaging server system 108 supports various services and operations provided to the messaging client application 104. Such operations include sending data to, receiving data from, and processing data generated by messaging client application 104. As examples, the data may include message content, client device information, geographic location information, media annotations and overlays, message content persistence conditions, social network information, and live event information. Data exchange within messaging system 100 is invoked and controlled via functions available to a User Interface (UI) of messaging client application 104.

Turning now specifically to messaging server system 108, application Programming Interface (API) server 110 is coupled to application server 112 and provides a program interface. The application server 112 is communicatively coupled to a database server 118 that facilitates access to a database 120 in which data associated with messages processed by the application server 112 is stored.

The API server 110 receives and transmits message data (e.g., command and message payloads) between the client device 102 and the application server 112. In particular, the API server 110 provides a set of interfaces (e.g., routines and protocols) that the messaging client application 104 can call or query in order to invoke the functionality of the application server 112. The API server 110 exposes various functions supported by the application server 112, including account registration; a login function; sending a message from a particular messaging client application 104 to another messaging client application 104 via the application server 112; sending a media file (e.g., image or video) from messaging client application 104 to messaging server application 114 for possible access by another messaging client application 104; a setting of a media data collection (e.g., story); retrieving such a collection; retrieving a list of friends of the user of the client device 102; retrieving the message and the content; adding friends to the social graph and deleting friends from the social graph; the location of friends within the social graph; and open application events (e.g., associated with messaging client application 104).

Application server 112 hosts a plurality of applications and subsystems, including messaging server application 114, image processing system 116, and social networking system 122. The messaging server application 114 implements a variety of message processing techniques and functions, particularly in connection with aggregation and other processing of content (e.g., text and multimedia content) in messages that include information received from multiple instances of the messaging client application 104. As will be described in further detail, text and media content from multiple sources may be aggregated into a collection of content (e.g., a so-called story or gallery). Messaging server application 114 may then provide these sets to messaging client application 104. Other processor and memory intensive data may also be processed at the server side by messaging server application 114 in view of the hardware requirements of such processing.

The application server 112 also includes an image processing system 116 that is dedicated to performing various image processing operations, typically with respect to images or video received within the payload of a message at the message server application 114.

The social networking system 122 supports various social networking functions and services, and makes these functions and services available to the messaging server application 114. To this end, the social networking system 122 maintains and accesses an entity graph (e.g., entity graph 304 in FIG. 3) within the database 120. Examples of functions and services supported by social-networking system 122 include identification of other users of messaging system 100 that are associated with or "follow" by a particular user, as well as identification of interests and other entities of the particular user.

Application server 112 is communicatively coupled to database server 118, which facilitates access to database 120, with data associated with messages processed by messaging server application 114 stored in database 120.

Fig. 2 is a block diagram illustrating further details regarding messaging system 100, according to an example embodiment. In particular, messaging system 100 is shown to include messaging client application 104 and application server 112, which in turn contain a plurality of subsystems, namely ephemeral timer system 202, collection management system 204, annotation system 206, and tracking-based imaging system 210.

The ephemeral timer system 202 is responsible for temporary access to content allowed by the messaging client application 104 and the messaging server application 114. To this end, the short timer system 202 includes a plurality of timers that selectively display and enable access to messages and associated content via the messaging client application 104 based on a duration and display parameters associated with the message or collection of messages (e.g., SNAPCHATs). Further details regarding the operation of the transient timer system 202 are provided below.

The collection management system 204 is responsible for managing collections of media (e.g., collections of text, images, video, and audio data). In some examples, a collection of content (e.g., a message including images, video, text, and audio) may be organized as an "event gallery" or "event story. Such a collection may be made available for a specified period of time, such as the duration of an event related to the content. For example, content related to a concert may be made available as a "story" for the duration of the concert. The collection management system 204 may also be responsible for publishing icons that provide notifications of the presence of a particular collection to the user interface of the messaging client application 104.

The collection management system 204 also includes a planning interface (curation interface) 208, which planning interface 208 allows the collection manager to manage and plan specific collections of content. For example, the curation interface 208 enables an event organizer to curate a collection of content related to a particular event (e.g., delete inappropriate content or redundant messages). In addition, the collection management system 204 employs machine vision (or image recognition techniques) and content rules to automatically organize the collection of content. In some embodiments, the user may be paid the compensation to include the user-generated content in the collection. In such a case, the planning interface 208 operates to automatically pay such users to use their content.

The annotation system 206 provides various functions that enable a user to annotate or modify or edit media content associated with a message. For example, the annotation system 206 provides functionality related to the generation and distribution of media overlays for messages processed by the messaging system 100. The annotation system 206 is operable to provide media overlays (e.g., snapcat geographic filters or filters) to the messaging client application 104 based on the geographic location of the client device 102. In another example, annotation system 206 is operable to provide media overlays to messaging client application 104 based on other information, such as social network information of a user of client device 102. The media overlay may include audio and visual content and visual effects. Examples of audio and visual content include pictures, text, logos, animations and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects may be applied to media content items (e.g., photos) at the client device 102. For example, the media overlay includes text that may be overlaid on top of the photo generated by the client device 102. In another example, the media overlay includes an identification of a location (e.g., a Venetian beach), a name of a live event, or a name of a merchant's overlay (e.g., a beach cafe). In another example, the annotation system 206 uses the geographic location of the client device 102 to identify a media overlay that includes the name of the merchant at the geographic location of the client device 102. The media overlay may include other indicia associated with the merchant. The media overlay may be stored in database 120 and accessed through database server 118.

In one example embodiment, the annotation system 206 provides a user-based distribution platform that enables a user to select a geographic location on a map and upload content associated with the selected geographic location. The user may also specify a particular environment in which to provide particular media overlays to other users. The annotation system 206 generates a media overlay that includes the uploaded content and associates the uploaded content with the selected geographic location.

In another example embodiment, the annotation system 206 provides a merchant-based posting platform that enables merchants to select particular media overlays associated with geographic locations via a bidding process. For example, the annotation system 206 associates the media overlay of the highest bidding merchant with the corresponding geographic location for a predefined amount of time.

The tracking-based imaging system 210 is configured to apply complex image processing in real-time so that a user can view effects and create a sequence of images. The image sequence may be annotated using the annotation system 206 and published as a short message 502, which will be discussed in further detail below with reference to FIG. 5.

Fig. 3 is a schematic diagram illustrating data 300 that may be stored in database 120 of messaging server system 108, according to some example embodiments. Although the contents of database 120 are shown as including multiple tables, it will be appreciated that the data may be stored in other types of data structures (e.g., as an object-oriented database).

Database 120 includes message data stored in message table 314. Entity table 302 stores entity data including entity map 304. The entities for which records are maintained within the entity table 302 may include individuals, corporate entities, organizations, objects, places, events, and the like. Whatever the type, any entity involved with the data stored by messaging server system 108 may identify the entity. Each entity has a unique identifier and an entity type identifier (not shown).

Entity map 304 also stores information about relationships and associations between entities. For example, such relationships may be social, professional (e.g., working at a common company or organization), interest-based, or activity-based.

Database 120 also stores annotation data, including in the example form of filters, in annotation table 312. Filters with data stored in annotation table 312 are associated with and applied to video (data stored in video table 310) and/or images (data stored in image table 308). In one example, the filter is an overlay that is displayed overlaid on the image or video during presentation to the recipient user. The filters may be of various types, including user-selected filters from a filter library that is presented to the sending user by messaging client application 104 when the sending user is composing a message. Other types of filters include geo-location filters (also referred to as geo-filters) that may be presented to a sending user based on geographic location. For example, based on geographic location information determined by a Global Positioning System (GPS) unit of the client device 102, neighborhood-specific or location-specific geographic location filters may be presented within the messaging client application 104 user interface. Another type of filter is a data filter that may be selectively presented to the sending user by messaging client application 104 based on other inputs or information collected by client device 102 during the message creation process. Examples of data filters include a current temperature at a particular location, a current speed at which a sending user travels, a battery life of the client device 102, or a current time.

Other annotation data that may be stored within the image table 308 is so-called "shot" data. A "shot" may be a real-time special effect and sound that may be added to an image or video.

As described above, video table 310 stores video data that, in one embodiment, is associated with a message for which a record is maintained within message table 314. Similarly, image table 308 stores image data associated with the message for which message data is stored in message 314. The entity table 302 may associate various annotations from the annotation table 312 with various images and videos stored in the image table 308 and the video table 310.

Story table 306 stores data regarding a collection of messages and associated image, video, or audio data that is compiled into a collection (e.g., a SNAPCHAT story or gallery). Creation of a particular collection may be initiated by a particular user (e.g., any user for whom records are maintained in entity table 302). A user may create a "personal story" in the form of a collection of content that has been created and transmitted/broadcast by the user. To this end, the user interface of messaging client application 104 may include user-selectable icons to enable the sending user to add particular content to his or her personal story.

Collections may also constitute "live stories," which are collections of content from multiple users that are created manually, automatically, or using a combination of manual and automatic techniques. For example, a "live story" may constitute a policy stream of user-submitted content from various locations and events. Users whose client devices 102 have location services enabled and are co-located or at a particular time may be presented with options via the user interface of messaging client application 104 to contribute content to a particular live story. The real-time story may be identified to the user by messaging client application 104 based on the user's location. The end result is a "live story" told from a community perspective.

Another type of collection of content is referred to as a "location story" that enables users whose client devices 102 are located within a particular geographic location (e.g., within a university or college campus) to contribute to the particular collection. In some embodiments, the contribution to the location story may require a second level of authentication to verify that the end user belongs to a particular organization or other entity (e.g., is a student in a university campus).

Fig. 4 is a diagram illustrating the structure of a message 400 generated by a messaging client application 104 for communicating with another messaging client application 104 or a messaging server application 114, in accordance with some embodiments. The contents of a particular message 400 are used to populate a message table 314 stored within database 120 that is accessible to messaging server application 114. Similarly, the content of message 400 is stored in memory as "in-send" or "in-transmit" data for client device 102 or application server 112. Message 400 is shown to include the following components:

● Message identifier 402: a unique identifier that identifies the message 400.

● Message text payload 404: text is to be generated by the user via the user interface of the client device 102 and is included in the message 400.

● Message image payload 406: image data captured by the camera component of the client device 102 or retrieved from the memory of the client device 102 and included in the message 400.

● Message video payload 408: video data captured by the camera component or retrieved from a memory component of the client device 102 is included in the message 400.

● Message audio payload 410: audio data captured by a microphone or retrieved from a memory component of the client device 102 is included in the message 400.

● Message annotation 412: annotation data (e.g., filters, tags, or other enhancements) represents annotations to be applied to the message 400 to the message image payload 406, the message video payload 408, or the message audio payload 410.

● Message duration parameter 414: parameter values in seconds indicate the amount of time that the contents of message 400 (e.g., message image payload 406, message video payload 408, and message audio payload 410) will be presented to or made accessible to the user by messaging client application 104.

● Message geographic location parameter 416: geographic location data (e.g., latitude and longitude coordinates) associated with the content payload of message 400. A plurality of message geographic location parameter 416 values may be included in the payload, each of which is associated with a respective content item included in the content (e.g., a particular image in the message image payload 406, or a particular video in the message video payload 408).

● Message story identifier 418: an identifier value that identifies one or more content collections (e.g., a "story") associated with a particular content item in the message image payload 406 of the message 400. For example, multiple images within the message image payload 406 may each be associated with multiple content sets using an identifier value.

● Message tag 420: one or more tags, each tag indicating a subject of content included in the message payload. For example, where a particular image included in the message image payload 406 depicts an animal (e.g., a lion), a tag value indicating the relevant animal may be included within the message tag 420. The tag value may be generated manually based on user input or may be generated automatically using, for example, image recognition.

● Message sender identifier 422: an identifier (e.g., a messaging system identifier, an email address, or a device identifier) indicating a user of the client device 102, a message 400 is generated on the client device 102 and the message 400 is sent therefrom.

● Message recipient identifier 424: an identifier (e.g., a messaging system identifier, an email address, or a device identifier) indicating the user of the client device 102 to which the message 400 is addressed.

The contents (e.g., values) of the various components of the message 400 may be pointers to locations in a table where the content data values are stored. For example, the image value in the message image payload 406 may be a pointer (or address) to a location within the image table 308. Similarly, values in message video payload 408 may point to data stored in video table 310, values stored in message notes 412 may point to data stored in notes table 312, values stored in message story identifier 418 may point to data stored in story table 306, and values stored in message sender identifier 422 and message receiver identifier 424 may point to user records stored within entity table 302.

Fig. 5 is a schematic diagram illustrating an access restriction process 500 according to which access to content (e.g., ephemeral message 502 and associated multimedia data payload) or a collection of content (e.g., ephemeral message story 504) may be time-limited (e.g., ephemeral).

The ephemeral message 502 is shown as being associated with a message duration parameter 506, the value of the display duration parameter 506 determining the amount of time that the ephemeral message 502 will be displayed by the transceiving client application 104 to the receiving user of the ephemeral message 502. In one embodiment, where messaging client application 104 is a SNAPCHAT application client, ephemeral message 502 may be viewed by the receiving user for up to 10 seconds, depending on the amount of time specified by the sending user using message duration parameter 506.

The message duration parameter 506 and the message receiver identifier 424 are shown as inputs to a message timer 512, the message timer 512 being responsible for determining the amount of time that the ephemeral message 502 is displayed to a particular receiving user identified by the message receiver identifier 424. In particular, the ephemeral message 502 will be displayed to the relevant receiving user only for a period of time determined by the value of the message duration parameter 506. The message timer 512 is shown providing output to a more general ephemeral timer system 202, which ephemeral timer system 202 is responsible for the overall timing of the display of content (e.g., ephemeral message 502) to the receiving user.

Transient message 502 is shown in fig. 5 to be included in a transient message story 504 (e.g., a personal snapchar story or an event story). Ephemeral message story 504 has an associated story duration parameter 508, and the value of story duration parameter 508 determines the duration that ephemeral message story 504 is available and accessible to a user of messaging system 100. For example, story duration parameter 508 may be a duration of a concert, where ephemeral message story 504 is a collection of content related to the concert. Alternatively, a user (owning user or curating user) may specify the value of story duration parameter 508 when performing the setting and creation of ephemeral message story 504.

In addition, each ephemeral message 502 within ephemeral message story 504 has an associated story participation parameter 510 whose value determines the duration that ephemeral message 502 will be accessible in the context of ephemeral message story 504. Thus, a particular ephemeral message 502 may "expire" and become inaccessible in the context of ephemeral message story 504 before ephemeral message story 504 itself expires in accordance with story duration parameter 508. Story duration parameter 508, story participation parameter 510, and message recipient identifier 424 each provide input to story timer 514, which timer 514 is operative to determine whether or not ephemeral message 502 of a particular ephemeral message story 504 is to be displayed to a particular receiving user, and, if so, for how long. Note that ephemeral message story 504 also knows the identity of the particular receiving user due to message recipient identifier 424.

Thus, story timer 514 is operative to control the overall lifetime of the associated ephemeral message story 504 and the individual ephemeral messages 502 included in ephemeral message story 504. In one embodiment, each ephemeral message 502 within ephemeral message story 504 remains visible and accessible for a period of time specified by story duration parameter 508. In another embodiment, some ephemeral messages 502 may expire within the context of ephemeral message story 504 based on story participation parameter 510. Note that even in the context of ephemeral message story 504, the respective display duration parameter 506 may determine the duration that ephemeral message 502 is displayed to the receiving user for the particular ephemeral message 502. Thus, the display duration parameter 506 determines the duration that a particular ephemeral message 502 is displayed to a receiving user, regardless of whether the receiving user views the ephemeral message 502 within the context of the ephemeral message story 504 or externally.

Ephemeral timer system 202 may also remove a particular ephemeral message 502 from ephemeral message story 504 based on determining that the associated story participation parameter 510 has been exceeded. For example, when the sending user has established a story participation parameter 510 for 24 hours from release, ephemeral timer system 202 will remove the relevant ephemeral message 502 from ephemeral message story 504 after the specified 24 hours. Ephemeral timer system 202 is also used to remove ephemeral message stories 504 when story participation parameters 510 of each ephemeral message 502 within ephemeral message stories 504 have expired, or when ephemeral message stories 504 themselves have expired according to story duration parameters 508.

In some use cases, the creator of a particular ephemeral message story 504 may specify an uncertain story duration parameter 508. In this case, the expiration of the story participation parameter 510 of the last remaining ephemeral message 502 in ephemeral message story 504 will determine when ephemeral message story 504 itself expires. In this case, a new ephemeral message 502 with a new story participation parameter 510 added to ephemeral message story 504 effectively extends the lifetime of ephemeral message story 504 to be equal to the value of story participation parameter 510.

In response to the ephemeral timer system 202 determining that the ephemeral message experience 504 has expired (e.g., is no longer accessible), the ephemeral timer system 202 communicates with the messaging system 100 (e.g., in particular, the messaging client application 104) such that the indicia (e.g., icon) associated with the relevant ephemeral message story 504 is no longer displayed within the user interface of the messaging client application 104. Similarly, when the ephemeral timer system 202 determines that the message duration parameter 506 of a particular ephemeral message 502 has expired, the ephemeral timer system 202 causes the messaging client application 104 to no longer display a flag (e.g., icon or text identification) associated with the ephemeral message 502.

FIG. 6 illustrates example internal functional components of a tracking-based imaging system 210, according to some example embodiments. As shown, the tracking-based imaging system 210 includes an image engine 605, an editing engine 610, a tracking engine 615, a display engine 620, and a pipeline engine 625. The image engine 605 is configured to generate one or more images using the client device 102 (e.g., using an image sensor) and display the images on a display device of the client device 102. According to some example embodiments, the image engine 605 is further configured to receive instructions for applying an image effect on one or more images. Image effects may require CNN-based image segmentation, CNN-based object tracking, or other computationally intensive image processing operations.

According to some example embodiments, the editing engine 610 is configured to apply a learning scheme (such as a convolutional neural network) to one or more images. The tracking engine 615 is configured to track image features across multiple images, for example using a flowsheet. For example, the tracking engine 615 may use a light flow scheme to create a light flow graph that describes how image features (e.g., pixels displaying image features) are transformed or moved from one image to a subsequent image. The tracking engine 615 is also configured to apply the flowsheet to the image to generate a new image. The display engine 620 manages the combining of images generated by the editing engine 610 and the tracking engine 615 into a modified image sequence that can be displayed in real-time on the client device 102 and published as the ephemeral message 502. The pipeline engine 625 manages the orchestration of the editing engine 610 operating from a first pipeline (e.g., thread) and the tracking engine 615 operating from a second pipeline (e.g., additional threads), as discussed in further detail below.

FIG. 7 illustrates a flowchart of an example method 700 for implementing tracking compensated image effects, according to some example embodiments. In operation 705, the image engine 605 displays a sequence of images on a display of a client device (e.g., the client device 102). For example, the image engine 605 captures video using a client device camera. The image engine 605 then displays the video on a display device (e.g., screen) of the client device in real time or near real time (e.g., without significant delay between the object being imaged and displayed on the screen of the client device). At operation 710, the image engine 605 receives an instruction to visually modify an image sequence. In particular, for example, a user of the client device may select a User Interface (UI) button that generates instructions for visually modifying the image sequence. The instructions may require effects generated by Convolutional Neural Networks (CNNs), such as image segmentation, object classification, object tracking, human recognition, or other image schemes.

At operation 715, the machine learning engine applies the machine learning scheme to the keyframes of the image sequence. In some example embodiments, the key frame is every nth image in the sequence of images; for example, every two images (n=2), every three images, every five images, and so on. In some example embodiments, N is preconfigured depending on the type of client device. For example, if editing engine 610 detects that it is executing from a latest model client device (e.g., iPhone7 as month 8 2017), then n=2; and if editing engine 610 detects that it is executing from an older model of client device (e.g., iPhone4 as in 2017, 8), N may be set higher than 2, e.g., n=5. In some example embodiments, which image is a key frame (and thus processed by the editing engine 610) varies depending on how fast the editing engine 610 may operate from its asynchronous pipeline, as discussed in further detail below.

At operation 720, the tracking engine 615 generates a flowsheet between the current image and the last image generated by the editing engine 610. The current image is an image of the sequence of images that has not yet been processed by the editing engine 610 or tracking engine 615. In some example embodiments, the flowsheet is an optical flow diagram that includes a vector transformation (e.g., transformation, scaling) that describes how pixels of an image feature move from one image to a subsequent image.

At operation 725, the tracking engine 615 generates lost frames using the flowsheet. Lost frames are frames between key frames that the editing engine 610 has skipped or has no opportunity to process (e.g., because the editing engine 610 is busy processing earlier images). In some example embodiments, the tracking engine 615 generates a given missing frame by applying the flowsheet to the last modified image generated by the editing engine 610.

In operation 730, the display engine 620 displays the final image sequence on the display device in real time. As described above, some of the images in the final image sequence are generated by slower schemes (e.g., CNN) and some of the images are generated by faster schemes (e.g., optical flow) and combined during the run (on-the-fly) to increase overall speed.

Fig. 8 illustrates a flowchart of an example method 800 for real-time tracking compensation processing of an image, according to some example embodiments. In those example embodiments, the editing engine 610 and the tracking engine 615 operate asynchronously (e.g., concurrently, in parallel) in different pipelines. For example, the editing engine 610 may apply CNN to generate an image mask in a first thread and the tracking engine 615 to generate an image mask in a second thread. Editing engine 610 typically applies a more intensive computing scheme (e.g., CNN) than tracking engine 615 applies; thus, although they are working for the same image sequence, the editing engine 610 typically lags the tracking engine 615 by multiple frames. The amount of hysteresis allowed varies depending on the speed at which the editing engine 610 completes its processing. The tracking engine 615 is configured to compensate for the changing hysteresis of the editing engine 610 by generating a flowsheet and applying the flowsheet to the last image output by the editing engine 610, thereby generating a new modified image of the current frame. FIG. 8 illustrates an example of a dynamic asynchronous process according to some example embodiments. In fig. 8, "K" refers to the original image sequence (e.g., image sequence 1105), so IMG1 is K1, IMG2 is K2, and so on. Further, in fig. 8, K having no number is the current frame because it is the next image to be processed. Further, N tracks are sequential, so K-N is some image before the current frame (e.g., K-1 is the image immediately before the current image K).

In operation 805, the image engine 605 generates a sequence of images, e.g., K1, K2, K3, etc. At operation 810, the editing engine 610 begins processing each image to generate modified images M1, M2, M3, etc. For example, editing engine 610 applies the trained CNN to K1 to generate M1, and so on. At operation 815, the pipeline engine 625 determines whether the editing engine 610 is lagging. For example, the pipeline engine 625 determines whether the editing engine 610 is still processing an image prior to the current frame, where the current frame is the next image to process. If the editing engine 610 has no hysteresis, the method 800 returns to operation 810, where the editing engine 610 applies the CNN to the next image.

If the editing engine 610 lags, then in operation 820, the pipeline engine 625 instantiates a second pipeline (e.g., thread) to generate a flowsheet. In particular, a flow graph is generated from the second pipelined tracking engine 615 between the current frame and the last frame for which the editing engine 610 has generated output. For example, if the editing engine 610 finally generated an image mask five frames after the current frame, a flowsheet is generated between the original image backed five frames and the current image. The flowsheet thus captures changes from the editing engine 610 currently lagging the current frame to be processed. At operation 825, the tracking engine 615 applies the flow graph to the actual final output of the editing engine 610 to generate an output (e.g., mask) of the current frame. The operations permitted for 810, 815, 820, and 825 vary depending on how fast the engines in each pipeline can operate. In operation 830, the final image sequence is displayed on a display device in real time or near real time.

As discussed, in some example embodiments, the final image sequence may be another image sequence generated using the output of the editing engine 610 and the tracking engine 615. For example, an image mask generated asynchronously by two pipelines may be applied to replace the color of the hat from red to blue in the final image sequence. In some example embodiments, the output of the editing engine 610 and tracking engine 615 is the image (e.g., not an image mask) displayed at operation 830. For example, editing engine 610 may apply a CNN-based artistic style to an image (e.g., style transfer), and tracking engine 615 mimics the artistic style and collates and displays the engine's output in real-time.

Turning to fig. 12, fig. 12 illustrates a data flow 1200 for refining an image mask according to some example embodiments. The frame memory 1235 stores images of an original image sequence (e.g., image sequence 1105 of fig. 11), while the mask memory 1240 stores image masks (e.g., modified image sequence 1110 of fig. 11). Frame memory 1235 and mask memory 1240 may be a rolling buffer that tracks the last N frames, e.g., the last 100 frames before the current frame, which is the next frame to process.

The current frame 1205 is input into the tracking engine 615, which generates a new mask 1220 using the current frame 1205 and a previous frame from the frame memory 1235, as described above, (e.g., the previous frame is a frame with a mask). The new mask 1220 is stored in the mask memory 1240 so that it can be used to generate a final image (e.g., a girl with blue caps instead of red caps).

The new mask 1220 may be rough because it is an approximation generated using the light stream 1210. To improve the accuracy of the new mask 1220, the tracking engine 615 implements a guide filter 1225 that uses the current frame 1205 as a guide to refine and correct errors in the new mask 1220. The refined mask 1230 is then output from the guide filter 1225 for further processing (e.g., for recoloring the hat from red to blue).

Fig. 13 is a block diagram illustrating an example software architecture 1306, which example software architecture 1306 may be used in conjunction with the various hardware architectures described herein. FIG. 13 is a non-limiting example of a software architecture, and it should be appreciated that many other architectures can be implemented to facilitate the functionality described herein. The software architecture 1306 may execute on hardware, such as the machine 1400 of fig. 14, including processors, memory, and I/O components, etc. Representative hardware layer 1352 is shown to represent, for example, machine 1400 of fig. 14. Representative hardware layer 1352 includes a processing unit 1354 with associated executable instructions 1304. Executable instructions 1304 represent executable instructions of software architecture 1306, including implementations of the methods, components, etc. described herein. Hardware layer 1352 also includes memory/storage 1356, which also has executable instructions 1304. Hardware layer 1352 may also include other hardware 1358.

In the example architecture of fig. 13, the software architecture 1306 may be conceptualized as a stack of layers, with each layer providing specific functionality. For example, the software architecture 1306 may include layers such as an operating system 1302, libraries 1320, framework/middleware 1318, applications 1316, and presentation layer 1314. In operation, the application 1316 and/or other components within the layer may call the API call 1308 through a software stack and receive a response in the form of message 1312. The layers shown are representative in nature and not all software architectures have all layers. For example, some mobile or dedicated operating systems may not provide framework/middleware 1318, while other operating systems may provide such layers. Other software architectures may include other or different layers.

The operating system 1302 may manage hardware resources and provide common services. Operating system 1302 may include, for example, kernel 1322, services 1324, and drivers 1326. The kernel 1322 may act as an abstraction layer between hardware and other software layers. For example, core 1322 may be responsible for memoryManagement, processor management (e.g., scheduling), component management, networking, security settings, etc. The service 1324 may provide other common services for other software layers. The driver 1326 is responsible for controlling or interfacing with the underlying hardware. For example, the number of the cells to be processed, the drivers 1326 include a display driver, a camera driver, Driver, flash memory driver, serial communication driver (e.g., universal Serial Bus (USB) driver), -in>Drivers, audio drivers, power management drivers, etc., depending on the hardware configuration.

Library 1320 provides a common infrastructure used by applications 1316 and/or other components and/or layers. Library 1320 provides functionality that allows other software components to perform tasks in a simpler manner than by interfacing directly with the underlying operating system 1302 functionality (e.g., kernel 1322, services 1324, and/or drivers 1326). The library 1320 may include a system library 1344 (e.g., a C-standard library), which system library 1344 may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, and the like. Further, libraries 1320 may include API libraries 1346, such as media libraries (e.g., libraries that support presentation and manipulation of various media formats such as MPEG4, H.264, MP3, AAC, AMR, JPG, or PNG), graphics libraries (e.g., openGL framework that may be used to present 2D and 3D graphics content on a display), databases (e.g., SQLite that may provide various relational database functions), web libraries (e.g., webKit that may provide web browsing functions), and the like. The library 1320 may also include a variety of other libraries 1348 to provide many other APIs for applications 1316 and other software components/modules.

Framework/middleware 1318 provides a higher level of public infrastructure that can be used by applications 1316 and/or other software components/modules. For example, the framework/middleware 1318 may provide various Graphical User Interface (GUI) functions, advanced resource management, advanced location services, and the like. The framework/middleware 1318 can provide a wide variety of other APIs that can be utilized by the applications 1316 and/or other software components/modules, some of which can be specific operating systems 1302 or platforms.

Applications 1316 include built-in applications 1338 and/or third party applications 1340. Examples of representative built-in applications 1338 may include, but are not limited to, a contact application, a browser application, a book reader application, a location application, a media application, a messaging application, and/or a gaming application. Third party applications 1340 may include the use of ANDROID by entities other than the vendor of the particular platform ^TM Or IOS ^TM An application developed by a Software Development Kit (SDK) and may be mobile software running on a mobile operating system, such as an IOS ^TM 、ANDROID ^TM WINDOWS, or other mobile operating system. Third party application 1340 may call API call 1308 provided by a mobile operating system, such as operating system 1302, to facilitate the functionality described herein.

Applications 1316 may use built-in operating system functionality (e.g., kernel 1322, services 1324, and/or drivers 1326), libraries 1320, and framework/middleware 1318 to create a user interface to interact with a user of the system. Additionally, in some systems, the user may interact with the user through a presentation layer, such as presentation layer 1314. In these systems, the application/component "logic" may be separate from aspects of the application/component that the user interacts with.

Fig. 14 is a block diagram illustrating components of a machine 1400 capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more of the methods discussed herein, according to some example embodiments. In particular, FIG. 14 shows a diagrammatic representation of machine 1400 in the example form of a computer system, for which instructions 1416 (e.g., software, programs, applications, applets, applications, or other executable code) may be used to cause machine 1400 to perform any one or more of the methodologies discussed herein. Accordingly, instructions 1416 may be used to implement the modules or components described herein. Instructions 1416 transform the generic non-programmed machine 1400 into a specific machine 1400 that is programmed to perform the functions described and illustrated in the manner described. In alternative embodiments, machine 1400 operates as a standalone device or may be coupled (e.g., networked) to other machines. In a network deployment, machine 1400 may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 1400 may include, but is not limited to, a server computer, a client computer, a Personal Computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a Personal Digital Assistant (PDA), an entertainment media system, a cellular telephone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a network appliance, a network router, a network switch, a network bridge, or any machine capable of executing instructions 1416, in turn or otherwise, that specify actions to be taken by machine 1400. Furthermore, while only a single machine 1400 is illustrated, the term "machine" shall also be taken to include a collection of machines that individually or jointly execute instructions 1416 to perform any one or more of the methodologies discussed herein.

Machine 1400 may include a processor 1410, memory/storage 1430, and I/O components 1450, which may be configured to communicate with each other, such as through bus 1402. Memory/storage 1430 may include memory 1432, such as main memory or other memory and storage unit 1436, both of which processor 1410 may access via bus 1402. The memory unit 1436 and the memory 1432 store instructions 1416 embodying any one or more of the methodologies or functions described herein. The instructions 1416 may also reside, completely or partially, within the memory 1432, within the storage unit 1436, within one of the processors 1410 (e.g., within a processor cache accessible to the processor unit 1412 or 1414), or any suitable combination thereof, during execution thereof by the machine 1400. Accordingly, the memory 1432, the storage unit 1436, and the memory of the processor 1410 are examples of machine-readable media.

The I/O component 1450 can include a variety of components to receive input, provide output, generate output, send information, exchange information, capture measurements, and the like. The particular I/O components 1450 included in a particular machine 1400 will depend on the type of machine. For example, a portable machine such as a mobile phone would likely include a touch input device or other such input mechanism, while a headless server machine would likely not include such a touch input device. It will be appreciated that the I/O component 1450 may include many other components not shown in fig. 14. The I/O components 1450 are grouped according to functionality for simplicity of the following discussion, and the grouping is in no way limiting. In various example embodiments, the I/O components 1450 may include an output component 1452 and an input component 1454. The output component 1452 may include visual components (e.g., a display such as a Plasma Display Panel (PDP), a Light Emitting Diode (LED) display, a Liquid Crystal Display (LCD), a projector, or a Cathode Ray Tube (CRT)), acoustic components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, and so forth. The input components 1454 may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optoelectronic keyboard, or other alphanumeric input components), point-based input components (e.g., a mouse, touchpad, trackball, joystick, motion sensor, or other pointing instrument), tactile input components (e.g., physical buttons, a touch screen providing the location and/or force of a touch or touch gesture, or other tactile input components), audio input components (e.g., a microphone), and the like.

In further example embodiments, the I/O components 1450 may include a biometric component 1456, a motion component 1458, an environmental component 1460, or a location component 1462, among a variety of other components. For example, the biometric component 1456 may include components that detect expressions (e.g., hand expressions, facial expressions, acoustic expressions, body gestures, or eye tracking), measure biometric signals (e.g., blood pressure, heart rate, body temperature, perspiration, or brain waves), identify a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), and so forth. The motion component 1458 may include an acceleration sensor component (e.g., accelerometer), a gravity sensor component, a rotation sensor component (e.g., gyroscope), and so forth. The environmental components 1460 may include, for example, an illumination sensor component (e.g., photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas sensor that detects the concentration of hazardous gases or measures contaminants in the atmosphere for safety), or other components that may provide an indication, measurement, or signal corresponding to the surrounding physical environment. The position components 1462 may include position sensor components (e.g., GPS receiver components), altitude sensor components (e.g., altimeters or barometers that may detect barometric pressure), orientation sensor components (e.g., magnetometers), and so forth.

Communication may be implemented using a variety of techniques. The I/O components 1450 may include a communication component 1464, the communication component 1464 operable to couple the machine 1400 to the network 1480 or device 1470 via the coupler 1482 and coupler 1472, respectively. For example, communication component 1464 may include a network interface component or other suitable device to interface with network 1480. In the other instance of the present invention, the communication components 1464 may include wired communication components, wireless communication components, cellular communication components, near Field Communication (NFC) components, and so forth,Components (e.g.)>Low power consumption)/(f)>Components, and other communication components that provide communication via other means. Device 1470 may be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via USB).

Further, the communication component 1464 may detect an identifier or include components operable to detect an identifier. For example, the communication component 1464 may include a Radio Frequency Identification (RFID) tag reader component, NFC smart tag detection groupAn optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multidimensional barcodes such as Quick Response (QR) codes, aztec codes, data matrices, dataglyph, maxiCode, PDF418, ultra codes, UCC RSS-2D barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying a marked audio signal). In addition, various information may be derived via communications component 1464, such as by Internet Protocol (IP) geolocation, by Location of signal triangulation, location of NFC beacon signals that may indicate a particular location via detection, and so forth.

Vocabulary list

Herein, "carrier wave signal" refers to an intangible medium capable of storing, encoding or carrying instructions 1416 for execution by the machine 1400, and includes digital or analog communication signals or other intangible medium to facilitate communication of such instructions 1416. The instructions 1416 may be transmitted or received over the network 1480 using a transmission medium via a network interface device and using any of a variety of well-known transmission protocols.

"client device" in this context refers to any machine 1400 that interfaces with a communication network 1480 to obtain resources from one or more server systems or other client devices 102. Client device 102 may be, but is not limited to, a mobile phone, desktop computer, laptop computer, PDA, smart phone, tablet, super book, netbook, multiprocessor system, microprocessor-based or programmable consumer electronics system, game console, set top box, or any other communication device that a user may use to access network 1480.

Herein, "communication network" refers to one or more portions of network 1480, which may be an ad hoc network, an intranet, an extranet, a Virtual Private Network (VPN), a Local Area Network (LAN), a Wireless LAN (WLAN), a Wide Area Network (WAN), a Wireless WAN (WWAN), a Metropolitan Area Network (MAN), the internet, a portion of the internet, a public switched telephone network (PS) TN), plain Old Telephone Service (POTS) network, cellular telephone network, wireless network, and,A network, another type of network, or a combination of two or more such networks. For example, the network or a portion of the network 1480 may include a wireless or cellular network, and the coupling may be a Code Division Multiple Access (CDMA) connection, a global system for mobile communications (GSM) connection, or another type of cellular or wireless coupling. In this example, the coupling may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1 xRTT), evolution data optimized (EVDO) technology, general Packet Radio Service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, third generation partnership project (3 GPP) including 3G, fourth generation wireless (4G) networks, universal Mobile Telecommunications System (UMTS), high Speed Packet Access (HSPA), worldwide Interoperability for Microwave Access (WiMAX), long Term Evolution (LTE) standards, and other standards defined by various standards-setting organizations, other remote protocols, or other data transmission technologies.

In this context, a "ephemeral message" refers to a message 400 that is accessible for a limited duration of time. The ephemeral message 502 may be text, images, video, or the like. The access time of the ephemeral message 502 may be set by the message sender. Alternatively, the access time may be a default setting or a setting specified by the recipient. Regardless of the setup technique, the message 400 is temporary.

Herein, "machine-readable medium" refers to a component, device, or other tangible medium capable of temporarily or permanently storing instructions 1416 and data, and may include, but is not limited to, random Access Memory (RAM), read Only Memory (ROM), cache memory, flash memory, optical media, magnetic media, cache memory, other types of storage (e.g., erasable programmable read-only memory (EPROM)), and/or any suitable combination thereof. The term "machine-readable medium" shall be taken to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) that are capable of storing instructions 1416. The term "machine-readable medium" shall also be taken to include any medium or combination of multiple media that is capable of storing instructions 1416 (e.g., code) for execution by the machine 1400 such that the instructions 1416, when executed by the one or more processors 1410 of the machine 1400, cause the machine 1400 to perform any one or more of the methodologies described herein. Thus, a "machine-readable medium" refers to a single storage device or apparatus, as well as a "cloud-based" storage system or storage network that includes multiple storage devices or apparatus. The term "machine-readable medium" does not include signals themselves.

"component" in this document refers to a device, physical entity, or logic having boundaries defined by function or subroutine calls, branch points, APIs, or other techniques that provide partitioning or modularization of specific processing or control functions. Components may be combined with interfaces of other components through their interfaces to perform a machine process. A component may be a packaged functional hardware unit designed for use with other components and a portion of a program that typically performs the specified function of the relevant function. The components may constitute software components (e.g., code embodied on a machine-readable medium) or hardware components. A "hardware component" is a tangible unit capable of performing certain operations and may be configured or arranged in some physical manner. In various example embodiments, one or more computer systems (e.g., a standalone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., processor 1412 or set of processors 1410) may be configured by software (e.g., an application or application portion) as hardware components that operate to perform certain operations as described herein. The hardware components may also be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include specialized circuitry or logic permanently configured to perform certain operations. The hardware component may be a special purpose processor such as a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC). The hardware components may also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. For example, the hardware components may include software that is executed by a general purpose processor or other programmable processor. Once such software is configured, the hardware components become the particular machine (or particular component of machine 1400) that is uniquely customized to perform the configured functions, and are no longer general purpose processor 1410. It should be appreciated that the decision to mechanically implement a hardware component in a dedicated and permanently configured circuit, or in a temporarily configured circuit (e.g., configured by software) may be driven by cost and time considerations. Thus, the phrase "hardware component" (or "hardware-implemented component") should be understood to include a tangible entity, be it a physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) entity that operates in some manner or performs some of the operations described herein.

Considering embodiments in which hardware components are temporarily configured (e.g., programmed), each hardware component need not be configured or instantiated at any one time. For example, where the hardware components include a general-purpose processor 1412 configured by software as a special-purpose processor, the general-purpose processor 1412 may be configured as different special-purpose processors (e.g., including different hardware components) at different times, respectively. The software configures the particular processor 1412 or 1410 accordingly, e.g., to form particular hardware components at one time instance and to form different hardware components at different time instances.

A hardware component may provide information to and receive information from other hardware components. Thus, the described hardware components may be considered to be communicatively coupled. Where multiple hardware components exist simultaneously, communication may be achieved through signaling (e.g., through appropriate circuitry and buses) between two or more of the hardware components. In embodiments where multiple hardware components are configured or instantiated at different times, communication between the hardware components may be achieved, for example, by storing and retrieving information in a memory structure accessible to the multiple hardware components. For example, one hardware component may perform an operation and store an output of the operation in a memory device communicatively coupled thereto. Another hardware component may then access the memory device at a later time to retrieve and process the stored output. The hardware component may also initiate communication with an input or output device and may operate on a resource (e.g., a collection of information).

Various operations of the example methods described herein may be performed, at least in part, by one or more processors 1410 that are temporarily configured (e.g., via software) or permanently configured to perform related operations. Whether temporarily or permanently configured, such a processor 1410 may constitute processor-implemented components that operate to perform one or more of the operations or functions described herein. As used herein, "processor-implemented components" refers to hardware components implemented using one or more processors 1410. Similarly, the methods described herein may be implemented, at least in part, by a processor, with the particular processor 1412 or 1410 being examples of hardware. For example, at least some of the operations of one method may be performed by one or more processors 1410 or processor-implemented components. In addition, one or more processors 1410 may also operate in a "cloud computing" environment or as "software as a service" (SaaS) to support the execution of related operations. For example, at least some of the operations may be performed by a set of computers (as examples of a machine 1400 that includes a processor 1410), which may be accessed via a network 1480 (e.g., the internet) and via one or more suitable interfaces (e.g., APIs). The performance of certain operations may be distributed among processors 1410, residing not only within a single machine 1400, but also across multiple machines 1400. In some example embodiments, the processor 1410 or processor-implemented components may be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other example embodiments, the processor 1410 or processor-implemented components may be distributed across multiple geographic locations.

"processor" herein refers to any circuit or virtual circuit (physical circuit emulated by logic executing on the actual processor 1412) that manipulates data values in accordance with control signals (e.g., "commands," "operation code," "machine code," etc.) and generates corresponding output signals that will be used to operate the machine 1400. The processor may be, for example, a Central Processing Unit (CPU), a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an ASIC, a Radio Frequency Integrated Circuit (RFIC), or any combination thereof. The processor 1410 may also be a multi-core processor 1410 having two or more independent processors 1412, 1414 (sometimes referred to as "cores") that may execute instructions 1416 simultaneously.

"timestamp" in this context refers to a sequence of characters or coded information that identifies when an event occurs, such as a given date and time, sometimes as accurate as a fraction of a second.

A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the patent and trademark office patent file or records, but otherwise reserves all copyright rights whatsoever. The following claims apply to the software and data as described below and in the accompanying drawings which form a part hereof: copyright 2017, snap INC. All rights are reserved.

Claims

1. A method, comprising:

generating, using one or more processors of the device, a video sequence comprising a plurality of frames;

generating a modified image in a first pipeline by using a machine learning scheme on the video sequence, the first pipeline comprising a previous frame having a corresponding modified previous frame and a current frame having no corresponding modified current frame in the first pipeline;

generating, at the device, a mapping between the previous frame and the current frame in a second pipeline; and

a corresponding modified current frame is generated by applying the mapping to the modified previous frame.

2. The method of claim 1, wherein the previous frame is separated from the current frame by a plurality of other frames in the video sequence.

3. The method of claim 1, wherein the second pipeline is asynchronous to the first pipeline.

4. The method of claim 3, wherein the first pipeline and the second pipeline are implemented on different threads of the one or more processors of the device.

5. The method of claim 1, wherein the machine learning scheme is trained to apply image manipulation functions, and wherein the modified previous frame exhibits the image manipulation.

6. The method of claim 5, wherein the mapping is applied to approximate the image manipulation in the modified previous frame.

7. The method of claim 1, wherein the map is a flowsheet describing image feature changes in the video sequence.

8. The method of claim 1, further comprising:

displaying a modified video sequence on the device, the modified video sequence comprising the modified previous frame and the modified current frame.

9. The method of claim 8, wherein the modified video sequence organizes the modified previous frame and the modified current frame.

10. The method of claim 1, wherein the machine learning scheme is a convolutional neural network.

11. The method of claim 10, wherein the convolutional neural network is trained to perform segmentation.

12. The method of claim 1, wherein the video sequence is generated by an image sensor of the device.

13. A system, comprising:

one or more processors of the machine; and

a memory storing instructions that, when executed by the one or more processors, cause the machine to perform operations comprising:

Generating, using the one or more processors, a video sequence comprising a plurality of frames;

generating, on the machine, a mapping between the previous frame and the current frame in a second pipeline; and

14. The system of claim 13, wherein the previous frame is separated from the current frame by a plurality of other frames in the video sequence.

15. The system of claim 13, wherein the second pipeline is asynchronous to the first pipeline.

16. The system of claim 15, wherein the first pipeline and the second pipeline are implemented on different threads of the one or more processors of the machine.

17. The system of claim 13, wherein the machine learning scheme is trained to apply image manipulation functions, and wherein the modified previous frame exhibits the image manipulation.

18. The system of claim 17, wherein the mapping is applied to approximate the image manipulation in the modified previous frame.

19. The system of claim 13, wherein the map describes image feature variations in the video sequence.

20. A non-transitory machine-readable medium embodying instructions that, when executed by a machine, cause the machine to perform operations comprising: