KR102624635B1 - 3D data generation in messaging systems - Google Patents

Abstract

The target technology receives image data and depth data. The target technology selects an augmented reality content generator that supports three-dimensional (3D) effects. The subject technology applies 3D effects to image data and depth data based at least in part on a selected augmented reality content generator. The target technology uses a processor to generate messages containing information related to the applied 3D effects, image data, and depth data.

Description

3D data generation in messaging systems

This application is based on U.S. Provisional Patent Application No. 62/893,037, filed on August 28, 2019, titled “GENERATING 3D DATA IN A MESSAGING SYSTEM,” filed on August 28, 2019 This U.S. Provisional Patent Application No. 62/893,046, titled “BEAUTIFICATION TECHNIQUES FOR 3D DATA IN A MESSAGING SYSTEM,” filed on August 28, 2019, is entitled “EFFECTS FOR 3D DATA IN A MESSAGING SYSTEM.” Patent Application No. 62/893,048, claims priority to U.S. Provisional Patent Application No. 62/893,050, entitled “PROVIDING 3D DATA FOR MESSAGES IN A MESSAGING SYSTEM,” filed August 28, 2019; Each of these contents is incorporated herein by reference in its entirety for all purposes.

With the increased use of digital images, the availability of portable computing devices, the availability of increased capacity in digital storage media, and the increased bandwidth and accessibility of network connections, digital images are a part of everyday life for an increasing number of people. It has become.

To facilitate the identification of discussion of any particular element or act, the most significant digit or digits in a reference number refer to the figure number in which the element is first introduced.
1 is a schematic representation of a network environment in which the present disclosure may be deployed, in accordance with some example embodiments.
2 is a schematic representation of a messaging client application, according to some example embodiments.
3 is a schematic representation of a data structure as maintained in a database, according to some example embodiments.
4 is a diagrammatic representation of a message, according to some example embodiments.
5 is a flow diagram of an access restriction process according to some example embodiments.
Figure 6 is a schematic diagram illustrating the structure of message annotations, as described in Figure 4, containing side information corresponding to a given 3D message, according to some embodiments.
7 is a block diagram illustrating various modules of an annotation system, according to some example embodiments.
8 is a flow diagram illustrating a method for generating a 3D message, according to some example embodiments.
9 is a flow diagram illustrating a method for performing transformation passes for processing image and depth data that may be performed in conjunction with a method for generating a 3D message, according to some example embodiments.
10 is a flow diagram illustrating a method for performing beautification of image and depth data that may be performed in conjunction with a method for generating a 3D message, according to some example embodiments.
11 is a flow diagram illustrating a method for updating a view of a 3D message in response to motion data that may be performed in conjunction with a method for generating a 3D message, according to some example embodiments.
12 illustrates a carousel for selecting and applying an augmented reality content generator to media content (e.g., an image or video) and presenting the applied augmented reality content generator to a messaging client application or messaging system, according to some embodiments. (carousel) illustrates example user interfaces.
13 is an example illustrating capturing image information and generating a 3D message on a display of a client device, according to some example embodiments.
Figure 14 is an example illustrating a raw depth map and a packed depth map, according to some example embodiments.
Figure 15 is an example illustrating a depth inpainting mask and depth inpainting, according to some example embodiments.
16 shows a 3D attachment rendered in response to particles, reflections on a graphical object (e.g., glasses), and motion data (e.g., motion data from a gyroscopic sensor), according to some example embodiments. and examples of 3D effects illustrating dynamic 3D attachments and post effects rendered in response to motion data.
17 is an example of a 3D effect illustrating dynamic artificial lighting rendered in response to motion data, and reflections on glasses, 3D attachments, and an animated sprite background rendered in response to motion data, according to some example embodiments. /This is an example of 3D effects that illustrate refraction.
18 shows an example of a controlled particle system (e.g., an animated projectile) rendered in response to motion data, and 3D effects illustrating 2D and 3D attachments, and to motion data, according to some example embodiments. Here is an example of 3D effects that illustrate joint animation for 3D attachments (eg, bunny ears) that are rendered in response.
19 is an example of 3D effects illustrating sprites, reflections on glasses, 2D and 3D attachments rendered in response to motion data, and glasses and particles rendered in response to motion data, according to some example embodiments. , and 3D effects illustrating reflection/refraction on an animated background.
20 is an example of 3D effects illustrating an animated foreground and an attachment occluding a user's face rendered in response to motion data, and dynamic artificial lighting, particles rendered in response to motion data, according to some example embodiments. , and 3D effects illustrating reflection/refraction on glasses.
21 is an example of a 3D effect illustrating retouches, post effects, 3D attachments and particles rendered in response to motion data, and 3D attachments, sprites and particles rendered in response to motion data, according to some example embodiments. This is an example of 3D effects illustrating particles.
Figure 22 is a block diagram illustrating a software architecture in which the present disclosure may be implemented, in accordance with some example embodiments.
Figure 23 is a schematic representation of a machine in the form of a computer system within which a set of instructions can be executed to cause the machine to perform any one or more of the methodologies discussed, according to some example embodiments.

Users with a range of interests from various locations can capture digital images of various subjects and make the captured images available to others through networks such as the Internet. To enhance users' experiences with digital images and provide a variety of features, computing devices capture under a wide range of changing conditions (e.g., changes in image scales, noise, lighting, motion, or geometric distortion). Being able to perform image processing operations on a variety of objects and/or features can be difficult and computationally intensive.

As discussed herein, the destination infrastructure supports the creation and sharing of interactive 3D media, referred to herein as 3D messages, across the various components of the messaging system. The infrastructure described herein allows other forms of 3D media to be presented across target systems, allowing depth-based media to be shared across messaging systems and along with photo and video messages. In example embodiments described herein, messages may enter the system from a live camera or from storage (e.g., if 3D messages and/or other messages are stored in memory or a database). The target system supports motion sensor input, manages transmission and storage of depth data, and loading of external effects and asset data.

As described herein, a three-dimensional (3D) message refers to an interactive 3D image that includes at least image and depth data. In an example embodiment, in addition to traditional image textures, the 3D message is rendered using an object system to visualize the spatial details/geometry of what the camera sees. When a viewer interacts with this 3D message by moving the client device, the movement triggers a corresponding change in perspective from which images and geometry are rendered to the viewer.

As referred to herein, the phrases “augmented reality experience,” “augmented reality content item,” and “augmented reality content generator” refer to image modifications, filters, and lenses (LENSES), as further described herein. , includes or refers to various image processing operations corresponding to media overlay, conversion, etc.

As referred to herein, 3D augmented reality content generator refers to real-time special effects and/or sounds that can be added to messages, AR effects, such as 3D animated graphic elements, and/or other 3D content. Modify image and/or depth data.

1 is a block diagram illustrating an example messaging system 100 for exchanging data (e.g., messages and associated content) over a network. Messaging system 100 includes multiple instances of client devices 102, each of which hosts multiple applications, including messaging client applications 104. Each messaging client application 104 is communicatively coupled to other instances of messaging client application 104 and messaging server system 108 over a network 106 (e.g., the Internet).

Messaging client applications 104 may communicate and exchange data with other messaging client applications 104 and with messaging server system 108 over network 106 . Data exchanged between messaging client applications 104 and between messaging client applications 104 and messaging server system 108 include functions (e.g., commands that invoke functions) as well as payload data ( For example, text, audio, video or other multimedia data).

Messaging server system 108 provides server-side functionality to specific messaging client applications 104 over network 106. Although certain functions of messaging system 100 are described herein as being performed by messaging client application 104 or by messaging server system 108, certain functions within messaging client application 104 or messaging server system 108 The location of functionality is a design choice. For example, it may be technically feasible to initially place certain technology and functionality within the messaging server system 108, but later transfer this technology and functionality to the messaging client application 104 when the client device 102 has sufficient processing capacity. It may be desirable.

Messaging server system 108 supports various services and operations provided to messaging client application 104. Such operations include sending data to, receiving data from, and processing data generated by messaging client application 104. This data may include, for example, message content, client device information, geolocation information, media annotations and overlays, message content persistence conditions, social network information, and live event information. Data exchange within messaging system 100 is initiated and controlled through functions available through user interfaces (UIs) of messaging client application 104.

Referring now specifically to messaging server system 108, an application program interface (API) server 110 is coupled to application server 112 and provides a programmatic interface. Application server 112 is communicatively coupled to database server 118, which facilitates access to database 120 where data associated with messages processed by application server 112 is stored.

The Application Program Interface (API) server 110 receives and transmits message data (e.g., commands and message payloads) between the client device 102 and the application server 112. Specifically, application program interface (API) server 110 provides interfaces (e.g., routines and protocols) that can be called or queried by messaging client application 104 to invoke the functionality of application server 112. s) is provided. The application program interface (API) server 110 provides account registration, login functionality, transfer of messages through the application server 112 from one messaging client application 104 to another messaging client application 104, and messaging client application 110. Transfer of media files (e.g., images or videos) from 104 to messaging server application 114, and collection of media data (e.g., for possible access by other messaging client applications 104). , Stories), retrieval of the user's list of friends on the client device 102, retrieval of such collections, retrieval of messages and content, addition and deletion of friends to the social graph, location of friends in the social graph, and ( It exposes various functions supported by the application server 112, including, for example, opening application events (related to the messaging client application 104).

Application server 112 hosts a number of applications and subsystems, including messaging server application 114, image processing system 116, and social networking system 122. Messaging server application 114 may perform multiple message processing, particularly related to aggregation and other processing of content (e.g., text and multimedia content) included in messages received from multiple instances of messaging client application 104. Implement technologies and functions. As will be described in greater detail, text and media content from multiple sources may be aggregated into collections of content (e.g., called stories or galleries). These collections are then made available by the messaging server application 114 to the messaging client application 104. Processing of other processor- and memory-intensive data may also be performed server-side by the messaging server application 114, taking into account hardware requirements for such processing.

Application server 112 also typically includes an image processing system 116 dedicated to performing various image processing operations on images or video received within the payload of a message in messaging server application 114.

Social networking system 122 supports various social networking functions and services and makes these functions and services available to messaging server application 114. To this end, social network system 122 maintains and accesses entity graph 304 (shown in FIG. 3) within database 120. Examples of functions and services supported by social networking system 122 include, but are not limited to, identification of other users of messaging system 100 with which a particular user has relationships or “follows,” and also identification of other entities and functions of a particular user. Include interests.

Application server 112 is communicatively coupled to database server 118, which facilitates access to database 120 where data associated with messages processed by messaging server application 114 is stored.

2 is a block diagram illustrating additional details regarding messaging system 100, according to example embodiments. Specifically, the messaging system 100 is shown as including a messaging client application 104 and an application server 112, which in turn may include a number of subsystems, namely a short-lived timer system 202, a collection management system 204 ) and annotation system 206.

The short-term timer system 202 is responsible for enforcing temporary access to content permitted by the messaging client application 104 and messaging server application 114. To this end, the short-lived timer system 202 selectively displays messages and associated content via the messaging client application 104 based on the duration and display parameters associated with the message or collection of messages (e.g., a story). Contains a number of timers that display and enable access to them. Additional details regarding the operation of short-term timer system 202 are provided below.

Collection management system 204 is responsible for managing collections of media (eg, collections of text, images video and audio data). In some examples, a collection of content (eg, messages containing images, video, text, and audio) may be organized into an “event gallery” or “event story.” Such collections may be made available for a specified period of time, such as the duration of the event to which the content relates. For example, content about a music concert may be made available as a “story” for the duration of the music concert. Collection management system 204 may also be responsible for posting icons in the user interface of messaging client application 104 that provide notification of the existence of a particular collection.

Collection management system 204 further includes a curation interface 208 that allows collection managers to manage and curate specific collections of content. For example, curation interface 208 allows an event organizer to curate a collection of content related to a particular event (e.g., remove inappropriate content or duplicate messages). Additionally, collection management system 204 automatically curates content collections using machine vision (or image recognition technology) and content rules. In certain embodiments, a reward may be paid to a user for including user-generated content in a collection. In such cases, curation interface 208 operates to automatically pay such users for using their content.

Annotation system 206 provides various functions that allow a user to annotate or otherwise modify or edit media content associated with a message. For example, annotation system 206 provides functionality related to the creation and posting of media overlays for messages processed by messaging system 100. The annotation system 206 operationally supplies media overlays or supplements (e.g., image filters) to the messaging client application 104 based on the geolocation of the client device 102. In another example, the annotation system 206 operationally supplies a media overlay to the messaging client application 104 based on other information, such as social network information of the user of the client device 102. Media overlays can include audio and visual content and visual effects. Examples of audio and visual content include photos, text, logos, animations, and sound effects. Examples of visual effects include color overlaying. Audio and visual content or visual effects may be applied to media content items (e.g., photos) on client device 102. For example, a media overlay may include text that may be overlaid over a photo taken by client device 102. In other examples, the media overlay includes the identification of a location overlay (eg, Venice beach), the name of a live event, or the name of a merchant overlay (eg, Beach Coffee House). In another example, annotation system 206 uses the geolocation of client device 102 to identify a media overlay that includes the name of a merchant at the geolocation of client device 102. The media overlay may include other indicia associated with the merchant. Media overlays may be stored in database 120 and accessed through database server 118.

In one example embodiment, annotation system 206 provides a user-based publishing platform that allows users to select geolocations on a map and upload content associated with the selected geolocation. to provide. Users can also specify situations in which certain media overlays should be provided to other users. Annotation system 206 creates a media overlay that contains the uploaded content and associates the uploaded content with the selected geolocation.

In another example embodiment, annotation system 206 provides a merchant-based publishing platform that allows merchants to select specific media overlays associated with a geolocation through a bidding process. For example, annotation system 206 associates a media overlay of the highest bidding merchant with a corresponding geolocation for a predefined amount of time.

FIG. 3 is a schematic diagram illustrating data structures 300 that may be stored in database 120 of messaging server system 108, according to some example embodiments. Although the contents of database 120 are shown as including a number of tables, it will be appreciated that data may be stored in other types of data structures (eg, as an object-oriented database).

Database 120 contains message data stored in message table 314. The entity table 302 stores entity data including the entity graph 304. Entities for which records are maintained within entity table 302 may include individuals, corporate entities, organizations, objects, places, events, etc. Any entity, regardless of type, about which messaging server system 108 stores data may be a recognized entity. Each entity has a unique identifier as well as an entity type identifier (not shown).

The entity graph 304 further stores information regarding relationships and associations between entities. Such relationships may be, for example, social, professional (eg, work in a general corporation or organization), interest-based, or activity-based.

Database 120 also stores annotation data in annotation table 312 in the example form of filters. Filters, for which data is stored in annotation table 312, are associated with and applied to videos (for which data is stored in video table 310) and/or images (for which data is stored in image table 308). do. In one example, filters are overlays that are displayed overlaid on an image or video during presentation to a recipient user. Filters can be of various types, including user-selected filters from a gallery of filters that are presented to the sending user by messaging client application 104 when the sending user is composing a message. Other types of filters include geolocation filters (also called geo-filters), which can be presented to the sending user based on geographic location. For example, neighborhood or specific location-specific geolocation filters may be presented within the user interface by messaging client application 104 based on geolocation information determined by the GPS unit of client device 102. Another type of filter is a data filter that can be selectively presented to the sending user by the messaging client application 104 based on information or other inputs collected by the client device 102 during the message creation process. Examples of data filters include the current temperature at a particular location, the current speed at which the sending user is traveling, battery life for the client device 102, or the current time.

Other annotation data that may be stored within image table 308 are augmented reality content generators (eg, corresponding to applying LENSES, augmented reality experiences, or augmented reality content items). Augmented reality content generators can be real-time special effects and sounds that can be added to images or videos.

As described above, augmented reality content generators, augmented reality content items, overlays, image transformations, AR images, and similar terms refer to modifications that may be made to videos or images. This involves real-time modification of the image as it is captured using the device sensor and then displayed on the device's screen with the modifications. This includes modifications to stored content, such as video clips in a gallery that can also be modified. For example, on a device that accesses multiple augmented reality content generators, a user can use a single video clip with multiple augmented reality content generators to see how different augmented reality content generators will modify the stored clip. For example, multiple augmented reality content generators applying different pseudorandom motion models can be applied to the same content by selecting different augmented reality content generators for the content. Similarly, real-time video capture can be used with modifications shown to demonstrate how video images currently being captured by the device's sensors may modify the captured data. Such data may simply be displayed on a screen and not stored in memory, or content captured by device sensors may be recorded and stored in memory with or without modifications (or both). In some systems, a preview feature can show how different augmented reality content generators will look simultaneously within different windows within the display. This could, for example, allow multiple windows with different pseudo-random animations to be viewed simultaneously on the display.

Accordingly, data and various systems that use augmented reality content generators or other such transformation systems to modify content using this data can be used to create objects (e.g., faces, hands, bodies, cats, dogs, surfaces, objects, etc.), tracking these objects as they leave, enter, and move around the field of view in video frames, and modifying or transforming these objects as they are tracked. . In various embodiments, different methods may be used to achieve such transformations. For example, some embodiments may involve creating a three-dimensional mesh model of an object or objects, and using a transform and animated texture of the model within the video to accomplish the transformation. In other embodiments, tracking of points on an object may be used to place an image or texture (which may be two-dimensional or three-dimensional) at the tracking position. In still other embodiments, neural network analysis of video frames may be used to place images, models, or textures in content (e.g., images or frames of video). Accordingly, augmented reality content generators provide both the images, models, and textures used to create transformations of the content, as well as the additional modeling and analysis information needed to achieve these transformations with object detection, tracking, and placement. refers to

Real-time video processing can be performed on any type of video data (eg, video stream, video file, etc.) stored in the memory of any type of computerized system. For example, a user can load video files and save them to the device's memory, or use the device's sensors to generate a video stream. In addition, any objects, such as human faces and parts of the human body, animals, or inanimate objects such as chairs, cars, or other objects, can be processed using computer animation models.

In some embodiments, when a particular modification is selected with content to be converted, the elements to be converted are identified by the computing device and then detected and tracked if they are present in a frame of video. The elements of the object are modified according to the request for modification, thus transforming the frames of the video stream. Transformation of frames of a video stream can be performed by different methods for different types of transformation. For example, for transformations of frames that primarily refer to changing the shapes of elements of an object, characteristic points for each of the elements of the object can be identified (e.g. using Active Shape Model (ASM) or other known methods). ) is calculated. A mesh is then created based on the characteristic points for each at least one element of the object. This mesh is used in the next stage to track the elements of the object in the video stream. In the tracking process, the referenced mesh for each element is aligned with the position of each element. Next, additional points are created on the mesh. A first set of points is created for each element based on the modification request, and a second set of points is created for each element based on the first set of points and the modification request. Frames of the video stream can then be transformed by modifying elements of the object based on the mesh and the sets of first and second points. In such a method, the background of the modified object can likewise be altered or distorted by tracking and modifying the background.

In one or more embodiments, transformations that change some regions of an object using elements of the object may be performed by calculating characteristic points for each element of the object and generating a mesh based on the calculated characteristic points. there is. Points are created on the mesh, and then various areas are created based on the points. The elements of the object are then tracked by aligning the region for each element with the position for each of at least one element, and the properties of the regions can be modified based on the request for modification, thus transforming the frames of the video stream. . Depending on the specific request for modification, the properties of the mentioned areas may be transformed in different ways. These modifications include changing the color of areas; removing at least some of the regions from frames of the video stream; including one or more new objects in areas based on the modification request; and modifying or distorting elements of an area or object. In various embodiments, any combination of such modifications or other similar modifications may be used. For specific models to be animated, some characteristic points can be selected as control points to be used to determine the overall state-space of options for model animation.

In some embodiments of a computer animation model that transforms image data using face detection, faces are detected on the image using a specific face detection algorithm (eg, Viola-Jones). Afterwards, the Active Shape Model (ASM) algorithm is applied to the facial area of the image to detect facial feature reference points.

In other embodiments, other methods and algorithms suitable for face detection may be used. For example, in some embodiments, features are located using landmarks that represent distinguishable points present in most of the images under consideration. For example, for facial landmarks, the location of the left eye pupil may be used. When the initial landmark is not identifiable (eg, if the person has an eye patch), auxiliary landmarks may be used. Such landmark identification procedures may be used for any such objects. In some embodiments, the set of landmarks forms a shape. Shapes can be expressed as vectors using the coordinates of points in the shape. One shape is aligned to another shape by a similarity transformation (allowing translation, scaling, and rotation) that minimizes the average Euclidean distance between shape points. The average shape is the average of the aligned training shapes.

In some embodiments, a search begins for landmarks from an average shape aligned to the position and size of the face as determined by the global face detector. This search then repeats the steps of proposing a temporary shape by adjusting the positions of the shape points by template matching of the image texture around each point, followed by matching the temporary shape to the global shape model until convergence occurs. . In some systems, individual template matches are unreliable, and the shape model pools the results of weak template matchers to form a stronger overall classifier. The entire search is repeated at each level in the image pyramid, from coarse to fine resolution.

Embodiments of the conversion system include capturing an image or video stream on a client device (e.g., client device 102) and converting it locally on the client device 102, while maintaining an appropriate user experience, computation time, and power consumption. Complex image manipulation can be performed. Complex image manipulation involves changing size and shape, conveying emotion (e.g., changing a face from a frown to a smile), conveying status (e.g., aging a subject, decreasing apparent age, changing gender), conveying style, and graphics. component applications, and any other suitable image or video manipulation implemented by a convolutional neural network configured to run efficiently on client device 102.

In some example embodiments, a computer animation model for converting image data is created when a user uses a client device 102 with a neural network running as part of a messaging client application 104 running on the client device 102. It can be used by a system to capture an image or video stream of a user (e.g., a selfie). A conversion system operating within messaging client application 104 determines the presence of a face in an image or video stream and provides modification icons associated with a computer-animated model for converting the image data, or a computer-animated model, as described herein. It can exist as something associated with an interface. Modification icons contain changes that may be the basis for modifying the user's face within an image or video stream as part of a modification operation. Once a modification icon is selected, the transformation system initiates the process of transforming the user's image (e.g., creating a smiling face on the user) to reflect the selected modification icon. In some embodiments, the modified image or video stream may be presented in a graphical user interface displayed on the mobile client device as soon as the image or video stream is captured and the specified modification is selected. The transformation system may implement a complex convolutional neural network on a portion of the image or video stream to generate and apply selected modifications. That is, users can capture an image or video stream and, once the edit icon is selected, receive a presentation of the modified results in real time or near real time. Additionally, modifications can be persistent as long as the video stream is being captured and the selected modification icon remains toggled. Machine training neural networks can be used to enable these modifications.

In some embodiments, a graphical user interface presenting modifications performed by the conversion system may provide additional interaction options to the user. These options may be based on the interface used to initiate content capture and selection of a particular computer animation model (eg, launching from a content creator user interface). In various embodiments, editing may continue after the initial selection of the editing icon. The user can toggle the correction on or off by tapping or otherwise selecting the face being modified by the transformation system, and save the correction for later viewing or browsing to other areas of the imaging application. When multiple faces are modified by the transformation system, the user can toggle modification on or off globally by tapping or selecting a single face to be modified and displayed within the graphical user interface. In some embodiments, individual faces may be individually modified among a group of multiple faces or such modifications may be individually toggled by tapping or selecting an individual face or series of individual faces displayed within a graphical user interface. .

In some example embodiments, a graphics processing pipeline architecture is provided that allows different augmented reality experiences (e.g., AR content generators) to be applied to corresponding different layers. This graphics processing pipeline is extensible to provide multiple augmented reality experiences embedded in synthetic media (e.g., images or video) for rendering by messaging client application 104 (or messaging system 100). Provides a rendering engine.

As mentioned above, video table 310 stores video data associated with messages for which, in one embodiment, records are maintained in message table 314. Similarly, image table 308 stores image data associated with messages for which message data is stored in entity table 302. Entity table 302 may associate various annotations from annotation table 312 with various images and videos stored in image table 308 and video table 310.

Story table 306 stores data about collections of messages and associated image, video or audio data, compiled into a collection (e.g., story or gallery). Creation of a particular collection may be initiated by a particular user (e.g., each user for whom a record is maintained in entity table 302). A user may create a “personal story” in the form of a collection of content created and transmitted/broadcast by that user. To this end, the user interface of the messaging client application 104 may include user-selectable icons to enable the sending user to add specific content to his or her personal story.

Collections may also constitute “live stories,” which are collections of content from multiple users created manually, automatically, or using a combination of manual and automated techniques. For example, “Live Stories” may constitute a curated stream of user-submitted content from various locations and events. Users whose client devices are location services enabled and who are at a common location event at a particular time may be presented with the option to contribute content to a particular live story, for example, via the user interface of the messaging client application 104. . Live stories may be identified to the user by messaging client application 104 based on their location. The end result is a “live story” told from a community perspective.

An additional type of content collection is a “location story,” which allows users with client devices 102 located within a specific geographic location (e.g., a college or university campus) to contribute to a specific collection. "It is known. In some embodiments, contributions to location stories may require a second degree of authentication to verify that the end user belongs to a particular organization or other entity (e.g., is a student on a college campus).

4 is a schematic diagram illustrating the structure of a message 400 generated by a messaging client application 104 for communication to an additional messaging client application 104 or a messaging server application 114, according to some embodiments. am. The content of a particular message 400 is used to populate a message table 314 stored within database 120, accessible by messaging server application 114. Similarly, the content of message 400 is stored in memory as “in-transit” or “in-flight” data of client device 102 or application server 112. Message 400 is shown as including the following components:

· Message identifier 402: A unique identifier that identifies the message 400.

· Message text payload 404: Text generated by the user through the user interface of the client device 102 and included in the message 400.

· Message image payload 406: Image data captured by a camera component of client device 102 or retrieved from a memory component of client device 102 and included in message 400.

· Message video payload 408: Video data captured by a camera component or retrieved from a memory component of client device 102 and included in message 400.

· Message audio payload 410: Audio data captured by a microphone or retrieved from a memory component of client device 102 and included in message 400.

· Message annotation 412: Annotation data (e.g., filter, stickers or other reinforcers).

· Message duration parameter 414: The content of the message (e.g., message image payload 406, message video payload 408, message audio payload 410) is transmitted via messaging client application 104. A parameter value that indicates the amount of time, in seconds, that will be presented or made accessible to the user.

· Message geolocation parameters 416: Geolocation data (e.g., latitude and longitude coordinates) associated with the content payload of the message. Multiple message geolocation parameter 416 values may be included in the payload, each of which may be associated with content (e.g., a specific image within the message image payload 406, or a specific image within the message video payload 408). related to the content items included in the video).

· Message Story Identifier 418: An identifier value that identifies one or more content collections (e.g., “stories”) with which a particular content item within the message image payload 406 of message 400 is associated. For example, multiple images within message image payload 406 may each be associated with multiple content collections using identifier values.

· Message tag 420: Each message 400 may be tagged with multiple tags, each of which represents the subject of the content included in the message payload. For example, if a particular image included in message image payload 406 depicts an animal (e.g., a lion), a tag value representing the relevant animal may be included within message tag 420. Tag values may be generated manually based on user input, or automatically using image recognition, for example.

· Message sender identifier 422: An identifier representing the user of the client device 102 through which the message 400 was created and from which the message 400 was sent (e.g., a messaging system identifier, email address, or device identifier) ).

· Message recipient identifier 424: An identifier representing the user of client device 102 to which message 400 is addressed (e.g., messaging system identifier, email address, or device identifier).

The content (e.g., values) of the various components of message 400 may be pointers to locations within a table where the content data values are stored. For example, an image value in the message image payload 406 may be a pointer to a location (or its address) in the image table 308. Similarly, values in message video payload 408 can point to data stored in video table 310, values stored in message annotation 412 can point to data stored in annotation table 312, and message story Values stored in identifier 418 may point to data stored in story table 306, and values stored in message sender identifier 422 and message recipient identifier 424 may point to user records stored in entity table 302. there is.

5 illustrates an example in which access to content (e.g., a short-lived message 502, and an associated multimedia payload of data) or a collection of content (e.g., a short-lived message group 504) may be time-limited (e.g. Here is a schematic diagram illustrating an access-restriction process 500, which can be (e.g., short term).

Short-term message 502 is shown to be associated with a message duration parameter 506, the value of which determines how short-term message 502 will be displayed by messaging client application 104 to the user receiving short-term message 502. Decide on the amount of time. In one embodiment, the short-lived message 502 may be visible to the receiving user for up to 10 seconds, depending on the amount of time the sending user specifies using the message duration parameter 506.

The message duration parameter 506 and message recipient identifier 424 are shown as inputs to a message timer 512, which determines whether a short-lived message 502 will be received at a specific reception identified by message recipient identifier 424. Responsible for determining the amount of time a side is visible to the user. In particular, short-lived messages 502 will only be visible to the relevant recipient user for a period of time determined by the value of the message duration parameter 506. Message timer 512 is shown providing output to the more general short-term timer system 202, which is responsible for the overall timing of the display of content (e.g., short-term message 502) to a receiving user.

Short-term message 502 is shown in FIG. 5 as being included within a short-term message group 504 (e.g., a collection of messages in a personal story or event story). Short-term message group 504 has an associated group duration parameter 508, the value of which determines the time duration for which short-term message group 504 is presented and accessible to users of messaging system 100. Group duration parameter 508 may be, for example, the duration of a music concert, where short-lived message group 504 is a collection of content related to that concert. Alternatively, a user (either an owning user or a curator user) may specify a value for the group duration parameter 508 when performing setup and creation of a short-lived message group 504.

Additionally, each short-term message 502 within a short-term message group 504 has an associated group participation parameter 510, the value of which determines whether the short-term message 502 will be accessible within the context of the short-term message group 504. Determines the duration of time. Accordingly, a particular short-term message group 504 is “expired” and inaccessible within the context of the short-term message group 504 before the short-term message group 504 itself expires in terms of the group duration parameter 508. It can be done. The group duration parameter 508, group participation parameter 510, and message recipient identifier 424 each provide inputs to the group timer 514, which, during operation, first sets Determines whether a particular short-term message 502 will be displayed to a particular recipient user and, if so, for how long. Note that the short-term message group 504 also recognizes the identity of a particular recipient user as a result of the message recipient identifier 424.

Accordingly, the group timer 514 operationally controls the overall lifetime of the associated short-term message group 504, as well as the individual short-term messages 502 included in the short-term message group 504. In one embodiment, each and every short-term message 502 within a short-term message group 504 remains visible and accessible for a time-period specified by the group duration parameter 508. In a further embodiment, a particular short-term message 502 may expire within the context of a short-term message group 504 based on the group participation parameter 510. Note that the message duration parameter 506 may still determine the duration of time that a particular short-term message 502 is displayed to the receiving user, even within the context of a short-term message group 504. Accordingly, the message duration parameter 506 determines whether a particular short-term message 502 will be sent to the receiving user, regardless of whether the receiving user is viewing the short-term message 502 within or outside the context of the short-term message group 504. Determines the duration of time displayed.

The short-lived timer system 202 may furthermore, during operation, remove a particular short-lived message 502 from the short-lived message group 504 based on a determination that it has exceeded the associated story participation parameter 510 . For example, when a sending-side user has established a group participation parameter 510 of 24 hours from posting, the short-term timer system 202 may transmit related short-term messages 502 from the short-term message group 504 after the specified 24 hours. ) will be removed. The short-lived timer system 202 may also determine when the group participation parameter 510 for each and every short-lived message 502 within the short-lived message group 504 has expired, or when the short-lived message group 504 itself has expired the group duration parameter. Operates to remove a short-lived message group 504 when it expires with respect to 508 .

In certain use cases, the creator of a particular short-term message group 504 may specify an infinite group duration parameter 508. In this case, the expiration of the group participation parameter 510 for the last remaining short-term message 502 within the short-term message group 504 will determine when the short-term message group 504 itself expires. In this case, the new short-term message 502 added to the short-term message group 504, together with the new group participation parameter 510, causes the lifetime of the short-term message group 504 to be equal to the value of the group participation parameter 510. Extend effectively.

In response to the short-term timer system 202 determining that the short-term message group 504 has expired (e.g., is no longer accessible), the short-term timer system 202 sends the messaging system 100 (and For example, specifically communicate with the messaging client application 104 so that an indication (e.g., an icon) associated with the relevant short-lived message group 504 is no longer displayed within the user interface of the messaging client application 104. do. Similarly, when the short-term timer system 202 determines that the message duration parameter 506 for a particular short-term message 502 has expired, the short-term timer system 202 causes the messaging client application 104 to send the short-term message ( 502) so as not to display any further indications (e.g., icons or text identifications) associated therewith.

As discussed herein, the destination infrastructure supports the creation and sharing of interactive 3D media, referred to herein as 3D messages, across the various components of messaging system 100. The infrastructure described herein allows other forms of 3D media to be provided across target systems, allowing depth-based media to be shared across messaging system 100 and along with photo and video messages. In example embodiments described herein, messages may enter the system from a live camera or from storage (e.g., if 3D messages and/or other messages are stored in memory or a database). The target system supports motion sensor input, manages transmission and storage of depth data, and loading of external effects and asset data.

As mentioned above, a three-dimensional (3D) message refers to an interactive 3D image that includes at least image and depth data. In an example embodiment, in addition to traditional image textures, the 3D message is rendered using an object system to visualize the spatial details/geometry of what the camera sees. When a viewer interacts with this 3D message by moving the client device, the movement triggers a corresponding change in perspective from which images and geometry are rendered to the viewer.

In one embodiment, the target system provides 3D effects that work with other components of the system to process depth data, including particles, shaders, and 2D assets that can populate different depth planes within messages. and 3D geometry. In an example, this would allow the target system to occlude and respond to user interactions (e.g., as detected through motion sensor data) by changing the physical placement and visual appearance of assets within these messages, and It makes it possible to provide effects. In an example, 3D effects as described herein are rendered in real-time to the user, which also allows different particles and layers to be placed in different positions for each user viewing those particles and layers. Let it be done.

As discussed herein, 2D attachments or 3D attachments are images (e.g., sprites) or geometry (e.g., 3D attachments) that can be attached to a 3D effect in a similar manner to those attached to an augmented reality content generator. corresponds to an object).

As described herein, facial effects refer to beautification, facial retouching, stretching and other effects that can be applied to an image containing a face to transform or beautify the face through an augmented reality content generator and/or other media overlay. do.

As referred to herein, gyro-based interaction refers to a type of interaction in which the rotation of a given client device is used as input to change aspects of an effect (e.g., changing the color of light in a scene). (rotate the phone along the x-axis).

As referred to herein, 3D augmented reality content generator refers to real-time special effects and/or sounds that can be added to 3D messages and modify image and/or depth data.

In an embodiment, when a user initiates capture of a 3D message, the target system applies a bundled or dynamic depth effect augmented reality content generator using a mesh generated based on depth information and camera images. To recreate the same effect after storing the 3D message in the cloud (e.g., in database 120 of messaging server system 108), raw input data or generated meshes, augmented reality content generators, etc. are stored. In an example, raw input data is stored in addition to the camera image to reduce storage requirements. The discussion below relates to example data stored in connection with such 3D messages in accordance with some embodiments.

FIG. 6 illustrates the structure of message annotations 412 containing additional information corresponding to a given 3D message, according to some embodiments, generated by messaging client application 104, as described above in FIG. 4. This is a schematic diagram. In an embodiment, the content of a particular message 400, including the additional data shown in Figure 6, is used to populate a message table 314 stored within database 120 for a given 3D message, which is then used by the messaging server application ( 114) and/or messaging client application 104. As illustrated in the example embodiment, message annotation 412 includes the following components corresponding to data for the 3D message:

· Augmented reality content generator identifier 652: Identifier of the augmented reality content generator used in the message (e.g., animations and/or effects including 3D effects, LENSES, filters, etc.)

· Message identifier (654): Identifier of the message

· Asset identifiers 656: A set of identifiers for assets in the message. For example, individual asset identifiers may be included for multiple assets as determined by a particular 3D augmented reality content generator. In an embodiment, these assets are created by a 3D augmented reality content generator on the sender side, uploaded to messaging server application 114, and used by a 3D augmented reality content generator on the recipient side to recreate the message. Examples of typical assets include:

o Original still RGB image captured by the camera

o A combined depth map and portrait segmentation mask that provides a 3D capture of the user isolated from the background. In an embodiment and further described herein, this is generated by render passes in a 3D augmented reality content generator from raw depth data and portrait segmentation, where the portrait segmentation produces a multi-channel image for transmission (e.g. , RGB channels with an alpha channel).

o Blurred background image to be placed behind the 3D user capture. In an embodiment, this is generated by render passes in an augmented reality content generator that uses a portrait segmentation mask to perform inpainting of the image content behind the user.

o 3D depth data (further mentioned below)

o Portrait segmentation mask (further mentioned below)

· Depth data 658: Raw depth data (e.g. 640x360 with 16 bits and/or depth map

· Mask data 660: data corresponding to a portrait segmentation mask based on raw depth data and/or depth map.

· Metadata (662) corresponding to additional metadata, including but not limited to:

o 3D message metadata attached to camera image metadata

· Camera-specific data

· Focal distance

· Main points

· External camera data

· Quaternary number representing rotation between two cameras

· Conversion between two cameras

· Other camera information (e.g. camera position)

o Augmented reality content generator ID of the 3D depth effect in the corresponding augmented reality content generator

o Media properties that indicate that the message has depth

o Augmented Reality Content Generator asset metadata corresponding to the Augmented Reality Content Generator

Although not shown in FIG. 6 , in an exemplary embodiment, a given 3D message also includes the following data (e.g., as previously described with respect to FIG. 4 ): 1) a placeholder 2D image; (e.g. 2D photos with information corresponding to 3D effects), and 2) standard 2D overlays applied to 3D messages (e.g. filters based on geolocation, stickers, captions, etc.). Accordingly, in an embodiment, the 3D message includes metadata corresponding to configuration data (e.g., camera specific data, attached object positions) for the augmented reality content generator and references to assets stored in association with the 3D message. Contains a placeholder image.

In an example, a user can select from multiple augmented reality content generators that result in different visual processing of the raw data provided by the camera. In the example, after an augmented reality content generator (involving 3D data) is selected in messaging client application 104, the camera captures raw depth data, and camera images. In the example, this raw data is provided to components to render a view of the message containing depth data. Additionally, in an embodiment, this raw data is used by message client application 104 (or components thereof) to generate a portrait segmentation mask.

In an embodiment, the augmented reality content generator may be configured to store assets that are uploaded to the messaging server application 114, and other data attached to the message (any other data the recipient needs to reconstruct the effect, e.g., field of view). information, configuration parameters, etc.).

In an example, the sender creates a standard 2D image message that includes augmented reality content generator metadata containing information that is later used by the recipient to reconstruct the 3D message. This is the ID of the 3D message augmented reality content generator (e.g. the recipient will also download and run the same augmented reality content generator used by the sender), the 3D message ID (e.g. all assets will be stored in this specific 3D message) (to associate with), and any structured data (e.g., numbers, text, vectors and matrices) embedded directly in the metadata and any referencing larger assets stored in the messaging server application 114. Contains asset IDs and configuration data generated by the 3D augmented reality content generator itself, including a number of asset IDs (e.g., images, videos, 3D meshes).

In an embodiment, facial data processing occurs only on the sender side. The results of this processing are then stored as configuration data by the 3D message augmented reality content generator (e.g. 3D transformation of the head as a transformation matrix), and this configuration data is retrieved and used at the recipient side (e.g. , the recipient does not reprocess the face data from the original image). This allows the receiving device to render #D for display.

As an example, the recipient receives a standard 2D image message, but due to the presence and content of the augmented reality content generator metadata, the recipient reconstructs the 3D message based on this metadata. This entails first fetching the 3D message augmented reality content generator using that ID, and then fetching all assets associated with the 3D message ID. The recipient loads the 3D message augmented reality content generator, which itself includes logic to request appropriate assets and data and reassemble the 3D message. In the example, the 3D message asset will not contain information to perform further processing with respect to the given media overlay, so that, for example, if a sticker (e.g., an overlaid image) has been applied on top of the 3D message, the recipient may You will receive a basic unknown 3D message asset, and stickers will be applied as media overlays by the recipient.

FIG. 7 is a block diagram 700 illustrating various modules of annotation system 206, according to some example embodiments. The annotation system 206 includes an image and depth data receiving module 702, a sensor data receiving module 704, an image and depth data processing module 706, a 3D effects module 708, a rendering module 710, and a sharing module ( 712), and an augmented reality content generator module 714. The various modules of annotation system 206 are configured to communicate with each other (e.g., via a bus, shared memory, or switch). Any one or more of these modules may be implemented using one or more computer processors 750 (e.g., by configuring such one or more computer processors to perform the functions described for those modules), so that the computer processor ( 750) (e.g., a processor set provided by client device 102).

Any one or more of the described modules may be hardware alone (e.g., one or more of the computer processors 750 of a machine (e.g., machine 2300)) or a combination of hardware and software. For example, any described module of annotation system 206 may be implemented using one or more of computer processors 750 (e.g., For example, it may physically comprise an array of, or a subset of, one or more computer processors of a machine (e.g., machine 2300). As another example, any module of annotation system 206 may include: software that configures an arrangement of one or more computer processors 750 (e.g., among one or more computer processors of a machine (e.g., machine 2300)) to perform the operations described herein for that module; hardware, or both. Accordingly, different modules of the annotation system 206 may include different arrangements of such computer processors 750 or a single arrangement of such computer processors 750 at different times. Moreover, any two or more modules of the annotation system 206 may be combined into a single module, and the functions described herein for a single module may be subdivided among multiple modules. Additionally, various According to example embodiments, modules described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices.

Image and depth data reception module 702 receives images and depth data captured by client device 102. For example, the image is a picture captured by an optical sensor (e.g., camera) of client device 102. The image includes one or more real-world features, such as a user's face or real-world object(s) detected in the image. In some embodiments, the image includes metadata that describes the image. For example, depth data may be derived from rays emitted from a light emitting module directed at an object (e.g., a user's face) with features having different depths (e.g., eyes, ears, nose, lips, etc.). Contains data corresponding to a depth map containing underlying depth information. For example, a depth map is similar to an image, but instead of each pixel providing a color, the depth map measures the distance of that part of the image from the camera (e.g., absolutely, or relative to other pixels in the depth map). About) indicates.

The sensor data receiving module 704 receives sensor data from the client device 102. Sensor data is any type of data captured by sensors on client device 102. In examples, sensor data may include motion of client device 102 collected by a gyroscope, a touch sensor (e.g., a touchscreen), GPS, or describing the current geographic location and/or movement of client device 102. It may include touch inputs or gesture inputs from other sensors of client device 102. As another example, sensor data may include temperature data representative of the current temperature detected by a sensor of client device 102. As another example, sensor data may include light sensor data that indicates whether client device 102 is in a dark or bright environment.

Image and depth data processing module 706 performs operations on received image and/or depth data. For example, various image processing and/or depth processing operations are performed by image and depth data processing module 706, which are discussed further herein.

The 3D effects module 708 performs various operations based on 3D algorithms or techniques corresponding to animations and/or provides visual and/or audio effects to the received image and/or depth data, which This is further explained in the specification.

Rendering module 710 performs rendering of a 3D message for display by messaging client application 104 based on data provided by at least one of the modules described above.

Sharing module 712 generates 3D messages for storage in and/or transmission to messaging server system 108. Sharing module 712 enables sharing of 3D messages with other users of messaging server system 108.

Augmented reality content generator module 714, in embodiments, causes display of selectable graphical items presented in a carousel arrangement. By way of example, a user may use various inputs to rotate selectable graphical items on and off the display screen in a manner that corresponds to a carousel that provides a periodic view of graphical items. A carousel arrangement allows multiple graphical items to occupy a specific graphical area on the display screen. In an example, the augmented reality content generators may be organized into respective groups for inclusion on a carousel arrangement, thereby enabling rotating through the augmented reality content generators by group.

Given a 3D message, a 3D model of the subject or scene may be captured using the embodiments described herein. These 3D models can be combined with augmented reality content generator(s), such as (LENSES and AR effects) and 3D effects, and shared within the target system to provide additional interactivity for the viewer. elements can be provided.

In embodiments described herein, depth and image data can be used to add Z-axis dimensions (e.g., depth dimensions) to conventional 2D photographs (e.g., X-axis and Y-axis dimensions). ) 3D face and scene reconstruction can be performed. This format allows the viewer to interact with the 3D message, changes the angle/viewpoint at which the 3D message is rendered by the target system, and affects the particles and shaders used to render the 3D message.

In an example, viewer interaction input comes from movement while viewing a 3D message (e.g., from a motion sensor of the device displaying the 3D message to the viewer), which in turn determines how content, particles, and shaders are rendered. It is converted into a change in perspective. Interaction can also come from on-screen touch gestures and other device motion.

FIG. 8 is a flow diagram illustrating a method 800 of generating a 3D message, according to some example embodiments. Method 800 is such that the operations of method 800 may be performed, in part or in full, by messaging client application 104, particularly with respect to each of the components of annotation system 206 described above in FIG. 7. may be implemented as computer readable instructions for execution by one or more computer processors; Accordingly, method 800 is described below by way of example and with reference thereto. However, it will be appreciated that at least some of the operations of method 800 may be deployed on a variety of other hardware configurations and method 800 is not intended to be limited to messaging client application 104.

At operation 802, the image and depth data receiving module 702 receives image data and depth data captured by an optical sensor (e.g., a camera) of the client device 102. In an example, to create a 3D message, the user selects the 3D message camera mode in the messaging client application 104, which causes the camera to capture raw depth data and a portrait segmentation mask along with the camera image.

In operation 804, 3D effects module 708 selects a 3D effect. In an example, a 3D effect may be selected based on user input corresponding to selection of a 3D augmented reality content generator, for example, as provided in the user interface of messaging client application 104.

In operation 806, 3D effects module 708 applies the selected 3D effect to the image data and/or depth data. In an example, the selected 3D effect includes logic that enables processing image data and/or depth data.

In operation 808, rendering module 710 renders a view of the 3D message using the applied 3D effect. In an example, the rendering module provides a view of the 3D message based on the applied 3D effect displayed by messaging client application 104. As further described herein, a viewer of a 3D message may provide additional inputs (e.g., movement data and/or touch inputs) that cause the 3D message to be updated and re-rendered in response to such inputs. You can.

In operation 810, sharing module 712 generates a 3D message containing 3D effect data. As previously discussed, the 3D message may include the information described in Figures 6 and/or 4, which allows the 3D message to be reconstructed and rendered by a viewer of the 3D message upon receipt of the 3D message.

In operation 812, sharing module 712 stores or transmits the generated 3D message to messaging server system 108. In an example, messaging client application 104 transmits a 3D message to messaging server system 108 so that the 3D message can be stored and/or viewed at a later time by a particular recipient or viewer of the 3D message.

In an embodiment, in a scenario where a given 3D message is received (e.g., by a sender client device sharing the 3D message to a recipient client device), at operation 802 to render a view of the received 3D message ( Similar operations described for 808 may be performed (e.g., prior to operations 810 and 812 to thereby generate a 3D message and/or store or transmit a 3D message).

The discussion below relates to example embodiments for sharing 3D messages and/or storing such 3D messages in persistent storage (e.g., database 120).

In embodiments, a user may initiate a process (e.g., by selecting a command provided in the user interface of messaging client application 104) to store a 3D message in database 120 of messaging server system 108. You can. In this example, image and depth data are stored, as well as information as previously described in Figure 6 (e.g., a 3D augmented reality content generator for loading 3D messages).

In an embodiment, a user (e.g., the sender of a 3D message) may use (e.g., the messaging client application 104) to send 3D messages to a set of recipients (e.g., one or more recipients of the 3D message). You can initiate the process (by selecting a command provided in the user interface). In an embodiment, messaging client application 104 may provide prompts and/or messages informing users about 3D messages and how they differ from photos and videos (eg, 2D messages).

In embodiments, a given 3D message may be stored on messaging server system 108 and then exported. For example, when a user selects an export command for a selected 3D message, each augmented reality content generator corresponding to the 3D effect associated with the 3D message is retrieved, and the 3D effect is looped to create a video that can be looped. It is applied to image data through . In this video, the 3D mesh can be rotated 360 degrees to complete the loop over a certain period of time.

9 illustrates a method 900 of performing transformation passes for processing image and depth data that may be performed in conjunction with the method 800 for generating a 3D message as described above, according to some example embodiments. This is an illustrative flow chart. Method 900 is such that the operations of method 900 may be performed, in part or in full, by messaging client application 104, particularly with respect to each of the components of annotation system 206 described above in FIG. 7. may be implemented as computer readable instructions for execution by one or more computer processors; Accordingly, method 900 is described below by way of example and with reference thereto. However, it will be appreciated that at least some of the operations of method 900 may be deployed on a variety of other hardware configurations and method 900 is not intended to be limited to messaging client application 104.

At operation 902, the image and depth data reception module 702 receives image data and depth data captured by an optical sensor (e.g., a camera) of the client device 102. In an example, the depth data includes a depth map corresponding to the image data. In embodiments, image data (e.g., a color frame) is resized (e.g., as described further herein) if the height of the image data exceeds a certain size (e.g., 2048 pixels). improves processing utilization (for image processing operations) and ensures better compatibility with a wider variety of client devices.

In an example embodiment, machine learning techniques and heuristics may be used to generate depth maps in instances where a given client device does not include appropriate hardware (e.g., a depth-sensing camera) to enable capturing depth information. It is used to create These machine learning techniques can train a machine learning model from training data from shared 3D messages (or other image data) in the example. These heuristics include generating a depth map of a person using face tracking and portrait segmentation.

In an example embodiment, the machine learning techniques described above may use a neural network model to generate a depth map. For depth estimation, the input to the neural network is an RGB image and the output is a depth image. In the example, the neural network generates a depth map with lower resolution than the RGB image. 3D effects rendered using such a depth map may, in examples, be limited by the lower resolution of the depth map. In particular, fine details (e.g., hair) may be difficult to preserve at the lower resolution of this depth map. Therefore, as discussed further herein, the subject technology takes advantage of these potential drawbacks associated with depth maps to create more 3D effects that appear more natural and less artificial when rendered and presented to a viewing user of a 3D message. It provides various technologies to solve the problem.

In an example embodiment, multi-view stereo computer vision techniques are used to generate depth maps from multiple images or video as a user moves a camera about the scene.

In another embodiment, a neural network model may be utilized by a client device to generate segmentation mask(s), which then perform inpainting of the background and corresponding depth map of a given image. It is used for, and is further discussed herein.

At operation 904, image and depth data processing module 706 generates a depth map using at least the depth data. As discussed further below, a second depth map, referred to as a packed depth map, can be generated based at least in part on the depth map for additional technical advantages.

At operation 906, image and depth data processing module 706 generates a segmentation mask based at least in part on the image data. In an embodiment, the image and depth data processing module 706 uses a convolutional neural network to determine a segmentation mask to apply to all pixels to assign the pixel to a particular object class (e.g., face/portrait or background). Dense prediction operations are performed, where prediction is made for the segmentation mask, and a segmentation mask is determined based on groupings of the classified pixels (e.g., face/portrait or background). Alternatively, the segmentation mask may be generated by hardware of client device 102 (e.g., a neural network processor or other machine learning oriented processor) and then received as included in the raw input data.

At operation 908, the image and depth data processing module 706 performs background inpainting and blurring of the received image data using at least a segmentation mask to generate background inpainted image data. In an example, image and depth data processing module 706 may use a background inpainting technique to remove a portrait (e.g., containing a user's face) from the background and blur the background to focus on the person in the frame. Perform. In an embodiment, some of the processing described above (eg, transform passes) is used for depth and color textures, while other image processing is used for color texture (eg, blurring the background). In an embodiment, processing (e.g., transformation passes) is chained for rendering to a target, and the processed depth map and color texture are rendered in such a way that they are consumed by the effect(s).

At operation 910, image and depth data processing module 706 generates a depth inpainting mask based at least in part on the depth map. In an example, the depth map may correspond to the previously mentioned packed depth map. In an example, the image and depth data processing module 706 uses a depth inpainting mask to clean up artifacts in the depth map. Alternatively, the image and depth data processing module 706 may instead use the segmentation mask mentioned above to inpaint the depth map (e.g., prior to generating the depth inpainting mask).

At operation 912, image and depth data processing module 706 performs inpainting of the depth map using a depth inpainting mask to generate an inpainted depth map. As previously mentioned, the inpainted depth map corresponds to a post-processed depth map in which artifacts, if any, have been removed from the original depth map. In an example, the post-processed depth map includes segmentation applied to the alpha channel of the depth map (e.g., a channel other than the channels that define color values for pixels in the image).

At operation 914, image and depth data processing module 706 generates a depth normal map based at least in part on the depth map. In an embodiment, the depth map in this operation may correspond to the previously mentioned packed depth map. In an example, the image and depth data processing module 706 provides a post-processed foreground image by applying 3D effect(s) to the foreground region of the image data using the depth normal map.

At operation 916, the rendering module 710 creates a view of the 3D message using at least the background inpainted image, the inpainted depth map, and the post-processed foreground image, which are assets included in the generated 3D message. Create. In an example, rendering module 710 renders a view of a 3D message for display by messaging client application 104.

In an embodiment, a client device (e.g., client device 102) receives a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item being sent to an augmented reality content generator that includes a 3D effect. Respond. The client device captures image data using at least one camera of the client device. The client device generates depth data using a machine learning model based at least in part on the captured image data. The client device applies 3D effects to the image data and depth data based at least in part on the augmented reality content generator.

In an example, image data is captured with two or more cameras. Alternatively,

Image data is captured using dual pixel autofocus from a single camera (depth information can be derived using multiple images captured by dual pixel autofocus). As previously discussed, the machine learning model can be a deep neural network or a convolutional neural network that provides predictions of depth data based on the captured image data, and the machine learning model receives the captured image data as input and Produces a map as output. In some implementations, the machine learning model runs on a neural network processor or graphics processing unit of the client device.

The discussion below includes, in embodiments, components of the annotation system 206, such as image and depth data processing module 706, 3D effects module 708, and/or rendering module 710, and perform rendering and/or It relates to various “cameras” that perform various operations for processing image and/or depth data together with the generation of 3D messages.

In an embodiment, the scene camera contains most of the effect(s). This is where all 3D or graphical attachments, gyro-based interactions and particles are added and/or configured. In an example, attachments are configured on the transmitting and receiving side by storing positions, rotations, etc. through persistent storage.

In an embodiment, the facial effects camera includes facial effects that can be applied to both color and depth textures or to a color texture. For effects that affect both color and depth map textures, facial effects are rendered at the depth inpainting layer. In an example, this is used for a facial stretching effect.

In an embodiment, for facial effects that only affect the color texture, those facial effects are placed on a separate layer within the same camera to prevent them from being applied on the depth map. Examples of such effects are the face retouch effect and the face mask effect.

In an embodiment, the composite camera renders the output of the scene camera and applies any effects (e.g., color filters or screen space particles) that apply to the entire scene at the end of the pipeline.

Beautification techniques include the retouching of image data, including region(s) of the image data corresponding to a representation of the face (e.g., facial image data), which can achieve results similar to plastic surgery or makeup in the physical world. refers to image processing manipulations (e.g., “beautification manipulations”) related to For example, these beautification techniques include slimming the cheeks, enlarging the eyes, smoothing the skin, brightening teeth or skin, removing blemishes or wrinkles, and changing eye color. It is possible to modify facial image data in the digital domain, such as reducing sagging skin, enhancing skin color, adding facial tattoos or markings, etc. Therefore, a given beautification technique can improve the aesthetic appeal of facial images. It is useful to provide beautification of facial image data in an automated manner to avoid tedious (e.g. manually selected or performed by the user) interactions from the user, thereby presenting a rendering of the facial beautification. This leads to a more convenient and efficient process.

Additionally, when facial image data must be modified, regions of the image data must be selected for modification in a precise manner to avoid visual artifacts that may result in lower quality or aesthetically unpleasing applications of beautification techniques. Therefore, it is advantageous to use portrait segmentation masks, discussed further herein, to more accurately apply a given beautification technique to facial image data.

In the target system, the selected AR content generator may include at least one beautification technique applied to the image data and depth data to cause a beautification effect that may be provided to a display (e.g., rendering) of the generated 3D message. there is.

10 is a flow diagram illustrating a method 1000 of performing beautification of image and depth data that may be performed in conjunction with the method 900 for generating a 3D message as described above, according to some example embodiments. . Method 1000 may be performed in part or in whole by messaging client application 104, particularly with respect to each of the components of annotation system 206 described above in Figure 7. may be implemented as computer readable instructions for execution by one or more computer processors; Accordingly, method 1000 is described below by way of example and with reference thereto. However, it will be appreciated that at least some of the operations of method 1000 may be deployed on a variety of other hardware configurations, and method 1000 is not intended to be limited to messaging client application 104.

As described above, the subject system may be configured to perform facial effects (e.g., beautification, enables the application of facial retouching, stretching and other effects). As discussed herein, “beautification” refers to analyzing images according to user-provided criteria and modifying the images to meet the criteria. Such criteria may include X, Y and Z values associated with the color and transparency of pixels within the image. “Beautification operations” refers to a set of image processing operations to perform beautification of facial image data.

At operation 1002, 3D effects module 708 selects, at a client device (e.g., client device 102), a plurality of selections (e.g., in an interface as further discussed in Figure 12 below). Receives a selection of selectable graphic items from available graphic items. In an example, the selectable graphical item is or corresponds to an augmented reality content generator for applying a 3D effect, and the 3D effect includes at least one beautification operation to be performed in conjunction with the 3D effect.

Examples of such beautification operations include facial retouching, which includes a number of features to retouch the user's face, such as smoothing skin, whitening teeth, sharpening eyes, and brightening eyes. Another example of a beautification manipulation includes a facial stretching effect that enables stretching points of the user's face. Another example of beautification manipulation includes changing the color of the user's eyes and/or creating eye reflections. Other examples of beautification manipulation include facial liquification effects that warp the face into a sphere. Other examples of beautification manipulations include face insertion effects that map facial features (e.g., eyes) to other areas of the face. It is understood that other types of beautification techniques are contemplated and within the scope of the target system.

Beautification manipulations may modify image data to increase skin smoothness, adjust lighting, and modify the color of facial image data. Exemplary approaches for achieving the aforementioned image effects include portrait segmentation, portrait fusion, color correction, Gaussian mixture model (GMM), Gaussian filter, Bayesian segmentation, skin color detection, bidirectional filter, HSV color descriptor, wavelet transform, It includes using gradient domain image processing, Poisson image cloning, Lee filter, edge-preserving smoothing filter, blurring, noise reduction, blemish removal, feature detection and extraction, etc. Different approaches may be used depending on the target technology to perform a given beautification operation. For example, machine learning models can be applied to beautification operations such as convolutional neural networks, generative adversarial networks, etc. These machine learning models can be used to preserve facial feature structures, smooth blemishes, remove wrinkles, or preserve facial skin texture in facial image data.

At operation 1004, image and depth data reception module 702 captures image data and depth data from an optical sensor of a client device (e.g., client device 102). In an embodiment, in response to selection of a selectable graphics time corresponding to a particular 3D effect, client device 102 initiates operations at image and depth data reception module 702 to receive captured image data and depth data. can do. As discussed herein, raw input data from such cameras may include captured image data and depth data and, in some embodiments, may utilize the hardware capabilities of the client device (e.g., a GPU or neural network processor). etc.) may also include a portrait segmentation mask created using.

At operation 1006, 3D effects module 708 applies a 3D effect, including at least one beautification manipulation, to image data and depth data. In an embodiment, as part of applying a 3D effect, 3D effects module 708 performs a beautification operation on at least a region of image data containing facial image data, wherein the beautification operation includes smoothing pixels within the region. , lighting adjustments, or color modifications. Beautification manipulation also includes the use of machine learning models to preserve facial feature structures, smooth blemishes, remove wrinkles, or preserve facial skin texture in facial image data included in the captured image data.

At operation 1008, sharing module 712 generates a 3D message based at least in part on an applied 3D effect that includes at least one beautification manipulation. In embodiments, when a 3D message is stored in data 120 of messaging server system 108, upon subsequent viewing by a recipient, the 3D message is rendered with an applied beautification manipulation, among other assets discussed herein. , Information (e.g., metadata) corresponding to the applied beautification technique and post-processed image data is included in the 3D message.

At operation 1010, rendering module 710 renders a view of the 3D message that renders a view of the 3D message based at least in part on an applied 3D effect that includes at least one beautification operation. In an alternative embodiment, operations 1008 and 1010 are performed where a view of the 3D message is initially displayed (e.g., to provide a preview of the 3D message) and then the 3D message is displayed as further described herein. It is recognized that this may be performed in a different order to produce the included assets and metadata.

11 illustrates a method 1100 for updating a view of a 3D message in response to motion data that may be performed in conjunction with the method 800 for generating a 3D message as described above, according to some example embodiments. This is a flow chart. Method 1100 may be performed in part or in full by messaging client application 104, particularly for each of the components of annotation system 206 described above in FIG. 7. may be implemented as computer readable instructions for execution by one or more computer processors; Accordingly, method 1100 is described below by way of example and with reference thereto. However, it will be appreciated that at least some of the operations of method 1100 may be deployed on a variety of other hardware configurations, and method 1100 is not intended to be limited to messaging client application 104.

In operation 1102, sensor data reception module 704 receives motion data from a motion sensor (e.g., gyroscope, motion sensor, touch screen, etc.). In an embodiment, messaging client application 104 receives sensor data captured by sensors on client device 102, such as location or motion sensors.

At operation 1104, 3D effects module 708 updates the view of the 3D message based on the motion data. In an embodiment, in response to motion data corresponding to a change in the roll/yaw/pitch orientation of the client device, the 3D message has a corresponding change in how it is rendered by the 3D effects module 708 (e.g. An input of -10 degrees roll will shift the perspective of the content to +10 degrees roll). In an embodiment, in response to not receiving motion data (e.g., roll/yaw/pitch) for a certain period of time (e.g., 3 seconds), 3D effects module 708 configures the 3D effects module 708 to indicate depth and parallax. Updates the view of the 3D message by showing an animation with subtle shifts in pitch, roll, and yaw. Additionally, in response to movement, the above-described animation will stop and the response to input is processed by the 3D effects module 708.

In an embodiment, as described above, additional 3D effects and/or augmented reality content generators (e.g., media overlays) may be applied to the image and/or depth data, which depth data may be combined with an image texture. , changing the properties of both the geometry of the depth map as well as the depth dimension at the front of the reconstructed model.

At operation 1106, rendering module 710 renders an updated view of the 3D message. The updated view in 3D is provided for display on a display of a client device (e.g., client device 102).

12 illustrates selecting an augmented reality content generator and applying it to media content (e.g., an image or video) and sending the applied augmented reality content generator to a messaging client application 104 (or messaging system 100), according to some embodiments. )) illustrate example user interfaces showing a carousel for presentation.

In embodiments of such a user interface, selectable graphical items 1250 may be displayed on a display screen of a given computing device (e.g., client device 102) for which a portion or subset of selectable graphical items 1250 are displayed. It can be presented as a Russell array. By way of example, a user may use various inputs to rotate selectable graphical items on and off the display screen in a manner that corresponds to a carousel that provides a periodic view of graphical items. Accordingly, the carousel arrangement provided in the user interface allows multiple graphical items to occupy a particular graphical area on the display screen.

In an example, respective AR experiences corresponding to different AR content generators may be organized into respective groups for inclusion on a carousel arrangement, thereby enabling rotation by group through media overlays. Although the carousel interface is provided as an example, it is understood that other graphical interfaces may be used. For example, a set of augmented reality content generators may include a graphical list, scrolling list, scrolling graphics, or other graphical interface that allows navigation through various graphical items for selection, etc. As used herein, a carousel interface refers to the display of graphical items in an arrangement similar to a circular list, thus allowing navigation through the circular list based on user inputs (e.g., touch or gestures). Allows selection and scrolling of graphical items. In an example, a set of graphical items may be presented on a horizontal (or vertical) line or axis, with each graphical item represented as a specific thumbnail image (or icon, avatar, etc.). At any one time, some of the graphical items within the carousel interface may be hidden. If the user wishes to view hidden graphical items, in an example, the user may input user input (e.g., touch, gesture, etc.) to scroll the graphical items in a specific direction (e.g., left, right, up, or down, etc.). ) can be provided. Subsequent views of the carousel interface are then displayed, where animations are provided or rendered to present one or more additional graphical items for inclusion on the interface, some of the previously presented graphical items being displayed in this subsequent view. You can die in In embodiments, in this manner, a user may navigate back and forth through a set of graphical items in a circular manner. Accordingly, it will be appreciated that a carousel interface can optimize screen space by displaying only a subset of images from a set of graphical items in a circular view.

As described herein, augmented reality content generators may be included on a carousel arrangement (or other interface as previously discussed), thereby enabling rotation through the augmented reality content generators. Additionally, augmented reality content generators may be selected for inclusion based on various signals including, for example, time, date, geolocation, metadata associated with media content, etc. In the carousel arrangement of user interface examples in Figure 12, each augmented reality content generator is selected from the available augmented reality content generators provided by the target system.

In the description below, selectable graphical items correspond to respective augmented reality content generators applied to media content. As shown in user interface 1200, selectable graphical items 1250 corresponding to the carousel arrangement include selectable graphical items 1251 in the display screen of the electronic device (e.g., client device 102). do. For example, a swipe gesture may be received via a touch screen of client device 102 and, in response to receiving the swipe gesture, navigation through selectable graphical items to facilitate selection of a particular augmented reality content generator. This becomes possible. Selectable graphical items 1251 are selected through touch input by the user (e.g., tapping, or releasing the touch at the end of a gesture). In this example, selectable graphical item 1251 corresponds to a specific augmented reality content generator that includes 3D effects.

In the second example of FIG. 12 , upon selection of selectable graphics item 1251, 3D effects 1260, 3D effects 1262 and 3D effects 1264 are displayed for display on client device 102. It is rendered. In this example, 3D effects 1260 include particles that are spatially rendered and moving in response to sensor information (e.g., gyroscopic data, etc.) on the viewer's electronic device (e.g., client device 102). -These are based effects. 3D effects 1262 may include color filtering and shader effects, which may also move in response to sensor information. 3D effects 1264 include, in some examples, a 3D attachment (e.g., a headband of roses) that refers to a wearable 3D object or model of some type, shape(s), color, texture, etc. Accordingly, the corresponding augmented reality content generator includes a 3D object rendered approximating the facial image data from the image data. Examples of color filtering include a daylight effect that matches the time of day for a location that corresponds to the location where the message was created (eg, based on location metadata included with the message). Examples of shader effects include, but are not limited to: liquid moving around the screen, glitter effects, bloom effects, rainbow effects, and changing background based on movement.

In the third example of FIG. 12 , the user provides movement of the client device 102 in the display of 3D effects 1270 and 3D effects 1272 in user interface 1200 . In this example, 3D effects 1270, 3D effects 1272, and 3D effects 1274 are respectively the 3D effects 1260, 3D effects 1262, and 3D effects discussed in the second example ( 1264) are different versions. 3D effects 1270, 3D effects 1272, and 3D effects 1274 were rendered for display in response to movement of the client device 102 (e.g., motion data from a gyroscopic sensor), The view of the 3D effects described above is updated (e.g., re-rendered) in response to newly received motion data, which may change the perspective of the scene being viewed by the viewer. For example, particles, color filtering, and/or 3D attachments change in response to motion data.

In an embodiment, a given client device (e.g., client device 102) selects a set of augmented reality content generators from available augmented reality content generators based on metadata associated with each augmented reality content generator and metadata The data includes information indicating that the corresponding augmented reality content generator includes at least a 3D effect, and the set of augmented reality content generators includes at least one augmented reality content generator without a 3D effect and at least one augmented reality content generator with a 3D effect. Includes a real-life content generator. The client device receives a selection of a selectable graphical item from a list of selectable graphical items, the selectable graphical item including an augmented reality content generator that includes a 3D effect. The client device captures image data and depth data using at least one camera of the client device. The at least one camera includes a first camera and a second camera, the first camera has a first focal length, the second camera has a second focal length, and the first focal length and the second focal length are different. Additionally, the client device applies 3D effects to the image data and depth data based at least in part on the augmented reality content generator.

In an embodiment, the disparity map is determined at least in part by the distance between the first pixel from the first image captured by the first camera and the second pixel from the second image captured by the second camera. is generated based on the first pixel and the second pixel corresponding to the same object. A disparity map is an image where each pixel contains a distance value between a pixel from a first image and a corresponding pixel from a second image. First pixels of a first object in the disparity map have a brightness greater than second pixels of a second object in the disparity map, and the first pixel has depth values that are less than second depth values of the second pixel. Also, in the example, the depth map is generated based at least in part on the disparity map.

In an implementation, the above-described 3D effects and corresponding metadata are included in a message that can be provided (e.g., shared) with other users of messaging system 100. This other user can receive the message and, when accessed, view the message for display on the receiving client device. The receiving client device renders the 3D effect for display as specified in the received message, using similar or identical components as described in Figure 7 above. Additionally, this other user may, in response, provide a movement to the receiving client device that initiates a re-rendering of the 3D effect in which the perspective of the scene being viewed by the viewer changes based on the provided movement.

13 is an example illustrating capturing image information and generating a 3D message on a display of a client device, according to some example embodiments.

In a first example, view 1300 is provided for display on a display of a client device (eg, client device 102). View 1300 includes an image of a portrait representation of the user (e.g., including a face). Selectable graphical elements 1305 are provided for display in view 1300. In an embodiment, selectable graphical element 1305 corresponds to an augmented reality content generator for creating 3D messages and applying 3D effects and other image processing operations as discussed further herein. Upon selection of a selectable graphical element 1305, the second interface initiates operations (as described elsewhere herein) to generate a 3D message for display (e.g., depth capturing capabilities). may be presented including an interface for capturing images (using a front-facing camera lens on a client device 102 having an image).

In a second example, view 1350 displays a 3D message captured in the first example within view 1300 with a depth effect that introduces blurring in background areas of the image (e.g., behind a portrait of the user). Includes. This display in view 1350 may be updated to render 3D effects associated with 3D messages in response to receiving sensor data (e.g., motion data, gyroscopic sensor data, etc.) where the user is moving the client device. . In an example, depending on the relative position of the client device to the viewing user, the 3D effect may be updated for presentation on the client device's display to take into account the position change. For example, when the client device's display is tilted in a particular way to a first position, one set of 3D effects may be rendered and presented for display, and when the client device is moved to a different position, a second set of 3D effects may be rendered and presented for display. Effects can be rendered to update the image and indicate changes in the viewing perspective, providing a more 3D viewing experience for the viewing user.

The discussion below describes a given 3D message (e.g., as shown in view 1350) for rendering (e.g., as a preview on a client device, or on a receiving device that is different from the client device), according to some embodiments. It is about the various technologies used to create.

Figure 14 is an example illustrating a raw depth map and a packed depth map, according to some example embodiments. The examples below are performed by a given client device as part of generating a given 3D message using at least raw input data (e.g., image data and depth data) provided by the client device's camera.

In a first example, an example raw depth map 1400 is generated by a given client device (e.g., client device 102) based on raw data captured by a camera of the client device. This raw data may include image data (eg, photographic images) and depth data from a camera. In an example, a client device converts a single channel floating point texture to a raw depth map enabling multiple passes of processing without loss of precision. A client device can spread (e.g., transmit or transform portions of data) a single channel floating point texture to multiple lower precision channels, illustrated in the second example as packed depth map 1450. In an embodiment, raw depth map 1400 and packed depth map 1450 have a lower resolution (e.g., a lower number of total pixels) than the raw image data captured by the client device's camera.

The client device performs operations to separate a foreground with a given object (e.g., a portrait of a user) within a given image from a background within the same image. In an embodiment, the client device generates a segmentation mask using at least the raw depth map 1400 or the packed depth map 1450 described above. Alternatively, in an example, a segmentation mask may be included in raw data captured by a camera when the capabilities of the client device include generating a segmentation mask as part of the image capturing process.

Using the segmentation mask, the client device performs a diffusion-based inpainting technique to remove foreground objects from the background in the image, thereby creating a background inpainted image (e.g., without foreground objects). In an example, diffusion-based inpainting techniques attempt to fill missing areas (e.g., foreground objects) by propagating image content from the boundary into the interior of the missing area. Removing the foreground object in this way is advantageous, at least because, after rendering the 3D message, when the camera of the client device is moved, the foreground object is not removed (e.g., when the above operations are not performed). otherwise, a "ghost" of (part of) the image of the subject may be visible in the background (e.g., resulting in an undesirable visual effect).

Additionally, after rendering the 3D message, the client device is moved and in areas of the image that have (large) depth changes (e.g., between foreground and background corresponding to part of the side of the user's face), a segmentation mask and If the inpainting technique is not performed as described herein, stretching artifacts may appear in the portion of the image that contains the user's face, and the boundaries of the user's face or head may appear smeared between the foreground and background of the image. ) may appear as such. Additionally, without performing the techniques described herein, a hard (e.g., visually distinct) boundary between the foreground and background of an image may appear as an undesirable visual effect when the client device is moved, causing the image to become more distorted. It makes it look artificial, unrealistic, distorted, and exaggerated.

Figure 15 is an example illustrating a depth inpainting mask and depth inpainting, according to some example embodiments. The examples of FIG. 15 may be performed in conjunction with the examples of FIG. 14 (e.g., after the operations of FIG. 14 are performed).

In an embodiment, a client device performs the same diffusion-based inpainting technique (e.g., as described above for background inpainting), using a depth inpainting mask to create a depth map (e.g., a packed Extend the foreground boundary of the depth map (1450). As shown in the first example, the depth inpainting mask 1500 is created using at least a depth map. In an example, the depth inpainting mask 1500 may be a deep convolutional neural network that performs edge detection and classification of each pixel in the depth map, an encoder-decoder architecture for segmentation, a fully convolutional network, a feature map, and a deconvolution network. It can be determined using approaches applied to depth maps, including machine learning techniques such as solution networks and unsupervised feature learning.

In a second example, image data 1550 (e.g., an inpainted depth map) shows the result of inpainting a depth map using a depth inpainting mask 1500. This is done, in examples, to improve the appearance (e.g., render more accurately) of hair, ears, or shoulders. As previously mentioned, the depth map may be of lower resolution than the image data (e.g., RBG image), and 3D effects or image processing operations applied to the depth map may be limited by the low resolution. To better preserve fine details such as hair, the depth map inpainting technique described above is provided by Target Technology. By using the depth map and depth inpainting mask 1500, the client device determines specific areas of the image (e.g., areas with missing or bad data) as shown in image data 1550 and It can be filled.

In an example, the client device determines a depth map based at least in part on the depth data and generates a depth inpainting mask corresponding to an area of the depth map that includes facial depth data; At least perform depth map inpainting of the depth map using the generated depth inpainting mask. The depth map may be a packed depth map in an embodiment.

In an embodiment, a client device generates a depth normal map by determining a plane that fits around the neighborhood of each depth pixel in the depth map. This is advantageous for determining how light should interact with the surface to achieve interesting lighting effects and beautification effects, for example. In the example, the generated depth normal map is a low resolution image, but can be used effectively to provide these effects in a 3D message. In the example, the normal map uses RGB information (e.g., corresponding to the ") can determine the direction in which the polygon is oriented, where the client device uses the determined orientation of the surface normal to determine how to shade the polygon. In other words, a normal map is an image that stores the direction at each pixel, and the directions are called normals. The red, green, and blue channels of the image can be used by the client device to control the direction of each pixel's normal, and the normal map can be used to mimic high-resolution details for a low-resolution image.

In an example, the client device generates a normal map of the depth map for applying a lighting effect to facial image data in the image data and applies a lighting effect to the facial image data based at least in part on the normal map. In an example, the lighting effect includes at least two different colors, a first color from the two different colors is applied to a first portion of the facial image data, and a second color from the two different colors is applied to the facial image data. Applies to the second image.

Using the generated normal map, the client device applies beautification techniques (discussed further herein), lighting effects, and other image processing techniques to create 3D effects that appear convincing and natural to the viewing user of the 3D message. can do. The client device generates a post-processed foreground image based at least on the techniques described above involving the generated normal map.

The client device generates a 3D message that includes various assets such as a post-processed foreground image, a post-processed depth map, an inpainted background image, and other metadata included (e.g., as described above). . In embodiments, a device receiving a 3D message may use the included assets to render a view of the 3D message by creating a foreground mesh and a background mesh. The foreground mesh may be generated using metadata related to the post-processed depth map and camera-specific metadata (e.g., lens information discussed earlier, etc.). The background mesh can be created using at least an inpainted background image.

The following discussion of Figures 16-21 are examples of 3D effects and other graphical effects presented for display on a given client device (e.g., client device 102) using at least some of the techniques described above. .

16 shows a 3D attachment rendered in response to particles, reflections on a graphical object (e.g., glasses), and motion data (e.g., motion data from a gyroscopic sensor), according to some example embodiments. and examples of 3D effects illustrating dynamic 3D attachments and post effects rendered in response to motion data.

In the first example of Figure 16, the view 1600 of 3D effects is updated (e.g., re-rendered) in response to newly received motion data that may change the perspective of the scene being viewed by the viewer. For example, particles, reflections and/or 3D attachments change in view 1600 in response to motion data.

As further shown, a second example of 3D effects illustrates dynamic 3D attachments and view 1650 post effects that are rendered in response to motion data (e.g., motion data from a gyroscopic sensor). In this second example of Figure 13, the view of 3D effects is updated (e.g., re-rendered) in response to newly received motion data that may change the perspective of the scene being viewed by the viewer. For example, 3D text (eg, “Santa Monica”) changes positions in response to motion data.

17 is an example of a 3D effect illustrating dynamic artificial lighting rendered in response to motion data, and reflections on glasses, 3D attachments, and an animated sprite background rendered in response to motion data, according to some example embodiments. /This is an example of 3D effects that illustrate refraction.

In the first example of Figure 17, the view 1700 of 3D effects is updated (e.g., re-rendered) in response to newly received motion data that may change the perspective of the scene being viewed by the viewer. For example, artificial lighting on the face changes in response to motion data.

A second example of a 3D effect in FIG. 17 is a view 1750 showing reflections/refractions on glasses, 3D attachments, and animated sprite backgrounds rendered in response to motion data (e.g., motion data from a gyroscopic sensor). ) is an example. In the second example of Figure 17, the view of 3D effects is updated (e.g., re-rendered) in response to newly received motion data that may change the perspective of the scene being viewed by the viewer. For example, reflections/refractions on glasses and animated sprite backgrounds change in response to motion data.

18 is an example of 3D effects illustrating a controlled particle system (e.g., an animated projectile), and 2D and 3D attachments rendered in response to motion data, according to some example embodiments. This is an example of 3D effects that illustrate joint animation for 3D attachments (e.g., bunny ears) rendered in response to (e.g., motion data from a gyroscopic sensor).

In the first example of Figure 18, the view 1800 of 3D effects is updated (e.g., re-rendered) in response to newly received motion data that may change the perspective of the scene being viewed by the viewer. For example, the animation of a controlled particle system changes and attachments move in response to motion data.

In the second example of Figure 18, the view 1850 of 3D effects is updated (e.g., re-rendered) in response to newly received motion data that may change the perspective of the scene being viewed by the viewer. For example, the animation of a 3D attachment changes in response to movement data.

19 shows 3D effects illustrating sprites, reflections on glasses, and 2D and 3D attachments rendered in response to motion data (e.g., motion data from a gyroscopic sensor), according to some example embodiments. Examples of 3D effects illustrating glasses, particles, and reflections/refractions on an animated background that are rendered in response to motion data.

In the first example of Figure 19, the view 1900 of 3D effects is updated (e.g., re-rendered) in response to newly received motion data that may change the perspective of the scene being viewed by the viewer. For example, reflections on glasses, sprites, and attachments change in response to movement data.

In the second example of Figure 19, the view 1950 of 3D effects includes reflections/refractions on the glasses, particles, and background that change in response to motion data.

20 is an example of 3D effects illustrating an animated foreground and an attachment obscuring a user's face rendered in response to motion data (e.g., motion data from a gyroscopic sensor), according to some example embodiments. and examples of 3D effects illustrating dynamic artificial lighting, particles, and reflections/refractions on glasses rendered in response to motion data.

In the first example of Figure 20, view 2000 shows an occlusion effect (eg, an ice or freeze effect) on a foreground change in response to motion data.

In the second example of Figure 20, view 2050 illustrates dynamic artificial lighting, particles, and reflection/refraction on glasses rendered in response to motion data (e.g., motion data from a gyroscopic sensor). Shows 3D effects.

21 is an example of a 3D effect illustrating retouches, post effects, 3D attachments and particles rendered in response to motion data, and 3D attachments, sprites and particles rendered in response to motion data, according to some example embodiments. This is an example of 3D effects illustrating particles.

In the first example of Figure 21, view 2100 shows sprites (e.g., petals from flowers) being animated and particles changing in response to motion data.

In the second example of Figure 21, view 2150 shows a 3D attachment (e.g., a mask) that is animated and changes position, sprites are animated, and particles change in response to motion data.

Figure 22 is a block diagram illustrating an example software architecture 2206 that may be used with the various hardware architectures described herein. 22 is a non-limiting example of a software architecture, and it will be appreciated that many other architectures may be implemented to facilitate the functionality described herein. Software architecture 2206 may be executed on hardware, such as machine 2300 of FIG. 23, which includes, inter alia, processors 2304, memory 2314, and (input/output) I/O components 2318. there is. A representative hardware layer 2252 is illustrated and may represent, for example, machine 2300 of FIG. 23. Representative hardware layer 2252 includes processing unit 2254 with associated executable instructions 2204. Executable instructions 2204 represent executable instructions of software architecture 2206, including implementation of methods, components, etc. described herein. Hardware layer 2252 also includes memory/storage 2256, which are memory and/or storage modules, which also have executable instructions 2204. Hardware layer 2252 may also include other hardware 2258.

In the example architecture of Figure 22, software architecture 2206 can be conceptualized as a stack of layers, with each layer providing specific functionality. For example, software architecture 2206 may include layers such as operating system 2202, libraries 2220, frameworks/middleware 2218, applications 2216, and presentation layer 2214. . In operation, applications 2216 and/or other components within the layers invoke API calls 2208 through the software stack and receive responses as one or more messages 2212 in response to the API calls 2208. can do. The layers illustrated are representative in nature and not all software architectures have all layers. For example, some mobile or special purpose operating systems may not provide frameworks/middleware 2218, while others may provide such a layer. Other software architectures may include additional or different layers.

Operating system 2202 may manage hardware resources and provide common services. Operating system 2202 may include, for example, a kernel 2222, services 2224, and drivers 2226. Kernel 2222 may serve as an abstraction layer between hardware and other software layers. For example, the kernel 2222 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. Services 2224 may provide other common services for different software layers. Drivers 2226 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 2226 may include a display driver, camera driver, Bluetooth® driver, flash memory driver, serial communication driver (e.g., Universal Serial Bus (USB) driver), Wi-Fi® driver, depending on the hardware configuration. , audio drivers, power management drivers, etc.

Libraries 2220 provide common infrastructure used by applications 2216 and/or other components and/or layers. Libraries 2220 may be used in an easier manner than for other software components to interface directly with base operating system 2202 functionality (e.g., kernel 2222, services 2224, and/or drivers 2226). It provides functionality that allows you to perform tasks. Libraries 2220 may include system libraries 2244 (e.g., C standard library) that may provide functions such as memory allocation functions, string manipulation functions, math functions, etc. In addition, libraries 2220 include media libraries (e.g., libraries to support presentation and manipulation of various media formats such as MPREG4, H.264, MP3, AAC, AMR, JPG, PNG), graphics libraries (e.g., the OpenGL framework, which can be used to render 2D and 3D in graphical content on a display), database libraries (e.g., SQLite, which can provide various relational database functions), web libraries (e.g., For example, it may include API libraries 2246 such as WebKit), which can provide web browsing functionality. Libraries 2220 may also include a wide variety of other libraries 2248 that provide many different APIs to applications 2216 and other software components/modules.

Frameworks/middleware 2218 (sometimes referred to as middleware) provides a higher-level, common infrastructure that can be used by applications 2216 and/or other software components/modules. For example, frameworks/middleware 2218 may provide various graphical user interface (GUI) functions, high-level resource management, high-level location services, etc. Frameworks/middleware 2218 may provide a broad spectrum of other APIs that can be used by applications 2216 and/or other software components/modules, some of which may be specific to operating system 2202 Or it can be platform specific.

Applications 2216 include built-in applications 2238 and/or third-party applications 2240. Examples of representative built-in applications 2238 may include, but are not limited to, contact applications, browser applications, book reader applications, location applications, media applications, messaging applications, and/or gaming applications. Third-party applications 2240 may include applications developed using the ANDROID ^™ or IOS ^™ software development kit (SDK) by an entity other than the vendor of a particular platform, such as IOS ^™ , ANDROID ^™ , WINDOWS® Phone. It may be mobile software that runs on a mobile operating system, such as , or other mobile operating systems. Third-party applications 2240 may invoke API calls 2208 provided by a mobile operating system (e.g., operating system 2202) to facilitate the functionality described herein.

Applications 2216 may utilize built-in operating system functions (e.g., kernel 2222, services 2224, and/or drivers 2226) to create user interfaces for interacting with users of the system. , libraries 2220, and frameworks/middleware 2218 may be used. Alternatively or additionally, in some systems, interaction with the user may occur through a presentation layer, such as presentation layer 2214. In these systems, the application/component “logic” can be separated from aspects of the application/component that interact with the user.

23 illustrates some example embodiments that may read instructions from a machine-readable medium (e.g., a machine-readable storage medium) and perform any one or more of the methodologies discussed herein. A block diagram illustrating components of machine 2300, according to . Specifically, Figure 23 shows a schematic representation of a machine 2300 in an example form of a computer system, within which instructions are provided to cause the machine 2300 to perform any one or more of the methodologies discussed herein. 2310 (e.g., software, program, application, applet, app, or other executable code) may be executed. Accordingly, instructions 2310 may be used to implement modules or components described herein. Instructions 2310 transform a general, non-programmed machine 2300 into a specific machine 2300 that is programmed to perform the described and illustrated functions in the described manner. In alternative embodiments, machine 2300 may operate as a standalone device or be coupled (e.g., networked) to other machines. In a networked deployment, machine 2300 may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 2300 is a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smartphone, A mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or by machine 2300. It may include, but is not limited to, any machine capable of executing instructions 2310 sequentially or otherwise specifying the actions to be taken. Additionally, although only a single machine 2300 is illustrated, the term “machine” can also refer to a collection of machines that individually or jointly execute instructions 2310 to perform any one or more of the methodologies discussed herein. should be considered to include.

Machine 2300 includes processors 2304 (e.g., processors 2308 through processor 2312), memory/storage 2306, and May include I/O components 2318. Memory/storage 2306 may include memory 2314, such as main memory, or other memory storage, and a storage unit 2316, both of which are accessible to processors 2304, e.g., via bus 2302. can do. Storage unit 2316 and memory 2314 store instructions 2310 implementing any one or more of the methodologies or functions described herein. Instructions 2310 may also, during their execution by machine 2300, fully or partially, within memory 2314, within storage unit 2316, within at least one of processors 2304 (e.g., within the processor's cache memory), or any suitable combination thereof. Accordingly, memory 2314, storage unit 2316, and memory of processors 2304 are examples of machine-readable media.

I/O components 2318 may include a wide variety of components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, etc. The specific I/O components 2318 included in a particular machine 2300 will depend on the type of machine. For example, a portable machine, such as a mobile phone, will likely include a touch input device or other such input mechanism, whereas a headless server machine will probably not include such a touch input device. It will be appreciated that I/O components 2318 may include many other components not shown in FIG. 23 . I/O components 2318 are grouped according to functionality merely to simplify the discussion below and this grouping is in no way limiting. In various example embodiments, I/O components 2318 may include output components 2326 and input components 2328. Output components 2326 may include visual components (e.g., a display such as a plasma display panel (PDP), light emitting diode (LED) display, liquid crystal display (LCD), projector, or cathode ray tube (CRT)), acoustics, etc. Components (e.g., speakers), haptic components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. Input components 2328 may include alphanumeric input components (e.g., a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input component), point-based input components (e.g., a mouse, touchpad, trackball, joystick, motion sensor, or other pointing device), tactile input components (e.g., a physical button, a touch screen that provides the position and/or force of a touch or touch gesture, or other tactile input) component), audio input components (e.g., microphone), etc.

In further example embodiments, I/O components 2318 may include biometric components 2330, motion components 2334, environmental environmental components 2336, or positional components, among a wide range of other components. May include components 2338. For example, biometric components 2330 may detect expressions (e.g., hand expressions, facial expressions, vocal expressions, body gestures, or eye tracking) and biometric signals (e.g., blood pressure, It may include components such as measuring heart rate, body temperature, sweat, or brain waves) and identifying a person (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or electroencephalography-based identification). Motion components 2334 may include acceleration sensor components (e.g., accelerometer), gravity sensor components, rotation sensor components (e.g., gyroscope), etc. Environmental components 2336 include, for example, light sensor components (e.g., a photometer), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, and a pressure sensor. Components (e.g., a barometer), acoustic sensor components (e.g., one or more microphones to detect background noise), proximity sensor components (e.g., infrared sensors to detect nearby objects), gas sensor (e.g., gas detection sensors to detect concentrations of hazardous gases for safety or measure pollutants in the atmosphere), or provide indications, measurements, or signals corresponding to the surrounding physical environment. May contain other components. Position components 2338 include position sensor components (e.g., a GPS receiver component), altitude sensor components (e.g., an altimeter or a barometer that detects barometric pressure from which altitude can be derived), and orientation sensor components. (e.g., magnetometer), etc. may be included.

Communication can be implemented using a wide variety of technologies. I/O components 2318 may include communication components 2340 operable to couple machine 2300 to network 2332 or devices 2320 via coupling 2324 and coupling 2322, respectively. You can. For example, communication component 2340 may include a network interface component, or other suitable device for interfacing with network 2332. In further examples, communication components 2340 include wired communication components, wireless communication components, cellular communication components, near field communication (NFC) components, Bluetooth® components (e.g., Bluetooth® Low Energy), May include Wi-Fi® components, and other communication components that provide communication via other modalities. Devices 2320 may be other machines or any of a wide variety of peripheral devices (eg, peripheral devices coupled via USB).

Moreover, communication components 2340 can detect identifiers or include components operable to detect identifiers. For example, communication components 2340 may include Radio Frequency Identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., 1-1, such as a Universal Product Code (UPC) bar code) For detecting dimensional bar codes, multi-dimensional bar codes such as Quick Response (QR) codes, Aztec Code, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar code, and other optical codes. optical sensors), or acoustic detection components (e.g., microphones to identify tagged audio signals). Additionally, a variety of information, such as location via Internet Protocol (IP) geo-location, location via Wi-Fi® signal triangulation, location via NFC beacon signal detection that can indicate a specific location, etc., can be transmitted to the communicating components (2340). ) can be derived through.

The following discussion concerns various terms or phrases mentioned throughout the subject disclosure.

“Signal media” refers to any intangible medium capable of storing, encoding or carrying instructions for execution by a machine, including digital or analog communication signals or other intangible media to facilitate communication of software or data. Includes. The term “signal medium” should be considered to include any form of modulated data signal, carrier wave, etc. The term “modulated data signal” refers to a signal that has one or more of its characteristics set or changed in such circumstances to encode information within the signal. The terms “transmission medium” and “signal medium” mean the same thing and may be used interchangeably in this disclosure.

“Communication Network” refers to an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), and wireless WAN (WWAN). , metropolitan area network (MAN), Internet, part of the Internet, part of the Public Switched Telephone Network (PSTN), plain old telephone service (POTS) network, cellular telephone network, wireless network, Wi-Fi® network, other types of networks , or a combination of two or more such networks. For example, a network or portion of a network may include a wireless or cellular network and the combination may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile communications (GSM) connection, or another type of cellular or wireless combination. there is. In this example, the combination is 3GPP (which includes Single Carrier Radio Transmission Technology (1xRTT), Evolution-Data Optimized (EVDO) technology, General Packet Radio Service (GPRS) technology, Enhanced Data rates for GSM Evolution (EDGE) technology, and 3G. third Generation Partnership Project), fourth generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Long Term Evolution (LTE) standards, various standards It may implement any of a variety of types of data transmission technologies, such as other long-distance protocols, or other data transmission technologies, as defined by configuration organizations.

“Processor” refers to control signals (e.g., “instructions,” “op codes,” “machine code,” etc.) that manipulate data values and corresponding output signals applied to operate the machine. Refers to any circuit or virtual circuit (a physical circuit emulated by logic running on an actual processor) that generates Processors include, for example, a Central Processing Unit (CPU), Reduced Instruction Set Computing (RISC) processor, Complex Instruction Set Computing (CISC) processor, Graphics Processing Unit (GPU), Digital Signal Processor (DSP), and Application Specification (ASIC). It may be an integrated circuit (RFIC), a radio-frequency integrated circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor, having two or more independent processors (sometimes referred to as “cores”) that can execute instructions simultaneously.

“Machine-storage medium” means single or multiple storage devices and/or media (e.g., a centralized or distributed database, and/or associated caches and servers). Accordingly, the term may be understood to include, but is not limited to, solid-state memories, including memory internal or external to processors, and optical and magnetic media. Specific examples of machine-storage media, computer-storage media, and/or device-storage media include, for example, semiconductor memory devices, such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ), FPGA, and non-volatile memory, including flash memory devices; Magnetic disks such as internal hard disks and removable disks; magneto-optical disk; and CD-ROM and DVD-ROM disks. The terms “machine-storage media”, “device-storage media”, and “computer-storage media” mean the same thing and may be used interchangeably in this disclosure. The terms “machine-storage media,” “computer-storage media,” and “device-storage media” specifically refer to “signals,” at least some of which exclude carrier waves, modulated data signals, and other such media. Included under the term “medium”.

“Component” refers to a device, physical entity, or logic with boundaries defined by function or subroutine calls, branch points, APIs, or other technologies that provide partitioning or modularization of specific processing or control functions. do. Components can be combined with other components through their interfaces to execute machine processes. A component may be a packaged functional hardware unit designed to be used with other components and usually a portion of a program that performs a specific among related functions. Components may constitute either software components (eg, code implemented on a machine-readable medium) or hardware components. A “hardware component” is a tangible unit that can perform certain operations and be configured or arranged in a certain physical way. In various example embodiments, one or more computer systems (e.g., a stand-alone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or group of processors) A hardware component that operates to perform specific operations as described herein may be configured by software (e.g., an application or application portion). Hardware components may also be implemented mechanically, electronically, or in any suitable combination thereof. For example, a hardware component may include dedicated circuitry or logic that is permanently configured to perform specific operations. The hardware component may be a special-purpose processor, such as a field-programmable gate array (FPGA) or an application specific integrated circuit (ASIC). A hardware component may also include programmable logic or circuitry that is temporarily configured by software to perform specific operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware components become specific machines (or specific components of a machine) uniquely tailored to perform the configured functions and are no longer general-purpose processors. Recognizing that the decision to implement a hardware component mechanically, in a dedicated, permanently configured circuit, or in a temporarily configured circuit (e.g., configured by software) may be driven by cost and time considerations. something to do. Accordingly, the phrase "hardware component" (or "hardware-implemented component") refers to a tangible entity, i.e., physically constructed, permanently configured (or configured) to operate in a particular way or perform the particular operations described herein. It should be understood to encompass entities that are either hardwired (e.g., hardwired) or temporarily constructed (e.g., programmed). Given embodiments in which hardware components are temporarily configured (eg, programmed), there is no need for each of the hardware components to be configured or instantiated at any one instance of time. For example, if a hardware component includes a general-purpose processor configured by software to be a special-purpose processor, the general-purpose processor may be configured at different times by each of the different special-purpose processors (e.g., comprising different hardware components). It can be configured. The software thus configures a particular processor or processors, for example, to configure a particular hardware component at one instance of time and a different hardware component at a different instance of time. Hardware components can provide information to and receive information from other hardware components. Accordingly, the described hardware components may be considered to be communicatively coupled. In cases where multiple hardware components exist simultaneously, communication may be accomplished through signal transmission between or between two or more of the hardware components (e.g., via appropriate circuits and buses). In embodiments where multiple hardware components are configured or instantiated at different times, communication between such hardware components may include, for example, storage and retrieval of information within memory structures that are accessible to the multiple hardware components. It can be achieved through For example, a hardware component may perform an operation and store the output of that operation in a memory device that is communicatively coupled thereto. Additional hardware components can then access the memory device to retrieve and process the stored output. Hardware components can also initiate communication with input or output devices and manipulate resources (eg, collections of information). The various operations of the example methods described herein may be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented components operative to perform one or more operations or functions described herein. As used herein, “processor-implemented component” refers to a hardware component implemented using one or more processors. Similarly, the methods described herein may be at least partially processor-implemented, with a particular processor or processors being an example of hardware. For example, at least some of the operations of the method may be performed by one or more processors or processor-implemented components. Moreover, the one or more processors may also operate to support performance of related operations in a “cloud computing” environment or as “software as a service” (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines comprising processors), which may be performed over a network (e.g. the Internet) and through one or more suitable interfaces (e.g. For example, it can be accessed via API). Performance of certain of the operations may reside within a single machine, as well as be distributed among processors deployed across multiple machines. In some example embodiments, processors or components implemented by a processor may be located in a single geographic location (eg, within a home environment, an office environment, or a server farm). In other example embodiments, processors or components implemented by a processor may be distributed across multiple geographic locations.

“Carrier signal” refers to any intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, including digital or analog communication signals or other intangible media to facilitate communication of such instructions. Includes. Instructions may be transmitted or received over a network using a transmission medium via a network interface device.

“Computer-readable media” refers to both machine-storage media and transmission media. Accordingly, the terms include both storage devices/media and carrier waves/modulated data signals. The terms “machine-readable medium,” “computer-readable medium,” and “device-readable medium” mean the same thing and may be used interchangeably in this disclosure.

“Client Device” refers to any machine that interfaces to a communications network to obtain resources from one or more server systems or other client devices. Client devices include mobile phones, desktop computers, laptops, personal digital assistants (PDAs), smartphones, tablets, ultrabooks, netbooks, laptops, multi-processor systems, microprocessor-based or programmable consumer electronics, gaming consoles, and set-tops. It may be, but is not limited to, a box, or any other communication device that a user may use to access the network. In the subject disclosure, a client device is also referred to as an “electronic device.”

“Short-term message” refers to a message that is accessible for a time-limited duration. Short-term messages can be text, images, videos, etc. Access times for short-lived messages may be set by the message sender. Alternatively, the access time may be a default setting or a setting specified by the recipient. Regardless of the configuration technique, messages are temporary.

Claims (80)

As a method,
Using a processor, receiving image data and depth data;
selecting, using the processor, an augmented reality content generator corresponding to a three-dimensional (3D) effect;
Using the processor, applying the 3D effect to the image data and the depth data based at least in part on the selected augmented reality content generator, applying the 3D effect to the image data and the depth data comprising: :
generating a depth map using at least the received depth data;
generating a packed depth map based at least in part on the depth map;
generating a segmentation mask based at least on the received image data, and
performing background inpainting and blurring of the received image data using at least the segmentation mask to generate background inpainted image data; and
Using a processor, generating a message containing information about the applied 3D effect, the image data, and the depth data,
The steps for generating the packed depth map are:
Converting a single-channel floating point texture to a raw depth map; and
generating a plurality of channels based at least in part on the raw depth map,
The performing the background inpainting includes performing a diffusion-based inpainting technique to fill the missing area by propagating image content from the boundary between the missing area and the background area to the interior of the missing area,
The method of claim 1, wherein the background area includes a specific area of the image data without a foreground subject, and the missing area includes the foreground subject.
delete delete delete According to paragraph 1,
The multiple channels have lower precision than the single channel floating point texture, and the method undergoes multiple image processing operations without losing additional precision. According to paragraph 1,
generating a depth inpainting mask based at least in part on the depth map; and
The method further comprising performing inpainting of the depth map using the depth inpainting mask to generate an inpainted depth map. According to clause 6,
The inpainted depth map expands a foreground region of the depth map, and the expanded foreground region corresponds to at least one region of the image data corresponding to a representation of a portion of the face corresponding to hair, ears, or shoulders. Method used to perform image processing operations. According to paragraph 1,
generating a depth normal map based at least in part on the depth map; and
The method further comprising providing a post-processed foreground image by applying the 3D effect to a foreground region of the image data using the depth normal map. According to paragraph 1,
further comprising transmitting the message to a messaging server for storing the message, wherein the message includes a set of assets, the set of assets being a post-processed foreground image, a post-processed depth map, and a background. How to include inpainted images. According to clause 9,
Receiving a command to initiate sharing of the stored message on at least one receiving device; and
The method further comprising transmitting the stored message at at least one receiving device. As a system,
processor; and
Contains memory containing instructions,
The instructions, when executed by the processor, cause the processor to:
Receiving, using a processor, image data and depth data;
selecting, using the processor, an augmented reality content generator corresponding to a three-dimensional (3D) effect;
Using the processor, applying the 3D effect to the image data and the depth data based at least in part on the selected augmented reality content generator, wherein applying the 3D effect to the image data and the depth data comprises: :
generating a depth map using at least the received depth data;
generating a packed depth map based at least in part on the depth map;
generating a segmentation mask based at least on the received image data, and
At least performing background inpainting and blurring of the received image data using the segmentation mask to generate background inpainted image data; and
cause, using a processor, to perform operations including generating a message containing information regarding the applied 3D effect, the image data, and the depth data;
The operation of generating the packed depth map is:
Converting a single-channel floating point texture to a raw depth map; and
generating a plurality of channels based at least in part on the raw depth map,
The operation of performing the background inpainting includes performing a diffusion-based inpainting technique to fill the missing area by propagating image content from the boundary between the missing area and the background area to the interior of the missing area,
The system of claim 1, wherein the background area includes a specific area of the image data without a foreground subject, and the missing area includes the foreground subject. A non-transitory computer-readable medium containing instructions, comprising:
The instructions, when executed by a computing device, cause the computing device to:
Receiving, using a processor, image data and depth data;
selecting, using the processor, an augmented reality content generator corresponding to a three-dimensional (3D) effect;
Using the processor, applying the 3D effect to the image data and the depth data based at least in part on the selected augmented reality content generator, wherein applying the 3D effect to the image data and the depth data comprises: :
generating a depth map using at least the received depth data;
generating a packed depth map based at least in part on the depth map;
generating a segmentation mask based at least on the received image data, and
At least performing background inpainting and blurring of the received image data using the segmentation mask to generate background inpainted image data; and
cause, using a processor, to perform operations including generating a message containing information regarding the applied 3D effect, the image data, and the depth data;
The operation of generating the packed depth map is:
Converting a single-channel floating point texture to a raw depth map; and
generating a plurality of channels based at least in part on the raw depth map,
The operation of performing the background inpainting includes performing a diffusion-based inpainting technique to fill the missing area by propagating image content from the boundary between the missing area and the background area to the interior of the missing area,
wherein the background area includes a specific area of the image data without a foreground subject, and the missing area includes the foreground subject. As a method,
Receiving, at a client device, a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item comprising an augmented reality content generator for applying a 3D effect, the 3D effect comprising at least one Contains US dollar manipulation -;
capturing image data and depth data using a camera of the client device;
Applying the 3D effect to the image data and the depth data, including the at least one beautification operation based at least in part on the augmented reality content generator, wherein the beautification operation is as part of applying the 3D effect. The steps to be performed and apply the 3D effect are:
generating a depth map using at least the depth data;
generating a packed depth map based at least in part on the depth map;
generating a segmentation mask based at least on the image data, and
performing background inpainting and blurring of the image data using at least the segmentation mask to generate background inpainted image data;
generating a 3D message based at least in part on the applied 3D effect including the at least one beautification manipulation; and
rendering a view of the 3D message based at least in part on the applied 3D effect including the at least one beautification manipulation,
The steps for generating the packed depth map are:
Converting a single-channel floating point texture to a raw depth map; and
generating a plurality of channels based at least in part on the raw depth map,
The performing the background inpainting includes performing a diffusion-based inpainting technique to fill the missing area by propagating image content from the boundary between the missing area and the background area to the interior of the missing area,
The method of claim 1, wherein the background area includes a specific area of the image data without a foreground subject, and the missing area includes the foreground subject. According to clause 13,
The steps of applying the 3D effect to the image data and the depth data are:
performing the beautification operation on at least a region of the image data containing facial image data, wherein the beautification operation includes at least one of smoothing, lighting adjustment, or color modification. method. According to clause 13,
The method of claim 1, wherein the beautification manipulation further comprises using a machine learning model to preserve facial feature structures, smooth blemishes, remove wrinkles, or preserve facial skin texture. According to clause 13,
generating a depth inpainting mask corresponding to a region of the depth map containing facial depth data; and
The method further comprising performing depth map inpainting of the depth map using at least the generated depth inpainting mask.
delete According to clause 13,
The packed depth map has a lower resolution than the image data. According to clause 16,
generating a normal map of the depth map to apply a lighting effect to facial image data of the image data; and
The method further comprising applying the lighting effect to the facial image data based at least in part on the normal map. According to clause 19,
The lighting effect includes at least two different colors, a first color from the two different colors is applied to a first portion of the facial image data, and a second color from the two different colors is applied to the facial image data. Method applied to a second image of data. According to clause 13,
The method further comprising causing display of an interface comprising the plurality of selectable graphical items, each selectable graphical item corresponding to a respective augmented reality content generator of the set of augmented reality content generators. According to clause 21,
Receiving a swipe gesture through a touch screen of the client device; and
In response to receiving the swipe gesture, the method further includes causing navigation through the plurality of selectable graphical items. As a system,
processor; and
Contains memory containing instructions,
The instructions, when executed by the processor, cause the processor to:
Receiving, at a client device, a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item comprising an augmented reality content generator for applying a 3D effect, the 3D effect comprising at least one Contains US dollar manipulation -;
capturing image data and depth data using a camera of the client device;
Applying the 3D effect to the image data and the depth data, including the at least one beautification operation based at least in part on the augmented reality content generator, wherein the beautification operation is as part of applying the 3D effect. The operations performed and applying the 3D effect are:
generating a depth map using at least the depth data;
generating a packed depth map based at least in part on the depth map;
generating a segmentation mask based at least on the image data, and
At least performing background inpainting and blurring of the image data using the segmentation mask to generate background inpainted image data;
generating a 3D message based at least in part on the applied 3D effect including the at least one beautification manipulation; and
perform operations including rendering a view of the 3D message based at least in part on the applied 3D effect including the at least one beautification operation,
The operation of generating the packed depth map is:
Converting a single-channel floating point texture to a raw depth map; and
generating a plurality of channels based at least in part on the raw depth map,
The operation of performing the background inpainting includes performing a diffusion-based inpainting technique to fill the missing area by propagating image content from the boundary between the missing area and the background area to the interior of the missing area,
The system of claim 1, wherein the background area includes a specific area of the image data without a foreground subject, and the missing area includes the foreground subject. A non-transitory computer-readable medium containing instructions, comprising:
The instructions, when executed by a computing device, cause the computing device to:
Receiving, at a client device, a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item comprising an augmented reality content generator for applying a 3D effect, the 3D effect comprising at least one Contains US dollar manipulation -;
capturing image data and depth data using a camera of the client device;
Applying the 3D effect to the image data and the depth data, including the at least one beautification operation based at least in part on the augmented reality content generator, wherein the beautification operation is as part of applying the 3D effect. The operations performed and applying the 3D effect are:
generating a depth map using at least the depth data;
generating a packed depth map based at least in part on the depth map;
generating a segmentation mask based at least on the image data, and
At least performing background inpainting and blurring of the image data using the segmentation mask to generate background inpainted image data;
generating a 3D message based at least in part on the applied 3D effect including the at least one beautification manipulation; and
perform operations including rendering a view of the 3D message based at least in part on the applied 3D effect including the at least one beautification operation,
The operation of generating the packed depth map is:
Converting a single-channel floating point texture to a raw depth map; and
generating a plurality of channels based at least in part on the raw depth map,
The operation of performing the background inpainting includes performing a diffusion-based inpainting technique to fill the missing area by propagating image content from the boundary between the missing area and the background area to the interior of the missing area,
wherein the background area includes a specific area of the image data without a foreground subject, and the missing area includes the foreground subject. As a method,
Selecting a set of augmented reality content generators from a plurality of available augmented reality content generators based on metadata associated with each augmented reality content generator, wherein the metadata indicates that the corresponding augmented reality content generator includes at least a 3D effect. and information indicating that the set of augmented reality content generators includes at least one augmented reality content generator without a 3D effect and at least one augmented reality content generator with a 3D effect;
Receiving, at a client device, a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item comprising an augmented reality content generator that includes a 3D effect;
capturing image data and depth data using at least one camera of the client device; and
Applying the 3D effect to the image data and the depth data based at least in part on the augmented reality content generator, wherein applying the 3D effect includes:
generating a depth map using at least the depth data;
generating a packed depth map based at least in part on the depth map;
generating a segmentation mask based at least on the image data, and
At least performing background inpainting and blurring of the image data using the segmentation mask to generate background inpainted image data,
The steps for generating the packed depth map are:
Converting a single-channel floating point texture to a raw depth map; and
generating a plurality of channels based at least in part on the raw depth map,
The performing the background inpainting includes performing a diffusion-based inpainting technique to fill the missing area by propagating image content from the boundary between the missing area and the background area to the interior of the missing area,
The method of claim 1, wherein the background area includes a specific area of the image data without a foreground subject, and the missing area includes the foreground subject. According to clause 25,
generating a 3D message based at least in part on the applied 3D effect; and
The method further comprising rendering a view of the 3D message based at least in part on the applied 3D effect. According to clause 25,
The at least one camera includes a first camera and a second camera, the first camera has a first focal length, the second camera has a second focal length, the first focal length and the second camera Focal distances vary in different ways. According to clause 27,
A disparity map is based at least in part on the distance between a first pixel from a first image captured by the first camera and a second pixel from a second image captured by the second camera. Generated, wherein the first pixel and the second pixel correspond to the same object. According to clause 28,
The disparity map includes an image where each pixel includes a distance value between a pixel from the first image and a corresponding pixel from the second image. According to clause 29,
First pixels of a first object in the disparity map have a brightness greater than second pixels of a second object in the disparity map, and the first pixels have a depth value less than second depth values of the second pixels. How to have them. According to clause 28,
A method wherein a depth map is generated based at least in part on the disparity map. According to clause 25,
A method wherein the augmented reality content generator includes a beautification operation. According to clause 32,
The method of claim 1, wherein the augmented reality content generator includes a 3D object rendered to approximate facial image data from the image data. According to clause 25,
The method further comprising causing display of an interface comprising a plurality of selectable graphical items, each selectable graphical item corresponding to a respective augmented reality content generator of the set of augmented reality content generators. As a system,
processor; and
Contains memory containing instructions,
The instructions, when executed by the processor, cause the processor to:
Selecting a set of augmented reality content generators from a plurality of available augmented reality content generators based on metadata associated with each augmented reality content generator, wherein the metadata specifies that the corresponding augmented reality content generator includes at least a 3D effect. and information indicating that the set of augmented reality content generators includes at least one augmented reality content generator without a 3D effect and at least one augmented reality content generator with a 3D effect;
At a client device, receiving a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item comprising an augmented reality content generator that includes a 3D effect;
capturing image data and depth data using at least one camera of the client device; and
Applying the 3D effect to the image data and the depth data based at least in part on the augmented reality content generator, wherein applying the 3D effect includes:
generating a depth map using at least the depth data;
generating a packed depth map based at least in part on the depth map;
generating a segmentation mask based at least on the image data, and
At least performing background inpainting and blurring of the image data using the segmentation mask to generate background inpainted image data -
Perform actions including,
The operation of generating the packed depth map is:
Converting a single-channel floating point texture to a raw depth map; and
generating a plurality of channels based at least in part on the raw depth map,
The operation of performing the background inpainting includes performing a diffusion-based inpainting technique to fill the missing area by propagating image content from the boundary between the missing area and the background area to the interior of the missing area,
The system of claim 1, wherein the background area includes a specific area of the image data without a foreground subject, and the missing area includes the foreground subject. A non-transitory computer-readable medium containing instructions, comprising:
The instructions, when executed by a computing device, cause the computing device to:
Selecting a set of augmented reality content generators from a plurality of available augmented reality content generators based on metadata associated with each augmented reality content generator, wherein the metadata specifies that the corresponding augmented reality content generator includes at least a 3D effect. and information indicating that the set of augmented reality content generators includes at least one augmented reality content generator without a 3D effect and at least one augmented reality content generator with a 3D effect;
At a client device, receiving a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item comprising an augmented reality content generator that includes a 3D effect;
capturing image data and depth data using at least one camera of the client device; and
Applying the 3D effect to the image data and the depth data based at least in part on the augmented reality content generator, wherein applying the 3D effect includes:
generating a depth map using at least the depth data;
generating a packed depth map based at least in part on the depth map;
generating a segmentation mask based at least on the image data, and
At least performing background inpainting and blurring of the image data using the segmentation mask to generate background inpainted image data -
Perform actions including,
The operation of generating the packed depth map is:
Converting a single-channel floating point texture to a raw depth map; and
generating a plurality of channels based at least in part on the raw depth map,
The operation of performing the background inpainting includes performing a diffusion-based inpainting technique to fill the missing area by propagating image content from the boundary between the missing area and the background area to the interior of the missing area,
wherein the background area includes a specific area of the image data without a foreground subject, and the missing area includes the foreground subject. As a method,
Receiving, at a client device, a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item comprising an augmented reality content generator that includes a 3D effect;
capturing image data using at least one camera of the client device;
generating depth data using a machine learning model based at least in part on the captured image data; and
Applying the 3D effect to the image data and the depth data based at least in part on the augmented reality content generator, wherein applying the 3D effect includes:
generating a depth map using at least the depth data;
generating a packed depth map based at least in part on the depth map;
generating a segmentation mask based at least on the image data, and
At least performing background inpainting and blurring of the image data using the segmentation mask to generate background inpainted image data,
The steps for generating the packed depth map are:
Converting a single-channel floating point texture to a raw depth map; and
generating a plurality of channels based at least in part on the raw depth map,
The performing the background inpainting includes performing a diffusion-based inpainting technique to fill the missing area by propagating image content from the boundary between the missing area and the background area to the interior of the missing area,
The method of claim 1, wherein the background area includes a specific area of the image data without a foreground subject, and the missing area includes the foreground subject. According to clause 37,
A method wherein the image data is captured by two or more cameras. According to clause 37,
Wherein the image data is captured using dual pixel autofocus from a single camera. According to clause 37,
The method of claim 1, wherein the machine learning model includes a deep neural network or a convolutional neural network that provides prediction of depth data based on the captured image data. According to clause 40,
The method of claim 1, wherein the machine learning model receives the captured image data as input and generates a depth map as output. According to clause 40,
The method of claim 1, wherein the machine learning model is executed on a neural network processor or graphics processing unit of the client device. According to clause 37,
generating a 3D message based at least in part on the applied 3D effect;
rendering a view of the 3D message based at least in part on the applied 3D effect;
Receiving motion data from a motion sensor of the client device;
updating a view of the 3D message based at least in part on the received motion data; and
The method further comprising rendering an updated view of the 3D message. According to clause 37,
The at least one camera includes a first camera and a second camera, the first camera has a first focal length, the second camera has a second focal length, the first focal length and the second camera Focal distances vary in different ways. According to clause 37,
further comprising selecting a set of augmented reality content generators from the plurality of available augmented reality content generators based on metadata associated with each augmented reality content generator, wherein the metadata determines that the corresponding augmented reality content generator is at least A method comprising information indicating that it includes a 3D effect, wherein the set of augmented reality content generators includes at least one augmented reality content generator without a 3D effect and at least one augmented reality content generator with a 3D effect. According to clause 45,
The method further comprising causing display of an interface comprising a plurality of selectable graphical items, each selectable graphical item corresponding to a respective augmented reality content generator of the set of augmented reality content generators. As a system,
processor; and
Contains memory containing instructions,
The instructions, when executed by the processor, cause the processor to:
At a client device, receiving a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item comprising an augmented reality content generator that includes a 3D effect;
capturing image data using at least one camera of the client device;
generating depth data using a machine learning model based at least in part on the captured image data; and
Applying the 3D effect to the image data and the depth data based at least in part on the augmented reality content generator, wherein applying the 3D effect includes:
generating a depth map using at least the depth data;
generating a packed depth map based at least in part on the depth map;
generating a segmentation mask based at least on the image data, and
At least performing background inpainting and blurring of the image data using the segmentation mask to generate background inpainted image data -
Perform actions including,
The operation of generating the packed depth map is:
Converting a single-channel floating point texture to a raw depth map; and
generating a plurality of channels based at least in part on the raw depth map,
The operation of performing the background inpainting includes performing a diffusion-based inpainting technique to fill the missing area by propagating image content from the boundary between the missing area and the background area to the interior of the missing area,
The system of claim 1, wherein the background area includes a specific area of the image data without a foreground subject, and the missing area includes the foreground subject. A non-transitory computer-readable medium containing instructions, comprising:
The instructions, when executed by a computing device, cause the computing device to:
At a client device, receiving a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item comprising an augmented reality content generator that includes a 3D effect;
capturing image data using at least one camera of the client device;
generating depth data using a machine learning model based at least in part on the captured image data; and
Applying the 3D effect to the image data and the depth data based at least in part on the augmented reality content generator, wherein applying the 3D effect includes:
generating a depth map using at least the depth data;
generating a packed depth map based at least in part on the depth map;
generating a segmentation mask based at least on the image data, and
At least performing background inpainting and blurring of the image data using the segmentation mask to generate background inpainted image data -
Perform actions including,
The operation of generating the packed depth map is:
Converting a single-channel floating point texture to a raw depth map; and
generating a plurality of channels based at least in part on the raw depth map,
The operation of performing the background inpainting includes performing a diffusion-based inpainting technique to fill the missing area by propagating image content from the boundary between the missing area and the background area to the interior of the missing area,
wherein the background area includes a specific area of the image data without a foreground subject, and the missing area includes the foreground subject. delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete

Images