KR102485626B1 - Generating a stitched data stream - Google Patents

Abstract

Systems and methods wherein a server computer receives a plurality of messages from a plurality of user computing devices, each message of the plurality including a data stream, and a sequence of messages of the plurality of messages associated with a similar geographic location and time period. Determine a subset, determine a set of messages from the subset of messages based on a matching score for each pair of messages, and for each message of the set of messages based on the time period for each message. stitching together a set of messages to create a stitched data stream from data streams, wherein the stitched data stream comprises data streams overlapping in time periods such that there can be more than one data stream during a given time period. Contains messages with

Description

GENERATING A STITCHED DATA STREAM}

This patent application claims the benefit of priority to U.S. Application Serial No. 15/470,004, filed on March 27, 2017, and also claims priority to U.S. Patent Application No. 15,470,025, filed on March 27, 2017; , the contents of these applications are incorporated herein by reference in their entirety.

A messaging system may receive millions of messages from users who wish to share media content such as audio, images, and video between user devices (eg, mobile devices, personal computers, etc.). The media content of these messages may be associated with common geolocations, common time periods, common events, and the like.

The various of the accompanying drawings represent only exemplary embodiments of the present disclosure and should not be considered as limiting its scope.
1 is a block diagram illustrating an example messaging system for exchanging data (eg, messages and associated content) over a network, in accordance with some example embodiments.
2 is a block diagram illustrating further details regarding a messaging system, in accordance with some demonstrative embodiments.
3 is a schematic diagram illustrating data that may be stored in a database of a messaging server system, in accordance with some demonstrative embodiments.
4 is a schematic diagram illustrating the structure of a message, in accordance with some embodiments, generated by a messaging client application for communication.
FIG. 5 illustrates a set of user interfaces in which access to content (eg, short-lived messages and associated multimedia payloads of data) or collections of content (eg, short-lived message stories) can be time-limited (eg, short-lived). ), a schematic diagram illustrating the access-restriction process.
6 is a flow diagram illustrating aspects of a method according to some example embodiments.
7 shows an example of a spectrogram according to some example embodiments.
8 shows an example of detected maximum values of a spectrogram, in accordance with some demonstrative embodiments.
9 shows an example of maxima of a spectrogram linked together to a final audio fingerprint, in accordance with some demonstrative embodiments.
10 and 11 each show a visual representation of a stitched data stream comprising a plurality of messages received from a plurality of user devices, according to some demonstrative embodiments.
12 is a flow diagram illustrating aspects of a method according to some example embodiments.
13-15 each depict an example user interface of a user computing device in accordance with some example embodiments.
16 is a block diagram illustrating an example of a software architecture that may be installed on a machine, according to some demonstrative embodiments.
17 is a diagrammatic representation of a machine in the form of a computer system within which a set of instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein, in accordance with an illustrative embodiment. exemplify

The systems and methods described herein relate to handling media content items to be shared between devices via a messaging system. For example, a user may wish to share one or more videos, images, and the like with one or more other users. These messages may be associated with common audio content such as songs, concerts, voices, and the like. Embodiments described herein are directed to stitching together data streams from a plurality of messages received from a plurality of user computing devices to create dense audio stitches associated with a common audio timeline for the data streams. provide a mechanism. For example, audio stitching is used to automatically stitch together a set of messages or data streams associated with the same audio content. Given a set of messages or data streams, the system described herein extracts audio fingerprints for each message and uses the audio fingerprint across all pairs of messages to identify messages associated with the same audio content. Audio matches are created by matching , and then audio stitches are created by finding paths through the set of audio matches. The stitched audio or stitched data stream can then be provided to one or more user computing devices to allow users to view the stitched data stream.

A stitched data stream may include messages with overlapping data streams in time periods such that there may be more than one data stream during a given time period. As the user views the stitched data stream, exemplary embodiments allow the user to switch to a variety of different data streams during any given time period. For example, a user may be watching a lead vocalist at a concert and then switch views to see a guitarist, or a drummer, or an audience member at that point in the concert. In this way, a user can switch between alternating views at any given time period in a common audio timeline.

1 is a block diagram illustrating a networked system 100 (eg, a messaging system) for exchanging data (eg, messages and associated content) over a network. The networked system 100 includes a number of client devices 102, each hosting a number of client applications 104. Each client application 104 is communicatively coupled to other instances of the client application 104 and the server system 108 via the network 106 .

Client device 102 may also be referred to herein as a user device or user computing device. Client devices 102 include mobile phones, desktop computers, laptops, personal digital assistants (PDAs), smartphones, tablets, ultrabooks, netbooks, laptops, multiprocessor systems, microprocessor-based or programmable consumer electronics devices, game consoles, may include, but is not limited to, a set top box, a computer in a vehicle, or any other communication device that a user may use to access the networked system 100. In some embodiments, client device 102 may include a display module (not shown) that displays information (eg, in the form of user interfaces). In further embodiments, client device 102 may include one or more of touch screens, accelerometers, gyroscopes, cameras, microphones, global positioning system (GPS) devices, and the like. Client device 102 is used to create media content items such as video, images (eg, photos), audio, and to send and receive messages containing such media content items to and from other users. It may be the user's device. Elements of these media content from multiple messages can then be stitched together as described in more detail in the embodiments described below.

One or more users may interact with the client device 102 (eg, a person, machine, or other means of interacting with the client device 102). In example embodiments, a user may not be part of system 100, but may interact with system 100 via client device 102 or other means. For example, a user may provide input (eg, touch screen input or alphanumeric input) to client device 102 , and the input may be transmitted over network 106 to system 100 (eg, a server system). 108, etc.). In such cases, other entities within system 100, in response to receiving input from the user, may communicate information over network 106 to client device 102 to be presented to the user. In this way, a user may use client device 102 to interact with various entities within system 100 .

System 100 may further include a network 106 . One or more portions of network 106 may be an ad hoc network, intranet, extranet, virtual private network (VPN), local area network (LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), urban area. Network (MAN), part of the Internet, part of the Public Switched Telephone Network (PSTN), cellular telephone network, wireless network, WiFi network, WiMax network, another type of network, or a combination of two or more such networks. can be

Client device 102 communicates with other entities in system 100 via a web client (e.g., a browser such as the Internet Explorer® browser developed by Microsoft® Corporation of Redmond, Wash.) or one or more client applications 104. You can access various data and applications provided by As noted above, the client device 102 may be a web browser, messaging application, email application, e-commerce site application, mapping or location application, media content editing application, media content viewing application, and the like, but not limited thereto. may include one or more client applications 104 (also referred to as “apps”) that are not

In one example, client application 104 is a messaging application that allows a user to take a photo or video, add a caption, or otherwise edit the photo or video, and then send the photo or video to another user. can The message may be ephemeral and may be removed from the receiving user device after viewing or after a predetermined amount of time (eg, 10 seconds, 24 hours, etc.). Short-lived messages refer to messages that are accessible for a time-limited duration. A short-lived message can be text, images, video, and other such content that can be stitched together according to embodiments described herein. The access time for short-lived messages can be set by the sender of the message. Alternatively, the access time may be a default setting or a setting specified by the recipient. Regardless of the setting technique, messages are temporary.

Messaging applications may further allow users to create galleries. A gallery may be a collection of photos and videos that other users may view by other users who "follow" the user's gallery (eg, subscribe to view and receive updates from the user's gallery). can A gallery can also be short-lived (eg, lasting 24 hours, lasting for the duration of an event (eg, during a music concert, sporting event, etc.), or lasting for some other predetermined amount of time).

A short message can be associated with a message duration parameter, the value of which determines the amount of time the short message will be displayed by the client application 104 to a receiving user of the short message. A short-lived message may further be associated with a message receiver identifier and a message timer. A message timer may be responsible for determining the amount of time that a short-lived message is visible to a particular recipient user identified by the message receiver identifier. For example, a short-lived message may be visible to the relevant receiving user only for a period of time determined by the value of the message duration parameter.

In another example, a messaging application may allow a user to store photos and videos and create a gallery that is not short-lived and can also be sent to other users. For example, gather photos and videos from a recent vacation and share them with friends and family.

In some embodiments, one or more client applications 104 may be included on a given one of the client devices 102, and as needed, data and/or processing capabilities not available locally (eg, system 100 for accessing location information, authenticating users, validating payment methods, accessing media content stored on servers, synchronizing media content between client devices 102 and server computers, etc. ) (e.g., server system 108) may be configured to locally provide at least some of the user interface and functionality of the application 104 configured to communicate with other entities within the server system 108. Conversely, one or more applications 104 may not be included on the client device 102, which then uses its web browser to access the system 100 (e.g., the server system 108). It can access one or more applications hosted on different entities.

Server system 108 may provide server-side functionality to one or more client devices 102 over a network 106 (eg, the Internet or a wide area network (WAN)). The server system 108 includes an application programming interface (API) server 110, an application server 112, a messaging server application 114, and a media content processing server 116, a social networking system 122, and data stream stitching system 124, each of which can be communicatively coupled to each other and to one or more data storage(s), such as database(s) 120.

Server system 108 may be a cloud computing environment according to some example embodiments. Server system 108, and any servers associated with server system 108, may be associated with a cloud-based application in one exemplary embodiment. One or more database(s) 120 may include unprocessed media content, original media content from users (eg, high-quality media content), and processed media content (eg, client devices 102 and of media content formatted for sharing and viewing on client devices 102), stitched audio data streams, user information, user device information, and the like. One or more database(s) 120 may include cloud-based storage external to server system 108 (eg, hosted by one or more third-party entities external to server system 108 ). Although storage devices are shown as database(s) 120, it is understood that system 100 may access and store data in storage devices such as database 120, blob storage, and other types of storage methods. do.

Accordingly, each client application 104 can communicate and exchange data with other client applications 104 and with the server system 108 over the network 106 . Data exchanged between the messaging client applications 104 and between the client application 104 and the server system 108 includes functions (eg, instructions to invoke functions) as well as payload data (eg, For example, text, audio, video or other multimedia data).

Server system 108 provides server-side functionality to specific messaging client applications 104 over network 106 . While certain functions of system 100 are described herein as being performed by either client application 104 or server system 108, the location of particular functionality within either client application 104 or server system 108. It will be appreciated that is a design option. For example, it is technically desirable to initially deploy certain technology and functionality within the server system 108, but later transfer the technology and functionality to the client application 104 when the client device 102 has sufficient processing capacity. can do.

Server system 108 supports various services and operations provided to client applications 104 . Such operations include sending data to, receiving data from, and processing data generated by the client application 104 . This data may include, for example, message content, client device information, geolocation information, media annotations and overlays, message content persistence conditions, social network information, and live event information, date and timestamp. Data exchange within the networked system 100 is initiated and controlled through functions available through the user interfaces (UIs) of the client application 104 .

In the server system 108 , an application program interface (API) server 110 is coupled to the application server 112 and provides a programmatic interface to the application server. Application server 112 is communicatively coupled to database server 118, which facilitates access to database 120 where data associated with messages processed by application server 112 is stored.

API Server 110 The server receives and transmits message data (eg, instructions and message payloads) between the client device 102 and the application server 112 . Specifically, API server 110 provides a set of interfaces (e.g., routines and protocols) that can be invoked or queried by client application 104 to invoke functionality of application server 112. provides The API server 110 provides account registration, login functionality, sending of messages from a particular client application 104 to another client application 104 via the application server 112, and from a client application 104 to a messaging application server. transmission of media files (eg images or videos) to 114, and establishment of a collection of media data (eg stories) for possible access by another client application 104, a client device 102, search the user's list of friends, search such collections, search messages and content, add and delete friends to the social graph, locate friends in the social graph, (e.g., the client application 104 ) exposes various functions supported by the application server 112, including opening and application events related to ), and so forth.

Application server 112 hosts a number of applications and subsystems, including messaging server application 114 , media content processing system 116 , social networking system 122 , and data stream stitching system 124 . Messaging server application 114 processes multiple messages, particularly related to aggregation and other processing of content (eg, text and multimedia content) included in messages received from multiple instances of messaging client application 104. Implement technologies and functions. As described in more detail, text and media content from multiple sources can be aggregated into collections of content (eg, called stories or galleries). These collections are then made available to the client application 104 by the messaging server application 114 . Processing of other processor and memory intensive data may also be performed server-side by the messaging server application 114, taking into account the hardware requirements for such processing.

Application server 112 also includes a media content processing system 116 dedicated to performing various media content processing operations, typically with respect to images or video received within the payload of a message in messaging server application 114. do. Media content processing system 116 may access one or more data storage (e.g., database(s) 120) to retrieve stored data for use in processing media content and to store results of the processed media content. can

The social networking system 122 supports various social networking functions and services and makes these functions and services available to the messaging server application 114 . To this end, social-networking system 122 maintains and accesses entity graph 304 in database 120 . Examples of functions and services supported by the social-networking system 122 include identification of other users of the networked system 100 that a particular user has a relationship with or “follows,” and identification of other entities and interests of a particular user. include matters

Application server 112 is communicatively coupled to database server 118, which facilitates access to one or more database(s) 120 where data associated with messages processed by messaging server application 114 is stored. let it

Messaging server application 114 may be responsible for creating and delivering messages between users of client devices 102 . Messaging application server 114 may utilize any of a number of message delivery networks and platforms to deliver messages to users. For example, messaging application server 114 may be electronic mail (email), instant message (IM), Short Message Service (SMS), text, facsimile, or voice (e.g., VoIP (VoIP) Voice over IP) messages over wired (eg Internet), plain old telephone service (POTS), or wireless networks (eg mobile, cellular, WiFi, Long Term Evolution (LTE), Bluetooth) can be conveyed

Data stream stitching system 124 stitches together data streams included in a plurality of messages received from a plurality of user computing devices (eg, client devices 102 ), as described in more detail below. It can be responsible for creating data streams.

2 is a block diagram illustrating additional details regarding a system 100 according to example embodiments. Specifically, the system 100 is shown as including a messaging client application 104 and an application server 112, which in turn includes several subsystems, namely short-term timer system 202, collection management system 204 , and annotation system 206.

The short-term timer system 202 is responsible for enforcing temporary access to content allowed by the messaging client application 104 and the messaging server application 114. To this end, the short-term timer system 202 displays messages and associated content via the messaging client application 104 based on the duration and display parameters associated with a message or collection of messages (eg, a SNAPCHAT story). It includes a number of timers that selectively display and enable access to. Additional details regarding the operation of the short term timer system 202 are provided below.

Collection management system 204 is responsible for managing collections of media (eg, collections of text, image video, and audio data). In some examples, a collection of content (images, video, messages including text and audio) can be organized into an “event gallery” or “event story”. Such a collection may be made available for a specified period of time, such as the duration of an event to which the content relates. For example, content related to a music concert may be made available as a "story" for the duration of that music concert. The collection management system 204 may also be responsible for posting an icon to the user interface of the messaging client application 104 to provide notification of the existence of a particular collection.

The collection management system 204 further includes a curation interface 208 that allows a collection manager to manage and curate a particular collection of content. For example, the curation interface 208 enables an event organizer to curate a collection of content related to a particular event (eg, delete inappropriate content or duplicate messages). Additionally, the collection management system 204 automatically curates the content collection using machine vision (or image recognition technology) and content rules. In certain embodiments, rewards (e.g., money or non-monetary credits or points, travel miles, access to artwork or specialized lenses, etc.) may be paid to the user. In such cases, the curation interface 208 operates to automatically pay such users for using their content.

Annotation system 206 provides various functions that enable a user to annotate or otherwise revise or edit media content associated with a message. For example, annotation system 206 provides functions related to the creation and publishing of media overlays for messages processed by networked system 100 . The annotation system 206 operatively supplies a media overlay (eg, a SNAPCHAT filter) to the messaging client application 104 based on the geolocation of the client device 102 . In another example, annotation system 206 operatively supplies media overlays to messaging client application 104 based on other information, such as social network information of a user of client device 102 . Media overlays can include audio and visual content and visual effects. Examples of audio and visual content include pictures, texts, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects may be applied to media content items (eg, photos) on the client device 102 . For example, a media overlay includes text that can be overlaid on top of a photo taken and created by the client device 102 . In another example, the media overlay includes a location identification overlay (eg, Venice Beach), the name of the live event, and a vendor name overlay (eg, Beach Coffee House). In another example, the annotation system 206 uses the geolocation of the client device 102 to identify a media overlay that includes the merchant's name at the geolocation of the client device 102 . The media overlay may include other indicia associated with the merchant. Media overlays are stored in database 120 and can be accessed through database server 118 .

In one exemplary embodiment, the annotation system 206 provides a user-based posting platform that enables users to select a geolocation on a map and upload content associated with the selected geolocation. A user can also specify situations in which a particular media overlay should be presented to other users. The annotation system 206 creates a media overlay that includes the uploaded content and associates the uploaded content with the selected geolocation.

In another exemplary embodiment, the annotation system 206 provides a merchant-based publishing platform that enables merchants to select specific media overlays associated with geolocations through a bidding process. For example, the annotation system 206 associates the media overlay of the highest bidder with a corresponding geolocation for a predefined amount of time.

3 is a schematic diagram 300 illustrating data that may be stored in database 120 of server system 108, according to certain example embodiments. Although the contents of database 120 are shown as including a number of tables, it will be appreciated that data may be stored in other types of data structures (eg, as an object oriented database).

Database 120 includes message data stored in message table 314 . Entity table 302 stores entity data including entity graph 304 . Entities for which records are maintained in entity table 302 may include individuals, company entities, organizations, objects, places, events, and the like. Any entity about which server system 108 stores data, regardless of type, may be a recognized entity. Each entity has a unique identifier as well as an entity type identifier (not shown).

Entity graph 304 further stores information about relationships and associations between entities. Such relationships may be social, professional (eg, work in a general corporation or organization) interest-based or activity-based, by way of example only.

Database 120 also stores annotation data in annotation table 312 in the form of exemplary filters. Filters, for which data is stored in annotation table 312, are videos (for which data is stored in video table 310) and/or images (for which data is stored in image table 308). related to and applicable to them. In one example, filters are overlays that are displayed overlaid on an image or video during presentation to a receiving user. Filters can be of various types, including user-selected filters from a gallery of filters that are presented to the sending user by the messaging client application 104 when the sending user is composing a message. Other types of filters include geolocation filters (also known as geofilters) that can be presented to a sending user based on geographic location. For example, geolocation filters specific to a neighborhood or a particular location may be presented within a user interface by messaging client application 104 based on geolocation information determined by a GPS unit of client device 102 . Another type of filter is a data filter that can be selectively presented to the sending user by the messaging client application 104 based on information gathered by the client device 102 during the message creation process or other inputs. . Examples of data filters include the current temperature at a particular location, the current rate at which the sending user is moving, the battery life for the client device 102, or the current time of day.

Another annotation data that may be stored within the image table 308 is so-called "lens" data. “Lens” can be real-time special effects and sounds that can be added to an image or video.

As noted above, video table 310, in one embodiment, stores video data associated with messages for which records are maintained in message table 314. Similarly, image table 308 stores image data associated with messages for which message data is stored in entity table 302 . Entity table 302 can associate various annotations from annotations table 312 with various images and videos stored in image table 308 and video table 310 .

The story table 306 stores data relating to a collection of messages and associated image, video or audio data that is compiled into a collection (eg, a SNAPCHAT story or gallery). Creation of a particular collection may be initiated by a particular user (eg, each user for whom a record is maintained in the entity table 302 ). A user can create a "personal story" in the form of a collection of content created and transmitted/broadcast by that user. To this end, the user interface of the messaging client application 104 may include a user selectable icon to enable the sending user to add specific content to their personal story.

A collection may also constitute a “live story,” a collection of content from multiple users created manually, automatically, or using a combination of manual and automated techniques. For example, a “live story” may constitute a curated stream of user-submitted content from various locations and events. Users whose client devices 102 are location service capable and who are at a common location event at a particular time have the option to contribute content to a particular live story, for example, through the user interface of the messaging client application 104. can be presented The live story may be identified to the user by the messaging client application 104 based on their location. The end result is a "live story" told from a community perspective.

An additional type of content collection is a "location story" that enables a user whose client device 102 is located within a particular geographic location (eg, college or university campus) to contribute to a particular collection. ) is known as In some embodiments, contributions to a location story may require a second degree of authentication to verify that the end user belongs to a particular organization or other entity (eg, is a student on a comprehensive college campus).

4 is a schematic diagram illustrating the structure of a message 400, according to some of some embodiments, generated by a client application 104 for communication to a further client application 104 or messaging server application 114. . The content of a particular message 400 is used to populate a message table 314 stored in database 120, accessible by messaging server application 114. Similarly, the content of message 400 is stored in memory as “in-transit” or “in-flight” data of client device 102 or application server 112 . Message 400 is shown to include the following components:

• Message identifier 402: A unique identifier that identifies the message 400.

· Message text payload 404: Text generated by the user via the user interface of the client device 102 and included in the message 400.

· Message image payload 406: image data captured by the camera component of client device 102 or retrieved from memory of client device 102 and included in message 400.

· Message video payload 408: Video data captured by the camera component or retrieved from the memory component of the client device 102 and included in the message 400.

· Message audio payload 410: Audio data captured by the microphone or retrieved from a memory component of the client device 102 and included in the message 400.

Message Annotations 412: Annotation data representing annotations to be applied to the message image payload 406, message video payload 408, or message audio payload 410 of the message 400 (e.g. filters , stickers or other reinforcers).

Message duration parameter 414: for which the content of the message 400 (e.g., message image payload 406, message video payload 408, message audio payload 410) is determined by the messaging client application A parameter value indicating the amount of time, in seconds, to be presented or made accessible to the user via (104).

· Message Geolocation Parameters 416: Geolocation data associated with the content payload of the message 400 (eg, latitude and longitude coordinates). A number of message geolocation parameter 416 values may be included in the payload, each of which may be a specific image within the content (eg, a specific image within the message image payload 406, or a specific image within the message video payload 408). video) is associated with respect to content items included in it.

• Message Story Identifier 418: Identifier values that identify one or more content collections (eg, “stories”) with which a particular content item within the message image payload 406 of the message 400 is associated. For example, each of a plurality of images within the message image payload 406 may be associated with a plurality of content collections using identifier values.

· Message tag 420: Each message 400 may be tagged with a plurality of tags, each representing a subject of content included in the message payload. For example, if a specific image included in the message image payload 406 depicts an animal (eg, a lion), a tag value representing the related animal may be included in the message tag 420 . Tag values can be manually generated based on user input, or automatically generated using, for example, image recognition.

Message sender identifier 422: an identifier representing the user of the client device 102 on which the message 400 was created and from which the message 400 was sent (e.g., a messaging system identifier, email address, or device identifier) .

Message Recipient Identifier 424: An identifier representing the user of the client device 102 to which the message 400 is addressed (eg, a messaging system identifier, email address, or device identifier).

The contents (eg, values) of the various components of message 400 may be pointers to locations within a table where content data values are stored therein. For example, the image value in the message image payload 406 can be a pointer to (or its address) a location within the image table 308. Similarly, values in message video payload 408 can point to data stored in video table 310, values stored in message annotations 412 can point to data stored in annotations table 312, and , values stored in message story identifier 418 may point to data stored in story table 306, and values stored in message sender identifier 422 and message recipient identifier 424 may point to user stored in entity table 302. Records can be pointed to.

FIG. 5 illustrates an aspect in which access to content (e.g., short message 502, and an associated multimedia payload of data) or content collection (e.g., short message story 504) may be time-restricted (e.g., is a schematic diagram illustrating an access-limiting process 500, which can be, for example, in the short term.

The short-term message 502 is shown as being associated with a message duration parameter 506, the value of which is the amount of time the short-term message 502 will be displayed by the client application 104 to the receiving user of the short-term message 502. determine the amount In one embodiment, if the client application 104 is a SNAPCHAT application client, a short-term request is made by the receiving user for up to 10 seconds, depending on the amount of time the sending user specifies using the message duration parameter 506. Message 502 is viewable.

The message duration parameter 506 and the message recipient identifier 424 are shown as being inputs to a message timer 512, where the short message 502 receives a specific message identified by the message recipient identifier 424. Responsible for determining the amount of time shown to the end user. In particular, a short-lived message 502 will only be visible to the relevant receiving user for a period of time determined by the value of the message duration parameter 506 . The message timer 512 is shown providing an output to a more generalized short-term timer system 202 that is responsible for the overall timing of the display of content (e.g., the short-term message 502) to the receiving user.

A short-term message 502 is shown in FIG. 5 as being included within a short-term message story 504 (eg, a personal SNAPCHAT story, or an event story). The short message story 504 has an associated story duration parameter 508, the value of which determines the duration of time for which the short message story 504 is presented and accessible to users of the networked system 100. . For example, the story duration parameter 508 can be the duration of a music concert, where the short message story 504 is a collection of content related to that concert. Alternatively, a user (owning user or curator user) may specify a value for the story duration parameter 508 when performing setup and creation of a short message story 504 .

Additionally, each short-lived message 502 within the short-lived message story 504 has an associated story participation parameter 510, the value of which is the length of time the short-lived message 502 will be accessible within the context of the short-lived message story 504. determine the duration. Thus, a particular short-lived message story 504 "expires" and is inaccessible within the context of the short-lived message story 504 before the short-lived message story 504 itself expires in terms of the story duration parameter 508. it can be done The story duration parameter 508, the story participation parameter 510, and the message recipient identifier 424 each provide input to the story timer 514, which during operation, firstly, specifies the short-term message story 504. Determines whether the short message 502 will be displayed to a particular receiving user and, if so, for how long. Note that the short message story 504 also recognizes the identity of a particular recipient user as a result of the message recipient identifier 424 .

Accordingly, the story timer 514 controls, during operation, the overall lifetime of each short-term message 502 included in the short-term message story 504, as well as the associated short-term message story 504. In one embodiment, each and every short message 502 within the short message story 504 remains viewable and accessible for a period of time specified by the story duration parameter 508 . In a further embodiment, any short-lived message 502 may expire within the context of the short-lived message story 504 based on the story participation parameter 510 . Note that the message duration parameter 506, even within the context of the short-lived message story 504, can still determine the duration of time a particular short-lived message 502 is displayed to the receiving user. Accordingly, the message duration parameter 506 determines whether a particular short-term message 502 is sent to the receiving end, regardless of whether the recipient user is viewing that short-term message 502 within the context of the short-term message story 504 or outside it. Determine the duration of time displayed to the user.

The short-term timer system 202 may furthermore remove a particular short-term message 502 from the short-term message story 504 based on a determination that during operation it has exceeded an associated story engagement parameter 510 . For example, when the sending user establishes a story engagement parameter 510 of 24 hours from posting, the short-term timer system 202 removes the associated short-term message 502 from the short-term message story 504 after the specified 24 hours. something to do. The short-lived timer system 202 also determines when the story participation parameter 510 for each and every short-lived message 502 within the short-term message story 504 has expired, or when the short-term message story 504 itself has a story duration parameter. The aspect of 508 operates to purge the short-lived message story 504 in any of the cases when it has expired.

In some use cases, the creator of a particular short-lived message story 504 may specify an indefinite story duration parameter 508 . In this case, the expiration of the story participation parameter 510 for the last remaining short-term message 502 in the short-term message story 504 will determine when the short-term message story 504 itself expires. In this case, a new short-term message 502 added to the short-term message story 504, by means of the new story participation parameter 510, makes the lifetime of the short-term message story 504 equal to the value of the story participation parameter 510. extend effectively.

In response to the short-term timer system 202 determining that the short-term message story 504 has expired (e.g., is no longer accessible), the short-term timer system 202 may send the system 100 (and e.g. For example, specifically communicating with the messaging client application 104 to cause an indication (eg, icon) associated with the associated short message story 504 to no longer be displayed within the user interface of the client application 104 . Similarly, when the short-term timer system 202 determines that the message duration parameter 506 for a particular short-term message 502 has expired, the short-term timer system 202 causes the client application 104 to send the short-term message 502 ) to no longer display an indication associated with (eg, an icon or text identification).

6 is an aspect of a method 600 for generating a stitched data stream from a plurality of messages received from a plurality of user computing devices (eg, client devices 102), according to some demonstrative embodiments. It is a flow chart illustrating For example, server system 108 may determine that a subset of the plurality of messages is associated with a common audio content and to create a stitched data stream from the subset of the plurality of messages for an audio timeline associated with the common audio content. can For purposes of illustration, method 600 is described with respect to networked system 100 of FIG. 1 . It should be understood that method 600 may be practiced with other system configurations in other embodiments.

At operation 602, server system 108 (eg, via a server computer associated with data stream stitching system 124) receives data from a plurality of user computing devices (eg, client devices 102). receive a message from Each message of the plurality of messages may include media content such as images, data streams, and the like. In one example, the data stream may be a video taken by a user during a concert, speech, or other event that includes audio. Each message may also include geolocation data, a date and time for the message (eg, when the message was created or sent), data and a timestamp for the data stream or other media content (eg, the data stream or other media content of the message was captured or recorded), and other data.

At operation 604, the server computer determines a subset of messages, of the plurality of messages, that are associated with a similar geographic location and time period. For example, the server computer may analyze each message of the plurality of messages to determine the geolocation of each of the plurality of messages and the time period for each of the messages. In one example, the server computer may determine the geolocation of each of the plurality of messages using data included in the message, such as the global positioning system (GPS) coordinates of the user computing device from which the message was transmitted. In another example, the server computer determines whether the message is sent to the server based on a timestamp included in the message indicating the time the message was sent, or the media content (eg, data stream) included in the message was captured. The time period may be determined based on the time received by the computer, and the like. The server computer determines a geolocation associated with a region of predetermined GPS coordinates in which a subset of messages of the plurality of messages is the same and a time period within a predetermined time window (eg, 1 minute, 5 minutes, 10 minutes, 20 minutes, etc.) can include

In one example, server system 108 may pre-divide the world (eg, world map) into predefined grids. For each grid, the server computer may aggregate all of the messages with geolocation belonging to the respective grid within a specific time period (eg, every 10 minutes). Each grid may be the same size, or each grid size may vary based on the density of population within a region, the size of a town or city, and the like. Thus, a subset of messages may be a collection of messages having geographic locations within one particular grid that occurred within the same (eg, 10 minute) time period.

At operation 606, the server computer extracts audio fingerprints for each message in the subset of messages to be used to determine if there is common audio across messages in the subset of messages. For example, the server computer may analyze the audio included in each message (eg, audio for a data stream of the message) and calculate an audio fingerprint. There are a number of different methods that can be used to compute audio fingerprints. One example is extracting features from a spectrogram and using the extracted features in comparison with other spectrograms to find out if the important spectrograms match up and where exactly they match up. will be. Using the spectrogram is just one example of how to compute audio fingerprints. It is understood that other methods may be used alternatively or in combination.

In one example, the following response is returned from analyzing the audio contained in the message to compute the audio fingerprint:

The spectrogram representation may be computed using a Fast Fourier Transform (FFT) over windows in time. For example, parameters used in FFT windows may include:

SAMPLING_RATE=44100

WINDOW_SIZE=4096

OVERLAP_RATIO=0.5

7 shows an example of a spectrogram 700 . With the spectrogram, maxima (or peaks) can be extracted using a diamond maximal filter of size NEIGHBORHOOD_SIZE = 20. 8 shows an example of detected maximum values of spectrogram 800 .

After maxima are extracted, pairs of maxima are linked together to become the final audio fingerprint, as shown in spectrogram 900 of FIG. 9 . First, maxima can be classified by the time they occur. The respective maxima are then linked with the first FAN_VALUE maxima, if they fall within the MIN_TIME_DELTA and MAX_TIME_DELTA time frames. A fingerprint can be created using three values:

1. Frequency of first maxima (11 bits)

2. Frequency of second maxima (11 bits)

3. Time difference between two maxima (8 bits)

In this example, these three values are packed to be a single unsigned integer to form an audio fingerprint. This audio fingerprint is then associated with the time frame of the first maxima. This results in the following fingerprint hash:

TIME_DELTA | FREQ1 | FREQ2

8 bit|11 bit|11 bit

The parameters that can be used for hashing are:

FAN_VALUE = 15

MIN_TIME_DELTA = 1

MAX_TIME_DELTA = 255

In one example, audio fingerprints are designed in a noise resistant manner. Thus, as long as there is some common audio shared between the two data streams, the two data streams can still be effectively matched.

Returning to FIG. 6, in operation 608, the server computer groups subsets of messages into a plurality of message pairs so that each message in the plurality of subsets of messages can be compared to each other message. do. Each pair of messages may include a first message and a second message.

At operation 610, the server computer compares the audio fingerprint of a first message to the audio fingerprint of a second message in each pair of messages to determine a match score for each pair of messages. In this way, the server computer can match audio fingerprints between pairs of all messages within a subset of messages (eg, from audio content in each data stream in each message). If the matching score for a pair of messages is higher than a predetermined threshold, the server computer may consider the pair of messages to be a match (eg, contain portions of the same audio content), and each message in the pair is ordered You can determine exactly where it will be. In one example, a match can be found if enough fingerprints are matched in the correct temporal order.

In another example, the server computer can determine at what point two messages should be concatenated so that their intersecting audio tracks match up correctly. Matching can be done in spectrogram space and can be accurate to 1/20th of a second. For example, a first message may include the first 10 seconds of audio content of audio content (eg, from time 00:00 to time 00:10), and a second message may include from time 00:08 to time 00 can contain 5 seconds of audio content up to :13, and so on.

In one example, the server computer may determine if there are enough message pairs that are linked together so that they all become one common audio timeline. A common audio timeline is associated with specific audio content (eg, song, voice, concert, etc.). The common audio timeline may include the entire length of the audio content (eg, the entire song, voice, concert, etc.), or the common audio timeline may include a portion of the total length of the audio content (eg, the end of the song). 45 seconds, the middle part of the voice, the first 10 minutes of a concert, etc.).

A predetermined threshold may be used to determine if there are enough pairs of messages to be linked together so that they all become one common audio timeline. For example, a predetermined threshold may be a certain amount of time that linked messages must be (e.g., a minimum of 30 seconds, 1 minute, 5 minutes, etc.), a certain number of messages that must be linked (e.g., 2, 10, 20, 50, 100, etc.), or other thresholds. If there are enough pairs of messages to link together to become one common audio timeline, the server computer can stitch together the messages associated with the common audio timeline.

At operation 612, the server computer determines a set of messages from among the subset of messages associated with the common audio timeline. For example, a server computer may determine a set of messages based on a matching score for each pair of messages. In one example, the set of messages may include all messages within the subset of messages. In another example, a set of messages may be a further subset of a subset of messages (eg, messages in a subset of messages may not all be associated with common audio content, but instead with different audio content). may be associated with).

At operation 614, the server computer stitches the set of messages together to create a stitched data stream from the data streams for each message in the set of messages. For example, a server computer may stitch together a set of messages based on the time period for each message. For example, Message A may be for the time period 00:10-00:20 in the common audio timeline, Message B may be for the time period 00:20-00:23, and Message C may be for the time period 00:20-00:23. It could be for the period 00:22-00:31, and so on. Data streams may overlap in time periods such that there may be more than one data stream during a given time period.

In one example, the server computer may determine a start message for stitching. A start message can be the message that is occurring first in a common audio timeline for a set of messages. For example, it could be one message with a period of time in the audio timeline starting at 00:10, and another message with a period of time in the audio timeline starting at 00:15, and so on. becomes like The server computer determines which message is earliest in the audio timeline (eg, the one starting at time 00:10).

In one example, there may be more than one message with similar start times, any of which may be a start message. For example, the first message can start at time 00:10:01, the second message can start at 00:10:02, and the third message can start at 00:10:01. The server computer may perform an interest score (eg, based on user interest), or an interest score (eg, based on user interest), based on quality scores of the messages (eg, image clarity, audio clarity, steady state of the stream, etc.) indicating the level of quality of the message. Based on other means, or a combination of these methods, a starting message may be selected from these messages by random selection.

In one example, the server computer determines all possible starting points for a stitch by considering messages or data streams from messages that appear only in front of all matches. The server computer may then run a recursive breadth-first search starting at each starting point to create a dense audio stitch from the audio matches.

In another example, the server computer may optimize stitches. For example, optimized stitches may include the least number of messages or data streams from those needed to span an entire dense stitch. These can be efficiently calculated for each dense stitch using a greedy algorithm. For example, starting with a first message or data stream, the server computer can determine the longest message or data stream that intersects the first message to select a second message. The server computer can then determine the longest message or data stream that intersects the second message to select a third message, and so on until the entire common audio timeline is complete.

10 depicts a visual representation of a stitched data stream 1000 comprising multiple message or data streams 1002-1030 received from various users via multiple user devices. Each box 1002-1030 represents a message or data stream within stitched data stream 1000 received from a user. For example, each message or data stream within each box 1002-1030 may include a time period during a common audio timeline (e.g., 1002 is the first 10 seconds of the common audio timeline). 1004 can be the next 10 seconds of the common audio timeline, 1006 can be the next 5 seconds of the common audio timeline, etc.). In this example, stitched data stream 1000 includes messages from a live audio event.

11 shows another visual representation of a stitched data stream 1100 comprising multiple messages or data streams received from various users via multiple user devices. As in FIG. 10, each box in FIG. 11 represents a message or data stream within the stitched data stream 1100 received from the user. In this example, stitched data stream 1100 includes messages from a live sporting event.

In one example, a stitched data stream may include prioritized messages based on a random selection of the order of display in a timeline. In another example, a stitched data stream may include prioritized messages according to one or more rules. For example, a stitched data stream may include messages that are prioritized based on a quality score for display order in a common audio timeline. In one example, each message or data stream may be analyzed to determine a score for the quality of the data stream based on image quality, shakiness of the data stream, audio quality, light, and the like.

In another example, the stitched data stream may include prioritized messages based on a user or user profile of the user associated with the user computing device. For example, the stitched data stream may include messages from the user or other users associated with the user (eg, friends, family, other users "followed" by the user, etc.). The server computer may prioritize the selection of messages created or transmitted by the user or other users associated with the user.

Referring again to FIG. 6 , at operation 616 , the server computer provides the stitched data stream to one or more user computing devices. In one example, a server computer may transmit the stitched data stream to one or more computing devices. In another example, a server computer, such as through a link or other means, that allows a user computing device to access a stitched data stream on the server computer (eg, stream the stitched data stream from the server computer). , can provide access to stitched data streams from one or more computing devices.

One or more user devices may display the stitched data stream on the display of one or more user devices, as shown in the examples of FIGS. 13-15 . In one example, the stitched data stream is displayed on the user device as a continuous data stream transitioning from one message data stream to the next as a continuous common audio stream. In one example, the audio from each data stream of the stitched data stream is averaged to provide a better quality audio stream. Thus, the continuous common audio stream may include audio that is the average of the audio associated with each data stream in the stitched data stream to provide a better quality audio stream, or the continuous audio stream may include audio in a plurality of messages. It may contain the original audio received.

Exemplary embodiments allow a user to switch a display between alternating data streams during a given time period in a stitched data stream. Using the example illustrated in FIGS. 13-15 , a user may be viewing a display 1306 of a stitched data stream for a recent concert on computing device 1302 . Display 1306 may show a lead vocal at a concert, and the user may wish to see other views at that time in the audio timeline. In one example, display 1306 may somehow indicate that there are alternating views of the data stream currently being viewed (eg, via buttons 1304, highlighted frames, pop-up messages, or other indicators). The user moves the device (eg, shakes, tilts, etc.), touches the device's display 1306 (eg, touches a button or link on the display 1306), or in some other way he or she alternates display You can provide an indication that you want to see . After receiving an indication from the user, the user computing device 1302 may show an audience as shown in FIG. 14, a guitarist in a band as shown in FIG. 15, or an alternate view (e.g., another display). For example, a data stream) may be displayed. The stitched data stream will then play continuously from that view to the end of that view, then transition to the next view that aligns with that view, and so on. Accordingly, the display of the stitched data stream transitions to the next message data stream when the previous data stream ends or when the user indicates that he/she wishes to see the alternating display.

12 is a flow diagram illustrating aspects of a method 1200 for providing an alternating data stream for display on a user computing device (eg, client device 102 ), according to some demonstrative embodiments. For purposes of illustration, method 1200 is described with respect to networked system 100 of FIG. 1 . It should be understood that method 1200 may be practiced with other system configurations in other embodiments.

At operation 1202, the computing device receives a request for an alternating data stream. The computing device in these examples may be a user computing device (eg, client device 102) or a server computer (eg, a server computer associated with server system 118). For example, during display on a user computing device of a stitched data stream comprising a plurality of separate data streams associated with a common audio timeline, the user computing device or server computer may (e.g., the stitched data stream be connected to the user computing device). Active data currently being displayed on the computing device (depending on whether it is on the user computing device or on the server computer, whether streamed from the server computer, and/or whether functionality to provide alternating data streams exists on the user computing device or on the server computer) A request for an alternating data stream among a plurality of data streams other than the stream may be received. As described in more detail above, a plurality of data streams associated with a common audio timeline are received from a plurality of computing devices and stitched together to form a stitched data stream. Also as noted above, a common audio timeline may be for audio content, in one example, an average of the audio associated with multiple individual data streams of the stitched data streams.

In one example, a user may indicate a desire to switch to an alternate data stream on a display on the user computing device, as described above. For example, a user computing device may receive an indication from a user of the computing device to switch to an alternate data stream (eg, via motion input, touch input, etc.), as described above. In one example, the user computing device may send a request to server system 108 or, in another example, process the request itself. When a user computing device sends a request to server system 108, the computing device, such as a server computer, receives the request for an alternate data stream. A user computing device receives a request for an alternating data stream if the user computing device processes the request itself.

In operation 1204, the computing device determines a subset of a plurality of individual data streams of individual data streams of the stitched data stream associated with the time period of the active data stream in a common audio timeline. For example, the computing device can determine the time period of an active data stream based on data received in a request for an alternate data stream. In one example, the computing device can determine that an active data stream is associated with the time period 00:30 to 00:33 of the common audio timeline. A request for an alternating data stream may include a time period, or a current timestamp (e.g., 00:30:49, 00:32:10, etc.) of which the active data stream was being displayed when the request was sent. can include The computing device may analyze the plurality of data streams to determine which of the plurality of data streams includes a time period overlapping a time period of an active data stream.

In operation 1206, the computing device selects an alternating data stream from a subset of a plurality of individual data streams of the stitched data streams associated with the time period of the active data stream in a common audio timeline. In one example, to select an alternating data stream from the subset of the plurality of data streams, the computing device determines a quality for each data stream of the subset of the plurality of data streams of the stitched data streams associated with the time period of the active data stream. score can be determined. In this example, the computing device selects an alternate data stream based on a quality score for the alternate data stream. For example, the computing device may select the data stream with the highest quality score for an alternate data stream. The quality score, in one example, may be based on quality scores of videos or images within the data stream and/or interest scores of the data stream for a plurality of users. In another example, the computing device may randomly select an alternating data stream from a subset of the plurality of data streams.

At operation 1208, the computing device can provide an alternating data stream for displaying the computing device. In one example, a server computer may provide an alternating data stream to a user computing device for display to a user. In another example, a user computing device may display alternating data streams to a user.

The display of active data streams on the computing device may transition from a common audio timeline to alternating data streams on the computing device. In this way, the user can view alternating data streams as described above. The display of the stitched data stream continues from the alternating data stream and transitions to the next data stream of the plurality of data streams occurring after the alternating data stream has ended on the common audio timeline.

16 is a block diagram 1600 illustrating a software architecture 1602 that may be installed on any one or more of the devices described above. For example, in various embodiments, server systems 108, including server systems 110, 112, 114, 116, 118, 122, and 124, and client devices 102 may have software architecture 1602 ) may be implemented using some or all of the elements of 16 is only a non-limiting example of a software architecture, and it will be appreciated that many other architectures may be implemented to facilitate the functionality described herein. In various embodiments, software architecture 1602 is implemented by hardware, such as machine 1700 of FIG. 17 including processor 1710 , memory 1730 , and I/O components 1750 . In this example, software architecture 1602 can be conceptualized as a stack of layers where each layer can provide specific functionality. For example, software architecture 1602 includes layers such as operating system 1604 , libraries 1606 , frameworks 1608 , and applications 1610 . During operation, applications 1610 invoke application programming interface (API) calls 1612 via the software stack and send messages 1614 in response to API calls 1612, in accordance with some embodiments. receive

In various implementations, operating system 1604 manages hardware resources and provides common services. Operating system 1604 includes, for example, kernel 1620 , services 1622 , and drivers 1624 . Kernel 1620 acts as an abstraction layer between hardware and other software layers according to some embodiments. For example, kernel 1620 provides memory management, processor management (eg, scheduling), component management, networking, and security settings, among other functionality. Service 1622 can provide other common services for other software layers. According to some embodiments, driver 1624 is responsible for controlling or interfacing with the underlying hardware. For example, drivers 1624 may include display drivers, camera drivers, BLUETOOTH® or BLUETOOTH® Low Energy drivers, flash memory drivers, serial communication drivers (eg, Universal Serial Bus (USB) drivers). ), WI-FI® drivers, audio drivers, power management drivers, and the like.

In some embodiments, libraries 1606 provide a common low-level infrastructure utilized by applications 1610 . Libraries 1606 may include system libraries 1630 (e.g., the C standard library), which may provide functionality such as memory allocation functions, string manipulation functions, mathematical functions, and the like. . In addition, the libraries 1606 include media libraries (eg, Moving Picture Experts Group-4 (MPEG4), H.264 or Advanced Video Coding (AVC), Moving Picture Experts Group Layer-3 (MP3), AAC (Advanced Audio Coding), AMR (Adaptive Multi-Rate) audio codec, libraries that support the presentation and manipulation of various media formats such as JPEG or JPG (Joint Photographic Experts Group), PNG (Portable Network Graphics), graphics library (e.g., the OpenGL framework used to render graphical content to displays in two dimensions (2D) and three dimensions (3D)), database libraries (e.g., SQLite, which provides various relational database functions), API libraries 1632, such as web libraries (e.g., WebKit, which provides web browsing functionality), and the like Libraries 1606 also provide applications 1610 with many other APIs. ) may include a wide variety of other libraries 1634 provided.

Frameworks 1608 provide a high-level common infrastructure that can be utilized by applications 1610 according to some embodiments. For example, the frameworks 1608 provide various graphical user interface (GUI) functionality, high-level resource management, high-level location services, and the like. Frameworks 1608 may provide a wide range of other APIs that may be utilized by applications 1610 , some of which may be specific to a particular operating system 1604 or platform.

In an example embodiment, applications 1610 include home application 1650, contacts application 1652, browser application 1654, books reader application 1656, location application 1658, media application 1660, messaging applications 1662, game applications 1664, and a wide range of other applications such as third party applications 1666 and media content applications 1667. According to some embodiments, application 1610 is a program that executes functions defined in the program. Various programming languages are structured in various ways, such as object-oriented programming languages (eg, Objective-C, Java, or C++) or procedural programming languages (eg, C or assembly language). It can be adapted to create one or more of the applications 1610 . In certain instances, third party applications 1666 (e.g., applications developed using an ANDROID™ or IOS™ software development kit (SDK) by an entity other than the vendor of a particular platform) may be IOS™, ANDROID ™, WINDOWS® Phone, or another mobile operating system. In this example, third party application 1666 can invoke API calls 1612 provided by operating system 1604 to facilitate the functionality described herein.

As noted above, some embodiments may specifically include a messaging application 1662. In certain embodiments, this may be a standalone application that operates to manage communications with a server system, such as server system 108 . In other embodiments, this functionality may be integrated with another application, such as media content viewing application 1667. Messaging application 1662 may request and display various media content items, allowing a user to enter data related to media content items via a touch interface, keyboard, or using the camera device of machine 1700; I/O components 1750 may provide communication with server system 108 and reception and storage of media content items in memory 1730 . The presentation of media content items and user inputs associated with the media content items is messaging using different frameworks 1608, library 1606 elements, or operating system 1604 elements running on machine 1700. Can be managed by application 1662.

17 illustrates a machine 1700 capable of reading instructions from a machine readable medium (eg, a machine readable storage medium) to perform any one or more of the methodologies discussed herein, in accordance with some embodiments. It is a block diagram illustrating the components of Specifically, FIG. 17 shows a schematic diagram of a machine 1700 in an exemplary form of a computer system, within which machine 1700 may be used to perform any one or more of the methodologies discussed herein. instructions 1716 (eg, software, programs, applications 1610, applets, apps, or other executable code) may be executed. In an alternative embodiment, machine 1700 may operate as a stand-alone device or be coupled (eg, networked) to other machines. In a networked deployment, machine 1700 is a server system (108, 110, 112, 114, 116, 118, 122, 124, etc.) or in the capacity of a client device 102 in a server-client network environment, or peer-to-peer. It can act as a peer machine in a two-peer (or distributed) network environment. Machine 1700 is a server computer, client computer, personal computer (PC), tablet computer, laptop computer, netbook, personal digital assistant (PDA), entertainment media system, cellular phone, smart phone, mobile device, wearable device (eg eg, a smart watch), a smart home device (eg, a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or instructions that specify actions to be taken by the machine 1700. can be any machine capable of executing s 1716 sequentially or otherwise, but is not limited to these. Further, although only one machine 1700 is shown, the term "machine" refers to a set of machines 1700 that individually or jointly execute instructions 1716 to execute any one or more of the methodologies discussed herein. It can also be taken to include collections.

In various embodiments, machine 1700 includes processor 1710 , memory 1730 , and I/O components 1750 that may be configured to communicate with each other over bus 1702 . In an exemplary embodiment, a processor 1710 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor) (Digital Signal Processor, DSP), Application Specific Integrated Circuit (ASIC), Radio Frequency Integrated Circuit (RFIC), another processor, or any suitable combination thereof) may be, for example, instructions 1716 It includes a processor 1712 and a processor 1714 capable of executing. The term "processor" is intended to include multi-core processor 1710, which may include two or more independent processors 1712, 1714 (also referred to as "cores") capable of executing instructions 1716 concurrently. . 17 shows multiple processors 1710, the machine 1700 can be a single processor 1710 having a single core, a single processor 1710 having multiple cores (e.g., a multi-core processor 1710), multiple processors 1712 and 1714 with a single core, multiple processors 1710 and 1712 with multiple cores, or any combination thereof.

According to some embodiments, memory 1730 includes main memory 1732 , static memory 1734 , and storage unit 1736 accessible to processor 1710 over bus 1702 . The storage unit 1736 may include a machine readable medium 1738 having stored thereon instructions 1716 implementing any one or more of the methodologies or functions described herein. Instructions 1716 may also, during execution by machine 1700, wholly or at least partially within main memory 1732, within static memory 1734, within at least one of processors 1710 (e.g. , in the cache memory of the processor), or in any suitable combination thereof. Thus, in various embodiments, main memory 1732 , static memory 1734 , and processors 1710 are considered machine-readable media 1738 .

As used herein, the term "memory" refers to any machine-readable medium 1738 capable of temporarily or permanently storing data, such as random access memory (RAM), read-only memory (ROM), buffer memory, It is taken to include, but is not limited to, flash memory, and cache memory. Although machine-readable medium 1738 is shown as being a single medium in an exemplary embodiment, the term “machine-readable medium” is used to refer to a single medium or multiple mediums (e.g., central centralized or distributed databases, or associated caches and servers). The term “machine-readable medium” refers to instructions 1716 that, when executed by one or more processors (eg, processors 1710 ) of machine 1700 , are associated with machine 1700 instructions (e.g., instructions 1716) for execution by a machine (e.g., machine 1700) to cause ) to perform any one or more of the methodologies described herein. should also be taken to include any medium, or combination of multiple mediums, capable of storing Accordingly, a "machine-readable medium" refers to a single storage device or device, as well as a "cloud-based" storage system or storage network comprising a plurality of storage devices or devices. Accordingly, the term "machine readable medium" is intended to include, but is not limited to, solid state memory (e.g., flash memory), optical media, magnetic media, other non-volatile memory (e.g., Erasable Programmable Read-Memory (EPROM)), Only Memory)), or any suitable combination thereof. The term "machine-readable medium" specifically excludes non-statutory signals per se.

I/O components 1750 include a wide variety of components that receive input, provide output, generate output, transmit information, exchange information, capture measurements, and the like. In general, it will be appreciated that I/O component 1750 may include many other components not shown in FIG. 17 . I/O components 1750 are grouped according to functionality merely to simplify the discussion below, and this grouping is not limiting in any way. In various illustrative embodiments, I/O component 1750 includes an output component 1752 and an input component 1754. The output component 1752 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a CRT), an acoustic component (e.g., speakers), haptic components (eg, vibration motors), and other signal generators. Input components 1754 may include alphanumeric input components (eg, a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), a point-based input component (e.g., a mouse, touchpad, trackball, joystick, motion sensor, or other pointing device), tactile input components (e.g., a physical button, a touch screen that provides the position and force of touches or touch gestures, or other tactile input components), audio input components (eg, microphone), and the like.

In some further exemplary embodiments, I/O component 1750 may include biometric component 1756, motion component 1758, environment component 1760, or position component 1762, among a wide variety of other components. include For example, the biometrics component 1756 detects expressions (e.g., hand expressions, facial expressions, voice expressions, gestures, or eye tracking), and detects biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), identify a person (eg, voice identification, retina identification, face identification, fingerprint identification, or electroencephalography-based identification), and the like. Motion component 1758 includes an acceleration sensor component (eg, accelerometer), a gravity sensor component, a rotation sensor component (eg, gyroscope), and the like. Environment component 1760 may include, for example, a light sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (eg, one or more microphones to detect background noise), a proximity sensor component (eg, an infrared sensor to detect nearby objects), a gas sensor component (eg, a machine olfactory detection sensors, gas detection sensors that detect concentrations of dangerous gases for safety or measure pollutants in the air), or other components that can provide indications, measurements, or signals in response to the physical environment around them. can include The location component 1762 may include a location sensor component (eg, a GPS receiver component), an altitude sensor component (eg, an altimeter or barometer that detects air pressure from which an altitude may be derived), an orientation sensor component (eg, a magnetometer), etc. includes

Communication can be implemented using a wide variety of technologies. I/O component 1750 may include a communications component 1764 operable to couple machine 1700 to network 1780 or device 1770 via coupling 1782 and coupling 1772, respectively. there is. For example, communication component 1764 includes a network interface component or another device suitable for interfacing with network 1780 . In a further example, communication component 1764 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, a BLUETOOTH® component (e.g., BLUETOOTH® Low Energy), a WI-FI® component, and other communication components that provide communication through other aspects. Device 1770 may be another machine 1700 or any of a wide variety of peripheral devices (eg, peripheral devices coupled via a Universal Serial Bus (USB)).

Additionally, in some embodiments, communication component 1764 detects an identifier or includes a component operable to detect an identifier. For example, the communications component 1764 may include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., a one-dimensional barcode such as a universal product code (UPC) barcode, a QR (Quick Response) code, Aztec code, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra code, UCC RSS (Uniform Commercial Code Reduced Space Symbology)-2D barcode, and other optical codes (optical sensors that detect multi-dimensional barcodes), acoustic a detection component (eg, a microphone that identifies tagged audio signals), or any suitable combination thereof. In addition, location via internet protocol (IP) geo-location, location via WI-FI® signal triangulation, detecting a BLUETOOTH® or NFC beacon signal that may indicate a specific location. Various information may be derived via communication components 1764, such as a location via communication.

In various example embodiments, one or more portions of network 1780 may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network. (local area network, LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (WWAN), metropolitan area network (MAN), Internet, Internet part of a public switched telephone network (PSTN), plain old telephone service (POTS) network, cellular telephone network, wireless network, WI-FI® network, any other type of network; or a combination of two or more such networks. For example, network 1780 or portions of network 1780 may include a wireless or cellular network, and combination 1782 may include a Code Division Multiple Access (CDMA) connection, a Global System for Mobile communications (GSM) connection, or another type of cellular or wireless coupling. In this example, combining 1782 includes Single Carrier Radio Transmission Technology (1xRTT), Evolution-Data Optimized (EVDO) technology, General Packet Radio Service (GPRS) technology, Enhanced Data rates for GSM Evolution (EDGE) technology, 3G. Third Generation Partnership Project (3GPP), fourth generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Long Term Evolution (LTE) standards, including , others defined by various standards-setting organizations, other long-range protocols, or other data transmission technologies.

In an illustrative embodiment, instructions 1716 may be performed using a transmission medium over a network interface device (eg, a network interface component included in communication component 1764) and any of a number of well-known protocols. is transmitted or received over the network 1780 using one of (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, in another illustrative embodiment, instructions 1716 are transmitted or received using a transmission medium via coupling 1772 to devices 1770 (eg, peer-to-peer coupling). . The term "transmission medium" should be taken to include any intangible medium capable of storing, encoding, or transporting instructions 1716 for execution by machine 1700, whether digital or analog communication signals or other intangible medium that facilitates the communication of such software.

Also, the machine-readable medium 1738 is non-transitory (ie, does not have a transitory signal) in that it does not embody a propagating signal. However, labeling a machine-readable medium 1738 as “non-transitory” should not be interpreted to mean that the medium is not removable; Medium 1738 should be considered transportable from one physical location to another. Additionally, because machine-readable medium 1738 is tangible, medium 1738 can be considered to be a machine-readable device.

Throughout this specification, a plural instance may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed concurrently, and the operations are not necessarily performed in the order illustrated. Structures and functionality presented as separate components in example configurations may be implemented as a combined structure or component. Similarly, structures and functionality disclosed as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements are within the scope of the subject matter herein.

Although an overview of the subject matter of the present invention has been described with reference to specific exemplary embodiments, various modifications and changes may be made to these embodiments without departing from the broader scope of the embodiments of the present disclosure.

The embodiments illustrated herein have been described in sufficient detail to enable any person skilled in the art to practice the teachings disclosed herein. Other embodiments may be used and derived therefrom, so that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Accordingly, this detailed description should not be regarded in a limiting sense, and the scope of the various embodiments is defined only by the appended claims and the full scope of their equivalents.

As used herein, the term “or” may be interpreted in an inclusive or exclusive sense. Moreover, multiple instances may be provided for resources, operations, or structures described herein as a single instance. Additionally, the boundaries between various resources, operations, modules, engines, and data stores are somewhat arbitrary, and specific operations are illustrated in the context of specific example configurations. Other assignments of functionality are contemplated and may fall within the scope of various embodiments of the present disclosure. In general, structures and functionality presented as separate resources in the example configurations may be implemented as a combined structure or resource. Similarly, structure and functionality presented as a single resource may be implemented as separate resources. These and other variations, modifications, additions, and improvements are within the scope of the embodiments of the present disclosure as represented by the appended claims. Accordingly, the present specification and drawings are to be regarded in an illustrative rather than a limiting sense.

Claims (20)

A method performed by at least one processor, comprising:
causing display of the stitched data stream as a continuous data stream transitioning from one message data stream to a next message data stream as a continuous audio stream, wherein the stitched data stream is associated with common audio content and is associated with a plurality of computing devices comprising a plurality of individual message data streams from a plurality of messages received from;
during display of the stitched data stream, receiving a request for an alternate message data stream in the stitched data stream in place of the currently displayed active message data stream; and
the alternating message data stream at the end of the alternating message data stream transitioning the display of the active message data stream on a common audio timeline and the display of the stitched data stream to the next message data stream in the stitched data stream; Steps that cause a transition to display
How to include. 2. The method of claim 1, wherein the stitched data stream comprises overlapping message data streams in time periods of the common audio timeline resulting in more than one message data stream for at least one time period in the stitched data stream. More inclusive, how. 3. The method of claim 2, further comprising: before causing a transition to display the active message data stream to display the alternate message data stream;
selecting the alternating message data stream from the message data streams of the stitched data stream having the time period in the common audio timeline corresponding to the time period of the active message data stream by performing operations;
Including, the operations are
determine a quality score for each one of the message data streams having a time period in the common audio timeline corresponding to the time period of the active message data stream;
Based on the quality score for the alternating message data stream, selecting the alternating message data stream from the message data streams having the time period in the common audio timeline corresponding to the time period of the active message data stream. to do
Including, how. 4. The method of claim 3, wherein the quality score is based on at least one of a quality of video or images in the message data stream or an interest score of the message data stream for a plurality of users. 3. The method of claim 2, further comprising: before causing a transition to display the active message data stream to display the alternate message data stream;
randomly selecting the alternating message data stream from the message data streams having the time period in the common audio timeline corresponding to the time period of the active message data stream; selecting the alternating message data stream from the message data streams of the stitched data stream having the time period in the corresponding common audio timeline.
How to include. According to claim 5,
determining the time period of the active message data stream based on data received in the request for the alternate message data stream;
How to include more. 3. The method of claim 2, further comprising: before causing a transition to display the active message data stream to display the alternate message data stream;
selecting the alternate message data stream by analyzing the message data streams of the stitched data stream to determine message data streams having a time period overlapping with the time period of the active message data stream.
How to include. 2. The method of claim 1, wherein the common audio timeline is for audio content that is an average of audio associated with the plurality of individual message data streams of the stitched data stream. 2. The method of claim 1, wherein the common audio timeline is for original audio received in each message of the plurality of messages. 2. The method of claim 1, wherein each individual message data stream is a short-lived message data stream. 11. The method of claim 10, wherein the access time for each short-term message data stream is set by the sender of the short-term message data stream. 11. The method of claim 10, wherein the access time for each short-term message data stream is based on default settings or settings set by the recipient of the short-term message data stream. As a server computer,
processor; and
Computer readable medium coupled with the processor
wherein the computer readable medium includes stored instructions executable by the processor to cause a computing device to perform operations that:
cause the display of the stitched data stream as a continuous data stream transitioning from one message data stream to the next message data stream as a continuous audio stream, wherein the stitched data stream is associated with common audio content and from multiple computing devices including a plurality of individual message data streams from a plurality of received messages;
during display of the stitched data stream, receive a request for an alternate message data stream in the stitched data stream instead of the currently displayed active message data stream;
of the stitched data stream at the end of the alternating message data stream transitioning the display of the active message data stream on a common audio timeline and the display of the stitched data stream to the next message data stream in the stitched data stream. Causing a transition to display the alternating message data stream.
Including, server computer. 14. The method of claim 13, wherein the stitched data stream comprises overlapping message data streams in time periods of the common audio timeline resulting in more than one message data stream for at least one time period in the stitched data stream. Further comprising, a server computer. 15. The method of claim 14, wherein prior to causing a transition to display the active message data stream to display the alternating message data stream, the actions
determine a quality score for each one of the message data streams having a time period in the common audio timeline corresponding to the time period of the active message data stream;
Based on the quality score for the alternating message data stream, selecting the alternating message data stream from the message data streams having the time period in the common audio timeline corresponding to the time period of the active message data stream. to do
selecting the alternating message data stream from the message data streams of the stitched data stream having the time period in the common audio timeline corresponding to the time period of the active message data stream by performing operations comprising: thing
including,
and the quality score is based on at least one of a quality of video or images in the message data stream or an interest score of the message data stream for a plurality of users. 15. The method of claim 14, wherein prior to causing a transition to display the active message data stream to display the alternating message data stream, the operations
randomly selecting the alternating message data stream from the message data streams having the time period in the common audio timeline corresponding to the time period of the active message data stream; selecting the alternating message data stream from the message data streams of the stitched data stream having the time period in the corresponding common audio timeline.
Including, server computer. 17. The method of claim 16, wherein the operations
determining the time period of the active message data stream based on data received in the request for the alternate message data stream;
Further comprising a server computer. 15. The method of claim 14, wherein prior to causing a transition to display the active message data stream to display the alternating message data stream, the operations
selecting the alternating message data stream by analyzing the message data streams of the stitched data stream to determine message data streams having a time period overlapping with the time period of the active message data stream.
Including, server computer. 14. The method of claim 13, wherein the common audio timeline is for audio content that is an average of audio associated with the plurality of individual message data streams of the stitched data stream or original audio received in each message of the plurality of messages. For that, the server computer. A non-transitory computer-readable medium containing stored instructions executable by at least one processor to cause a computing device to perform operations comprising:
cause the display of the stitched data stream as a continuous data stream transitioning from one message data stream to the next message data stream as a continuous audio stream, wherein the stitched data stream is associated with common audio content and from multiple computing devices contains a plurality of individual message data streams from received messages;
during display of the stitched data stream, receive a request for an alternate message data stream in the stitched data stream instead of the currently displayed active message data stream;
of the stitched data stream at the end of the alternating message data stream transitioning the display of the active message data stream on a common audio timeline and the display of the stitched data stream to the next message data stream in the stitched data stream. Causing a transition to display the alternating message data stream.
A non-transitory computer readable medium comprising a.

Images