KR102387433B1 - Generating a stitched data stream - Google Patents

Abstract

The systems and methods provide that a server computer receives a plurality of messages from a plurality of user computing devices, each message of the plurality of messages comprising a data stream, of the messages of the plurality of messages associated with a similar geolocation and time period. determine a subset, determine a set of messages in the subset of messages based on a matching score for each pair of messages, and determine a set of messages in a subset of messages for each message in a set of messages based on a time period for each message. provides for stitching together a set of messages to create a stitched data stream from the data streams, wherein the stitched data stream overlaps in time periods such that there may be more than one data stream during a given time period. Contains messages with

Description

GENERATING A STITCHED DATA STREAM

This patent application claims priority to U.S. Application Serial No. 15/470,004, filed March 27, 2017, and also claims priority to U.S. Patent Application Serial No. 15,470,025, filed March 27, 2017; , the contents of these applications are incorporated herein by reference in their entirety.

A messaging system can receive millions of messages from users who wish to share media content such as audio, images, and video among user devices (eg, mobile devices, personal computers, etc.). The media content of these messages may be associated with a common geolocation, a common time period, a common event, and the like.

Various of the accompanying drawings represent only exemplary embodiments of the present disclosure and should not be construed as limiting the scope thereof.
1 is a block diagram illustrating an example messaging system for exchanging data (eg, messages and associated content) over a network, in accordance with some demonstrative embodiments.
2 is a block diagram illustrating additional details regarding a messaging system, in accordance with some demonstrative embodiments.
3 is a schematic diagram illustrating data that may be stored in a database of a messaging server system, in accordance with some demonstrative embodiments.
4 is a schematic diagram illustrating the structure of a message generated by a messaging client application for communication, in accordance with some embodiments;
5 illustrates that access to content (eg, short-term messages, and associated multimedia payloads of data) or content collections (eg, short-term message stories) may be time-limited (eg, short-lived); ), a schematic diagram illustrating the access-restriction process.
6 is a flowchart illustrating an aspect of a method in accordance with some demonstrative embodiments.
7 shows an example of a spectrogram in accordance with some demonstrative embodiments.
8 shows an example of detected maxima of a spectrogram, in accordance with some demonstrative embodiments.
9 shows an example of maxima of a spectrogram linked together in a final audio fingerprint, in accordance with some demonstrative embodiments.
10 and 11 each illustrate a visual representation of a stitched data stream comprising a plurality of messages received from a plurality of user devices, in accordance with some demonstrative embodiments.
12 is a flow diagram illustrating aspects of a method in accordance with some demonstrative embodiments.
13-15 each depict an example user interface of a user computing device in accordance with some example embodiments.
16 is a block diagram illustrating an example of a software architecture that may be installed on a machine, in accordance with some demonstrative embodiments.
17 is a schematic representation of a machine in the form of a computer system into which a set of instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein, in accordance with an illustrative embodiment; to exemplify

The systems and methods described herein relate to handling media content items to be shared between devices via a messaging system. For example, a user may wish to share one or more videos, images, and the like with one or more other users. These messages may be associated with common audio content such as songs, concerts, voices, and the like. Embodiments described herein provide for stitching together data streams from a plurality of messages received from a plurality of user computing devices to create dense audio stitches associated with a common audio timeline for the data streams. provide a mechanism. For example, audio stitching is used to automatically stitch together sets of messages or data streams associated with the same audio content. Given a set of messages or data streams, the system described herein extracts audio fingerprints for each message and spans all pairs of messages to identify messages associated with the same audio content. Audio stitches are created by creating audio matches by matching them, and then finding paths through the set of audio matches. The stitched audio or stitched data stream may then be provided to one or more user computing devices for users to view the stitched data stream.

A stitched data stream may include messages having overlapping data streams in time periods such that there may be more than one data stream during a given time period. As the user is viewing the stitched data stream, the example embodiments allow the user to switch to various other data streams for any given period of time. For example, a user may be watching the lead vocalist at a concert and then switch the view to see the guitarist, or drummer, or audience, etc. at that point in the concert. In this way, the user can switch between alternating views at any given period of time in a common audio timeline.

1 is a block diagram illustrating a networked system 100 (eg, a messaging system) for exchanging data (eg, messages and associated content) over a network. Networked system 100 includes multiple client devices 102 , each hosting multiple client applications 104 . Each client application 104 is communicatively coupled to other instances of the client application 104 and server system 108 via a network 106 .

Client device 102 may also be referred to herein as a user device or a user computing device. The client device 102 may be a mobile phone, desktop computer, laptop, personal digital assistant (PDA), smartphone, tablet, ultrabook, netbook, laptop, multiprocessor system, microprocessor-based or programmable consumer electronics device, game console, It may include, but is not limited to, a set top box, a computer in a vehicle, or any other communication device that a user may use to access the networked system 100 . In some embodiments, the client device 102 may include a display module (not shown) that displays information (eg, in the form of user interfaces). In further embodiments, the client device 102 may include one or more of touch screens, accelerometers, gyroscopes, cameras, microphones, global positioning system (GPS) devices, and the like. The client device 102 is used to create media content items such as video, images (eg, photos), audio, and to send and receive messages to and from other users that contain media content items such as such. It may be the user's device. Elements of such media content from multiple messages may then be stitched together as described in greater detail in the embodiments described below.

One or more users may interact with the client device 102 (eg, a person, machine, or other means of interacting with the client device 102 ). In example embodiments, a user may not be part of the system 100 , but may interact with the system 100 via a client device 102 or other means. For example, a user may provide input (eg, touch screen input or alphanumeric input) to the client device 102 , and the input may be provided to the system 100 (eg, a server system) via the network 106 . 108, etc.). In such a case, other entities within the system 100 may, in response to receiving input from the user, communicate information to the client device 102 over the network 106 for presentation to the user. In this way, a user may interact with various entities within the system 100 using the client device 102 .

System 100 may further include a network 106 . One or more portions of network 106 may include ad hoc networks, intranets, extranets, virtual private networks (VPNs), local area networks (LANs), wireless LANs (WLANs), wide area networks (WANs), wireless WANs (WWANs), urban areas network (MAN), part of the Internet, part of the Public Switched Telephone Network (PSTN), a cellular telephone network, a wireless network, a WiFi network, a WiMax network, another type of network, or a combination of two or more such networks can be

Client device 102 may communicate with other entities within system 100 via a web client (eg, a browser such as Internet Explorer® browser developed by Microsoft® Corporation of Redmond, Wash.) or one or more client applications 104 . access to a variety of data and applications provided by As noted above, the client device 102 may be configured to include, but is not limited to, a web browser, a messaging application, an e-mail application, an e-commerce site application, a mapping or location application, a media content editing application, a media content viewing application, and the like. It may include one or more client applications 104 (also referred to as “apps”) that do not.

In one example, the client application 104 is a messaging application that allows a user to take a photo or video, add a caption, or otherwise edit the photo or video, and then send the photo or video to another user. can The message may be ephemeral and may be removed from the receiving user device after view or after a predetermined amount of time (eg, 10 seconds, 24 hours, etc.). A short-lived message refers to a message that is accessible for a time-limited duration. A short-lived message may be text, images, video, and other such content that may be stitched together according to embodiments described herein. The access time for short-lived messages may be set by the sender of the message. Alternatively, the access time may be a default setting or a setting specified by the recipient. Regardless of the setting technique, the message is temporary.

The messaging application may further allow the user to create a gallery. A gallery is a collection of photos and videos that can be viewed by other users that other users "follow" the user's gallery (eg, subscribe to view and receive updates in the user's gallery). can A gallery may also be short-lived (eg, lasting 24 hours, lasting for the duration of the event (eg, during a music concert, sporting event, etc.), or lasting for another predetermined amount of time).

The short-term message may be associated with a message duration parameter, the value of which determines the amount of time the short-term message will be displayed by the client application 104 to the receiving user of the short-term message. The short-lived message may further be associated with a message receiver identifier and a message timer. A message timer may be responsible for determining the amount of time a short-lived message is shown to a particular recipient user identified by the message receiver identifier. For example, a short-lived message may be visible to the relevant recipient user only for a period of time determined by the value of the message duration parameter.

In another example, a messaging application may allow a user to store photos and videos and create a gallery that is not short-lived and that can also be sent to other users. For example, collect photos and videos from recent vacations and share them with friends and family.

In some embodiments, one or more client applications 104 may be included in a given one of the client devices 102 , and as needed, data and/or processing capabilities that are not locally available (eg, system 100 for accessing location information, authenticating users, validating payment methods, accessing media content stored on a server, synchronizing media content between client device 102 and a server computer, etc.) ) (eg, server system 108 ) may be configured to locally provide an application 104 configured to communicate with at least some of a user interface and functionality. Conversely, one or more applications 104 may not be included on the client device 102 , which then uses its web browser to access the system 100 (eg, the server system 108 ). One or more applications hosted on other entities may be accessed.

The server system 108 may provide server-side functionality to one or more client devices 102 over a network 106 (eg, the Internet or a wide area network (WAN)). The server system 108 includes an application programming interface (API) server 110 , an application server 112 , a messaging server application 114 , and a media content processing server 116 , a social networking system 122 , and data stream stitching. It may include system 124 , each of which may be communicatively coupled to each other and one or more data storage(s), such as database(s) 120 .

The server system 108 may be a cloud computing environment according to some demonstrative embodiments. The server system 108 , and any servers associated with the server system 108 may be associated with a cloud-based application in one exemplary embodiment. The one or more database(s) 120 may store raw media content, original media content from users (eg, high-quality media content), processed media content (eg, client devices 102 ) and storage devices that store information such as formatted media content for sharing and viewing on client devices 102), stitched audio data streams, user information, user device information, and the like. The one or more database(s) 120 may include cloud-based storage external to the server system 108 (eg, hosted by one or more third-party entities external to the server system 108 ). Although storage devices are shown as database(s) 120 , it is understood that system 100 may access and store data in storage devices such as database 120 , blob storage, and other types of storage methods. do.

Accordingly, each client application 104 may communicate and exchange data with other client applications 104 and with the server system 108 over the network 106 . Data exchanged between messaging client applications 104 and between client application 104 and server system 108 includes functions (eg, commands that invoke functions) as well as payload data (eg, for example, text, audio, video or other multimedia data).

Server system 108 provides server-side functionality to specific messaging client applications 104 over network 106 . Although specific functions of the system 100 are described herein as being performed by the client application 104 or by the server system 108 , the location of the specific functionality within the client application 104 or server system 108 . is a design option. For example, it is technically desirable to initially deploy certain technologies and functionality within the server system 108 , but later transfer these technologies and functionality to the client application 104 when the client device 102 has sufficient processing capacity. can do.

The server system 108 supports various services and operations provided to the client application 104 . Such operations include sending data to, receiving data from, and processing the data generated by the client application 104 . This data may include, for example, message content, client device information, geolocation information, media annotations and overlays, message content persistence conditions, social network information, and live event information, date and timestamp. Data exchange within the networked system 100 is initiated and controlled through functions available through user interfaces (UIs) of the client application 104 .

In the server system 108 , an application program interface (API) server 110 is coupled to the application server 112 and provides a programmatic interface to the application server. Application server 112 is communicatively coupled to database server 118 , which facilitates access to database 120 in which data associated with messages processed by application server 112 is stored.

The API server 110 server receives and transmits message data (eg, commands and message payloads) between the client device 102 and the application server 112 . Specifically, API server 110 is a set of interfaces (eg, routines and protocols) that can be called or queried by client application 104 to invoke functionality of application server 112 . provides API server 110 is responsible for account registration, login functionality, sending messages from one client application 104 to another client application 104 , via application server 112 , and messaging application server from client application 104 . Setting up a collection of media data (eg, a story) for transmission of media files (eg, image or video) to 114 , and possible access by another client application 104 , a client device retrieval of the user's list of friends of 102 , retrieval of such collections, retrieval of messages and content, addition and deletion of friends to and from the social graph, location of friends within the social graph, (eg, client application 104 ) ) exposes various functions supported by the application server 112 , including opening and application events, and the like.

Application server 112 hosts a number of applications and subsystems, including messaging server application 114 , media content processing system 116 , social networking system 122 , and data stream stitching system 124 . The messaging server application 114 handles multiple messages, particularly related to the aggregation and other processing of content (eg, text and multimedia content) included in messages received from multiple instances of the messaging client application 104 . Implement techniques and functions. As will be described in greater detail, text and media content from multiple sources may be aggregated into collections of content (eg, called stories or galleries). These collections are then made available to the client application 104 by the messaging server application 114 . Other processor and memory intensive processing of data may also be performed server-side by messaging server application 114, taking into account the hardware requirements for such processing.

The application server 112 also includes a media content processing system 116 dedicated to performing various media content processing operations, typically with respect to an image or video received within the payload of a message at the messaging server application 114 . do. The media content processing system 116 may access one or more data storage (eg, database(s) 120 ) to retrieve stored data for use in processing the media content and to store the results of the processed media content. can

The social networking system 122 supports various social networking functions and services and makes these functions and services available to the messaging server application 114 . To this end, social networking system 122 maintains and accesses entity graph 304 in database 120 . Examples of functions and services supported by social-networking system 122 include identification of other users of networked system 100 with which a particular user has a relationship or "follow", and identification of other entities and interests of a particular user. include matters

Application server 112 is communicatively coupled to database server 118 , which facilitates access to one or more database(s) 120 in which data associated with messages processed by messaging server application 114 are stored. make it

The messaging server application 114 may be responsible for the creation and delivery of messages between users of the client devices 102 . Messaging application server 114 may utilize any of a number of message delivery networks and platforms to deliver messages to users. For example, the messaging application server 114 may include electronic mail (e-mail), instant message (IM), Short Message Service (SMS), text, facsimile, or voice (e.g., VoIP (VoIP) Voice over IP)) messages over wired (eg, Internet), plain old telephone service (POTS), or wireless networks (eg, mobile, cellular, WiFi, Long Term Evolution (LTE), Bluetooth) can transmit

Data stream stitching system 124 is configured to stitch data streams included in a plurality of messages received from a plurality of user computing devices (eg, client devices 102 ), as described in more detail below. It can be responsible for creating data streams.

2 is a block diagram illustrating additional details regarding system 100 in accordance with example embodiments. Specifically, system 100 is shown including a messaging client application 104 and an application server 112 , which in turn include a number of several subsystems: short-lived timer system 202 , collection management system 204 . , and the annotation system 206 .

Short-lived timer system 202 is responsible for enforcing temporary access to content granted by messaging client application 104 and messaging server application 114 . To this end, the short-term timer system 202 posts messages and associated content via the messaging client application 104 based on display parameters and duration associated with the message, or collection of messages (eg, a SNAPCHAT story). It includes a number of timers that selectively display and enable access to them. Additional details regarding the operation of the short-term timer system 202 are provided below.

The collection management system 204 is responsible for managing collections of media (eg, collections of text, image video and audio data). In some examples, a collection of content (messages including images, video, text, and audio) may be organized into an “event gallery” or “event story”. Such a collection may be made available for a specified period of time, such as the duration of an event to which the content relates. For example, content related to a music concert may be made available as a “story” for the duration of the music concert. The collection management system 204 may also be responsible for posting an icon providing notification of the existence of a particular collection in the user interface of the messaging client application 104 .

Collection management system 204 further includes a curation interface 208 that allows collection managers to manage and curate specific collections of content. For example, the curation interface 208 enables an event organizer to curate a collection of content related to a particular event (eg, delete inappropriate content or duplicate messages). Additionally, the collection management system 204 automatically curates the content collection using machine vision (or image recognition technology) and content rules. In certain embodiments, to include user-generated content in a collection, a reward (eg, money or non-money credits or points, travel miles associated with a communication system or a third-party reward system); access to artwork or specialized lenses, etc.) may be paid to the user. In such cases, the curation interface 208 operates to automatically pay such users for using their content.

Annotation system 206 provides various functions that enable a user to annotate, otherwise modify, or edit media content associated with a message. For example, the annotation system 206 provides functions related to the creation and publication of media overlays for messages processed by the networked system 100 . The annotation system 206 operatively supplies a media overlay (eg, a SNAPCHAT filter) to the messaging client application 104 based on the geolocation of the client device 102 . In another example, the annotation system 206 operatively provides a media overlay to the messaging client application 104 based on other information, such as social network information of a user of the client device 102 . Media overlays can include audio and visual content and visual effects. Examples of audio and visual content include pictures, texts, logos, animations, and sound effects. Examples of visual effects include color overlays. Audio and visual content or visual effects may be applied to media content items (eg, photos) residing on the client device 102 . For example, the media overlay includes text that may be overlaid on top of a photo taken and created by the client device 102 . In another example, the media overlay includes a location identification overlay (eg, Venice Beach), a name of a live event, and a merchant name overlay (eg, Beach Coffee House). In another example, the annotation system 206 uses the geolocation of the client device 102 to identify a media overlay that includes the name of a merchant at the geolocation of the client device 102 . The media overlay may include other indicia associated with the merchant. Media overlays are stored in database 120 and may be accessed through database server 118 .

In one exemplary embodiment, the annotation system 206 provides a user-based publishing platform that enables users to select a geolocation on a map and upload content associated with the selected geolocation. The user may also specify situations in which a particular media overlay should be provided to other users. The annotation system 206 creates a media overlay that includes the uploaded content and associates the uploaded content with the selected geolocation.

In another exemplary embodiment, the annotation system 206 provides a merchant-based publishing platform that enables merchants to select a particular media overlay associated with a geolocation through a bidding process. For example, the annotation system 206 associates the media overlay of the highest bid merchant with a corresponding geolocation for a predefined amount of time.

3 is a schematic diagram 300 illustrating data that may be stored in a database 120 of a server system 108 , in accordance with certain example embodiments. Although the contents of database 120 are shown as including a number of tables, it will be appreciated that data may be stored in other types of data structures (eg, as object-oriented databases).

Database 120 includes message data stored in message table 314 . Entity table 302 stores entity data including entity graph 304 . Entities for which records are maintained in entity table 302 may include individuals, company entities, organizations, objects, places, events, and the like. Regardless of type, any entity for which the server system 108 stores data may be a recognized entity. Each entity has a unique identifier as well as an entity type identifier (not shown).

Entity graph 304 further stores information regarding relationships and associations between entities. Such relationships may be social, professional (eg, work in a general legal entity or organization) interest-based or activity-based, for example only, for example.

Database 120 also stores annotation data in annotation table 312 in the form of exemplary filters. Filters for which data is stored in annotation table 312 are videos (for which data is stored in video table 310 ) and/or images (for which data is stored in image table 308 ). related and applicable to them. In one example, filters are overlays that are displayed overlaid on an image or video during presentation to the receiving user. Filters can be of various types, including user-selected filters from a gallery of filters that are presented to the sending user by the messaging client application 104 when the sending user is composing a message. Other types of filters include geolocation filters (also known as geofilters) that can be presented to a transmitting user based on geographic location. For example, geolocation filters specific to a neighborhood or a particular location may be presented in the user interface by the messaging client application 104 based on geolocation information determined by the GPS unit of the client device 102 . Another type of filter is a data filter that may be selectively presented to a sending user by the messaging client application 104 based on information or other inputs collected by the client device 102 during the message generation process. . Examples of data filters include the current temperature at a particular location, the current speed at which the transmitting user is moving, the battery life for the client device 102 , or the current time.

Another annotation data that may be stored in the image table 308 is so-called “lens” data. "Lens" can be real-time special effects and sounds that can be added to an image or video.

As noted above, video table 310 stores video data associated with messages for which records are maintained in message table 314 , in one embodiment. Similarly, image table 308 stores image data associated with messages for which message data is stored in entity table 302 . Entity table 302 may associate various annotations from annotation table 312 with various images and videos stored in image table 308 and video table 310 .

The story table 306 stores data about a collection of messages and associated image, video or audio data, which are compiled into a collection (eg, a SNAPCHAT story or gallery). Creation of a particular collection may be initiated by a particular user (eg, each user for which a record is maintained in the entity table 302 ). A user may create a “personal story” in the form of a collection of content created and transmitted/broadcast by that user. To this end, the user interface of the messaging client application 104 may include a user selectable icon to enable the sending user to add specific content to their personal story.

Collections may also constitute “live stories,” which are collections of content from multiple users created manually, automatically, or using a combination of manual and automated techniques. For example, a “live story” may constitute a curated stream of user-submitted content from various locations and events. Users whose client devices 102 are location serviceable and are at a common location event at a specific time have the option of contributing content to a specific live story, for example, via the user interface of the messaging client application 104 . can be presented. The live story may be identified to the user by the messaging client application 104 based on its location. The end result is a "live story" told from a community perspective.

An additional type of content collection is a “location story” that enables a user whose client device 102 is located within a particular geographic location (eg, a college or university campus) to contribute to a particular collection. )" is known. In some embodiments, contributing to a location story may require a second degree of authentication to verify that the end user belongs to a particular organization or other entity (eg, is a student on a university campus).

4 is a schematic diagram illustrating the structure of a message 400 generated by a client application 104 for communication to a further client application 104 or messaging server application 114 , according to some of some embodiments. . The content of a particular message 400 is used to populate a message table 314 stored in the database 120 , accessible by the messaging server application 114 . Similarly, the content of message 400 is stored in memory as “in-transit” or “in-flight” data of client device 102 or application server 112 . Message 400 is shown to include the following components:

• Message Identifier 402 : A unique identifier that identifies the message 400 .

· Message text payload 404: text generated by a user through the user interface of the client device 102 and included in the message 400.

· Message Image Payload 406 : Image data captured by the camera component of the client device 102 or retrieved from the memory of the client device 102 and included in the message 400 .

• Message Video Payload 408 : Video data captured by the camera component or retrieved from the memory component of the client device 102 and included in the message 400 .

• Message Audio Payload 410 : Audio data captured by the microphone or retrieved from the memory component of the client device 102 and included in the message 400 .

Message annotations 412: annotation data (e.g. filters) representing annotations to be applied to message image payload 406, message video payload 408, or message audio payload 410 of message 400; , stickers or other reinforcements).

Message duration parameter 414: for which the content of message 400 (eg, message image payload 406, message video payload 408, message audio payload 410) is dependent on the messaging client application A parameter value indicating, in seconds, the amount of time presented or made accessible to the user via (104).

• Message Geolocation Parameters 416: Geolocation data (eg, latitude and longitude coordinates) associated with the content payload of message 400 . Multiple message geolocation parameter 416 values may be included in the payload, each of these parameter values being specific to the content (eg, a specific image in the message image payload 406 , or a specific image in the message video payload 408 ). video) with respect to the content items contained in it.

Message Story Identifier 418: Identifier values that identify one or more collections of content (eg, “stories”) with which a particular content item in the message image payload 406 of the message 400 is associated. For example, each of a plurality of images in the message image payload 406 may be associated with a plurality of content collections using identifier values.

• Message tag 420: Each message 400 may be tagged with a plurality of tags, each representing the subject of the content contained in the message payload. For example, if a specific image included in the message image payload 406 depicts an animal (eg, a lion), a tag value representing the related animal may be included in the message tag 420 . The tag values may be generated manually based on user input, or may be automatically generated using, for example, image recognition.

Message sender identifier 422: an identifier (eg, a messaging system identifier, email address, or device identifier) representing the user of the client device 102 from which the message 400 was generated and from which the message 400 was sent. .

• Message Recipient Identifier 424: an identifier representing the user of the client device 102 to which the message 400 is addressed (eg, a messaging system identifier, email address, or device identifier).

The contents (eg, values) of the various components of message 400 may be pointers to locations within a table in which content data values are stored. For example, the image value in the message image payload 406 may be a pointer (or its address) to a location within the image table 308 . Similarly, values in message video payload 408 may point to data stored in video table 310 , values stored in message annotations 412 may point to data stored in annotation table 312 and , the values stored in the message story identifier 418 may point to data stored in the story table 306 , and the values stored in the message sender identifier 422 and the message receiver identifier 424 are the user stored in the entity table 302 . You can point to records.

5 shows that access to content (eg, short-term message 502 , and associated multimedia payload of data) or content collection (eg, short-term message story 504 ) in that respect will be time-limited (eg, short-lived message story 504 ). It is a schematic diagram illustrating an access-restriction process 500 , which may be (eg, short term).

The short-term message 502 is shown associated with a message duration parameter 506 , the value of which is the time at which the short-term message 502 will be displayed by the client application 104 to the user receiving the short-term message 502 . determine the amount In one embodiment, when the client application 104 is a SNAPCHAT application client, it is short-term by the receiving user for up to 10 seconds, depending on the amount of time the sending user specifies using the message duration parameter 506 . Message 502 is viewable.

A message duration parameter 506 and a message recipient identifier 424 are shown as inputs to a message timer 512 , which indicates that a short-lived message 502 has received a particular receipt identified by the message recipient identifier 424 . Responsible for determining the amount of time shown to the side user. In particular, the short-lived message 502 will only be visible to the relevant recipient user for a period of time determined by the value of the message duration parameter 506 . Message timer 512 is shown providing an output to a more generalized short-term timer system 202 that is responsible for the overall timing of display of content (eg, short-term message 502) to a receiving user.

A short-term message 502 is illustrated in FIG. 5 as being included within a short-term message story 504 (eg, a personal SNAPCHAT story, or an event story). The short-term message story 504 has an associated story duration parameter 508 , the value of which determines the duration of time for which the short-term message story 504 is presented and accessible to users of the networked system 100 . . For example, the story duration parameter 508 may be the duration of a music concert, where the short-term message story 504 is a collection of content related to that concert. Alternatively, a user (owning user or curator user) may specify a value for story duration parameter 508 when performing setup and creation of short-lived message story 504 .

Additionally, each short-lived message 502 in the short-term message story 504 has an associated story engagement parameter 510 whose value is the amount of time the short-term message 502 will be accessible within the context of the short-term message story 504 . determine the duration. Thus, a particular short-term message story 504 "expires" within the context of the short-term message story 504 and is inaccessible before that short-term message story 504 itself expires in terms of the story duration parameter 508 . can be done The story duration parameter 508 , the story engagement parameter 510 , and the message recipient identifier 424 each provide inputs to the story timer 514 , which, in operation, first, Determines whether the short-term message 502 will be displayed to a particular recipient user and, if so, how long it will be displayed. Note that the short-lived message story 504 also recognizes the identity of a particular recipient user as a result of the message recipient identifier 424 .

Thus, the story timer 514 controls the overall lifetime of the individual short-term messages 502 included in the short-term message stories 504 as well as the associated short-term message stories 504 during operation. In one embodiment, each and every short-lived message 502 in the short-term message story 504 remains viewable and accessible for a period of time specified by the story duration parameter 508 . In further embodiments, some short-lived messages 502 may expire within the context of short-term message stories 504 based on story engagement parameters 510 . Note that the message duration parameter 506 can still determine the duration of time that a particular short-term message 502 is displayed to the receiving user, even within the context of the short-term message story 504 . Thus, the message duration parameter 506 determines whether a particular short-term message 502 is transmitted to the recipient side, regardless of whether the receiving-side user is viewing that short-term message 502 within the context of or outside the context of the short-term message story 504 . Determines the duration of time displayed to the user.

Short-lived timer system 202 may further remove certain short-lived messages 502 from short-lived message stories 504 based on a determination that during operation it has exceeded an associated story engagement parameter 510 . For example, when the sending user establishes a story engagement parameter 510 of 24 hours from a posting, the short-term timer system 202 removes the relevant short-term message 502 from the short-term message story 504 after a specified 24 hours. something to do. The short-lived timer system 202 is also configured to detect when the story engagement parameter 510 for each and every short-term message 502 in the short-term message story 504 has expired, or when the short-term message story 504 itself is a story duration parameter. Acts to remove the short-lived message story 504 in either case when it has expired in terms of 508 .

In some use cases, the creator of a particular short-lived message story 504 may specify an indefinite story duration parameter 508 . In this case, the expiration of the story participation parameter 510 for the last remaining short-term message 502 in the short-term message story 504 will determine when the short-term message story 504 itself expires. In this case, the new short-term message 502 added to the short-term message story 504 is made equal to the value of the story participation parameter 510 by the new story participation parameter 510 , so that the lifetime of the short-term message story 504 is equal to the value of the story participation parameter 510 . effectively extend

In response to the short-term timer system 202 determining that the short-term message story 504 has expired (eg, is no longer accessible), the short-term timer system 202 sends the system 100 (and for example Specifically, in communication with the messaging client application 104 ), such that an indication (eg, an icon) associated with the associated short-lived message story 504 is no longer displayed within the user interface of the client application 104 . Similarly, when the short-term timer system 202 determines that the message duration parameter 506 for a particular short-term message 502 has expired, the short-term timer system 202 causes the client application 104 to send the short-term message 502 ) to no longer display the indication (eg, icon or text identification) associated with it.

6 is an aspect of a method 600 for generating a stitched data stream from a plurality of messages received from a plurality of user computing devices (eg, client devices 102 ), in accordance with some demonstrative embodiments. It is a flowchart illustrating For example, the server system 108 may determine that the subset of the plurality of messages is associated with common audio content and to generate a stitched data stream from the subset of the plurality of messages for an audio timeline that is also associated with the common audio content. can For purposes of illustration, method 600 is described with respect to networked system 100 of FIG. 1 . It should be understood that method 600 may be practiced in other system configurations in other embodiments.

At operation 602 , the server system 108 (eg, via a server computer associated with the data stream stitching system 124 ) receives a plurality of user computing devices (eg, client devices 102 ) from a plurality of user computing devices (eg, client devices 102 ). receive a message from Each message of the plurality of messages may include media content such as images, data streams, and the like. In one example, the data stream may be a video taken by a user during a concert, voice, or other event that includes audio. Each message may also contain geolocation data, a date and time for the message (eg when the message was created or sent), data and timestamps for the data stream or other media content (eg, a data stream or other data such as when other media content of the message was captured or recorded), and the like.

In operation 604 , the server computer determines a subset of messages, of the plurality of messages, associated with a similar geolocation and time period. For example, the server computer may analyze each message of the plurality of messages to determine a geolocation of each of the plurality of messages and a time period for each of the messages. In one example, the server computer may determine the geolocation of each of the plurality of messages using data included in the message, such as global positioning system (GPS) coordinates of a user computing device from which the message was sent. In another example, the server computer is configured to execute the server computer based on a timestamp included in the message indicating a time when the message was sent, or a time when media content (eg, a data stream) included in the message was captured, or when the message was sent to the server. The time period may be determined based on the time received by the computer, and the like. The server computer determines a geolocation associated with an area of predetermined GPS coordinates in which a subset of the messages of the plurality of messages is the same and a time period within a predetermined time window (eg, 1 minute, 5 minutes, 10 minutes, 20 minutes, etc.) may include

In one example, the server system 108 may pre-partition the world (eg, a world map) into predefined grids. For each grid, the server computer may aggregate all messages with geolocations belonging to each grid within a certain period of time (eg, every 10 minutes). Each grid may be the same size, or each grid size may vary based on the density of the population within the area, the size of the town or city, and the like. Thus, a subset of messages may be a collection of messages having geolocations within one particular grid that occurred within the same (eg, 10 minutes) time period.

At operation 606 , the server computer extracts audio fingerprints for each message in the subset of messages to be used to determine if there is common audio across the messages in the subset of messages. For example, a server computer may analyze the audio contained in each message (eg, audio for a data stream of the message) and compute an audio fingerprint. There are a number of different methods that can be used to calculate audio fingerprints. One example is to extract features from a spectrogram and use the extracted features in comparison with other spectrograms to find out whether important spectrograms match up and where exactly they match up. will be. Using a spectrogram is just one example of how to calculate audio fingerprints. It is understood that other methods may be used alternatively or in combination.

In one example, the following response is returned from analyzing the audio included in the message to compute an audio fingerprint:

The spectrogram representation can be computed using a fast Fourier transform (FFT) over windows in time. For example, parameters used in the FFT window may include:

SAMPLING_RATE=44100

WINDOW_SIZE=4096

OVERLAP_RATIO=0.5

7 shows an example of a spectrogram 700 . With the spectrogram, the maxima (or peaks) can be extracted using a diamond maximal filter of size NEIGHBORHOOD_SIZE = 20. 8 shows an example of detected maxima of spectrogram 800 .

After the maxima have been extracted, the pairs of maxima are linked together to yield the final audio fingerprint, as shown in spectrogram 900 of FIG. 9 . First, the maxima can be sorted by the time they occur. The respective maxima are then linked with the first FAN_VALUE maxima thereafter if they are within the MIN_TIME_DELTA and MAX_TIME_DELTA time frames. A fingerprint can be created using three values:

1. Frequency of first maxima (11 bits)

2. Frequency of second maxima (11 bits)

3. Time difference between the two maximum values (8 bits)

In this example, these three values are packed into a single unsigned integer to form an audio fingerprint. This audio fingerprint is then associated with a time frame of first maxima. This results in the following fingerprint hash:

TIME_DELTA | FREQ1 | FREQ2

8 bit|11 bit|11 bit

The parameters that can be used for hashing are:

FAN_VALUE = 15

MIN_TIME_DELTA = 1

MAX_TIME_DELTA = 255

In one example, the audio fingerprints are designed in a noise-resistant manner. Thus, as long as there is some common audio shared between the two data streams, the two data streams can still be effectively matched.

6 , in operation 608 , the server computer groups the subset of messages into pairs of a plurality of messages, such that each message in the subset of the plurality of messages can be compared to a respective other message. do. Each pair of messages may include a first message and a second message.

At operation 610 , the server computer compares the audio fingerprint of the first message to the audio fingerprint of the second message in each pair of messages to determine a match score for each pair of messages. In this way, the server computer can match audio fingerprints between pairs of all messages within a subset of messages (eg, from the audio content in each data stream in each message). If the match score for the pair of messages is higher than a predetermined threshold, the server computer may consider the pair of messages to be a match (eg, comprising a portion of the same audio content), and each message in the pair is sorted You can decide exactly where to go. In one example, a match can be found if enough fingerprints are matched in the correct temporal order.

In another example, the server computer can determine at what point in time two messages should be linked, so that their intersecting audio tracks are matched up correctly. Matching can be done in spectrogram space and can be accurate to 1/20th of a second. For example, the first message may include the first 10 seconds of the audio content of the audio content (eg, from the time 00:00 to the time 00:10), and the second message may include the time from the time 00:08 to the time 00 It can contain 5 seconds of audio content up to :13, and so on.

In one example, the server computer may determine if there are enough message pairs that are linked together so that they all become one common audio timeline. A common audio timeline is associated with specific audio content (eg, songs, voices, concerts, etc.). The common audio timeline may include the entire length of the audio content (eg, the entire song, voice, concert, etc.), or the common audio timeline may include a portion of the total length of the audio content (eg, the end of the song). 45 seconds, the middle of the voice, the first 10 minutes of a concert, etc.).

A predetermined threshold may be used to determine if there are enough pairs of messages that are linked together such that they all become one common audio timeline. For example, a predetermined threshold may be a predetermined amount of time that messages must be linked (eg, at least 30 seconds, 1 minute, 5 minutes, etc.), a predetermined number of messages that must be linked (eg, 2, 10, 20, 50, 100, etc.), or some other threshold. If there are enough pairs of messages to link together to form one common audio timeline, the server computer can stitch together the messages associated with the common audio timeline.

At operation 612 , the server computer determines a set of messages among the subset of messages associated with the common audio timeline. For example, the server computer may determine the set of messages based on a match score for each pair of messages. In one example, the set of messages may include all messages in the subset of messages. In another example, a set of messages may be a further subset of a subset of messages (eg, messages in a subset of messages may not all be associated with common audio content, but instead have different audio content may be associated with).

At operation 614 , the server computer stitches the set of messages together to generate a stitched data stream from the data streams for each message in the set of messages. For example, the server computer may stitch together a set of messages based on a time period for each message. For example, message A may be for time period 00:10-00:20 in a common audio timeline, message B may be for time period 00:20-00:23, and message C may be for time period 00:10-00:23 It could be for the period 00:22-00:31, and so on. Data streams may overlap in time periods such that there may be more than one data stream during a given time period.

In one example, the server computer may determine a start message for stitching. The start message may be the first occurring message in the common audio timeline for the set of messages. For example, one message with a period of time in the audio timeline starting at 00:10, and another message having a period of time in the audio timeline starting at 00:15, etc. becomes like The server computer decides which message is the earliest in the audio timeline (eg the one starting at time 00:10).

In one example, there may be more than one message with similar start times, any of which may be a start message. For example, a first message may begin at time 00:10:01, a second message may begin at 00:10:02, and a third message may begin at 00:10:01. The server computer determines an interest score (eg, based on user interest), or The starting message may be selected from these messages by random selection, based on other means, or a combination of these methods.

In one example, the server computer determines all possible starting points for a stitch by considering messages or data streams from messages that appear only in front of all matches. The server computer may then run a recursive breadth-first search starting at each starting point to generate a dense audio stitch from the audio matches.

In another example, the server computer may optimize the stitches. For example, the optimized stitches may include the minimum number of messages or data streams from the messages needed to span the entire dense stitch. These can be computed efficiently for each tight stitch using a greedy algorithm. For example, starting with a first message or data stream, the server computer can determine the longest message or data stream that intersects the first message to select a second message. The server computer may then determine the longest message or data stream that intersects the second message to select the third message, and so on until the entire common audio timeline is complete.

10 shows a visual representation of a stitched data stream 1000 comprising a plurality of message or data streams 1002 - 1030 received from various users via a plurality of user devices. Each box 1002-1030 represents a message or data stream within the stitched data stream 1000 received from a user. For example, each message or data stream within each box 1002-1030 may include a period of time during a common audio timeline (eg, 1002 is the first 10 seconds of the common audio timeline). (1004 may be the next 10 seconds of the common audio timeline, 1006 may be the next 5 seconds of the common audio timeline, etc.). In this example, the stitched data stream 1000 includes messages from a live audio event.

11 illustrates another visual representation of a stitched data stream 1100 comprising a plurality of messages or data streams received from various users via a plurality of user devices. As in FIG. 10 , each box in FIG. 11 represents a message or data stream in the stitched data stream 1100 received from the user. In this example, the stitched data stream 1100 includes messages from a live sporting event.

In one example, the stitched data stream may include messages that are prioritized based on a random selection of an order of display in a timeline. In another example, the stitched data stream may include messages that are prioritized according to one or more rules. For example, the stitched data stream may include messages that are prioritized based on a quality score for display order in a common audio timeline. In one example, each message or data stream may be analyzed to determine a score for the quality of the data stream based on image quality, shakiness of the data stream, audio quality, light, and the like.

In another example, the stitched data stream can include messages that are prioritized based on the user or user profile of the user associated with the user computing device. For example, the stitched data stream may include messages from the user or from other users associated with the user (eg, friends, family, other users that the user "follows", etc.). The server computer may prioritize selection of messages generated or sent by the user or other users associated with the user.

Referring back to FIG. 6 , in operation 616 , the server computer provides the stitched data stream to one or more user computing devices. In one example, the server computer can transmit the stitched data stream to one or more computing devices. In another example, the server computer is configured such as via a link or other means that allows a user computing device to access a stitched data stream on the server computer (eg, stream a stitched data stream from the server computer). , may provide access to the stitched data stream from one or more computing devices.

The one or more user devices may display the stitched data stream on the display of the one or more user devices, as shown in the examples of FIGS. 13-15 . In one example, the stitched data stream is displayed on the user device as a continuous data stream transitioning from one message data stream to the next to a continuous common audio stream. In one example, the audio from each data stream of the stitched data stream is averaged to provide a better quality audio stream. Thus, a continuous common audio stream may include audio that is the average of the audio associated with each data stream in the stitched data stream to provide a better quality audio stream, or the continuous audio stream may contain audio in a plurality of messages. The received original audio may be included.

Exemplary embodiments allow a user to switch the display between alternating data streams for a given period of time in a stitched data stream. Using the example illustrated in FIGS. 13-15 , a user may be viewing a display 1306 of a stitched data stream for a recent concert on a computing device 1302 . Display 1306 may show the lead vocalist at a concert, and the user may wish to see other views at that time in the audio timeline. In one example, display 1306 can somehow indicate that there are alternating views of the currently viewed data stream (eg, via button 1304 , a highlighted frame, a pop-up message, or other indicator). The user moves the device (eg, shakes, tilts, etc.), touches the display 1306 of the device (eg, touches a button or link on the display 1306 ), or in another way that he or she alternates the display You can provide an indication that you want to see . After receiving the indication from the user, the user computing device 1302 may display an alternate view (eg, showing an audience as shown in FIG. 14 , a guitarist in a band as shown in FIG. 15 , or other displays). For example, a data stream) can be displayed. The stitched data stream will then continue to play from that view to the end of that view and then transition to the next view aligned with that view, and so on. Accordingly, the display of the stitched data stream transitions to the next message data stream when the previous data stream ends or when the user indicates that he or she wants to see an alternate display.

12 is a flow diagram illustrating aspects of a method 1200 for providing an alternating data stream for display on a user computing device (eg, client device 102 ), in accordance with some demonstrative embodiments. For purposes of illustration, method 1200 is described with respect to networked system 100 of FIG. 1 . It should be understood that method 1200 may be practiced in other system configurations in other embodiments.

At operation 1202 , the computing device receives a request for an alternating data stream. The computing device in these examples may be a user computing device (eg, client device 102 ) or a server computer (eg, a server computer associated with server system 118 ). For example, during display on a user computing device of a stitched data stream comprising a plurality of individual data streams associated with a common audio timeline, the user computing device or server computer (eg, the stitched data stream active data currently being displayed on the computing device (depending on whether it resides on or is streamed from a server computer, and/or whether functionality for providing an alternating data stream resides on the user computing device or the server computer). A request for an alternate data stream among a plurality of data streams other than the stream may be received. As described in greater detail above, a plurality of data streams associated with a common audio timeline are received from a plurality of computing devices and stitched together to form a stitched data stream. As also noted above, a common audio timeline may, in one example, be for audio content that is an average of audio associated with a plurality of individual data streams of stitched data streams.

In one example, the user may indicate a desire to switch to an alternating data stream on a display on the user computing device, as described above. For example, the user computing device may receive an indication from a user of the computing device to switch to an alternating data stream (eg, via motion input, touch input, etc.), as described above. In one example, the user computing device may send the request to the server system 108 , or in another example, may process the request itself. When a user computing device sends a request to the server system 108 , a computing device, such as a server computer, receives the request for an alternating data stream. If the user computing device processes the request itself, the user computing device receives the request for an alternating data stream.

At operation 1204 , the computing device determines a subset of a plurality of respective data streams of the respective ones of the stitched data streams associated with a time period of the active data stream in a common audio timeline. For example, the computing device may determine the time period of the active data stream based on data received in the request for the alternating data stream. In one example, the computing device can determine that the active data stream is associated with a time period 00:30 to 00:33 of the common audio timeline. The request for an alternating data stream may include a time period, or a current time stamp (eg, 00:30:49, 00:32:10, etc.) that the active data stream was being displayed when the request was sent. may include The computing device may analyze the plurality of data streams to determine which of the plurality of data streams includes a time period that overlaps a time period of the active data stream.

At operation 1206 , the computing device selects an alternating data stream from a subset of the plurality of individual data streams among the stitched data streams associated with a time period of the active data stream in a common audio timeline. In one example, to select an alternating data stream from a subset of the plurality of data streams, the computing device configures a quality for each data stream of a subset of the plurality of data streams of the stitched data stream associated with a time period of the active data stream. score can be determined. In this example, the computing device selects the alternating data stream based on a quality score for the alternating data stream. For example, the computing device may select the data stream with the highest quality score for the alternating data stream. The quality score may, in one example, be based on a quality score of the videos or images within the data stream and/or an interest score of the data stream for a plurality of users. In another example, the computing device may randomly select an alternating data stream from a subset of the plurality of data streams.

At operation 1208 , the computing device may provide an alternating data stream for displaying the computing device. In one example, the server computer may provide an alternating data stream to the user computing device for display to the user. In another example, the user computing device may display an alternating data stream to the user.

Display of the active data stream on the computing device may transition from a common audio timeline to an alternating data stream on the computing device. In this way, the user can view alternating data streams as described above. The display of the stitched data stream continues from the alternating data stream and transitions to the next data stream of the plurality of data streams occurring after the alternating data stream ends in a common audio timeline.

16 is a block diagram 1600 illustrating a software architecture 1602 that may be installed on any one or more of the devices described above. For example, in various embodiments, server systems 108 , including server systems 110 , 112 , 114 , 116 , 118 , 122 , and 124 , and client devices 102 are software architecture 1602 . ) may be implemented using some or all of the elements of 16 is only a non-limiting example of a software architecture, and it will be appreciated that many other architectures may be implemented to facilitate the functionality described herein. In various embodiments, software architecture 1602 is implemented by hardware, such as machine 1700 of FIG. 17 , including processor 1710 , memory 1730 , and I/O components 1750 . In this example, software architecture 1602 may be conceptualized as a stack of layers, each layer capable of providing specific functionality. For example, software architecture 1602 includes layers such as operating system 1604 , libraries 1606 , frameworks 1608 , and applications 1610 . In operation, applications 1610 initiate application programming interface (API) calls 1612 through the software stack and send messages 1614 in response to API calls 1612 , according to some embodiments. receive

In various implementations, the operating system 1604 manages hardware resources and provides common services. Operating system 1604 includes, for example, a kernel 1620 , services 1622 , and drivers 1624 . Kernel 1620 acts as an abstraction layer between hardware and other software layers in accordance with some embodiments. For example, kernel 1620 provides memory management, processor management (eg, scheduling), component management, networking, and security settings, among other functionality. Service 1622 may provide other common services for different software layers. Driver 1624 is responsible for controlling or interfacing with the underlying hardware, in accordance with some embodiments. For example, drivers 1624 include display drivers, camera drivers, BLUETOOTH® or BLUETOOTH® Low Energy drivers, flash memory drivers, serial communication drivers (eg, Universal Serial Bus (USB) drivers). ), WI-FI® drivers, audio drivers, power management drivers, and the like.

In some embodiments, libraries 1606 provide a low-level common infrastructure utilized by applications 1610 . Libraries 1606 may include system libraries 1630 (eg, the C standard library) that may provide functionality such as memory allocation functions, string manipulation functions, mathematical functions, and the like. . In addition, the libraries 1606 include media libraries (eg, Moving Picture Experts Group-4 (MPEG4), H.264 or Advanced Video Coding (AVC), Moving Picture Experts Group Layer-3 (MP3), AAC). Libraries supporting the presentation and manipulation of various media formats such as (Advanced Audio Coding), Adaptive Multi-Rate (AMR) audio codec, JPEG or JPG (Joint Photographic Experts Group), PNG (Portable Network Graphics), and graphic library (e.g., the OpenGL framework used to render graphical content to a display in two dimensions (2D) and three dimensions (3D)), database libraries (e.g. SQLite, which provides various relational database functions); It may include API libraries 1632, such as web libraries (eg, WebKit, which provides web browsing functionality), and the like. Libraries 1606 also provide many other APIs to applications 1610. ) may include a wide variety of other libraries 1634 .

Frameworks 1608 provide a high-level common infrastructure that can be utilized by applications 1610 in accordance with some embodiments. For example, frameworks 1608 provide various graphical user interface (GUI) functions, high-level resource management, high-level location services, and the like. Frameworks 1608 may provide a wide range of other APIs that may be utilized by applications 1610 , some of which may be specific to a particular operating system 1604 or platform.

In an exemplary embodiment, application 1610 includes home application 1650 , contacts application 1652 , browser application 1654 , book reader application 1656 , location application 1658 , media application 1660 , messaging applications 1662 , gaming applications 1664 , and a wide variety of other applications such as third party applications 1666 and media content applications 1667 . According to some embodiments, the application 1610 is a program that executes functions defined in the program. Various programming languages are structured in various ways, such as object-oriented programming languages (eg, Objective-C, Java, or C++) or procedural programming languages (eg, C or assembly language). It may be adapted to create one or more of the applications 1610 . In certain instances, third-party application 1666 (eg, an application developed using an ANDROID™ or IOS™ software development kit (SDK) by an entity other than a vendor of a particular platform) is an IOS™, ANDROID ™, WINDOWS® Phone, or mobile software running on a mobile operating system such as another mobile operating system. In this example, third-party application 1666 can invoke API calls 1612 provided by operating system 1604 to facilitate the functionality described herein.

As noted above, some embodiments may specifically include a messaging application 1662 . In certain embodiments, this may be a standalone application that operates to manage communications with a server system, such as server system 108 . In other embodiments, this functionality may be integrated with another application, such as the media content viewing application 1667 . Messaging application 1662 may request and display various media content items, the ability for a user to enter data related to media content items via a touch interface, a keyboard, or using a camera device of machine 1700; may provide for communication with server system 108 via I/O components 1750 , and reception and storage of media content items in memory 1730 . Presentation of media content items and user inputs associated with the media content items is messaging using different frameworks 1608 , library 1606 elements, or operating system 1604 elements running on machine 1700 . may be managed by the application 1662 .

17 illustrates a machine 1700 in accordance with some embodiments, capable of reading instructions from a machine-readable medium (eg, a machine-readable storage medium) to perform any one or more of the methodologies discussed herein. It is a block diagram illustrating the components of Specifically, FIG. 17 shows a schematic diagram of a machine 1700 in an exemplary form of a computer system, therein causing the machine 1700 to perform any one or more of the methodologies discussed herein. Instructions 1716 (eg, software, program, application 1610, applet, app, or other executable code) may be executed. In alternative embodiments, machine 1700 may operate as a standalone device or may be coupled (eg, networked) to other machines. In a networked deployment, machine 1700 is capable of server systems 108, 110, 112, 114, 116, 118, 122, 124, etc., or client devices 102 in a server-client network environment, or peer- It can operate as a peer machine in a two-peer (or distributed) network environment. Machine 1700 may be a server computer, client computer, personal computer (PC), tablet computer, laptop computer, netbook, personal digital assistant (PDA), entertainment media system, cellular phone, smart phone, mobile device, wearable device (eg, Instructions specifying actions to be taken by, for example, a smart watch), a smart home device (eg, a smart appliance), other smart devices, a web appliance, a network router, a network switch, a network bridge, or machine 1700 It can be, but is not limited to, any machine capable of executing steps 1716 sequentially or otherwise. Also, although only one machine 1700 is shown, the term “machine” refers to a machine 1700 executing instructions 1716, individually or jointly, to execute any one or more of the methodologies discussed herein. It is also taken to include collections.

In various embodiments, machine 1700 includes a processor 1710 , a memory 1730 , and an I/O component 1750 that may be configured to communicate with each other via a bus 1702 . In the exemplary embodiment, the processor 1710 (eg, central processing unit (CPU), reduced instruction set computing (RISC) processor, complex instruction set computing (CISC) processor, graphics processing unit (GPU), digital signal processor (Digital Signal Processor, DSP), Application Specific Integrated Circuit (ASIC), Radio Frequency Integrated Circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, instructions 1716 and a processor 1712 and a processor 1714 capable of executing The term “processor” is intended to include multi-core processor 1710, which may include two or more independent processors 1712, 1714 (also referred to as “cores”) capable of concurrently executing instructions 1716 . . Although FIG. 17 depicts multiple processors 1710, machine 1700 may include a single processor 1710 having a single core, a single processor 1710 having multiple cores (e.g., a multi-core processor 1710); may include multiple processors 1712 and 1714 with a single core, multiple processors 1710 and 1712 with multiple cores, or any combination thereof.

Memory 1730 includes main memory 1732 , static memory 1734 , and a storage unit 1736 , accessible by processor 1710 via bus 1702 , according to some embodiments. The storage unit 1736 can include a machine-readable medium 1738 having stored thereon instructions 1716 implementing any one or more of the methodologies or functions described herein. Instructions 1716 may also be, during execution by machine 1700 , fully or at least partially, in main memory 1732 , static memory 1734 , in at least one of processors 1710 (eg, , within the cache memory of the processor), or any suitable combination thereof. Accordingly, in various embodiments, main memory 1732 , static memory 1734 , and processors 1710 are considered machine-readable medium 1738 .

As used herein, the term “memory” refers to a machine-readable medium 1738 capable of temporarily or permanently storing data, including random access memory (RAM), read-only memory (ROM), buffer memory, Flash memory, and cache memory are taken to include, but are not limited to. Although machine-readable medium 1738 is depicted as being a single medium in the exemplary embodiment, the term "machine-readable medium" is intended to refer to a single medium or a plurality of media (eg, a central centralized or distributed databases, or associated caches and servers). The term “machine-readable medium” refers to machine 1700 when instructions 1716 are executed by one or more processors (eg, processors 1710 ) of machine 1700 . ) for execution by a machine (eg, machine 1700 ) (eg, instructions 1716 ) to cause it to perform any one or more of the methodologies described herein. It should also be taken to include any medium capable of storing the . Accordingly, “machine-readable medium” refers to a single storage device or device, as well as a “cloud-based” storage system or storage network including a plurality of storage devices or devices. Thus, the term "machine-readable medium" includes, but is not limited to, solid state memory (eg, flash memory), optical media, magnetic media, other non-volatile memory (eg, Erasable Programmable Read- Only Memory), or any suitable combination thereof. The term "machine-readable medium" specifically excludes non-statutory signals per se.

I/O component 1750 includes a wide variety of components that receive inputs, provide outputs, generate outputs, send information, exchange information, acquire measurements, and the like. In general, it will be appreciated that I/O component 1750 may include many other components not shown in FIG. 17 . I/O components 1750 are grouped according to functionality merely to simplify the discussion below, and this grouping is not limiting in any way. In various demonstrative embodiments, I/O component 1750 includes an output component 1752 and an input component 1754 . The output component 1752 may include a visual component (eg, a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a CRT), an acoustic component (eg, speakers), haptic components (eg, vibration motors), other signal generators, and the like. Input components 1754 include alphanumeric input components (eg, a keyboard, a touch screen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input components), a point-based input component (e.g., a mouse, touchpad, trackball, joystick, motion sensor, or other pointing device), tactile input components (e.g., a physical button, a touch screen that provides the location and force of touches or touch gestures; or other tactile input components), audio input components (eg, a microphone), and the like.

In some additional demonstrative embodiments, I/O component 1750 includes biometric component 1756 , motion component 1758 , environment component 1760 , or position component 1762 , among a wide variety of other components. include For example, the biometric component 1756 can detect an expression (eg, a hand expression, a facial expression, a voice expression, a gesture, or an eye tracking) and a biosignal (eg, blood pressure, heart rate, body temperature, sweat, or brain waves), identify a person (eg, voice identification, retina identification, face identification, fingerprint identification, or electroencephalogram-based identification), and the like. Motion component 1758 includes an acceleration sensor component (eg, an accelerometer), a gravity sensor component, a rotation sensor component (eg, a gyroscope), and the like. Environment component 1760 may include, for example, a lighting sensor component (eg, a photometer), a temperature sensor component (eg, one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (eg, For example, a barometer), an acoustic sensor component (eg, one or more microphones to detect background noise), a proximity sensor component (eg, an infrared sensor to detect nearby objects), a gas sensor component (eg, a machine olfactory sensors, gas detection sensors that, for safety purposes, detect concentrations of hazardous gases or measure airborne contaminants), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. may include The location component 1762 can be a location sensor component (eg, a GPS receiver component), an altitude sensor component (an altimeter or barometer that detects barometric pressure from which an altitude can be derived), an orientation sensor component (eg, a magnetometer), etc. includes

Communication may be implemented using a wide variety of technologies. I/O component 1750 may include a communication component 1764 operable to couple machine 1700 to network 1780 or device 1770 via coupling 1782 and coupling 1772, respectively. there is. For example, communication component 1764 includes a network interface component or another device suitable for interfacing with network 1780 . In a further example, communication component 1764 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, a BLUETOOTH® component (eg, BLUETOOTH® Low Energy), a WI-FI® component, and other communication components that provide for communication via other aspects. Device 1770 may be another machine 1700 or any of a wide variety of peripheral devices (eg, peripheral devices coupled via Universal Serial Bus (USB)).

Moreover, in some embodiments, communication component 1764 detects an identifier or includes a component operable to detect an identifier. For example, the communication component 1764 can be a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (eg, a one-dimensional barcode such as a Universal Product Code (UPC) barcode, a Quick Response) code, Aztec code, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra code, UCC RSS (Uniform Commercial Code Reduced Space Symbology)-2D barcode, and other optical codes) a detection component (eg, a microphone that identifies the tagged audio signal), or any suitable combination thereof. In addition, location via Internet Protocol (IP) geo-location, location via WI-FI® signal triangulation, detecting a BLUETOOTH® or NFC beacon signal that may indicate a specific location Various information may be derived via communication components 1764 , such as via location and the like.

In various demonstrative embodiments, one or more portions of network 1780 may include an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network. (local area network, LAN), wireless LAN (WLAN), wide area network (WAN), wireless WAN (wireless WAN, WWAN), metropolitan area network (MAN), Internet, Internet part of, part of the public switched telephone network (PSTN), plain old telephone service (POTS) networks, cellular telephone networks, wireless networks, WI-FI® networks, other types of networks; or a combination of two or more such networks. For example, network 1780 or portion of network 1780 may include a wireless or cellular network, and combination 1782 may include a Code Division Multiple Access (CDMA) connection, a Global System for Mobile communications (GSM) connection, or another type of cellular or wireless coupling. In this example, combining 1782 is 1xRTT (Single Carrier Radio Transmission Technology), EVDO (Evolution-Data Optimized) technology, GPRS (General Packet Radio Service) technology, EDGE (Enhanced Data rates for GSM Evolution) technology, 3G Third Generation Partnership Project (3GPP), fourth generation wireless (4G) networks, Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Long Term Evolution (LTE) standards including , other long-distance protocols, or other data transmission technologies, as defined by various standards-setting organizations, may implement any of various types of data transmission technologies.

In an exemplary embodiment, the instructions 1716 may be transmitted over a network interface device (eg, a network interface component included in communication component 1764 ) using a transmission medium and any of a number of well-known protocols. is transmitted or received over the network 1780 using one of (eg, Hypertext Transfer Protocol (HTTP)). Similarly, in another demonstrative embodiment, the instructions 1716 are transmitted or received using a transmission medium via coupling 1772 (eg, peer-to-peer coupling) to devices 1770 . . The term “transmission medium” should be taken to include any intangible medium capable of storing, encoding, or carrying instructions 1716 for execution by machine 1700 , digital or analog communication signals or other intangible media that facilitate communication of such software.

Further, the machine-readable medium 1738 is non-transitory (ie, has no transitory signal) in that it does not embody a propagated signal. However, labeling machine-readable medium 1738 as “non-transitory” should not be construed to mean that the medium is immovable; Media 1738 should be considered transportable from one physical location to another. Additionally, since machine-readable medium 1738 is tangible, medium 1738 may be considered to be a machine-readable device.

Throughout this specification, a plural instance may implement components, operations, or structures described as a single instance. Although individual acts of one or more methods are illustrated and described as separate acts, one or more of the individual acts may be performed concurrently, and the acts need not necessarily be performed in the order illustrated. Structure and functionality presented as separate components in an example configuration may be implemented as a combined structure or component. Similarly, structures and functionality disclosed as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements are within the scope of the subject matter herein.

Although the summary of the subject matter of the present invention has been described with reference to specific exemplary embodiments, various modifications and changes may be made to these embodiments without departing from the broader scope of the embodiments of the present disclosure.

The embodiments illustrated herein have been described in sufficient detail to enable any person skilled in the art to practice the teachings disclosed herein. Other embodiments may be used and derived therefrom, so that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Accordingly, this detailed description is not to be considered in a limiting sense, and the scope of various embodiments is defined only by the full scope of the appended claims and their equivalents.

As used herein, the term “or” may be interpreted in an inclusive or exclusive sense. Moreover, multiple instances may be provided for the resources, operations, or structures described herein as a single instance. Additionally, boundaries between various resources, operations, modules, engines, and data stores are somewhat arbitrary, and specific operations are illustrated in the context of specific example configurations. Other allocations of functionality are contemplated and may be within the scope of various embodiments of the present disclosure. In general, structures and functionality presented as separate resources in the example configurations may be implemented as a combined structure or resource. Similarly, structure and functionality presented as a single resource may be implemented as separate resources. These and other variations, modifications, additions, and improvements are within the scope of the embodiments of the present disclosure as represented by the appended claims. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

Claims (20)

As a method:
at the server computer, receiving a plurality of messages from a plurality of user computing devices, each of the plurality of messages comprising a data stream;
determining, by the server computer, a subset of messages of the plurality of messages associated with a similar geolocation and time period;
extracting, by the server computer, an audio fingerprint for each message in the subset of messages;
grouping, by the server computer, the subset of messages into a plurality of pairs of messages, each pair of messages comprising a first message and a second message;
comparing, by the server computer, audio fingerprints of the first message and the second message in each pair of messages to determine a matching score for each pair of messages;
determining, by the server computer, a set of messages among the subset of messages associated with a common audio timeline based on a matching score for each pair of messages;
a set of messages to generate, by the server computer, a stitched data stream from the data streams for each one of the set of messages based on a time period for each message corresponding to the common audio timeline stitching together , wherein the stitched data stream includes messages having data streams overlapping in time periods such that there is more than one data stream during at least one subset of a given time period; and
providing, by the server computer, the stitched data stream to one or more user computing devices. 2. The method of claim 1, wherein determining a subset of messages among the plurality of messages associated with a similar geolocation comprises a geolocation for each subset of messages associated with an area of predetermined global positioning system (GPS) coordinates having the same geolocation. A method comprising determining that According to claim 1,
further comprising determining a start message that is a message that occurs first in the common audio timeline;
wherein the set of messages are stitched together starting with the start message. 4. The method of claim 3, wherein determining a start message that is a message that occurs first in a timeline for the subset of messages selects the start message from a plurality of messages having similar time periods in the timeline based on a quality score. A method comprising the step of selecting. 4. The method of claim 3, wherein determining a start message that is a message that occurs first in a timeline for the subset of messages comprises randomly selecting the start message from a plurality of messages having similar time periods in the timeline. How to include. 2. The method of claim 1, wherein the stitched data stream includes messages that are prioritized based on a quality score for an order of display in the timeline. 2. The method of claim 1, wherein the stitched data stream comprises messages that are prioritized based on a random selection of an order of display in the timeline. The method of claim 1 , wherein the audio associated with the stitched data stream comprises audio that is an average of the audio associated with each data stream in the stitched data stream. As a server computer:
processor; and
A computer readable medium coupled with a processor, the computer readable medium causing a computing device to:
receiving a plurality of messages from a plurality of user computing devices, each of the plurality of messages comprising a data stream;
determining a subset of messages of the plurality of messages associated with a similar geolocation and time period;
extracting an audio fingerprint for each message in the subset of messages;
grouping the subset of messages into a plurality of pairs of messages, each pair of messages comprising a first message and a second message;
comparing audio fingerprints of the first message and the second message in each pair of messages to determine a matching score for each pair of messages;
determining a set of messages among the subset of messages associated with a common audio timeline based on the matching score for each pair of messages;
stitching the sets of messages together to produce a stitched data stream from the data streams for each one of the sets of messages based on a time period for each message corresponding to the common audio timeline. - the stitched data stream comprises messages having data streams overlapping in time periods such that there is more than one data stream during at least one subset of the given time period; and
stored instructions executable by the processor to cause it to perform operations comprising providing the stitched data stream to one or more user computing devices. 10. The method of claim 9, wherein determining a subset of messages of the plurality of messages associated with a similar geolocation comprises: determining that a geolocation for each subset of messages is associated with an area of the same predetermined GPS coordinates. server computer containing 10. The method of claim 9, wherein the operations are:
further comprising determining a start message that is a message that occurs first in the common audio timeline;
wherein the set of messages are stitched together starting with the start message. 12. The method of claim 11, wherein determining a start message that is a message that occurs first in a timeline for the subset of messages selects the start message from a plurality of messages having similar time periods in the timeline based on a quality score. A server computer comprising the step of selecting. 12. The method of claim 11, wherein determining a start message that is a message that occurs first in a timeline for the subset of messages comprises randomly selecting the start message from a plurality of messages having similar time periods in the timeline. server computer containing 10. The server computer of claim 9, wherein the stitched data stream includes messages that are prioritized based on a quality score for order of display in the timeline. 10. The server computer of claim 9, wherein the stitched data stream includes messages that are prioritized based on a random selection of an order of display in the timeline. 10. The server computer of claim 9, wherein the audio associated with the stitched data stream comprises audio that is an average of the audio associated with each data stream in the stitched data stream. A non-transitory computer-readable medium comprising:
Let the computing device:
receiving a plurality of messages from a plurality of user computing devices, each of the plurality of messages comprising a data stream;
determining a subset of messages of the plurality of messages associated with a similar geolocation and time period;
extracting an audio fingerprint for each message in the subset of messages;
grouping the subset of messages into a plurality of pairs of messages, each pair of messages comprising a first message and a second message;
comparing audio fingerprints of the first message and the second message in each pair of messages to determine a matching score for each pair of messages;
determining a set of messages among the subset of messages associated with a common audio timeline based on the matching score for each pair of messages;
stitching the sets of messages together to produce a stitched data stream from the data streams for each one of the sets of messages based on a time period for each message corresponding to the common audio timeline. - the stitched data stream comprises messages having data streams overlapping in time periods such that there is more than one data stream during at least one subset of the given time period; and
A non-transitory computer-readable medium comprising stored instructions executable by at least one processor to cause operations to be performed comprising providing the stitched data stream to one or more user computing devices. 18. The method of claim 17, wherein the operations are:
further comprising determining a start message that is a message that occurs first in the common audio timeline;
wherein the set of messages are stitched together beginning with the start message. 19. The method of claim 18, wherein determining the start message that is a message that occurs first in a timeline for the subset of messages comprises the start message from a plurality of messages having similar time periods in the timeline based on a quality score. A non-transitory computer-readable medium comprising the step of selecting 19. The method of claim 18, wherein determining the start message that is a message that occurs first in a timeline for the subset of messages comprises randomly selecting the start message from a plurality of messages having similar time periods in the timeline. A non-transitory computer readable medium comprising steps.

Images