JP7258857B2 - Modification of video data capture by the imaging device based on video data previously captured by the imaging device - Google Patents

Description

TECHNICAL FIELD This disclosure relates generally to capturing video data and, more particularly, to modifying video data capture based on previously captured video data.

It has become possible for client devices, online systems and networks to exchange greater amounts of content with each other than ever before. For example, the online system allows its users to exchange data such as: video data captured by different users via a client device associated with the user. In a specific example, the online system can establish video messaging between the user and additional users, allowing the users to exchange video data captured by their respective client devices in real time or near real time. obtain.

However, in providing video data, conventional client devices require the user to manually configure video capture. For example, a user of the client device may provide input to the client device to specify the focus of the client device's imager, specify the magnification of the client device's imager, or other parameters of the client device's imager. . In addition to requiring the user to provide initial input specifying the parameters of the client device's imager, conventional client devices also require the user to: Manually relocating the imaging device of the client device to a different part. Thus, the user manually selects and maintains the content that is captured by the imaging device of the client device and communicated to another client device. While the user can gain considerable control over the video data captured by the client device because of its reliance on user-provided input, conventional client devices allow a video messaging session during which video data capture takes place. The user cannot easily perform other tasks during this time. This is because conventional client devices cannot adjust video capture in response to changing circumstances without receiving one or more user-provided inputs.

Various client devices associated with users of the online system include one or more imaging devices. An imaging device included within the client device is configured to capture video data in a local area around the client device, including, for example, during a video call or when a user activates a video capture function. The client device also includes a controller coupled to the one or more imaging devices. The controller applies one or more models to the video data captured by the imaging device and modifies the video data captured by the imaging device and/or parameters of the imaging device based on application of the one or more models. This allows the controller to modify the captured video data based on features or content of video data previously captured by the imaging device.

The controller maintains and enforces one or more privacy settings for users and others captured within video data and other data in various embodiments. For example, the controller may be provided with default privacy settings, and the controller may not be able to identify the user until the user manually changes the privacy setting to allow the controller to make the user identification. Default privacy settings can extend to any video data, audio data, or even other data, and can allow users to choose whether or not the imaging device may identify any users in the data. Privacy settings also govern the communication of information from a client device to another entity (eg, another client device or a third party system). Various privacy settings allow users to control permissions regarding: user identification; and storage and sharing of any user-related data. Privacy settings can also be individually enforced for each individual. For example, a user who opts in for user-specific features cannot change default privacy settings for other users that may be incidentally captured within the local area around the client device.

In various embodiments, based on user privacy settings around user-specific activation by the client device, the controller applies one or more machine learning models to the video data captured by the imaging device to locating users included in the video data. The model applied to the captured video data by the controller 210 performs face tracking (in 2D or 3D), 2D pose tracking, 3D pose tracking, or any other suitable method to map parts of the human face. and human body parts can be identified. In various embodiments, the controller modifies captured video data or imaging device parameters to present the positioned user more prominently. For example, the controller crops the captured video data to remove portions of the video data that do not include at least one person. As another example, the controller corrects the focus of the imager to the human face and increases the magnification (eg, zoom) of the imager. In various embodiments, the user is given the option to: prevent any recordings (video, audio, etc.) from being stored locally on the client device and/or on the cloud; Delete any records if they have been saved.

In various embodiments, the one or more models applied by the controller apply one or more rules to modify the video data captured by the imaging device of the client device when the user specific functionality is enabled. do. For example, if the controller determines that a human face is facing the camera, the controller modifies the captured video data to make the person (positioned) in the video data appear more prominent. In another example, the controller determines the distance between the identified persons from the video data and the imaging device, and modifies the captured video data for persons whose determined distance from the imaging device is at least a lower bound. presents the doujin with at least a threshold dimension (e.g., with at least a threshold height or threshold width, or with at least a threshold percentage of the field of view of the imaging device), or a determined distance threshold distance from the imaging device People with less than 100 are displayed with at least a set of threshold dimensions.

The controller can receive data from other components of the client device and modify the captured video data based on characteristics of the received video data and data from other components of the client device. For example, an imaging device or client device includes an audio capture device, such as a microphone, configured to capture audio data from a local area around the client device. The controller is subject to user-selected privacy settings, but may process the captured audio data along with the captured video data in modifying the captured video data. In various embodiments, the controller applies one or more models to the captured audio data to determine locations within the captured video data, including the source of the audio data. A controller applies one or more models to locations within the captured video data that contain the source of the audio data. In response to applying a model that makes a positional determination (including sources of audio data, including people) within the captured video data, the controller modifies the captured video data to produce a captured video that includes sources of audio data. Positions within the data are presented more prominently, and the imager is repositioned to focus on the source of the audio data (e.g., increasing the field of view of the imager occupied by the source of the audio data or changing the audio data source position within the field of view). However, the controller does not modify the captured video data or relocate the imager in response to determining that a location within the captured video data that contains the audio data source does not contain a person. As another example, the controller may modify the captured video data or reposition the imaging device to make the particular person the controller determined to be the source of the captured audio data in the captured video data more prominent. Presenting (eg, increasing the field of view of the imager occupied by the source of the audio data or changing the position of the audio data source within the field of view of the imager). This allows the captured video data to more prominently display the user determined to be the speaker or provides audio data captured by the audio capture device of the client device.

In a video messaging session or other situation where the user has enabled the video capture and user identification features, the controller may apply one or more models to the captured video data after receiving information identifying the user of interest. is applied to modify one or more parameters of the imaging device to track the user of interest around the local area. For example, the controller may apply one or more facial recognition models to a person located within the captured video data, identified from previously captured video data (either received from the online system or based on information from the online system). identify a person's face that matches the face of the user of interest), and then refocus the imaging device to focus on the person whose face matches the face of the user of interest. Alternatively, the controller extracts a colormap from the captured video data containing the user of interest, refocuses the imaging device, and includes the extracted colormap in the video data captured by the imaging device. leave it alone.

Embodiments in accordance with the invention are disclosed in the accompanying claims, particularly directed to methods and computer program products, wherein features recited in one claim category, e.g. Products, systems, and storage media can be similarly entitled. Dependencies or back references to the appended claims are chosen for formal reasons only. However, since any subject matter derived by intentional back-reference (especially multiple claim dependency) to any preceding claim may be claimed as well, any combination of claim and features thereof is disclosed and claimed in the appended claim. Claims can be made regardless of the subordination chosen. Claimable subject matter includes not only the combination of features as set forth in the appended claims, but also any other combination of features of the claims, wherein each feature recited in a claim may be used in conjunction with any other feature or combination of the claims. It can be combined with other feature combinations. Also, any of the embodiments and features described or illustrated herein may be referred to in an individual claim and/or any embodiment or feature described or illustrated herein or any feature of an appended claim. and can be claimed in any combination.

The drawings depict various embodiments for illustrative purposes only. Those skilled in the art will appreciate that alternative embodiments of the structures and methods shown herein can be employed without departing from the principles described herein. will be easily recognized from

1 is a block diagram of a system environment in which an online system operates according to one embodiment; FIG. 1 is a block diagram of a client device according to one embodiment; FIG. 1 is a block diagram of an online system according to one embodiment; FIG. 4 is a flow chart of a method for selecting video data segments for presentation to users of an online system, according to an embodiment, which is based on the probability that an interested user's gaze is directed to different positions within the video data. It is based on a flow chart.

System Architecture FIG. 1 is a block diagram of system environment 100 for online system 140 . The system environment 100 illustrated by FIG. 1 comprises one or more client devices 110 , a network 120 , one or more third party systems 130 and an online system 140 . Also within system environment 100 shown in FIG. 1, controller 117 is coupled to client device 110 . In alternative configurations, different and/or additional components may be included within system environment 100 . For example, online system 140 is a social networking system, a content sharing network, or another system that provides content to users.

Client device 110 is one or more computing devices capable of receiving user input and transmitting and/or receiving data over network 120 . In one embodiment, client device 110 is a conventional computer system, such as a desktop or laptop computer. Alternatively, client device 110 may be a computer-capable device such as a personal digital assistant (PDA), mobile phone, smart phone, or another suitable device. Client device 110 is configured to communicate over network 120 . In one embodiment, client device 110 executes an application that allows a user of client device 110 to interact with online system 140 . For example, client device 110 executes a browser application to enable interaction between client device 110 and online system 140 over network 120 . In another embodiment, the client device 110 interacts with the online system 140 through an application programming interface (API) that runs on the client device's 110 native operating system, such as IOS® or ANDROID®. As further described below in connection with FIG. 2, the client device 110 captures image or video data of a local area around the client device 110 with a display device 115 configured to present content. and one or more imaging devices configured to.

Client device 110 is configured to communicate over network 120, which may be any combination of local and/or wide area networks using both wired and/or wireless communication systems. can include In one embodiment, network 120 uses standard communication technologies and/or protocols. For example, network 120 includes communication links using technologies such as: Ethernet, 802.11 family protocols, WiMAX (Worldwide Interoperability Specification for Microwave Access), 3G, 4G, CDMA (Code Division Multiple Access), DSL (Digital Subscriber Line), etc. Examples of networking protocols used to communicate over network 120 include Multiprotocol Label Switching (MPLS), Transmission Control Protocol/Internet Protocol (TCP/IP), Hypertext Transport Protocol (HTTP), Simple Mail Transfer Protocol. (SMTP), and File Transfer Protocol (FTP). Data exchanged over network 120 may be expressed using any suitable format, such as Hypertext Markup Language (HTML) or Extensible Markup Language (XML). In some embodiments, all or some of the communication links of network 120 may be encrypted using any suitable technique or techniques.

One or more third party systems 130 may be integrated into network 120 for communicating with online system 140, which is further described below in conjunction with FIG. In one embodiment, the third party system 130 is an application provider that conveys information describing an application for execution by the client device 110 or data for use by an application executing on the client device. 110 is an application provider. In other embodiments, third party system 130 provides content or other information for presentation through client device 110 . Third party system 130 may communicate information to online system 140 , such as information regarding advertisements, content, or applications provided by third party system 130 .

FIG. 2 is a block diagram of an embodiment of client device 117 . In the embodiment of FIG. 2, client device 110 includes display device 115 , imaging device 117 and controller 210 . However, in other embodiments, client device 110 includes components different from or in addition to those shown in FIG.

Display device 115 may be integrated into client device 110 or coupled with client device 110 . For example, display device 115 integrated with client device 110 may include a display screen included with client device 110 . Alternatively, display device 115 may be a monitor or other display coupled to client device 110 . The display device 115 presents image data or moving image data to the user. The image or video data presented by display device 115 is determined by the application running on client device 110 . A variety of applications may be included on the client device 110 and therefore the execution of different applications will change the content presented by the user with the display device 115 .

The imaging device 117 captures video data or images of the local area around the client device 110 and within the field of view of the imaging device 117 . In some embodiments, imaging device 117 includes one or more cameras, one or more video cameras, or any other device capable of capturing image or video data. Imaging device 117 may also include one or more filters (eg, filters used to increase the signal-to-noise ratio). Various parameters (eg, focal length, focus, frame rate, ISO, sensor temperature, shutter speed, aperture value, focus, etc.) configure the capture behavior of video or image data by the imaging device 117 . Thus, modifying one or more parameters of imaging device 117 modifies the video or image data captured by imaging device 117 after modifying the one or more parameters. Although FIG. 2 shows a single imaging device 117 included within client device 110, client device 110 may include any suitable number of imaging devices 117 in other embodiments. In various embodiments, the user is given the option to: prevent any recordings (video, audio, etc.) from being stored locally on the client device and/or on the cloud; Delete any records if they have been saved.

Controller 210 is coupled to imaging device 117 and includes storage coupled to the processor. In various embodiments, controller 210 is also coupled to display device 115 . Controller 210 includes instructions that, when executed by the processor, apply one or more models to video data captured by imaging device 117 . In various embodiments, one or more models are applied to video data, audio data, image data, or any combination of data included by imaging device 117 or in client device 110. /coupled} is captured by any other device. As described further below in conjunction with FIG. 4, the model applied to the captured video data by the controller 210 applies one or more rules to the features of the captured video data to determine the objects, people, and objects of the captured video data. , actions or any other suitable content. By applying the model and with one or more privacy settings, the controller 210 modifies the captured video data or modifies one or more parameters of the imaging device 117, which subsequently modifies the captured video data. . For example, the user can authorize the controller 210 to apply a model by which the user can be positioned within the captured video data based on characteristics of the captured video data, and the captured video data can be modified to make the positioned user more prominent. The additional video data can make the positioned user stand out more by standing or by modifying one or more parameters of the imaging device 117 (e.g., focal length, magnification or zoom, cropping of the captured video data). make it accompany The additional video data contains the positioned person more prominently and does this by presenting it as follows: positioned with at least a threshold dimension set (e.g. with at least a threshold height and a threshold width) presenting the person; presenting the positioned person over at least a threshold amount of the field of view of the imaging device 117 or over at least a threshold amount of frames of the captured video data; be presented at a designated position. However, the model applied by the controller 210 can identify any suitable component of the video data captured by the imaging device 117, modify the parameters of the imaging device 117, and modify the captured video data accordingly.

In various embodiments, client device 110 includes one or more audio capture devices, such as microphones. For example, client device 110 includes an array of microphones, which are configured to beamform in two or three dimensions. The audio capture device captures audio signals from different regions within the local area surrounding client device 110 . In various embodiments, one gastric anomaly audio capture device is coupled to the controller 210 and maintains information identifying different regions of the local area around the client device 110 . For example, the controller 210 may identify twenty-four 15 degree regions for the local area around the client device 110, which extend from a point within the client device 110 and span 360 degrees around the client device 110. Each part of the local area is identified.

One or more audio capture devices are coupled to controller 210 . In a manner that is subject to user privacy settings, controller 210 applies one or more models, such as machine learning models and other acoustic recognition models, to audio data captured from a region of the local area around client device 110. , apply. Controller 210 includes information identifying a user or object (e.g., television or mobile device) and applies one or more models to audio captured from a region of the local area around client device 110. determines whether the captured audio data includes audio data from the user or object identified by controller 210 or includes environmental noise. In some embodiments, the one or more models applied by controller 210 identify a user or an object identified by controller 210 from which one or more audio capture devices captured audio in a region. ,decide. In other embodiments, the client device 110 communicates audio data captured by one or more audio devices to the online system 140, which applies one or more models to identify the object to which the audio data is specified. Alternatively, determine whether it contains audio data captured from a user, or determine the specific user or object identified from which the audio data was captured. Online system 140 may provide client device 110 with an indication that the captured audio data includes audio data captured from an identified object or user, or may provide the specific identified data from which the audio data was captured. Provides information specifying an object or user. Controller 210 or online system 140 similarly determines whether this is the case for audio captured from other regions of the local area around client device 110 . Based on an identified object or user determination from which audio data is captured in different regions, the controller 210 modifies the placement of one or more audio devices to improve the quality of audio captured from one or more regions. Let For example, the controller 210 may relocate one or more audio capture devices to improve the quality of the audio that was obtained from regions around the local area from which the audio data was captured. Yes, either from a concrete user or from a concrete object. Similarly, the controller 210 can modify one or more parameters of the imaging device 117 based on the regions of the local area around the client device 110 where audio data has been captured from different users or objects. In various embodiments, one or more audio and imaging devices 117 may be directed to different portions of the local area around client device 110 . For example, the imaging device 117 is directed at an object described by a user, while the controller 210 directs one or more audio devices locally around the client device 110 from which audio data capture was made by a particular user. Point to the realm of the area.

In various embodiments, the online system 140 and the controller 210 of the client device 110 cooperatively and/or individually maintain and enforce one or more privacy settings, which privacy settings may For users or persons identified from data or other data. A user's or person's privacy settings determine how certain information associated with the user or person can be shared, and may be stored in association with information that identifies the user or person. In some embodiments, controller 210 retrieves privacy settings for one or more users maintained by online system 140 . In one embodiment, privacy settings specify certain information associated with a user and also specify with which other entities the specified information may be shared. Examples of entities with which information can be shared may include other users, applications, third party systems 130, or any entity that potentially has access to the information. Examples of information that may be shared by users include: image data including users or persons; audio data including audio captured from users or persons; video data including users or persons, and the like.

For example, in certain embodiments, the privacy settings allow the first user to specify (by opting out or not opting in) whether the online system 140 allows the following actions: for any purpose; Receiving, collecting, logging, or storing specific objects or information associated with a user. In certain embodiments, privacy settings may allow a first user to define whether a particular video or audio capture device, application, or process can: access, storage, and use of Privacy settings may allow a first user to opt-in or opt-out of a particular device, application, or process accessing, storing, or using objects or information. Online system 140 may obtain access to such information for the purpose of providing a particular function or service to a first user, while granting access to such information for any other purpose. Online system 140 need not have it. Before accessing, storing or using such objects or information, and before permitting such actions, online systems can prompt users to provide privacy settings, which privacy settings may be set. is about: What applications or processes (if they exist) can access, store, and use objects and information. As a non-limiting example, a first user can send a message to a second user via an application associated with an online social network (e.g., a messaging app), and such message is stored by online system 140. You can specify privacy settings that should not be set.

Privacy settings maintained and enforced by online system 140 and/or controller 210 can be associated with default settings. In various embodiments, the controller 210 may identify the user in captured video, audio, or other data unless the controller 210 obtains privacy settings from the user that allow the user to be identified to the controller 210. , not specified. For example, the privacy settings associated with a user may have default settings, unless the user manually changes the privacy settings to allow the controller 210 to identify the user. , the controller 210 is prevented from identifying the user so that the controller cannot identify the user. Also, in various embodiments, alternative privacy settings may be achieved by communicating user-identifying information from client device 110 to another entity (eg, another client device 110, online system 140, or third-party system 130). discipline. Alternate privacy settings have default settings that, in various embodiments, prevent transmission of information that identifies the user, unless the user manually modifies the alternate privacy settings to allow transmission. Controller 210 is prevented from transmitting information identifying the user to other entities. Controller 210 maintains one or more privacy settings for each user identified from captured video data and other data, allowing per-user control over transmission and identification of each user. In some embodiments, when controller 210 first identifies a person from the captured data, controller 210 prompts the person to provide privacy settings, and combines the provided privacy settings with information identifying the person. Store in association.

In various embodiments, for various components of online system 140 and/or client device 110 with features that may use the user's personal and biometric information as input for user authentication and experience personalization purposes, the user may You can choose to take advantage of the features of to enhance your experience when using the device and online system. As a non-limiting example, users can voluntarily provide personal information and biometric information to online system 140 . User privacy settings may specify that such information may only be used for certain processes, such as authentication, and such information shall not be shared with any third parties. or other processes or applications associated with online system 140. As another non-limiting example, online system 140 may provide the ability for users to contribute voiceprint recordings to online social networks. By way of non-limiting example, if a user wishes to take advantage of this feature of an online social network, the user may provide a voice recording of his/her voice to cause status updates on the online social network. . A recording of the voice input can be compared to the user's voiceprint to determine what the words uttered by the user are. A user's privacy settings may specify that such recordings are only available for voice input purposes (e.g., to authenticate users, to send voice messages, to use the voice-activated features of online social networks). to improve speech recognition), and such recordings may not be shared with any third party system or utilized by any other process or application associated with the online system 140. can be specified. As another non-limiting example, online system 140 may provide the ability for users to provide reference images (eg, facial profiles, etc.) to online social networks. Online social networks can compare reference images with later received image inputs (to authenticate users, tag users in photos, etc.). User privacy settings may specify that such recordings may only be used for limited purposes (e.g., authentication or tagging users in photos), and such recordings may not be shared with any third-party systems. It may be specified that it must not be used by other processes or applications associated with online system 140 . Any such restrictions on biometric data and/or other personal data may also apply to client device 110 .

Users may allow the following actions: capture of data; identification of users; and/or sharing and cross-application use of user-related data in one or more ways. For example, a user may pre-select various privacy settings prior to utilizing client device 110 functionality and/or performing an action on online system 140 . Another example is when the user performs an action for the first time or utilizes a feature of client device 110 and/or online system 140 for the first time and/or when the user has not performed an action or feature for a predetermined period of time. Can be prompted with a selection dialog if not in use. In yet another example, client device 110 and online system 140 may provide notification to the user when a particular feature requiring user data is activated or disabled due to user selection. and allow the user to make further choices via notifications. There are other suitable techniques for users to grant permissions.

In some embodiments, the controller 210 obtains information maintained by the online system 140 or from one or more third party systems 130 about the user identified from the captured video data. Yes, and such behavior is subject to privacy settings about the user. Based on the acquired information, as well as video, audio, image, and other data involving the user previously captured by the client device 110, the controller 210 may generate content for presentation to the user via the client device 110. can be generated. For example, controller 210 overlays content items from online system 140 that are associated with one or more objects identified by controller 210 from video or image data captured by client device 110 . Alternatively, online system 140 may maintain video data, audio data, image data, and other data including user and user received from client device 110 (or may be provided by online system 140 to one or more third parties). Based on the information (obtained from system 130), content is generated for the user, and pre-generated content is provided to client device 110 for presentation to the user.

FIG. 3 is a block diagram of the architecture of online system 140 . Online system 140 shown in FIG. 3 includes user profile store 305 , content store 310 , action logger 315 , action log 320 , edge store 325 , content selection module 330 and web server 335 . In other embodiments, online system 140 may include additional, fewer, or different components for various applications. Conventional components such as network interfaces, security features, load balancers, failover servers, management and network operations consoles, etc. are not shown so as not to obscure the details of this system configuration.

Each user of online system 140 is associated with a user profile stored in user profile store 305 . A user profile includes declarative information about the user that the user has explicitly shared, and may also include profile information inferred by online system 140 . In one embodiment, a user profile includes multiple data fields each describing one or more attributes of the corresponding online system user. Examples of information stored in user profiles include personal, demographic and other types of descriptive information such as work history, education, gender, hobbies and preferences, location, and the like. A user profile may also store other information provided by the user, such as images and videos. In some embodiments, a user's image can be tagged with information identifying the user of the online system that appears in the image, information identifying the image that the user has tagged and stored in the user's user profile. is. User profiles in user profile store 305 may also maintain references to actions by the corresponding user taken on content items in content store 310 and stored in action log 320 .

The profile maintained for the user also includes characteristics of one or more client devices 110 associated with the user, such that the online system 140 can later identify the user from the characteristics provided by the client device 110. become. For example, an application associated with online system 140 and running on client device 110 may transmit a device identifier or other information that uniquely identifies client device 110 to online system 140 in association with the user identifier. offer. Online system 110 stores the device identifier or other information that uniquely identifies client device 110 in a user profile maintained for the user for later online system 140 to uniquely identify the device identifier or client device 110 . Identification of the user is made when other information is received. Other characteristics of the client device 110 associated with the user may alternatively or additionally be contained within a user profile maintained by the user. For example, a user profile includes: the network address used by client device 110 to access network 120; the identifier of the application running on client device 110 from which online system 140 obtained information; type of client device 110 from which online system 140 obtained information (e.g., manufacturer name, model identifier of client device 110, etc.); execution on client device 110 from which online system 140 obtained information; operating system. However, online system 140 may store any suitable characteristics of client device 110 in the user profile, and online system 140 may maintain information about client device 110 utilized by the user corresponding to the user profile.

User profiles in the user profile store 305 are often associated with individuals and allow individuals to interact with each other through the online system 140, but user profiles can also be stored for entities such as businesses and organizations. This allows entities to establish a presence on online system 140 in order to connect and exchange content with users of other online systems. Entities can post information about their company, about their products, or provide other information to users of online system 140 using branded pages associated with the entity's user profile. Other users of the online system 140 may connect with the Brand Page to receive information posted on the Brand Page or receive information from the Brand Page. A user profile associated with a brand page may contain information about the entity itself that provides the user with background or informational data about that entity.

The content store 310 stores multiple objects each representing different types of content. Examples of content represented by objects include page posts, status updates, photos, videos, links, shared content items, game application achievements, check-in events for local businesses, branded pages, or other types of content. Contains content. Users of the online system can create objects that are stored by the content store 310, such as status updates, photos that the user has tagged for association with other objects in the online system 140, events, groups, or applications. can. In some embodiments, the object is received from a third party application or a third party application separate from online system 140 . In one embodiment, an object in content store 310 represents a piece of content or multiple content "items." As such, users of the online system are encouraged to communicate with each other by posting text and content items of various types of media to the online system 140 through various communication channels. This increases the amount of interaction between users and increases the frequency with which users interact within online system 140 .

One or more content items contained in the content store 310 include content creatives and bid amounts for presentation to users. Creatives are text, images, audio, video, or other suitable data presented to the user. In various embodiments, the creative also identifies pages of content. For example, the content item includes a link that specifies the network address of the landing page of the content intended for the user when the content item is accessed. If the content is presented to the user, the bid amount is included in the content item by the user, and if the content in the content item is presented to the user, the bid amount is included in the content item when presented. An expectation, such as financial compensation, provided by an advertiser to the online system 140 if the content receives user interaction or if some appropriate condition is met when the content in the content item is presented to the user. used to determine the value. For example, the bid amount included in the content item specifies the amount that the online system 140 will receive from a user who has provided the content item to the online system 140 when the content in the content item is displayed. In some embodiments, the expected value for the online system 140 of presenting content from a content item may be determined by multiplying the bid amount by the probability that the content of the content item will be accessed by a user.

Various content items may include goals that identify interactions that users associated with the content item would like other users to perform when presented with the content included in the content item. Exemplary purposes may include: installing an application associated with a content item; indicating preferences for a content item; sharing a content item with other users; interacting with an object associated with; making any other suitable interaction. When content from a content item is presented to a user of the online system, the online system 140 records interactions between the presented user and the content item or objects associated with the content item. In addition, online system 140 receives compensation from users associated with content items when online system users perform interactions with content items that satisfy goals contained in the content items.

Additionally, the content item may include one or more targeting criteria specified by the user who submitted the content item to online system 140 . The targeting criteria included in the content item request specify one or more characteristics of the user that are eligible to be presented with the content item. For example, targeting criteria are used to identify users with user profile information, edges, or actions that meet at least one targeting criteria. Thus, targeting criteria allow users to identify users with particular characteristics, simplifying subsequent delivery of content to individual users.

In one embodiment, targeting criteria may specify actions or connection types between users and other users or objects of online system 140 . Targeting criteria may also specify interactions between users and objects running outside of online system 140 , such as on third-party system 130 . For example, targeting criteria may include sending messages to other users, using applications, joining groups, leaving groups, attending events, generating event descriptions, selling products or services using online marketplaces. Identifies the user who performed a particular action, such as making a purchase or review, requesting information from a third party system 130, installing an application, or any other suitable action. By including actions in the targeting criteria, users can further narrow down who is suitable for presenting content items. As another example, targeting criteria identify users who have connections to other users or objects, or who have a particular type of connection to other users or objects.

Based on privacy settings, action logger 315 may be permitted to receive communications regarding user actions internal and/or external to online system 140 and may populate information regarding user actions into action log 320 . . Examples of actions include adding a connection to another user, sending a message to another user, uploading an image, reading a message from another user, including viewing content posted by other users and attending events posted by other users. Furthermore, since many actions may involve objects and one or more specific users, these actions are also associated with specific users and stored in action log 320 .

Based on privacy settings, action log 320 is used by online system 140 to track user actions on online system 140 and actions on third-party systems 130 that communicate information to online system 140 . may have permission from the user to do so. A user can interact with various objects on online system 140 and information describing these interactions is stored in action log 320 . Examples of interactions with objects include: commenting on posts; sharing links; checking in to physical locations via client device 110; accessing content items; Make any other appropriate interaction. Further examples of interactions with objects on online system 140 that are included in action log 320 may include: commenting on photo albums; communicating with users; establishing connections with objects. participating in an event; joining a group; creating an event; approving an application; using an application; to enter a transaction); Additionally, action log 320 may record user interactions with advertisements on online system 140 and other applications running on online system 140 . In some embodiments, data from the action log 320 is used to infer the interests or preferences of the user to enhance the interests contained in the user's user profile and further refine the user's preferences. make it understandable.

The action log 320 can also store user actions taken on a third party system 130 such as an external website and communicated to the online system 140, which are subject to the user's privacy settings. For example, it is possible for an e-commerce website to identify users of the online system 140 because the e-commerce website can recognize users of the online system 140 through social plug-ins. Because users of online system 140 are uniquely identifiable, as in the previous example, e-commerce websites may communicate information about user actions outside online system 140 to online system 140 for association with the user. can be done. As such, the action log 320 records information about actions the user takes on the third-party system 130, including browsing history of web pages, advertisements involved, purchases made, and other patterns from shopping and purchases. can be done. Additionally, actions performed by a user via an application associated with the third party system 130 and running on the client device 110 are recorded in the action log 320 and communicated by the application to the action logger 315 for association with the user. can be

In one embodiment, edge store 225 stores information describing connections between users and other objects on online system 140 as edges. Some edges can be user-defined, allowing users to specify relationships to other users. For example, a user can generate edges with other users, such as friends, co-workers, partners, etc., that match the user's real-life relationships. A user may, for example, express an interest in a page on online system 140, share a link with other users of online system 140, or comment on posts made by other users of online system 140. Other edges are generated when interacting with objects in the online system 140 .

Edges can include various features that characterize interactions between users, interactions between users and objects, or interactions between objects, respectively. For example, features included in an edge could be the rate of interaction between two users, how recently the two users interacted with each other, the rate or amount of information a user obtained about an object, or whether a user Describe the number and types of comments posted about the object. Features can also represent information that describes a particular object or user. For example, characteristics may represent information describing the level of interest a user has in a particular topic, the rate at which the user is logged into the online system 140, or demographic information about the user. Each feature can be associated with a source object or user, a target object or user, and a feature value. A feature can be specified as a value-based expression that describes a source object or user, a target object or user, or an interaction between a source object or user and a target object or user. As such, an edge may be represented as one or more feature expressions.

Edge store 325 also stores information about edges, such as affinity scores for objects, interests, and other users. A relevance score, or "relevance," may be calculated by the online system 140 over time to estimate a user's interest in objects or other users within the online system 140 based on actions taken by the user. . A user's relevance may be calculated by the online system 140 over time to estimate a user's interest in objects, topics, or other users within the online system 140 based on actions taken by the user. No. 12/978,265, filed Dec. 23, 2010; U.S. Patent Application No. 13/690,254, filed Nov. 30, 2012; No. 13/689,969, filed November 30, 2012, and U.S. Patent Application No. 13/690,088, filed November 30, 2012, wherein each Entirely incorporated by reference. In one embodiment, multiple interactions between a user and a particular object can be stored as an edge in edge store 225 . Alternatively, each interaction between the user and a particular object is stored as a separate edge. In some embodiments, connections between users can be stored in the user profile store 305, or the user profile store 305 can access the edge store 325 to determine connections between users.

Content selection module 330 selects one or more content items for transmission to client device 110 to be presented to the user. Content items suitable for presentation to the user are obtained from the content store 210 or from another source by a content selection module 230 that selects one or more of the content items for presentation to the viewing user. . A content item suitable for presentation to a user is a content item associated with at least a threshold number of targeting criteria satisfied by the user's characteristics, or a content item associated with no targeting criteria. In various embodiments, the content selection module 330 includes content items suitable for presentation to the user in one or more selection processes that identify a set of content items for presentation to the user. For example, the content selection module 330 determines the degree of relevance of various content items to the user based on attributes associated with the user by the online system 140 and based on the user's relevance to individual content items. . The degree of relevance of a content item to a user is based on the degree of quality of the content item to the user and is based on the creative contained in the content item and the landing page content identified by the links within the content item. be able to. Based on the degree of relevance, content selection module 330 selects content items for presentation to the user. As an additional example, the content selection module 330 selects content items having the highest degree of relevance, or having at least a threshold degree of relevance, for presentation to the user. Alternatively, the content selection module 330 ranks the content items based on their degree of relative relevance and selects the content items at the highest position in the ranking or at least a threshold position in the ranking. Select for presentation to the user.

Content items suitable for presentation to a user may include content items associated with a bid amount. The content selection module 330 uses the bid amount associated with the content item when selecting content for presentation to the user. In various embodiments, the content selection module 330 determines the expected value associated with various content items based on their bid amounts and determines the content item associated with the maximum expected value, or at least a threshold for presentation. Select the content item associated with the expected value. The expected value associated with the content item represents the expected amount of compensation to online system 140 for presenting the content item. For example, the expected value associated with a content item is the bid amount for the content item multiplied by the likelihood that a user will interact with the content item. The content selection module 330 may rank the content items based on the bid amounts associated with the content items and select those content items that are at least a threshold position in the ranking for presentation to the user. In some embodiments, the content selection module 330 provides a unified ranking of both non-bid related content items and bid related content items based on the bid amount and the degree of relevance associated with the content items. rank in. Based on the unified ranking, content selection module 330 selects content for presentation to the user. Selecting bid-relevant and non-bid-relevant content items through unified ranking is further described in U.S. patent application Ser. No. 13/545,266, filed July 10, 2012. and this patent application is incorporated herein by reference in its entirety.

For example, content selection module 330 receives a request to present a feed of content to a user of online system 140 . The feed contains content items, such as stories describing actions associated with users of other online systems connected to the user. Content selection module 330 accesses one or more of user profile store 305, content store 310, action log 320, and edge store 325 to obtain information about the user. For example, information describing actions associated with other users connected to the user or other data associated with users connected to the user is obtained. Content items from the content store 310 are retrieved and analyzed by the content selection module 330 to identify candidate content items suitable for presentation to the user. For example, content items associated with multiple users who are not connected to the user, or stories associated with multiple users with less than a threshold relevance to the user are discarded as candidate content items. Based on various criteria, the content selection module 330 selects one or more identified content items as candidate content items for presentation to the identified user. The selected content items are included in the feed of content presented to the user. For example, the feed of content includes at least a threshold number of content items describing actions associated with a plurality of users connected to the user via online system 140 .

In various embodiments, the content selection module 330 presents content to the user via a feed containing a plurality of content items selected for presentation to the user. Content selection module 330 may also determine the order in which the selected content items are presented via the feed. For example, the content selection module 330 orders content items in the feed based on the likelihood that a user will interact with various content items.

Based on the user's action or consent, the content selection module 330 receives video data captured by the imaging device 117 included in the client device 110 associated with the user of the online system and also receives video data. is sent to the receiving client device 110 for presentation to the viewing user via the display device 115 . The online system 140 can receive a request from the client device 110 identifying the viewing user, and can later provide the video data from the client device 110 to the receiving client device 110 in response to receiving approval from the viewing user. Conversely, online system 140 receives a request from a viewing user via recipient client device 110 and, in response to receiving approval from the user, forwards video data received from client device 110 to the recipient. provided to the client device 110; This allows different users of the online system 140 to exchange video data captured by the client device 110 associated with the user via the online system 140 .

The content selection module 330 can also receive instructions from the viewing user via the receiving client device 110 and send one or more instructions to the client device 110 . Based on the received instructions, the client device 110 modifies the video data captured after receiving the instructions, or modifies one or more parameters of the imaging device 117 based on the instructions. Accordingly, the client device 110 modifies the captured video data based on one or more instructions from the receiving client device 110 to render the modified video data or the video data captured by the imaging device 117 using the modified parameters. , to the content selection module 330, which transmits the video data to the receiving client device 110, which is further described below in connection with FIG. This allows the viewing user to make modifications or adjustments to the video data captured by the client device 110 , which is provided to the viewing user via the receiving client device 110 .

In various embodiments, content selection module 330 enforces one or more privacy settings of users of online system 140 . A user's privacy settings determine how certain information associated with the user can be shared, and can be stored in the user's user profile in user profile storage 305 . In one embodiment, privacy settings specify certain information associated with a user and also specify with which other entities the specified information may be shared. Examples of entities with which information can be shared may include other users, applications, third party systems 130, or any entity that can potentially access the information. Examples of information that a user may share include: user profile information (such as profile picture), phone number associated with the user, who the user is connected to, video data including the user, and video data taken by the user. Actions (such as adding connections), changing user profile information and the like. In various embodiments, online system 140 maintains privacy settings associated with users, which have default settings that prevent other entities from accessing or receiving content associated with users. and allows users to modify different privacy settings, thereby permitting other entities designated by the user to access or search for content corresponding to the modified privacy settings.

Privacy settings specifications can be provided at different granularity levels. In one embodiment, privacy settings may specify certain information to be shared with other users. For example, a privacy setting may specify a work phone number or a certain set of related information (such as a profile picture, home phone number and personal information including status). Alternatively, privacy settings can apply to all information associated with a user. A specification of the set of entities that can access a particular piece of information can also be specified at various levels of granularity. Various sets of entities with which information can be shared may include, for example: all users connected to the user, a set of users connected to the user, additional users connected to the user connected to the user. , all applications, all third party systems 130, a specific third party system 130, or all external systems.

One embodiment uses enumerations on entities to specify entities that are allowed to access specified information or to specify the types of information presented to different entities. For example, a user can specify the types of actions that are communicated to other users or to designated groups of users. Alternatively, users can specify action types and other information that will not be exposed or presented to other users.

Content selection module 330 determines whether particular information associated with the user can be accessed by other users connected to the user via online system 140, third-party system 130, and/or other application entities. Contains logic to determine Based on the user's privacy settings, the content selection module 330 may communicate information associated with the user (including information about actions taken by the user) to other users, third-party systems 130, applications, or other entities. determine whether access to the For example, the content selection module 230 uses the user's privacy settings to determine whether video data containing the user may be presented to another user. This allows a user's privacy settings to specify which users and other entities are allowed to receive: data about the user's actions; other data associated with the user.

Web server 335 links online system 140 to one or more client devices 110 and to one or more third party systems 130 over network 120 . Web server 335 serves web pages and other content, such as JAVA, FLASH, XML, and the like. The web server 335 can process messages, such as instant messages, queued messages (e.g., email), text messages, Short Message Service (SMS) messages, or other messages between the online system 140 and the client device 110 . can receive and forward messages sent using any suitable messaging technology. A user can send a request to web server 335 to upload information (eg, an image or video) stored in content store 310 . In addition, the web server 335 may be used to send data directly to a native client device operating system such as IOS®, ANDROID®, WEBOS®, or BlackBerry® OS. Application programming interface (API) functionality can be provided.

Modification of Video Data Capture Based on Features of Previously Captured Video Data FIG. 4 is an interaction diagram for one embodiment taken by imager 117 based on features of previously captured video. FIG. 4 is an interaction diagram of a method for modifying video data capture behavior made by device 117. FIG. In various embodiments, the steps described in connection with FIG. 4 may be performed in different orders. Also, in some embodiments, the method may include different and/or additional steps than those shown in FIG.

The imaging device 117 is included in the sending client device 110A, which is further described above in connection with FIGS. 1 and 2, to capture 405 video data of the local area around the sending client device 110A. do. Images captured by imaging device 117 are communicated to controller 210, which is contained within sending client device 110A (which in other embodiments is coupled to client device 110). In various embodiments, the user (e.g., through pre-selected privacy settings and/or prompted selections) tells the controller 210 to specify one or more features of the video captured 405 by the imaging device 117. Approval may be given for applying machine learning models to locate persons contained within the captured video data. In various embodiments, the controller 210 modifies the animation data to show the positioned user more prominently and transmits the modified animation data to the online system 140 . Positioned users are more prominently presented, and do this by presenting: within the video data with at least a threshold dimension set (e.g., with at least a threshold height and a threshold width) presenting; presenting over at least a threshold amount of the field of view of the imaging device 117 or over at least a threshold amount of frames of the captured video data; or presenting at one or more specific locations within the captured video data. For example, the controller 210 crops the captured video data to remove portions of the video data that do not include at least one person. As another example, controller 210 increases the zoom (also referred to as magnification) for portions of video data that include people. To modify the captured video data, the controller 210 can modify the video data after it has been captured by the imaging device 117 or modify one or more parameters of the imaging device 117 to determine how the imaging device 117 performs. can modify whether to capture (405) video data.

In various embodiments, controller 210 applies one or more methods to position the person in the captured video data. However, the controller 210 can similarly position objects (eg, appliances, furniture, products) by applying one or more models to the captured video data. Although the examples below relate to applying models to video data, one or more of the models may be applied to video data, audio data, image data, any other data captured by the client device 110, or any of them. can be applied to any combination of Controller 210 may use any suitable model or combination of models to locate the person in the video data captured 405 by imaging device 117 . The model applied to the captured video data by the controller 210 performs face tracking (in 2D or 3D), 2D pose tracking, 3D pose tracking, or any other suitable method to map parts of the human face. and human body parts can be identified. Similarly, the model applied by the controller 210 can identify objects from captured video data. In some embodiments, upon user authorization, controller 210 communicates with online system 140 to more specifically identify an object or person based on information obtained from online system 140; In other embodiments, the controller 210 maintains a model locally and identifies different objects or people from the captured video data. Based on the application of one or more models, the controller 210 modifies the cropping or zooming on the captured video data that includes the particular portion of the positioned user's body to make the particular portion of the positioned user's body more prominent. It can be displayed standing up. For example, if one or more models identify a person's face, the controller modifies the captured video data to remove portions of the video data that do not include the person's face. If the application of one or more models positions multiple people within the captured video data, controller 210 modifies the captured video data so that different portions of the video show different people. For example, the controller 210 divides the captured video data into grids, each area of the grid displaying one or more parts of a different person. In other embodiments, controller 210 increases the magnification (ie, zoom) of imaging device 117 for portions of video data that include portions of people. Accordingly, the controller 210 can crop a portion of the captured video data or increase the magnification (i.e., zoom) of a portion of the captured video data, thereby modifying the video data so that one person within the captured video data The part of the person described above can be presented more conspicuously. Also, when modifying the captured video data based on application of one or more models, the controller 210 may apply the one or more models to stabilize the modified video data and position one or more of the positioned persons. The part can be presented in higher quality.

Based on the privacy settings, the controller 210 applies one or more models that position the identified person's body parts to one or more parameters of the imaging device 117 or video data captured by the imaging device 117. Modifications can be made to modify the body parts of the positioned person contained within the captured video data. For example, the controller 210 finds different joints of the body of the identified person, and then modifies the captured video data or the parameters of the imaging device 117 to determine the joints corresponding to the different parts of the positioned person's body in the captured video data. can include Accordingly, the controller 210 may make corrections as to whether the captured video data includes the full body of the person, whether the person's head or the person's head and torso. The controller 210 can include various rules for modifying the human body parts of a person included in the captured video data, which can be based on: motion identified in previously captured video data; any other suitable features identified from the video data.

In various embodiments, one or more models applied by controller 210 apply one or more rules to modify video data captured 405 by imaging device 117 of sending client device 110A. For example, the controller 210 modifies the captured video data to make the person stand out more from the video data, such as modifying the zoom for the captured data for the part of the video data where the person is, or the video data where the person is. It does this by modifying the cropping for the part to remove non-human objects when the controller 210 determines that a human face is facing the camera. By way of example, the controller 210 modifies the captured video data to display the user's face more prominently, with the person's face directed toward the imaging device 117 (eg, one or more of the person's faces). is captured by the imaging device 117) and one or more models applied by the controller 210 are determined. In another example, the controller 210 determines the distance between persons in the video data and the imaging device 117 so that the person closest to the determined imaging device 117 is displayed prominently. Alternatively, the captured moving image data is modified so as to highlight persons whose determined distance from the imaging device is less than the threshold distance. In another example, the controller 210 applies one or more models to reposition the imaging device 117 as the person or object moves, such that the video data captured by the imaging device 117 is the movement of the person or object. be trackable.

One or more models applied by the controller 210 can modify the captured video data based on rules that take into account the locations of identified persons within the captured video data. In various embodiments, the user can give authorization to the controller 210 for positioning the user within the captured video, and the controller 210 applies a model to the captured video to allow the user's gaze to be reflected in the captured video data. can determine which position of the In response to determining that there are at least a threshold number or amount of positioned users with gazes directed to locations within the captured video data that include a particular person, the controller 210 modifies the captured video data. to show more of a particular person (eg, cropping captured video data to remove content other than a particular person, or increasing the magnification or zoom of a particular person). As another example, the controller 210 determines the distance between different persons in the video data and modifies the captured video data to more prominently display those who are within a threshold distance of each other. This allows the controller to modify the captured video data by cropping or zooming on the portion of the video data where the group of people is located. The controller 210 can also remove one or more frames from the captured video data based on objects or people identified within the captured video data. For example, if it is determined that less than a threshold number of objects or persons were identified in the captured video data, or that less than a threshold amount of object or person movement was identified in the captured video data, the controller 210 , remove frames from the captured video data before sending 410 the captured video data to the online system 140 . In another embodiment, the online system 140 removes frames from the video data received from the sending client device 110A, and uses the above conditions to pass the captured video data to the receiving client device 110B, as detailed below. Do this before sending (415) so that

The controller 210 can receive data from other components of the sending client device 110A and modify the captured video data based on characteristics of the received video data and data from other components of the sending client device 110A. . For example, imaging device 117 or client device 110 may include an audio capture device, such as a microphone, configured to capture audio data from a local area around client device 110 . The user may allow the controller 210 to process the captured audio data along with the captured video data in modifying the captured video data. In various embodiments, controller 210 applies one or more models to the captured audio data to determine locations within the captured video data, including the source of the audio data. The controller 210 applies one or more models to locations within the captured video data that contain the source of the audio data. In response to applying a model that makes a positional determination (including sources of audio data, including people) within the captured video data, the controller 210 modifies the captured video data to generate the captured video data, including sources of audio data. Presenting locations within the video data more prominently (i.e. zooming up for locations containing the source of the captured video data or cropping for locations containing the source of the captured video data to show objects other than the source of the captured video data) remove), or reposition the imaging device 117 to focus on the source of the audio data. However, in response to the controller 210 determining that a location within the captured video data that contains the audio data source does not contain a person, the controller does not modify the captured video data or reposition the imaging device 117 . As another example, the controller 210 may modify the captured video data or reposition the imaging device 117 to make the particular person that the controller 210 determined to be the source of the captured audio data in the captured video data more visible. This allows the captured video data to more prominently display the speaker or the person determined to provide the audio data captured by the audio capture device of the sending client device 110A (i.e. , Zoom up the position containing the person determined to be the audio data provider or crop the position containing the source of the captured video data to remove objects other than the person determined to be the audio data provider ).

In some embodiments, the controller 210 applies one or more models to allow the user to capture video data based on video data previously captured 405 by the imaging device 117 or the imaging device 117 of the client device 110 . You can also give permission to modify the parameters of For example, if the controller 210 has positioned multiple persons within the captured video data, the controller 210 may modify one or more parameters of the captured video data or the imaging device 117 so that each positioned person has the lowest position within the captured video. be prominently presented over a period of time (e.g., presented in video data with at least a threshold dimension set; across at least a threshold amount of the field of view of imaging device 117 or at least for frames of captured video data); presenting over a threshold amount; or presenting at one or more specific locations within the captured video data). As another example, subject to privacy settings, the controller 210 stores information identifying persons in the captured video data who have been prominently presented in the captured video data for at least a threshold period of time. If the controller 210 locates an additional person within the captured video data, the controller 210 compares the additional person with the stored information identifying the person who was prominently presented within the captured video data. In response to the additional person not being identified in the stored information, the controller 210 modifies the captured video data or modifies one or more parameters of the imaging device 117 so that the additional person is not identified in the threshold time period. To be displayed prominently across quantities. This allows the controller 210 to modify the video data such that each person positioned within the video data by the controller 210 is prominently displayed for at least a threshold amount of time.

The controller 210 may also modify parameters of the captured video data or the imaging device 117 in response to identifying movement of persons within the captured video data. For example, if one or more models applied to the captured video by the controller 210 determine that the positioned person is gesturing toward an object, the controller 210 modifies the captured video to match the positioned person. and objects to be presented prominently. For example, the captured video may show the positioned person prominently (e.g., displaying the positioned person with at least a set of threshold dimensions or displaying the positioned person for one or more frames of video data). display to occupy at least a threshold amount), and if the controller determines that the positioned person is gesturing toward the object, the controller 210 modifies the captured video so that the positioned person and the object are be presented. For example, the controller 210 can scale down the captured video data so that both positioned people and objects are included in the captured video. In another example, if the controller 210 applies one or more models to determine that the positioned person is holding an object, the controller 210 modifies one or more parameters of the captured video data or imaging device 117. so that the object is presented prominently (eg, changing the focus of the imager to the object and increasing the zoom of the imager 117). In some embodiments, the controller 210 can highlight the object held by the positioned person in the captured video data over a specific period of time, and modify the captured video data or imaging device 117 parameters to The positioned person can again be prominently presented in the captured video data.

Sending client device 110A transmits 410 video data from sending client device 110A to online system 140, which may be modified by controller 210 as further described below, and online system 140 transmits (415) the captured video data to the receiving client device 110B. Using the display device 115, the receiving client device 110B presents 420 the video data from the online system 140 to the viewing user of the online system 140. FIG. In various embodiments, a viewing user communicates a request to communicate with sending client device 110A from receiving client device 110B to online system 140 . Online system 140 communicates the request to sending client device 110A, and sending client device 110A provides online system 140 with a response. If the sending client device 110A provides authorization to the online system 140 in response to the request, the online system 140 captures (405) the video data captured 405 by the imaging device 117 of the sending client device 110A and provided to the online system 140. is provided to the receiving client device 110B for presentation on the display device 115, and vice versa.

In various embodiments, the user may direct the sending client device 110A (via predetermined privacy settings and user actions) to the persons positioned in the captured video by the controller 210. Corresponding one or more user-identifying information can be allowed to be provided to the online system 140 and sent 420 to the receiving client device 110B in which the one or more user-identifying information is included. accompanied by video data. Alternatively, to the online system 140, the sending client device 110A identifies the portion of the video data that includes the persons positioned by the controller 210, and the online system 140 identifies the persons positioned by the controller 210. portion of the video data with , to the stored image identifying the online system user. Based on the privacy setting, the online system 140 retrieves information identifying the user identified by the stored image, which image will appear on the sending client device 110A when the controller 210 locates the persons. The image determined by the online system 140 to have at least a threshold similarity with respect to the portion of the video data on the controller 210 . This allows the online system 140 to identify the users of the online system 140 included in the video data received from the sending client device 110A, if the users opt-in to such specific features. On-line system 140 includes the portion of the received video data with the person located by controller 210 as well as the image stored by on-line system 140 identifying the user (e.g., along with the profile image and identifying information in the user's user profile). The user applies one or more facial recognition processes or other identification processes to the images containing the user's face to be tagged, and the located person within the portion of the received video data is the online system user. can decide whether For example, online system 140 can augment the video data from sending client device 110A such that on the portion of the video data from sending client device 110A that includes a person identified by online system 140 as being an online system user: On the other hand, information identifying the online system user (eg, name, surname, email address, etc.) can be superimposed. The online system 140 transmits 420 the augmented video data to the receiving client device 110B. Alternatively, online system 140 generates information identifying online system users corresponding to persons in the video data received from sending client device 110A, and identifies online system users corresponding to persons in the video. The specified information is transmitted to the receiving client device 110B together with the video data. For example, the online system 140 may generate a list of names, surnames, user names, etc. for online system users corresponding to persons in the video data received from the client device 110A, and include this with the video data. Presented to the viewing user by the receiving client device 110B.

In some embodiments that allow users to use various user-related data to enhance the user experience of online system 140, online system 140 may use the viewing user and sender associated with recipient client device 110B. Identifies an online system user corresponding to a person in the video data received from the sending client device 110A, which can take into account the relationship between the users identified from the video data received from the client device 110A. This can be done at the time of generating the information to be used. For example, online system 140 generates identification information about online system users identified from video data received from sending client devices 110A that have connections to viewing users maintained by online system 140. do. In some embodiments, online system 140 provides information about online system users identified from video data received from sending client device 110A who do not have a connection to the viewing user maintained by online system 140. Specific information cannot be generated. Alternatively, the online system 140 visually displays the specific information of the online system user identified from the received video data connected to the viewing user and the specific information of the online system user not connected to the viewing user. distinguish between As another example, the online system 140 determines the viewing user relevance for each online system user identified from the video data received from the sending client device 110A and also determines the viewing user relevance identified from the video data. Modify the presentation of specific information for online system users. For example, the online system 140 generates information identifying the identified online system user from the received video data, which the online system 140 has at least a threshold relevance with respect to the viewing user. Visually distinguish identified online system users determined to be As another example, the online system 140 ranks information identifying online system users identified from the received video data, wherein viewing for online system users identified from the received video data. It does this based on user relevance. The online system 140 stores specific information about users who are online system users identified from the received video data and have at least a threshold ranking, and other online systems identified from the received video data. Information that is visually distinguishable from the user's specific information is generated. In another embodiment, the online system 140 generates information that presents information identifying online system users identified from the received video data. It does this based on the viewing user's relevance to the identified online system user.

The client device 110B on the receiving side presents (420) the video data from the online system 140 via the display device 115, whereby the client device 110A on the transmitting side presents the video data for presentation on the client device 110B on the receiving side. can provide. Information from the online system 140 that identifies the user of the online system 140 is presented along with the video data by the receiving client device 110B. From the presented video data, the receiving client device 110B receives (425) an interested user selection from the viewing user and transmits (430) information identifying the interested user to the online system 140. FIG. In various embodiments, a viewing user can select an object of interest. For example, the browsing user selects the information identifying the interested user from the information identifying the users of the online system 140 presented with the video data, and the receiving client device 110B and the receiving client device 110B are the interested users. , is sent 430 to online system 140 . The browsing user can select information identifying the interested user from the user-written information included in the moving image data and presented together with the moving image data by the receiving client device 110B. Alternatively, the viewing user selects the portion of the video data that includes the person, the receiving client device 110B identifies the selected portion of the video data presented to the online system 140, and the online system 140 selects the portion of the video data. Compare the content of the selection to stored images of the face and body associated with the user (e.g., images contained within the user profile or maintained by the online system 140 where various users are identified). to identify users associated with one or more stored images that match the content of the selected portion of the video as users of interest.

Online system 140 transmits 435 information identifying the user of interest to sending client device 110A. Based on the information identifying the user of interest and the characteristics of the objects contained in the video data captured 405 by the imaging device 117, the controller 210 may determine whether the captured video data or the imaging device 117 of the sending client device 110A are modified 440 to present the interested user more prominently. Modifications may be made after a notification about how the captured video is changed is displayed on the client device 110A for review and approval by the captured user, and/or prior approval (e.g., (approval via privacy settings), which relates to a list of pre-approved persons in control of the recipient client device 110B. Modifications that may be made to the imaging device 117 parameters include: increasing the magnification for portions of the video data that contain the user of interest; or removing portions of the video data that do not contain the user of interest. In various embodiments, the controller 210 of the sending client device 110A applies one or more models further described above to the captured video data to modify 440 the captured video data to better represent the user of interest. Present it prominently. The video data modified to highlight the user of interest is transmitted 445 from the sending client device 110A to the online system 140, where the online system 140 is presenting the user of interest prominently. The animation data is transmitted (450) to the receiving client device 110B, which presents (455) the modified animation data to the viewing user.

The controller 210 applies one or more models to the captured video data after receiving the information identifying the user of interest to modify one or more parameters of the imaging device 117 to adjust the number of users of interest around the local area. Track your target audience. For example, the controller 210 may apply one or more facial recognition models to the person(s) located within the captured video data (previously captured 405 based on information received from or from the online system 140). identifying a person's face that matches the face of the user of interest (identified from the video data obtained from the video data); focus on. Alternatively, the controller 210 extracts a color map from the captured video data containing the user of interest, repositions the focus of the imaging device 117, and inserts the extracted color map into the video data captured by the imaging device 117. remain included in the In some embodiments, controller 210 receives information identifying a user of interest and then modifies imaging device 117 to track the identified person or object, captured by imaging device 117 . moving image data to track the movement of an identified person or object.

In various embodiments, the viewing user provides additional instructions to the online system 140 for modifying the video data presented by the display device 115 of the receiving client device 110B, and the online system 140 provides the instructions to the sending client device 110B. Send to device 110A. Based on instructions from the online system 140, the controller 210 of the sending client device 110A modifies one or more parameters of the imaging device 117 or modifies the video data captured 405 by the imaging device 117. do. For example, instructions provided by a viewing user via the online system 140 may cause the controller 210 to configure the imager 117 of the sending client device 110A as follows: Allow the imaging device 117 to be repositioned as the user moves. As another example, instructions provided by a viewing user via an online system may result in video data captured 405 by imaging device 117 of sending client device 110A and presented via receiving client device 110B. Identifies an object in When the sending client device 110A receives instructions from the online system 140, the controller 210 modifies the captured video data or modifies one or more parameters of the image capture device 117 to send to additional client devices 110 thereafter. To ensure that the provided captured video data includes the specified object. This allows the viewing user to send a command to the sending client device 110A to modify the video data captured by the imaging device 117 of the sending client device 110A and presented to the viewing user via the receiving client device 110B. can provide. In various embodiments, a viewing user can provide instructions to the online system 140 for transmission to the sending client device 110A, thereby modifying the video data captured by the sending client device 110A, and This can be done without identifying the user of interest in the video data captured by 110A and presented 425 by the receiving client device 110B. This allows the viewing user to change the video data captured by the sending client device 110A.

Conclusion The above disclosure of exemplary embodiments is provided for purposes of illustration. They are not intended to be exhaustive, nor are they intended to limit patent rights to the precise forms disclosed. Persons of ordinary skill in the art can appreciate that many modifications and variations are possible in light of the above disclosure.

Some portions of this description describe embodiments of the invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to effectively convey the substance of their work to others skilled in the art. . These operations, although described functionally, computationally, or logically, will be understood to be implemented by computer programs or equivalent electrical circuitry, microcode, or the like. Moreover, it has also proven convenient at times, without loss of generality, to refer to these arrangements of operations as modules. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

any of the steps, operations, or processes described herein may be performed or implemented using one or more hardware or software modules, alone or in combination with other devices; is possible. In one embodiment, the software modules comprise computer program code executable by a computer processor to perform any or all of the steps, operations, or processes described. Implemented with a computer program product comprising a computer-readable medium.

Embodiments can also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, and/or it may be a general-purpose computing apparatus selectively activated or reconfigured by a computer program stored in the computer. can be provided. Such a computer program may be stored in a non-transitory, tangible computer-readable storage medium capable of being coupled to a computer system bus, or any type of medium suitable for storing electronic instructions. It is possible to Additionally, any computing system referred to herein may include a single processor, or may be a configuration employing a multi-processor design to increase computing capabilities.

Embodiments may also relate to products manufactured by the computing processes described herein. Such products may comprise information resulting from a computing process, which information is stored on a non-transitory tangible computer-readable storage medium and which is the computer program product or product described herein. Any embodiment of other data combinations can be included.

Finally, the language used herein has been chosen primarily for readability and instructional purposes and may not have been chosen to delineate or define patent rights. Accordingly, it is intended that the scope of patents be limited not by this detailed description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of embodiments is intended to be illustrative, not limiting, of the scope of patent, which is indicated in the following claims.

Embodiments in accordance with the invention are disclosed in the accompanying claims, particularly directed to methods and computer program products, wherein features recited in one claim category, e.g. Products, systems, and storage media can be similarly entitled. Dependencies or back references to the appended claims are chosen for formal reasons only. However, since any subject matter derived by intentional back-reference (especially multiple claim dependency) to any preceding claim may be claimed as well, any combination of claim and features thereof is disclosed and set forth in the appended claim. Claims can be made regardless of the subordination chosen. Claimable subject matter includes not only the combination of features as set forth in the appended claims, but also any other combination of the features of the claims, wherein each feature recited in a claim may be used in conjunction with any other feature or combination of the claims. It can be combined with other feature combinations. Also, any of the embodiments and features described or illustrated herein may be referred to in an individual claim and/or any embodiment or feature described or illustrated herein or any feature of an appended claim. and can be claimed in any combination.

According to one embodiment in accordance with the invention, a computer-implemented method may include the steps of:
providing captured video data for a local area within a field of view of an imaging device included in a client device, said video data including one or more users of an online system;
applying one or more models maintained by the client to features of the captured video data;
locating the one or more users of an online system included within the captured video data;
transmitting the video data to the online system;
receiving information from the online system identifying users of interest in the captured video data;
modifying one or more parameters of the imaging device to modify video data subsequently captured by the imaging device to crop or zoom portions of the video data containing the user of interest to better represent the user of interest; a highlighting step;
Sending video data subsequently captured by the imaging device using the modified one or more parameters to the online system.

modifying one or more parameters of the imaging device to modify video data subsequently captured by the imaging device to crop or zoom portions of the video data containing the user of interest to better represent the user of interest; The highlighting step may include the following steps:
increasing the magnification of the portion of the video data subsequently captured by the imaging device that includes the user of interest.

modifying one or more parameters of the imaging device to modify video data subsequently captured by the imaging device to crop or zoom portions of the video data containing the user of interest to better represent the user of interest; The highlighting step may include the following steps:
modifying the video data subsequently captured by the imaging device to remove one or more portions of the video data subsequently captured by the imaging device that do not include the user of interest step to do.

modifying one or more parameters of the imaging device to modify video data subsequently captured by the imaging device to crop or zoom portions of the video data containing the user of interest to better represent the user of interest; The highlighting step may include the following steps:
identifying the user of interest in video data captured by the imaging device;
Refocusing the imaging device to the user of interest.

Identifying the user of interest within video data captured by the imaging device may include the steps of:
Applying one or more facial recognition models to identify the user of interest in video data captured by the imaging device.

Identifying the user of interest within video data captured by the imaging device may include the steps of:
Extracting a color map from the portion of the captured video data that includes the user of interest.

According to one embodiment in accordance with the invention, a computer-implemented method may include the steps of:
receiving instructions from the online system to modify additional video data captured by the imaging device;
modifying the one or more parameters of the imaging device in response to the received instructions.

Modifying the one or more parameters of the imaging device in response to the received instructions may include:
Repositioning the imaging device as the user of interest moves within the field of view of the imaging device.

modifying one or more parameters of the imaging device to modify video data subsequently captured by the imaging device to crop or zoom portions of the video data containing the user of interest to better represent the user of interest; The highlighting step may include the following steps:
applying one or more models to the video data subsequently captured by the imaging device that identifies movement of the user of interest;
Modifying the field of view of the imaging device based on the identified movement of the user of interest.

According to one embodiment in accordance with the invention, a computer-implemented method may include the steps of:
receiving, at an online system, captured video data for a local area within a field of view of an imaging device included in a client device, said video data including one or more users of the online system; receiving;
retrieving information identifying the one or more users of the online system contained within the video data;
identifying users viewing the online system;
transmitting the video data and the information identifying the one or more users of the online system contained within the video data to an additional client device associated with the viewing user;
receiving information identifying users of interest of the one or more users of the online system included in the video data from the additional client device;
transmitting information identifying the user of interest to the client device;
receiving additional video data captured by the imaging device of the client device, the additional video data including a portion including the user of interest that has been cropped or zoomed to make it more prominent; a step;
transmitting the additional video data to the additional client device;

Retrieving information identifying the one or more users of the online system contained within the video data may include:
identifying a user of the online system included in the online system connected to the viewing user via the online system;
retrieving information identifying a user of the online system contained in the online system connected to the viewing user via the online system;

Retrieving information identifying the one or more users of the online system contained within the video data may include:
identifying a user of the online system included in the online system connected to the viewing user via the online system;
information specifying the user of the online system included in the moving image data, the user included in the moving image data connected to the browsing user via the online system being identified as the user via the online system; Generating information that visually distinguishes from users contained within the video data that are not connected to the viewing user.

Retrieving information identifying the one or more users of the online system contained within the video data may include:
determining the viewing user's relevance for each user of the online system included in the video data;
Generating information identifying each user of the online system included in the video data based on the viewing user's relevance to each user of the online system included in the video data.

The step of generating information specifying each user of the online system included in the video data based on the relevance of the viewing user to each user of the online system included in the video data is as follows: May include steps:
Generating information identifying each user of the online system included in the video data for the viewing users whose relevance is at least a threshold.

The step of generating information specifying each user of the online system included in the video data based on the relevance of the viewing user to each user of the online system included in the video data is as follows: May include steps:
ranking the users of the online system included in the video data based on the viewing user's relevance to the users of the online system included in the video data;
Generating information identifying each user of the online system included in the video data for users whose rank in the ranking is at least a threshold.

According to one embodiment in accordance with the invention, a computer-implemented method may include the steps of:
receiving instructions at the online system from the additional client device to modify video data captured by the imaging device of the client device;
sending said instructions to said client device;
receiving from the client device additional video data captured by the imaging device based on the instructions;
transmitting the additional video data from the online system to the additional client device;

In one embodiment according to the invention, in a computer program product comprising a computer readable storage medium having encoded instructions, the following steps may be taken by the processor when the instructions are executed by the processor:
providing captured video data for a local area within a field of view of an imaging device included in a client device, said video data including one or more users of an online system;
applying one or more models maintained by the client to features of the captured video data;
locating the one or more users of an online system included within the captured video data;
transmitting the video data to the online system;
receiving information from the online system identifying users of interest in the captured video data;
modifying one or more parameters of the imaging device to modify video data subsequently captured by the imaging device to crop or zoom portions of the video data containing the user of interest to better represent the user of interest; a highlighting step;
Sending video data subsequently captured by the imaging device using the modified one or more parameters to the online system.

modifying one or more parameters of the imaging device to modify video data subsequently captured by the imaging device to crop or zoom portions of the video data containing the user of interest to better represent the user of interest; The highlighting step may include the following steps:
modifying the video data subsequently captured by the imaging device to remove one or more portions of the video data subsequently captured by the imaging device that do not include the user of interest step to do.

modifying one or more parameters of the imaging device to modify video data subsequently captured by the imaging device to crop or zoom portions of the video data containing the user of interest to better represent the user of interest; The highlighting step may include the following steps:
identifying the user of interest in video data captured by the imaging device;
Refocusing the imaging device to the user of interest.

In one embodiment according to the invention, one or more computer-readable non-transitory storage media carry software operable when executed to perform the method according to the invention or any of the foregoing embodiments. can be materialized.

In one embodiment according to the invention, a system may comprise the following elements: one or more processors and at least one memory coupled to the processors, the memory containing instructions executable by the processors; A memory in which the processor is operable to implement instructions according to the present invention or any of the above-described embodiments as the processor executes the instructions.

In one embodiment according to the invention, a computer program product, preferably comprising a computer-readable non-transitory storage medium, performs the method according to the invention or any of the foregoing embodiments when executed on a data processing system. may be operable to

In one embodiment according to the invention, particularly when it is computer-implemented, the method may include the steps of:
maintaining one or more models on a client device, each model applying one or more rules based on features of the video data captured by the imaging device of the client device; determining which part of the body of a user of the online system is within the video data captured by
providing captured video data for a local area within the field of view of the imaging device contained within a client device;
locating the user included within the captured video data;
applying one or more of the maintained models to the captured video data; and modifying video data captured by the imaging device based on applying the one or more models to the user. cropping or zooming a portion of the video data that includes one or more portions of
sending said modified video data to said online system;

modifying the video data captured by the imaging device based on the application of the one or more models to crop or zoom the portions of the video data that include one or more of the user's portions; can contain:
identifying joints of the user's body;
Modifying video data captured by the imaging device to include joints corresponding to different parts of the user's body based on one or more rules contained in the one or more models.

modifying the video data captured by the imaging device based on the application of the one or more models to crop or zoom the portions of the video data that include one or more of the user's portions; can contain:
modifying said video data to include a selection of one or more from the following group: said user's head, said user's torso, said user's full body, and said one or more models; A group consisting of any combination thereof based on the application of one or more rules contained within.

The one or more rules may be based on one or more selected from the following group: content of video data previously captured by the imaging device; A group consisting of user movements and any combination thereof.

modifying the video data captured by the imaging device based on the application of the one or more models to crop or zoom the portions of the video data that include one or more of the user's portions; can contain:
determining from application of the one or more models that the user is gesturing toward an object within the local area;
Decreasing the magnification of the imaging device such that additional video data captured by the imaging device includes the user and the object to which the user is gesturing.

modifying the video data captured by the imaging device based on the application of the one or more models to crop or zoom the portions of the video data that include one or more of the user's portions; can contain:
determining from application of the one or more models that the user is holding an object within the local area;
Moving the vocal point of the imager to the object and increasing the magnification of the imager.

modifying the video data captured by the imaging device based on the application of the one or more models to crop or zoom the portions of the video data that include one or more of the user's portions; can contain:
Modifying one or more parameters of the imaging device that affect the field of view of the imaging device based on application of the one or more models.

The imaging device parameters may include the magnification of the imaging device.

modifying the video data captured by the imaging device based on the application of the one or more models to crop or zoom the portions of the video data that include one or more of the user's portions; can contain:
Modifying the video data captured by the imaging device to modify the magnification of the portion of the captured video data that includes the identified user.

Locating the user included within the captured video data may include the steps of:
receiving information identifying the user from the online system;
Based on the received information, identifying a portion of video data captured by the imaging device that includes the identified user.

In one embodiment according to the invention, in a computer program product comprising a computer readable storage medium having encoded instructions, the following steps may be taken by the processor when the instructions are executed by the processor:
maintaining one or more models on a client device, each model applying one or more rules based on features of the video data captured by the imaging device of the client device; determining which part of the user's body is within the video data captured by
providing captured video data for a local area within the field of view of the imaging device contained within a client device;
locating the user included within the captured video data;
applying one or more of the maintained models to the captured video data;
modifying video data captured by the imaging device based on application of the one or more models to crop or zoom portions of the video data that include one or more portions of the user;
sending said modified video data to said online system;

modifying the video data captured by the imaging device based on the application of the one or more models to crop or zoom the portions of the video data that include one or more of the user's portions; can contain:
identifying joints of the user's body;
Modifying video data captured by the imaging device to include joints corresponding to different parts of the user's body based on one or more rules contained in the one or more models.

modifying the video data captured by the imaging device based on the application of the one or more models to crop or zoom the portions of the video data that include one or more of the user's portions; can contain:
modifying said video data to include a selection of one or more from the following group: said user's head, said user's torso, said user's full body, and said one or more models; A group consisting of any combination thereof based on the application of one or more rules contained within.

The one or more rules may be based on one or more selected from the following group: content of video data previously captured by the imaging device; A group consisting of user movements and any combination thereof.

modifying the video data captured by the imaging device based on the application of the one or more models to crop or zoom the portions of the video data that include one or more of the user's portions; can contain:
determining from application of the one or more models that the user is gesturing toward an object within the local area;
Decreasing the magnification of the imaging device such that additional video data captured by the imaging device includes the user and the object to which the user is gesturing.

modifying the video data captured by the imaging device based on the application of the one or more models to crop or zoom the portions of the video data that include one or more of the user's portions; can contain:
determining from application of the one or more models that the user is holding an object within the local area;
Moving the focus of the imaging device to the object and increasing the magnification of the imaging device.

modifying the video data captured by the imaging device based on the application of the one or more models to crop or zoom the portions of the video data that include one or more of the user's portions; can contain:
Modifying one or more parameters of the imaging device that affect the field of view of the imaging device based on application of the one or more models.

The imaging device parameters may include the magnification of the imaging device.

modifying the video data captured by the imaging device based on the application of the one or more models to crop or zoom the portions of the video data that include one or more of the user's portions; can contain:
Modifying the video data captured by the imaging device to modify the magnification of the portion of the captured video data that includes the user.

Locating the user included within the captured video data may include the steps of:
receiving information identifying the user from the online system;
Based on the received information, identifying a portion of video data captured by the imaging device that includes the identified user.

In one embodiment according to the invention, one or more computer-readable non-transitory storage media carry software operable when executed to perform the method according to the invention or any of the foregoing embodiments. can be materialized.

In one embodiment according to the invention, a system may comprise the following elements: one or more processors and at least one memory coupled to the processors, the memory containing instructions executable by the processors; A memory in which the processor is operable to implement instructions according to the present invention or any of the above-described embodiments as the processor executes the instructions.

In one embodiment according to the invention, a computer program product, preferably comprising a computer-readable non-transitory storage medium, performs the method according to the invention or any of the foregoing embodiments when executed on a data processing system. may be operable to

In one embodiment according to the invention, particularly when it is computer-implemented, the method may include the steps of:
maintaining one or more models on a client device, each model applying one or more rules to locate a user within video data captured by an imaging device of the client device; determining and maintaining one or more of the positioned users to represent in data;
capturing video data for a local area within the field of view of the imaging device contained within a client device;
positioning a plurality of users within the captured video data;
applying one or more of the maintained models to the captured video data;
modifying the captured video data to indicate one or more of the positioned users based on application of the one or more of the maintained models;
transmitting the modified video data from the client device to an online system;

Modifying the captured video data to indicate one or more of the positioned users based on application of the one or more maintained models may include the steps of:
modifying one or more parameters of the imaging device based on application of the one or more of the maintained models;
Capturing additional video data for the local area using the modified one or more parameters of the imaging device.

Modifying the captured video data to indicate one or more of the positioned users based on application of the one or more maintained models may include the steps of:
determining where in the captured video data the positioned user's gaze is directed by application of one or more maintained models;
Correcting the focus of the imaging device to positioned users to which at least a threshold number of positioned users are gazed.

Modifying the captured video data to indicate one or more of the positioned users based on application of the one or more maintained models may include the steps of:
determining distances between different users positioned within the captured video data;
Modifying the focus of the imaging device so that additionally captured video data includes users within a threshold distance of each other.

Modifying the captured video data to indicate one or more of the positioned users based on application of the one or more maintained models may include the steps of:
capturing audio data via an audio capture device included in the client device;
determining the origin of the audio data within the local area by applying one or more of the maintained models to the captured audio data and the captured video data;
determining that the source of the audio data within the local area is a user;
Refocusing the imaging device to the source of the audio data in response to determining that the source of the audio data within the local area is the user.

Modifying the captured video data to indicate one or more of the positioned users based on application of the one or more maintained models may include the steps of:
identifying movement of the positioned user by applying one or more of the maintained models;
Modifying the field of view of the imaging device based on the identified movement of the positioned user.

Identifying motion of the positioned user by applying one or more of the maintained models may include the steps of:
identifying that the positioned user is gesturing toward an object contained within the local area;

Modifying the field of view of the imager may include the steps of:
Decreasing the magnification of the imaging device such that additional video data captured by the imaging device includes the positioned user and the object to which the positioned user is gesturing.

Identifying motion of the positioned user by applying one or more of the maintained models may include the steps of:
Determining from the application of the one or more maintained models that the positioned user is holding an object within the local area.

Modifying the field of view of the imager may include the steps of:
Moving the focus of the imaging device to the object and increasing the magnification of the imaging device.

Modifying the captured video data to indicate one or more of the positioned users based on application of the one or more maintained models may include the steps of:
Modifying the captured video data such that each positioned user is presented in the captured video data at least a threshold dimension and for at least a threshold period of time.

Modifying the captured video data to indicate one or more of the positioned users based on application of the one or more maintained models may include the steps of:
Modifying the captured video data, wherein each positioned user will be presented in the captured video data for at least a threshold period of time.

Modifying the captured video data to indicate one or more of the positioned users based on application of the one or more maintained models may include the steps of:
identifying a user's face in the captured video data by applying one or more of the maintained models;
Modifying the captured video data to remove portions of the captured video data that do not include the face of the user.

Modifying the captured video data to indicate one or more of the positioned users based on application of the one or more maintained models may include the steps of:
modifying the captured video data to include each of the positioned users;

Modifying the captured video data to indicate one or more of the positioned users based on application of the one or more maintained models may include the steps of:
Stabilizing the captured video data by applying the one or more maintained models.

Modifying the captured video data to indicate one or more of the positioned users based on application of the one or more maintained models may include the steps of:
comparing, wherein positioned users in said captured video data are included in said captured video data that have been previously presented with at least a threshold dimension in said captured video data. stored data describing the user; and comparing;
modifying the captured video data to prominently present the located user, wherein the identified user has at least a threshold dimension before being included in the captured video data. , in response to a comparing step that determines that the captured video data does not match.

In one embodiment according to the invention, in a computer program product comprising a computer readable storage medium having encoded instructions, the following steps may be taken by the processor when the instructions are executed by the processor:
maintaining one or more models on a client device, each model applying one or more rules to locate a user within video data captured by an imaging device of the client device; determining and maintaining one or more of the positioned users to be represented within the video data from data characteristics;
capturing video data for a local area within the field of view of the imaging device contained within a client device;
positioning a plurality of users within the captured video data;
applying one or more of the maintained models to the captured video data;
Modifying the captured animation data to indicate one or more of the positioned users based on application of the one or more of the maintained models.
transmitting the modified video data from the client device to an online system;

Modifying the captured video data to indicate one or more of the positioned users based on application of the one or more maintained models may include the steps of:
determining where in the captured video data the positioned user's gaze is directed by application of one or more maintained models;
Refocusing the imaging device to a positioned person having at least a threshold number of identified persons in their line of sight.

Modifying the captured video data to indicate one or more of the positioned users based on application of the one or more maintained models may include the steps of:
determining distances between different users positioned within the captured video data;
Modifying the focus of the imaging device so that additionally captured video data includes persons within a threshold distance of each other.

Modifying the captured video data to indicate one or more of the positioned users based on application of the one or more maintained models may include the steps of:
capturing audio data via an audio capture device included in the client device;
determining the origin of the audio data within the local area by applying one or more of the maintained models to the captured audio data and the captured video data;
determining that the source of the audio data within the local area is a located user;
Refocusing the imaging device to the source of the audio data in response to determining that the source of the audio data within the local area is the located user.

In one embodiment according to the invention, one or more computer-readable non-transitory storage media carry software operable when executed to perform the method according to the invention or any of the foregoing embodiments. can be materialized.

In one embodiment according to the invention, a system may comprise the following elements: one or more processors and at least one memory coupled to the processors, the memory containing instructions executable by the processors; A memory in which the processor is operable to implement instructions according to the present invention or any of the above-described embodiments as the processor executes the instructions.

In one embodiment according to the invention, a computer program product, preferably comprising a computer-readable non-transitory storage medium, performs the method according to the invention or any of the foregoing embodiments when executed on a data processing system. may be operable to

Claims (19)

A computer-implemented method for modifying video data captured by a client device (110) including an imaging device (117), wherein a controller (210) is coupled to the imaging device (117), the method comprising: ,
said imaging device (117) providing captured first video data for a local area within the field of view of said imaging device (117) , said first video data being one of an online social network system; comprising at least one user , and wherein the method comprises :
said controller (210) maintaining one or more privacy settings for said one or more users captured within said first video data;
Based on the one or more privacy settings of the one or more users, the controller (210) converts one or more models maintained by the client device (110) into the captured first animation. locating the one or more users of the online social network system included within the captured first video data by applying to data characteristics ;
sending said first video data including said one or more users positioned to said online social network system;
receiving , via the online social network system, information identifying users of interest from among the one or more users positioned within the captured first video data;
Based on the one or more privacy settings of the user of interest, the controller (210) modifies one or more parameters of the imaging device (117) to be subsequently captured by the imaging device (117). modifying the second moving image data to make the interested user stand out more by cropping or zooming a portion of the second moving image data including the interested user;
and sending the second video data subsequently captured by the imaging device (117) using the modified one or more parameters to the online social network system. Implementation method. 2. The computer-implemented method of claim 1, wherein one or more parameters of the imaging device (117) are modified to modify second video data subsequently captured by the imaging device (117) to The step of cropping or zooming a portion of the second video data containing the target user to make the user of interest stand out more,
A computer-implemented method comprising increasing the magnification of a portion of said second video data subsequently captured by said imaging device (117) containing said user of interest. 3. The computer-implemented method of claim 1 or claim 2, wherein one or more parameters of the imaging device (117) are modified to modify second video data subsequently captured by the imaging device (117). and cropping or zooming a portion of the second moving image data containing the interested user to make the interested user stand out more,
modifying the second moving image data subsequently captured by the imaging device (117) to provide one of the second moving image data subsequently captured by the imaging device (117) that does not include the interested user; A computer-implemented method comprising the step of removing said portion. 4. The computer-implemented method of any of claims 1-3, wherein one or more parameters of the imaging device (117) are modified to affect second video data subsequently captured by the imaging device (117). Cropping or zooming a portion of the second video data that includes the user of interest by modifying to make the user of interest stand out more,
identifying the user of interest in the second video data subsequently captured by the imaging device (117) ;
refocusing said imaging device (117) onto said user of interest. 5. The computer-implemented method of claim 4, wherein identifying the user of interest within the second video data subsequently captured by the imaging device (117) comprises:
A computer-implemented method, comprising applying one or more facial recognition models to identify the user of interest in the second video data subsequently captured by the imaging device (117) . 5. The computer-implemented method of claim 4, wherein identifying the user of interest within the second video data subsequently captured by the imaging device (117) comprises:
A computer-implemented method, comprising extracting a color map from the portion of the subsequently captured second video data that includes the user of interest. 7. The computer-implemented method of any of claims 1-6,
receiving instructions from the online social network system to modify the second video data subsequently captured by the imaging device (117) ;
and modifying said one or more parameters of said imaging device (117) in response to said received instructions. 8. The computer-implemented method of claim 7 , wherein modifying said one or more parameters of said imaging device (117) in response to said received instructions comprises:
A computer-implemented method comprising repositioning the imaging device (117) as the user of interest moves within the field of view of the imaging device (117 ). 9. The computer-implemented method of any of claims 1-8 , wherein one or more parameters of the imaging device (117) are modified to affect second video data subsequently captured by the imaging device (117). Cropping or zooming a portion of the second video data that includes the user of interest by modifying to make the user of interest stand out more,
applying one or more models identifying the movement of the user of interest to the second video data subsequently captured by the imaging device (117) ;
modifying the field of view of the imaging device (117) based on the identified movement of the user of interest. A computer-implemented method comprising:
receiving in an online system video data for a local area within the field of view of an imaging device (117) included in a client device (110) , said video data being sent to one or more users of the online system; receiving;
retrieving information identifying the one or more users of the online system contained within the video data;
identifying users viewing the online system;
transmitting said video data and said information identifying said one or more users of said online system contained within said video data to an additional client device (110) associated with said viewing user;
receiving information identifying users of interest of the one or more users of the online system included in the video data from the additional client device (110) ;
transmitting information identifying the user of interest to the client device (110) ;
receiving additional video data captured by the imaging device ( 117 ) of the client device (110 ), the additional video data including the user of interest cropped or zoomed to make it more prominent. receiving, including a portion;
and transmitting said additional video data to said additional client device (110) . 11. The computer-implemented method of claim 10 , wherein retrieving information identifying the one or more users of the online system contained within the video data comprises:
identifying users of the online system included in the online system connected to the viewing user via the online system;
retrieving, via the online system, information identifying users of the online system contained in the online system connected to the viewing user. 12. The computer-implemented method of claim 10 or claim 11 , wherein retrieving information identifying the one or more users of the online system contained within the video data comprises:
identifying users of the online system included in the online system connected to the viewing user via the online system;
Information specifying the user of the online system included in the video data, wherein the user included in the video data connected to the viewing user via the online system is identified via the online system generating information that visually distinguishes from users contained within the video data that are not connected to the viewing user. 13. The computer-implemented method of any of claims 10-12 , wherein retrieving information identifying the one or more users of the online system contained within the video data comprises:
determining the viewing user's relevance for each user of the online system included in the video data;
and generating information identifying each user of the online system included in the video data based on the viewing user's relevance to each user of the online system included in the video data. , a computer-implemented method. 14. The computer-implemented method of claim 13 , wherein information identifying each user of the online system included in the video data is associated with the viewing user for each user of the online system included in the video data. The step to generate based on
A computer-implemented method comprising generating information identifying each user of the online system included in the video data for the viewing users having at least a threshold relevance. 14. The computer-implemented method of claim 13 , wherein information identifying each user of the online system included in the video data is associated with the viewing user for each user of the online system included in the video data. The step to generate based on
ranking the users of the online system included in the video data based on the viewing user's relevance to the users of the online system included in the video data;
generating information identifying each user of the online system included in the video data whose position in the ranking is at least a threshold. 16. The computer-implemented method of any of claims 10-15 ,
receiving instructions from the additional client device (110) to modify the additional video data captured by the imaging device (117) of the client device (110) in the online system;
sending said instructions to said client device (110) ;
receiving from the client device (110) the additional video data captured by the imaging device (117 ) based on the instructions;
and transmitting said additional video data from said online system to said additional client device (110) . A computer program product comprising a computer readable storage medium having encoded instructions captured by a client device (110) including an imaging device (117) when executed by a processor. Execute a method for modifying video data, the method comprising:
providing captured first video data for a local area within the field of view of said imaging device (117) , said first video data including one or more users of an online social network system; and The method includes:
maintaining one or more privacy settings for the one or more users captured within the first video data;
applying one or more models maintained by the client device (110) to features of the captured first video data based on the one or more privacy settings of the one or more users ; locating the one or more users of the online social network system included in the captured first video data by:
sending said first video data including said one or more users positioned to said online social network system;
receiving , via the online social network system, information identifying users of interest from among the one or more users positioned within the captured first video data;
modifying one or more parameters of the imaging device (117) based on the one or more privacy settings of the user of interest to render second video data subsequently captured by the imaging device (117); modifying and cropping or zooming a portion of the second video data containing the user of interest to make the user of interest stand out more;
sending the second video data subsequently captured by the imaging device (117) using the modified one or more parameters to the online social network system ;
A computer program product characterized by : 18. The computer program product of claim 17 , wherein one or more parameters of the imaging device ( 117 ) are modified to modify second video data subsequently captured by the imaging device (117) to The step of cropping or zooming a portion of the second video data containing the target user to make the user of interest stand out more,
modifying the second moving image data subsequently captured by the imaging device (117) to provide one of the second moving image data subsequently captured by the imaging device (117) that does not include the interested user; A computer program product comprising the step of removing said portion. 19. The computer program product of claim 17 or claim 18 , wherein one or more parameters of said imaging device (117) are modified to modify second video data subsequently captured by said imaging device (117) . and cropping or zooming a portion of the second moving image data containing the interested user to make the interested user stand out more,
identifying the user of interest in the second video data subsequently captured by the imaging device (117) ;
refocusing said imaging device (117) to said user of interest.

Images