KR20200039814A - Modify capture of video data by the video capture device based on video data previously captured by the video capture device - Google Patents

Abstract

Various client devices include one or more image capture devices and displays configured to capture video data. Different users of the online system can allow client devices to exchange information captured by their respective video capture devices. Additionally, the client device modifies the captured video data based on users identified in the video data. For example, the client device changes parameters of the image capture device to more prominently display the user identified in the video data, and, based on the user's movement or gestures identified in the video data, changes the parameters of the image capture device. You can change more. The client device can apply a plurality of models to the captured video data to modify the captured video data, or to modify the subsequent capture of the video data by the video capture device.

Description

Modify capture of video data by the video capture device based on video data previously captured by the video capture device

The present disclosure relates generally to capturing video data, and in particular, to modifying the capture of video data based on previously captured video data.

Client devices, online systems, and networks increasingly allow users to exchange large amounts of content with each other. For example, an online system allows users to exchange video data captured by different users through client devices associated with the user. In a particular example, an online system can establish video messaging between a user and an additional user while allowing users to exchange video data captured by their respective client devices in real time or near real time.

However, conventional client devices when providing video data require the user to manually configure video video capture. For example, to identify the focus of the image capture device of the client device, to specify the enlargement of the image capture device of the client device, or to specify other parameters of the image capture device of the client device, The user of the client device provides inputs to the client device. In addition to providing initial inputs that identify the parameters of the client device's image capture device, to reposition to different portions of the local area within the field of view of the client device's image capture device, conventional The client devices of the user require the user to manually reposition the image capture device of the client device. Therefore, the user manually selects and maintains the content captured by the video capture device of the client device communicated to other client devices. This reliance on user-provided input gives the user considerable control over the video data captured by the client device, but conventional client devices allow the user to easily perform other video messaging while capturing video data through the client device. It does not allow to perform other tasks in the session because conventional client devices could not adjust video data capture as a change in conditions without receiving one or more inputs provided by the user. .

Various client devices associated with users of the online system include one or more video capture devices. The video capture device included in the client device is configured to capture video data in a local area around the client device, for example, during a video call or when a user turns on a video capture feature. Additionally, the client device includes a controller coupled to one or more video capture devices. The controller applies one or more models to video data captured by the image capture device. Then, based on the application of one or more models, parameters of the image capture device and / or video data captured by the image capture device are corrected. This allows the controller to modify the captured video data based on the content or characteristics of the video data previously captured by the image capture device.

The controller maintains and enforces one or more privacy settings for users or others captured in video or other data in various embodiments. For example, the controller can have a default privacy setting that prevents the controller from identifying the user until the user manually changes the privacy setting to allow the controller to identify the user. The default privacy setting also allows any user to select whether the video capture device allows any user in the data to be special in order to identify any user in the data, any captured video data, audio data, video data, or Can be extended to other data. Additionally, privacy settings also control the transmission of information from a client device to another entity (eg, another client device or a third party system). Various privacy settings allow the user to control the user's identification and control the storage and sharing of any user-related data. Privacy settings can also be enforced individually for each individual. For example, a user who opts in a user identification feature cannot change default privacy settings for other users that may be accidentally captured in a local area around the client device.

In various embodiments, based on the user's privacy selections that enable the client device to identify users, the controller may include video captured by the video capture device to locate users included in the captured video data. To the data, one or more machine learned models are applied. Models applied by the controller to the captured video data can perform facial tracking (two-dimensionally or three-dimensionally), two-dimensional pose tracking, three-dimensional pose tracking, or any other suitable In a manner, it may be performed to identify portions of the human face or portions of the human body. In various embodiments, the controller modifies the captured video data or parameters of the image capture device to more prominently present to the located users. For example, the controller crops captured video data to remove portions of video data that do not include at least one person. As another example, the controller corrects the focus of the image capture device on the human face, and increases the zoom (ie, zoom) of the image capture device. In various embodiments, users have options to prevent any records (video, voice, etc.) from being stored locally on the client device and / or in the cloud, and also delete any records if stored. Have options.

In various embodiments, when user identification is enabled, one or more models applied to the controller apply one or more rules to modify video data captured by the client device's image capture device. For example, if the controller also determines that a person's face is facing the camera, the controller modifies the captured video data in order to more prominently display the person located within the video data. In another example, the controller determines distances between people identified from the video data and the image capture device, and modifies the captured video data, so that a person with the minimum determined distance from the image capture device is at least thresholded by the video data. The dimensions of the value set are presented (e.g., at least a threshold height and a threshold width are presented, or using at least a threshold percentage of the field of view of the image capture device), or a determined distance from the image capture device People less than the threshold distance are displayed in at least the dimensions of the threshold set.

The controller can receive data from other components of the client device, and can modify the captured video data based on the characteristics of the video data and the data from other components of the client device. For example, an image capture device or client device includes an audio capture device, such as a microphone configured to capture audio data from a local area around the client device. The controller can process the captured audio data along with the captured video data when modifying the captured video data, subject to user-selected privacy settings. In various embodiments, the controller can apply one or more models to the captured audio data to determine a location within the captured video that includes the source of the audio data. The controller applies one or more models to a location in the captured video data that contains a source of audio data. In response to the application of models that determine that the location in the captured video data includes a source of audio data and includes a person, the controller can more prominently present the location in the captured video data that includes the source of audio data, Correct the captured video data, or reposition the video capture device to focus on the source of the audio data (eg, increase the field of view of the video capture device occupied by the source of the audio data, and the video capture device To change the location of the source of audio data within the field of view). However, the controller does not modify the captured video data or reposition the image capture device in response to determining that the location in the captured video data containing the source of the audio data does not include a person. As another example, the captured audio data while allowing the captured video data to display more prominently the user determined to be speaking, or otherwise providing the audio data captured by the audio capture device of the client device. To more prominently identify the identified person in the captured video data, which the controller determines as the source of, the controller modifies the captured video data or repositions the video capture device (e.g., the source of the audio data). Increase the field of view of the video capture device occupied by, and change the position of the source of the audio data in the field of view of the video capture device).

Identifying the user of interest to modify one or more parameters of the video capture device to follow the user of interest around the local area, in other situations where the user has turned on the video capture and user identification features, or in a video messaging session. After receiving the information, the controller may apply one or more models to the captured video data. For example, the controller may be configured to identify a person's face that matches the face of a user of interest (identified from video data previously received from the online system, or based on information from the online system), or Further facial recognition models are applied to people located in the captured video data, and subsequently the focus of the image capture device is repositioned to a person with a face matching the face of the user of interest. Alternatively, the controller extracts the color map from the captured video data containing the user of interest and repositions the focus of the image capture device, so that the extracted color map remains contained within the video data captured by the image capture device do.

Embodiments in accordance with the present invention are disclosed in the appended claims, particularly aimed at methods and computer program products, where one claim category, e.g., any feature referred to in a method, is claimed in another claim category, e.g. , Computer program products, systems, and storage media. Subsequent dependent or cited claims in the appended claims are selected for formal reasons only. However, any subject arising from an intentional recitation claim after any previous claims (in particular, multiple dependent claims) may also be claimed, so any combination of the claims and their features is disclosed and selected from the appended claims It can be claimed regardless of the dependent claims. The subject matter that may be claimed may include combinations of features presented in the appended claims, as well as any other combination of features in the claims, with each feature mentioned in the claims being any other in the claims. It can be combined with a feature or a combination of other features. Also, any of the features and embodiments depicted or described herein may be claimed in a separate claim, and / or any combination or combination of claims or appended claims described or described herein Any combination of any of the features can be claimed.

1 is a block diagram of a system environment operated by an online system according to an embodiment.
2 is a block diagram of a client device according to an embodiment.
3 is a block diagram of an online system according to an embodiment.
4 is a flow diagram of a method of selecting a segment of video data for presentation to a user of interest in an online system based on the possibilities of the user's gaze directed to different locations in the video data, according to an embodiment.
The drawings depict various embodiments for illustrative purposes only. Those skilled in the art will readily appreciate from the following description that alternative embodiments of the methods and structures illustrated herein can be used without departing from the principles described herein.

System architecture

1 is a block diagram of a system environment 100 for an online system 140. The system environment 100 illustrated by FIG. 1 includes one or more client devices 110, a network 120, one or more third party systems 130, and an online system 140. . Additionally, in the system environment 100 shown by FIG. 1, the controller 117 is connected to the client device 110. In alternative configurations, different and / or additional components can be included in the system environment 100. For example, online system 140 is a social networking system, a content sharing network, or other system that provides content to users.

The client devices 110 are one or more computing devices capable of transmitting and / or receiving data over the network 120 as well as receiving user input. In one embodiment, the client device 110 is a conventional computer system, such as a desktop computer or notebook computer. Alternatively, the client device 110 may be a computer-functional device, such as a personal digital assistant (PDA), mobile phone, smartphone, or other suitable device. The client device 110 is configured to communicate over the network 120. In one embodiment, client device 110 executes an application that allows a user of client device 110 to interact with online system 140. For example, client device 110 runs a browser application to enable interaction between client device 110 and online system 140 via network 120. In another embodiment, the client device 110 children OS (IOS®) or Android (ANDROID ^TM) and application programming interfaces on-line system 140 via a (API) operating on the native operating system of the client device 110, such And interact. As described further below in connection with FIG. 2, the client device 110 includes a display device 115 configured to present content, and to capture video data or images of a local area around the client device 110. It comprises one or more image capture devices configured.

The client device 110 communicates through the network 120, which may include any combination of local area and / or wide area networks, using wired and / or wireless communication systems. It is configured to. In one embodiment, network 120 uses standard communication technologies and / or protocols. For example, the network 120 includes Ethernet, 802.11, and WiMAX (worldwide interoperability for microwave access), 3G, 4G, code division multiple access (CDMA), digital subscriber Communication links using technologies such as digital subscriber line (DSL). Examples of networking protocols used to communicate over the network 120 include multiprotocol label switching (MPLS), transmission control protocol / Internet protocol (TCP / IP), hypertext transport protocol (HTTP), and simple mail. Transfer protocol (SMTP), and file transfer protocol (FTP). Data exchanged over the network 120 may be represented using any suitable format, such as hypertext markup language (HTML) or extensible markup language (XML). have. In some embodiments, some or all of the communication links of network 120 may be encrypted using any suitable technology or techniques.

One or more third party systems 130 may be connected to network 120 to communicate with online system 140, which is further described below in connection with FIG. In one embodiment, the third party system 130 communicates information describing applications for execution by the client device 110, or client devices 110 for use by an application running on the client device. ) Is an application provider that communicates data. In other embodiments, the third party system 130 provides content or other information for presentation via the client device 110. The third party system 130 may also communicate information, such as information, advertisements, or content, about the application provided by the third party system 130 to the online system 140.

2 is a block diagram of an embodiment of a client device 117. In the embodiment shown in FIG. 2, the client device 110 includes the display device 115 and the image capture device 117 as well as the controller 210. However, in other embodiments, the client device 110 includes different or additional components than those shown in FIG. 2.

The display device 115 may be integrated into the client device 110 or may be connected to the client device 110. For example, the display device 115 integrated in the client device 110 is a display screen included in the client device 110. Alternatively, the display device 115 is a monitor or other display connected to the client device 110. The display device 115 presents image data or video data to the user. The video or video data presented by the display device 115 is determined by an application running on the client device 110. Different applications can be included on the client device 110, such that the execution of different applications changes the content presented by the user by the display device 115.

The image capture device 117 captures images or video data of a local area around the client device 110 and a local area within the field of view of the image capture device 117. In some embodiments, image capture device 117 includes one or more cameras, one or more video cameras, or any other device capable of capturing image data or video data. Additionally, the image capture device 117 can include one or more filters (eg, used to increase the signal-to-noise ratio). Various parameters (e.g., focal length, focus, frame rate, ISO, sensor temperature, shutter speed, aperture, focal point, etc.) are image data or video data by the image capture device 117. Make up the capture. Therefore, modifying one or more parameters of the image capture device 117 is modifying video data or image data captured by the image capture device 117 after modification of the one or more parameters. 2 shows a single image capture device 117 included within the client device 110, but in other embodiments, the client device 110 includes any suitable number of image capture devices 117. In various embodiments, users have options to prevent any records (video, voice, etc.) from being stored locally in the client device and / or on the cloud, and also delete any records if stored. Have options.

The controller 210 includes a storage device connected to the image capture device 117 and connected to the processor. In various embodiments, the controller 210 is also connected to the display device 115. The controller 210 includes instructions that, when executed by the processor, apply one or more models to the video data captured by the image capture device 117. In various embodiments, one or more models are applied to video data, audio data, image data, or any combination of data captured by image capture device 117, connected to client device 110, or client It applies to any combination of data captured by any other devices included in the device. As described further below in connection with FIG. 4, the models applied to the video data captured by the controller 210 can be used for any other suitable content, objects, people, or movement of the captured video data. To identify, one or more rules are applied to the characteristics of the captured video data. Based on the application of the models, and subject to one or more privacy settings, the controller 210 modifies the captured video data or modifies one or more parameters of the image capture device 117, subsequently captured Video data is modified. For example, users may allow the controller 210 to apply models that locate users in the captured video data, based on characteristics of the captured video data, and more significantly include the located users. In order to modify the captured video data or modify one or more parameters of the image capture device 117 (eg, focal length, enlargement or zoom, cropping of the captured video data), additional video The data includes users more prominently located. At least the dimensions of the threshold set (eg, at least the threshold height or the threshold width) are presented to the located person, or at least in the field of view of at least a threshold amount of the image capture device 117 or at least of the captured video data By presenting a person located at a threshold amount of frames, or by presenting a person located at one or more specific locations in the captured video data, additional video data more significantly includes the person located. However, the models applied by the controller 210 can identify any suitable components of the video data captured by the image capture device 117, and thus can modify the parameters of the image capture device 117, Alternatively, the captured video data can be modified.

In various embodiments, client device 110 includes one or more audio capture devices, such as microphones. For example, the client device 110 includes an array of microphones configured for beamforming in two or three dimensions. The audio capture device captures audio signals from different areas within the local area around the client device 110. In various embodiments, one or more audio capture devices are coupled to the controller 210, which maintains information identifying different regions of the local area around the client device 110, eg, the controller 210 ) Identifies the areas of the 360-degree local area around the client device 110, while identifying the 24 15-degree areas of the local area around the client device 110 starting from a point in the client device 110.

One or more audio capture devices are connected to the controller 210. Subject to user-selected privacy settings, the controller 210 may record one or more models, such as machine learning models or other sound recognition models, from areas of the local area around the client device 110, audio data. Applies to. The controller 210 includes information identifying users or objects (eg, a television, mobile device), and one or more models are captured as audio captured from a region of the local area around the client device 110. The application of determines whether the captured audio data includes ambient noise or audio data from an object or user identified by controller 210. In some embodiments, one or more models applied by the controller 210 determine a specific user or specific object identified by the controller 210, from which one or more audio capture devices are located within the region. Is captured. In other embodiments, the client device 110 communicates the audio data captured by the one or more audio devices to the online system 140, which applies one or more models to the audio data, Determine whether to include audio data, captured from the identified object or user, or determine a specific identified user or object from which the audio data was captured. The online system 140 provides the client device 110 with an indication of whether the captured audio data includes the identified object or audio data captured from the user, or a specific identified object in which the audio data has been captured. Or, it provides an indication of whether or not to provide information specifying a user. The controller 210 or online system 140 similarly determines whether audio captured from other regions surrounds the local area of the client device 110. Based on the identified objects 'or users' decisions, where audio data in different regions has been captured, the controller 210 may perform one or more audio to improve the quality of the audio captured from one or more regions. Correct the positioning of devices. For example, controller 210 repositions one or more audio capture devices to improve the quality of audio captured from an area around a local area, where audio data is captured from a particular user or from a specified object. . Similarly, the controller 210 sets one or more parameters of the image capture device 117 based on regions of the local area around the client device 110 where audio data was captured from different users or objects. It can be repositioned or modified otherwise. The one or more audio devices and image capture device 117 may be directed to different portions of the local area around the client device 110 in various embodiments. For example, the image capture device 117 is directed to an object described by the user, while the controller 210 is a client device 110 in which audio data is captured by one or more audio capture devices by a specific user Directs you to the area of the surrounding local area.

The controller 210 of the client device 110 and the online system 140 cooperate and / or individually, in various embodiments, one or more for people or users, identified from captured video data or other data. Maintain and enforce the above privacy settings. The user or person's privacy settings determine how specific information associated with the user or person can be shared, and can be stored in association with information identifying the user or person. In some embodiments, controller 210 retrieves privacy settings for one or more users, maintained by online system 140. In one embodiment, privacy settings specify specific information associated with the user and identify other entities to which the specified information can be shared. Examples of entities in which information may be shared may include other users, applications, third-party systems 130, or any entities potentially accessing information. Examples of information that can be shared by the user include video data including the user or person, audio data including audio captured from the user or person, video data including the user or person, and the like.

For example, in certain embodiments, privacy settings can be determined by the first user (eg, by opt out, by not opt in), online system 140 is optional. For purposes of this, it may allow to specify whether information or special objects associated with the user can be received, collected, logged, or stored. In certain embodiments, the privacy settings determine whether the first user can access, store, or use certain objects or information associated with the user, certain video capture devices, audio capture devices, applications, or processes. Can be specified. Privacy settings can allow a first user to opt in or opt out of having information or objects that are accessed, stored, or used by specific devices, applications, or processes. Online system 140 may access such information for providing certain functions or services to a first user without the online system 140 having access to the information for any other purpose. have. Before accessing, storing, or using such objects or information, the online system can induce the user to provide privacy settings, which, if any, any applications or processes can Specifies whether an object or information can be accessed, stored, or used before allowing an action. By way of example, and not limitation, a first user may send a message to a second user through an application related to an online social network (eg, a messaging app), and those messages may be sent by the online system 140. You can specify privacy settings that should not be saved.

Privacy settings maintained and enforced by the online system 140 and / or controller 210 may be associated with default settings. In various embodiments, if the controller 210 does not obtain a privacy setting from a user that allows the controller 210 to identify the user, the controller 210 may capture video data, audio data, image data, or other It does not identify the user in the data. For example, the privacy settings associated with the user have default settings that prevent the controller 210 from identifying the user, unless the user manually changes the privacy settings to allow the controller 210 to identify the user, The controller 210 does not identify the user. Additionally, alternative privacy settings may, in various embodiments, be a client that identifies a user to another entity (eg, another client device 110, online system 140, third party system 130). The transmission of information from the device 110 is restricted. The alternative privacy setting prevents the controller 210 from transmitting information identifying the user to other entities unless the user manually modifies the alternative privacy setting to permit transmission, in various embodiments. It has a default setting that prevents the transmission of information identifying the user. The controller 210 allows one or more privacy settings for each user identified from the captured video data or other data, while allowing user-specific control of each user's identification and transmission. To maintain. In some embodiments, the controller 210 allows the person to provide privacy settings when the controller 210 first identifies the person from the captured data and stores the provided privacy settings in association with information identifying the person. Induces.

In various embodiments, the online system 140 and / or various components have functions that can use the user's personal information or biometric information as inputs for user-authentication or experience-personalization purposes. For field client device 110, the user may opt to use these functions to enhance their experience using the device and online system. By way of example, and not limitation, the user may voluntarily provide personal information or bio-measurement information to the online system 140. The user's privacy settings can specify that such information can be used only for certain processes, such as authentication, and such information cannot be shared with any third party, or other associated with the online system 140. It can further specify that it cannot be used for processes or applications. As another non-limiting example, the online system 140 may provide a function for a user to provide voice print recordings to an online social network. As a non-limiting example, if a user wants to take advantage of these features of an online social network, the user can provide a voice recording of his or her own voice to provide status updates on the online social network. have. To determine which words were spoken by the user, the recording of voice-input can be compared to the user's voice print. The user's privacy settings are such that voice recording improves speech recognition to authenticate voice-input purposes (eg, to use voice-operating features of online social networks, authenticating the user), sending voice messages. ), And can further specify that such a voice recording cannot be shared with any third-party system or used by other processes or applications associated with the online system 140. . As another non-limiting example, the online system 140 may provide a function for a user to provide a reference image (eg, a face profile) to an online social network. The online social network may compare the reference image to the subsequently received image input (eg, to authenticate the user, to tag the user in pictures). The user's privacy settings can specify that such voice recordings can only be used for a limited purpose (eg, authentication, tagging users in photos), and such voice recordings can be shared with any third party system. It can further specify that it cannot or cannot be used by other processes or applications associated with the system 140. Any of those restrictions regarding biomeasurement data and / or other personal data capture may also be applied to client device 110.

Users can authenticate data capture, identification of users, and / or sharing of user-related data and use between applications in one or more ways. For example, the user can pre-select various privacy settings before the users use the features of the client devices 110 and / or take actions on the online system 140. In other cases, when users first perform the action, when users use the features of the client devices 110 and / or the online system 140, and / or when the user has not performed the action or is predetermined Selective conversation can be induced, such as whether the feature has not been used for a period of time. In another example, to allow users to make additional choices through notifications, the client devices 110 and the online system 140 also have some features that require user data to be selected by the users. This can provide users with notifications of when to start operating or to be disabled. Other suitable ways for users to authorize are also possible.

In some embodiments, the controller 210 obtains information maintained by the online system 140, subject to privacy settings for the user, or one or more for the user identified from the captured video data. Obtain information from third-party systems 130. Based on the video data, audio data, image data, or other information, including the information obtained and the user previously captured by the client device 110, the controller 210 is presented to the user through the client device 110. You can create content to do. For example, the controller 210 is associated with one or more objects identified by the controller 210 from image data or video data captured by the client device 110, from the online system 140. Overlay content items. Alternatively, the online system 140 is a client and information maintained by the online system 140 for the user (or obtained from one or more third-party systems 130 by the online system 140). Create content for the user based on other data, video data, image data, or audio data, including the user received from the device 110, and provide the generated content to the client device 110 to present to the user do.

3 is a block diagram of the architecture of the online system 140. The online system 140 shown in FIG. 3 includes a user profile store 305, a content store 310, an action logger 315, an action log 320, and an edge store 325. And a content selection module 330 and a web server 335. In other embodiments, online system 140 may include additional, fewer, or different components for various applications. Conventional components, such as network interfaces, security functions, load balancers, failover severs, management and network operation consoles, and the like, provide details of the system architecture. Not to be ambiguous, not shown.

Each user of online system 140 is associated with a user profile, which is stored in user profile store 305. The user profile includes descriptive information about the user that has been clearly shared by the user, and may also include profile information deduced by the online system 140. In one embodiment, the user profile includes a plurality of data fields, each of which describes one or more attributes of the corresponding online system user. Examples of information stored in the user profile include electronic, demographic, and other types of descriptive information, such as work experience, educational history, gender, hobbies or preferences, address, and the like. The user profile may also store other information provided to the user, for example images or videos. In some embodiments, images of users may be tagged with information identifying online system users displayed in the image, and tagged with information stored in the user's user profile to identify images tagged by the user. Can be. The user profile in the user profile store 305 can also maintain criteria for actions, performed by the corresponding user on content items in the content store 310 and stored in the action log 320.

Additionally, the user profile maintained for the user allows one or more client devices associated with the user, while allowing the online system 140 to subsequently identify the user from the properties provided by the client device 110. It includes the characteristics of (110). For example, an application associated with the online system 140 and running on the client device 110 may have other information uniquely identifying the client device 110 to the online system 140 in association with a user identifier. Alternatively, a device identifier is provided. If the online system 140 receives other information or device identifiers that uniquely identify the client device 110, while allowing the user's subsequent identification, the online system 110 is able to locate the client device in the user profile maintained for the user ( 110) stores other information or device identifiers that uniquely identify. Other characteristics of client devices 110 associated with the user may alternatively or additionally be included in the user profile maintained by the user. For example, the user profile is the network 120, the identifier of the application running on the client device 110 on which the online system 140 has received the information, and the client device 110 on which the online system 140 has received the information. Type of client (eg, manufacturer, identifier of the model of the client device 110, etc.), and the client device 110 to access the operating system running on the client device 110 where the online system 140 has received the information. ). However, while allowing the online system 140 to maintain information about the client devices 110 used by the user corresponding to the user profile, the online system 140 can be used to view any of the client devices 110 in the user profile. It can store suitable characteristics of.

User profiles in the user profile store 305 allow individuals to interact with each other through the online system 140 and are often associated with individuals, but user profiles are also stored for entities such as businesses or organizations. Can be. This allows the entity to establish presence on the online system 140, in order to exchange and access content with other online system users. The entity may post information about itself, its products, or other information to users of the online system 140 using a brand page associated with the entity's user profile. Can provide. Other users of the online system 140 may access the brand page to receive information posted on the brand page, or to receive information from the brand page. The user profile associated with the brand page may include information about the entity itself, while providing information or background to the entity about the entity.

The content store 310 stores objects each representing various types of content. Examples of content represented by objects are page posts, status updates, photos, videos, links, shared content items, game application achievements, check-in events in local businesses, branded pages, or any other kind of content. Contains content. Online system users can view objects stored by the content store 310, such as status updates, pictures, events, groups or applications tagged by users to be associated with other objects in the online system 140. Can be created. In some embodiments, objects are received from third party applications or from third party applications remote from online system 140. In one embodiment, objects in content store 310 represent “content items” or fragments of content. Therefore, online system users are encouraged to communicate with each other by posting content items and text of various types of media to the online system 140 through various communication channels. This increases the amount of users' interaction with each other and the frequency with which users interact within the online system 140.

The one or more content items included in the content store 310 include creative, which is content for presentation to the user, and a bid amount. Creative is text, video, audio, video, or any other suitable data presented to the user. In various embodiments, the creative also specifies a page of content. For example, the content item includes a link that specifies the network address of the landing page of the content to which the user is directed when the content item is accessed. In the case where the user is presented with content, if the content in the content item is presented to the user, the content in the content item when presented receives user interaction, or any suitable condition when the content in the content item is presented to the user If satisfied, the bid amount is included in the content item by the user and is used to determine an expected value, such as a monetary reward, provided to the online system 140 by the advertiser. For example, the amount of beads included in the content item specifies the amount of money received by the online system 140 from a user who provided the content item to the online system 140 if the content within the content item is displayed. In some embodiments, the expected value for the online system 140 of presenting content from a content item may be determined by multiplying the amount of beads by the probability that the content of the content item is accessed by the user.

Various content items may include objects that identify interactions that a user associated with the content item wants other users to perform when the content included in the content item is presented. Examples of objects include installing an application associated with the content item, indicating preferences for the content item, sharing the content item with other users, interacting with the object associated with the content item, or any And performing other suitable interactions. Since the content from the content item is presented to online system users, the online system 140 logs interactions between users, presented with the content item or with objects associated with the content item. Additionally, since online system users perform interactions with a content item that satisfies an object included in the content item, the online system 140 is compensated by a user associated with the content item.

Additionally, the content item can include one or more targeting criteria, as specified by the user who provided the content item to the online system 140. The target criteria included in the content item request to specify one or more characteristics of users suitable for being presented with the content item. For example, target criteria are used to identify users with actions, user profile information, or edges that satisfy at least one of the target criteria. Therefore, target criteria allow the user to identify users with specific characteristics, while simplifying the subsequent distribution of content to different users.

In one embodiment, target criteria may specify types or actions of an object of the online system 140 or other user-to-user connections. Target criteria may also specify interactions between users and objects performed outside the online system 140, such as on the third party system 130. For example, target criteria include sending a message to another user, using an application, joining a group, leaving a group, participating in an event, creating an event description, and using an online marketplace Or identify users who have taken special actions, such as purchasing or reviewing services, requesting information from third-party system 130, installing an application, or performing any other suitable action. Including actions in target criteria allows users to further refine users who are eligible to be presented with a content item. As another example, target criteria identify users who have a connection to another user or object, or a special kind of connection to another user or object.

Based on privacy settings, the action logger 315 adds an action log 320 with information about the user actions, and the action logger 315 is a user inside and / or outside the online system 140. It may be authorized to receive communications for operations. Examples of actions include adding a connection to another user, sending a message to another user, uploading an image, reading a message from another user, watching content associated with another user, and other And attending events posted by the user. In addition, multiple actions may include an object and one or more special users, such actions are also associated with special users and stored in the action log 320.

Based on privacy settings, the action log 320 is online to track user actions on the online system 140, as well as actions on third-party systems 130 that communicate information to the online system 140. What is used by system 140 may be authorized by users. Users can interact with various objects on the online system 140, and information describing these interactions is stored in the action log 320. Examples of interactions with objects include commenting on posts, sharing links, checking in to physical locations through client device 110, accessing content items, and any Other suitable interactions are included. Additional examples of interactions with objects on the online system 140 included in the action log 320 include commenting on the photo album, communicating with the user, establishing a connection with the object, and events. This includes participating, joining a group, creating an event, authorizing an application, using an application, expressing preferences for an object ("likes" an object), and engaging in a transaction. . Additionally, the action log 320 can record user interactions with advertisements on the online system 140 as well as other applications running on the online system 140. In some embodiments, data from the action log 320 is used to infer the user's preferences or interests, while augmenting the interests included in the user's user profile and allowing a more complete understanding of user preferences. .

The action log 320 may also communicate user actions taken on a third party system 130, such as an external website, communicated to the online system 140, subject to the user's privacy settings. For example, an e-commerce website, via a social plug-in that enables an e-commerce website to identify a user of the online system 140, is an online system 140. ). Since users of the online system 140 are uniquely identifiable, e-commerce websites, such as the previous example, communicate information about user actions outside the online system 140 to the online system 140 for association with the user. Can deliver. Therefore, the action log 320 includes actions performed by users on the third-party system 130, including web page viewing histories, involved advertisements, purchases made, and other patterns from shopping and purchases. Information about can be recorded. Additionally, operations performed on the client device 110 and performed by the user through an application associated with the third party system 130 may be performed for association and recording with the user in the action log 320. , It can be communicated to the action logger (action logger) 315 by the application.

In one embodiment, the edge store 325 stores information describing connections between users and other objects on the online system 140 as edges. Some edges can be defined by users, allowing users to specify relationships with other users. For example, users can create edges with other users in parallel with their real-life relationships, such as friends, colleagues, partners, and the like. Other edges include users expressing interest on pages on the online system 140, sharing links with other users of the online system 140, and posts made by other users of the online system 140. Created when interacting with objects in online system 140, such as commenting on.

The edge may include various features, each representing characteristics of interactions between users, interactions between users and objects, or interactions between objects. For example, the features included in the edge are the rate of interaction between two users, how recently two users interacted with each other, the amount or percentage of information retrieved by one user for an object, Or describe the number and types of comments posted by the user for the object. Features can also represent information describing a particular object or user. For example, a feature may indicate the level of interest a user has on a particular topic, the rate at which the user logs into the online system 140, or information describing demographic information about the user. Each feature can be associated with a source object or user, a target object or user, and a feature value. A feature can be specified as a source object or as a representation based on values describing the user, target object or user, or interactions between the source object or user and the target object or user, so the edge is one or more features Can be expressed as expressions.

The edge store 325 also stores information about edges, such as affinity scores for objects, interests, and other users. The intimacy scores, or “intimacies”, are based on actions performed by the user, by the online system 140 to approach the user's interests in an object or in another user in the online system 140. Can be calculated over time. The user's intimacy is timed by the online system 140 to approximate the user's interests, in an object, or in a topic, in another user in the online system 140, based on actions performed by the user. Can be calculated for. Calculation of intimacy is based on U.S. patent application number 12 / 978,265 filed on December 23, 2010, U.S. patent application number 13 / 690,254 filed on November 30, 2012, filed on November 30, 2012 It is further described in U.S. Patent No. 13 / 689,969, and U.S. Patent No. 13 / 690,088 filed on November 30, 2012, each of which is incorporated herein by reference in its entirety. Multiple interactions between a user and a particular object may be stored as a single edge in the edge store 325 in one embodiment. Alternatively, each interaction between the user and a particular object is stored as a separate edge. In some embodiments, connections between users can be stored in user profile store 305, or user profile store 305 can access edge store 325 to determine connections between users.

The content selection module 330 selects one or more content items for communication to the client device 110 to be presented to the user. Content items eligible for presentation to the user are retrieved from the content store 310 or from other sources by the content selection module 330, selecting one or more content items to present to the viewing user. A content item eligible to be presented to a user is a content item associated with target criteria of at least a threshold number satisfied by the user's characteristics, or a content item not associated with a target criterion. In various embodiments, content selection module 330 includes, in one or more selection processes, content items eligible for presentation to a user, which identifies a set of content items for presentation to the user. For example, the content selection module 330 may relate the relevance of various content items to the user based on attributes associated with the user by the online system 140 or based on the user's affinity for different content items ( Determine the measures of relevance. The degree of relevance of the content item to the user is based on the degree of quality of the content item to the user, which is not only based on the creative contained in the content item, but also on the content of the landing page identified by links within the content item. You can. Based on the degrees of relevance, the content selection module 330 selects content items for presentation to the user. As an additional example, the content selection module 330 selects content items having the highest degree of relevance or at least a threshold level of relevance for presentation to the user. Alternatively, the content selection module 330 ranks content items based on their associated degrees of relevance and has the highest position in ranking to present to the user, or at least a threshold position in ranking. Select such content items.

Content items eligible for presentation to the user may include content items associated with the bead amounts. The content selection module 330 uses bead amounts associated with content items when selecting content for presentation to a user. In various embodiments, the content selection module 330 determines the expected value associated with various content items based on their bead amounts, and determines content items associated with the maximum expected value or at least the expected value of the threshold value. Choose to present. The expected value associated with the content item represents the expected amount of compensation to the online system 140 for presenting the content items. For example, the expected value associated with a content item is a product of the bead amount of the content item and the likelihood that the user will interact with the content item. The content selection module 330 can rank content items based on their associated bead amounts, and select content items having at least a threshold location in ranking to present to the user. In some embodiments, the content selection module 330, in unified ranking based on degrees of relevance and bead amounts associated with content items, content items and bead amounts not associated with bead amounts Ranks all of the content items associated with. Based on unified ranking, the content selection model 330 selects content for presentation to the user. Through unified ranking, selecting content items associated with bead quantities and content items not associated with bead quantities is further described in U.S. Patent No. 13 / 545,266 filed on July 10, 2012 And this patent is hereby incorporated by reference in its entirety.

For example, the content selection module 330 receives a request to present a feed of content to a user of the online system 140. The feed contains content items, such as stories, that describe actions associated with other online system users connected to the user. The content selection module 330 accesses one or more user profile stores 305, content stores 310, action logs 320, and edge stores 325 to retrieve information about the users. For example, information describing actions associated with other users connected to the user or other data associated with users connected to the user is retrieved. Content items from content store 310 are retrieved and analyzed by content selection model 330 to identify candidate content items eligible for presentation to the user. For example, content items associated with users who are not connected to the user or stories associated with users whose user has less than a threshold intimacy are discarded as candidate content items. Based on various criteria, the content selection model 330 selects one or more content items identified as candidate content items for presentation to the identified user. The selected content items are included in the feed of content presented to the user. For example, the feed of content includes at least a threshold number of content items that describe actions associated with users connected to the user through the online system 140.

In various embodiments, the content selection module 330 presents the content to the user through a feed, including a plurality of content items selected for presentation to the user. The content selection model 330 can also determine the order in which selected content items are presented through the field. For example, based on the possibility of a user interacting with various content items, content selection module 330 orders the content items in the feed.

Based on the user's action or permission, the content selection module 330 receives the video data captured by the image capture device 117 included in the client device 110 associated with the user of the online system, and the display device 115 ), The video data is transmitted to the receiving client device 110 to present to the viewing user. The online system 140 may receive a request from the client device 110 identifying the user being viewed, and in response to receiving permission from the user being viewed, the receiving client device from the client device 110. The video data is subsequently provided. Conversely, the online system 140, via the receiving client device 110, receives a request from the viewing user and subsequently receives a permission from the user, in response to receiving a request from the user. Video data is provided subsequent to the receiving client device 110. This allows different users of the online system 140 to exchange video data captured by the client devices 110 associated with the users through the online system 140.

Additionally, the content selection module 330 may receive instructions from the viewing user through the receiving client device 110 and send one or more instructions to the client device 110. Based on the received instructions, the client device 110 modifies the captured video data after receiving the instructions, or modifies one or more parameters of the image capture device 117 based on the instructions. Thus, the client device 110 modifies the captured video data, based on one or more instructions from the receiving client device 110, sends the modified video data, or uses the modified parameters. The video data captured by the video capture device 117 is transmitted to the content selection module 330, which transmits video data to the receiving client device 110, and is further described below in connection with FIG. . This allows the viewing user to modify or adjust the video data captured by the client device 110 and provided to the viewing user through the receiving client device 110.

The content selection module 330 enforces one or more privacy settings of users of the online system 140 in various embodiments. The user's privacy settings determine how special information associated with the user can be shared and stored in the user's user profile in the user profile store 305. In one embodiment, privacy settings specify special information associated with the user and identify other entities to which the specified information can be shared. Examples of entities for which information may be shared may include other users, applications, third-party systems 130, or any entity that may potentially access information. Examples of information that may be shared by the user include profile pictures, phone numbers associated with the user, user's connections to video data including the user, adding connections, and changing the user profile information to the user. And user profile information, such as actions taken. In various embodiments, the online system 140 maintains privacy settings associated with the user, corresponding to the modified privacy settings, with default settings that prevent other entities from accessing or receiving content associated with the user. Allows the user to modify different privacy settings to allow other entities designated by the user to search or access the content.

Privacy settings can be provided in different levels of granularity. In one embodiment, privacy settings can identify specific information to be shared with other users. For example, the privacy settings identify a particular set of related information, such as a work phone number or profile picture, a home phone number, and personal information including status. Alternatively, privacy settings can be applied to all information associated with the user. The designation of a set of entities that can access special information can also be specified in units of various levels. The various sets of entities to which information can be shared include, for example, all users connected to a user, a set of users connected to a user, additional users connected to users connected to the user, all applications Field, all third party systems 130, specific third party systems 130, or all external systems.

One embodiment uses a list of entities to identify types of information presented to different entities or to specify entities allowed to access the identified information. For example, a user can specify the types of operations that are communicated to a specified group of users or to other users. Alternatively, the user may specify types of actions or other information that is not presented or disclosed to other users.

The content selection module 330 may be accessed by other users, with which information associated with the user is connected to the user through the online system 140, third party system 130, and / or other applications and entities. Includes logic to determine if it can. Based on the user's privacy settings, the content selection module 330 includes information associated with the user, including information about actions taken by the user by another user, third-party system 130, application, or other entity. Decide if you are allowed to access. For example, the content selection module 230 uses the user's privacy settings to determine whether video data including the user can be presented to other users. This allows the user's privacy settings to specify, which other users or some other entities are allowed to receive data about the user's actions or other data associated with the user.

The web server 335 not only links the online system 140 to one or more client devices 110 via the network 120, but also to one or more third-party systems 130. The web server 335 not only serves web pages, but also serves other content such as Java (JAVA®), Flash (FLASH®), and XM (XML). The web server 335 can receive and route messages between the online system 140 and the client device 110, such as instant messages, queued messages (eg , Email), text messages, short message service (SMS) messages, or any other suitable messaging technique to receive or route the transmitted messages. The user can send a request to the web server 335 to upload information (eg, images or videos) stored in the content store 310. Additionally, the web server 335 is programmed for application to send data directly to native client device operating systems, such as IOS®, Android (ANDROID ^™ ), or BlackberryOS. It can provide an interface (API) function.

Modify video data capture based on previously captured characteristics of video data

4 is an interactive diagram of one embodiment of a method of modifying the capture of video data by the image capture device 117 based on the characteristics of the video previously captured by the image capture device 117. In various embodiments, the steps described in connection with FIG. 4 can be performed in different orders. Additionally, in some embodiments, the method may include different and / or additional steps than those shown in FIG. 4.

The image capture device 117 is connected to FIGS. 1 and 2 and is included in the transmitting client device 110A, as described above, and transmits video data of a local area around the transmitting client device 110A. Capture 405. Images captured by the image capture device 117 are communicated to the controller 210 included in the transmitting client device 110A (or connected to the client device 110 in other embodiments). In various embodiments, the user (eg, through pre-selected privacy settings and / or prompted choices) allows controller 210 to locate people included in the captured video data. , It may allow applying one or more machine-trained models to the characteristics of the video captured 405 by the image capture device 117. The controller 210 modifies the video data to more prominently locate the located users and, in various embodiments, transmits 410 the modified video data to the online system 140. The located user is presented in video data having at least a set of threshold dimensions (eg, at least with a threshold height or with a threshold width), or in the field of view of at least a threshold amount of the image capture device 117 Presented in at least a threshold amount of frames of video data, or more prominently at one or more specific locations within the captured video data. For example, the controller 210 crops the captured video data to remove portions of the video data that do not include at least one person. As another example, controller 210 increases the zoom (also referred to as magnification) of a portion of video data that includes a person. To modify the captured video data, the controller 210 can modify the video data after being captured by the image capture device 117, or how the image capture device 117 captures the video data (405). To correct, one or more parameters of the image capture device 117 may be modified.

In various embodiments, controller 210 applies one or more methods to locate people within the captured video data. However, the controller 210 can similarly locate objects (eg, household appliances, furniture, products) by applying one or more models to the captured video data. The following examples refer to the application of models to video data, but one or more models are video data, audio data, image data, any other data captured by client device 110, and any combination thereof. Can be applied to. The controller 210 can use any suitable model or combination of models to locate people within the video data captured 405 by the image capture device 117. Models applied to the captured video data by the controller 210 are facial tracking (two-dimensional or three-dimensional), two-dimensional poses to identify parts of the human body or parts of the human face. It may perform pose tracking, three-dimensional pose tracking, or any other suitable method. Similarly, the models applied by controller 210 can identify objects from the captured video data. In some embodiments, based on the user's permission, the controller 210 communicates with the online system 140 to more specifically identify objects or people based on information obtained from the online system 140. On the other hand, in other embodiments, the controller 210 maintains the models locally to identify different objects or people from the captured video data. Based on the application of one or more models, the controller 210 captures the video data, including certain parts of the located user's body, to more prominently display any part of the located user's body You can modify the cropping or zooming of. For example, when one or more models identify a human face, the controller modifies the captured video data to remove portions of the video data that do not include the human face. If the application of one or more models places a plurality of people in the captured video data, the controller 210 modifies the captured video data so that different parts of the video display different people. For example, the controller 210 divides the captured video data into a grid, where each area of the grid displays one or more portions of a different person. In other embodiments, the controller 210 increases the magnification (ie, zoom) of the image capture device 117 on a portion of video data that includes a portion of a person. Thus, the controller 210 can crop portions of the captured video data to modify the video data to more prominently present portions of one or more people located within the captured video data, or Alternatively, it is possible to increase magnification (ie, zoom) of portions of the captured video data. Additionally, based on the application of one or more models, when modifying the captured video data, the controller 210 modifies the video to present portions of one or more located people at higher quality. One or more models can be applied to stabilize the data.

Based on the privacy settings, the controller 210 also locates parts of the identified person's body to modify portions of the located person's body, included in the captured video data, and captures the image capture device ( One or more models may be applied to modify the video data captured by 117) or to modify one or more parameters of the image capture device 117. For example, the controller 210 positions different joints of the identified person's body to include joints corresponding to different parts of the located person's body, in the captured video data, and images Modify the parameters of the capture device 117 or the captured video data. Therefore, the controller 210 can modify whether the captured video data includes a human head, a human head and torso, or a whole body of a person. The controller 210 is included in the captured video data, based on the movement identified in the previously captured video data, any other suitable characteristics identified from the video data, or content included in the previously captured video data. It can include various rules for modifying parts of a person's body.

In various embodiments, the one or more models applied by the controller 210 are one or more to modify the video data captured 405 by the image capture device 117 of the transmitting client device 110A. Rules apply. For example, if the controller 210 also determines that the person's face is towards the camera, the controller 210 may remove the objects other than the person in order to more prominently display the person located from the video data. The captured video data is modified by correcting for cropping portions of the located video data, or by modifying the zoom of the data captured on a portion of the video data where the person is located. For example, if one or more models applied by the controller 210 determines that the human face is toward the image capture device 117 (eg, one or more specific features of the human face are image capture device If captured by (117), the controller 210 modifies the captured video data to display the user's face more prominently. In another example, the controller 210 determines the distances between the image capture device 117 and the people located in the video data, and more prominently displays a person with the minimum determined distance from the image capture device 117, Alternatively, the captured video data is corrected to display more prominently the people whose determined distance from the image capture device is less than the threshold distance. In another example, a controller to reposition the image capture device 117 as the person or object moves, while allowing video data captured by the image capture device 117 to track the movement of the object or person 210 applies one or more models.

One or more models applied by the controller 210 may modify the captured video data based on rules processing the locations of a plurality of people identified in the captured video data. In various embodiments, users can allow controller 210 to locate users in the captured video data, and controller 210 is a model for determining locations in the captured video data to which the eyes of the identified users are directed. Is applied to the captured video data. In response to determining that at least a threshold number or a threshold amount of positioned users has gazes directed to a location in the captured video data that includes the particular person, the controller 210 displays the particular person more prominently ( For example, modifying the captured video data to crop the captured video data to remove content different from a particular person, to increase zoom or magnification of the specific person). As another example, the controller 210 determines distances between different people located in the video data, and modifies the captured video data to display the people within each other's threshold distance more prominently, which the controller 210 A group of people is allowed to modify the captured video data by cropping or zooming a portion of the located video data. Additionally, the controller 210 can remove one or more frames from the captured video data, based on identified people or objects in the captured video data, eg, a number less than a threshold If objects or people are identified within the captured video data, or if an amount of movement less than a threshold of objects or people identified within the captured video data is determined, the controller 210 may display the captured video data. Prior to sending 410 to the online system 140, frames are removed from the captured video data. In other embodiments, the online system 140 sends the captured video data to the receiving client device 110B before sending 415 the captured video data, using the criteria described above, as described further below. Frames are removed from the video data received from device 110A.

The controller 210 can receive data from other components of the client device 110A being transmitted, and is based on characteristics of the video data and received data from other components of the client device 110A being transmitted. Thus, the captured video data can be corrected. For example, the video capture device 117 or the client device 110 includes an audio capture device, such as a microphone, configured to capture audio data from a local area around the client device 110. Users can allow the controller 210 to process the captured audio data, along with the captured video data, when modifying the captured video data. In various embodiments, the controller 210 applies one or more models to the captured audio data to determine a location within the captured video data that includes a source of audio data. The controller 210 applies one or more models to a location in the captured video data, including a source of audio data. In response to the application of models that determine that the location in the captured video data, including the source of the audio data, includes the person, the controller 210 is more prominent in determining the location in the captured video data that includes the source of audio data. For the sake of presentation, the captured video data may be modified to modify the captured video data (i.e., to increase the zoom of the location containing the source of the captured video data, or to remove objects other than the source of the captured video data) Cropping the location containing the source), or reposition the image capture device 117 to focus on the source of the audio data. However, the controller 210 does not modify the captured video data or restarts the video capture device 117 in response to determining that the location in the captured video data containing the source of the audio data does not include a person. Do not place. As another example, the controller 210 further refers to the person determined that the captured video data is being served, or otherwise, the audio data captured by the audio capture device of the client device 110A being transmitted. Of the captured video data to significantly display (i.e., to increase the zoom of that location, including the person determined to be providing audio data, or to remove objects other than the person determined to be providing audio data) The captured video data is displayed in order to more prominently locate a person located within the captured video data, which is determined by the controller 210 to be the source of the captured audio data, while allowing the location including the source to be cropped). Correct or reposition the image capture device 117.

In some embodiments, users may also, based on video data previously captured 405 by the image capture device 117, the controller 210, the image capture device 117 of the client device 110. To modify parameters or captured video data, one or more models may be allowed to apply. For example, if the controller 210 places a plurality of people in the captured video data, the controller 210 modifies the captured video data or one or more parameters of the image capture device 117, respectively. A person who has been markedly presented in the captured video data for a minimum amount of time (e.g., presented in video data at least as a set of threshold values, in a field of view of at least a threshold amount of the image capture device 117) Presented, or at least a threshold amount of frames of captured video data, or at one or more specific locations within the captured video data). As another example, subject to privacy settings, controller 210 stores information identifying people located in the captured video data, which have been remarkably presented in the captured video data, at least for a threshold amount of time. When the controller 210 places an additional person in the captured video data, the controller 210 compares the additional person with stored information that identifies people, which has been remarkably presented in the captured video data. In response to determining that the additional person is not identified by the stored information, the controller 210 modifies the captured video data or image capture device to significantly display the additional person for at least a threshold amount of time. Modify one or more parameters of (117). This allows the controller 210 to modify the video data, so that each person located in the video data by the controller 210 is markedly displayed for at least a threshold amount of time.

Additionally, the controller 210 may modify parameters of the video capture device 117 or captured video data in response to identifying movements of people located in the captured video data. For example, if one or more models applied to the video captured by the controller 210 determines that the located person is gesturing towards the object, the controller 210 modifies the captured video data. Thus, the located persons and objects are presented remarkably. As an example, if the captured video significantly displays the person who is located (e.g., a person who is located at least in dimensions of a threshold set, or at least one threshold amount of video data of one or more frames Occupied, displaying the positioned person), and if the controller determines that the located person is gesturing towards the object, the controller 210 modifies the captured video so that the located person and object are video data Is presented in For example, the controller 210 reduces the magnification of the captured video data, so that both the located person and the object are included in the captured video. In another example, if the controller 210 applies one or more models, which determine that the located person is holding the object, the controller 210 modifies one or more parameters of the image capture device 117 or Or by modifying the captured video data, the object is presented remarkably (eg, changing the focus of the image capture device 117 on the object and increasing the zoom of the image capture device 117). The controller 210 may, in some embodiments, more prominently present the object captured by the person located in the captured video data for a certain amount of time, then the parameters of the image capture device 117 Or it is possible to modify the captured video data, so that the located person is remarkably presented again by the captured video data.

The transmitting client device 110A transmits 410 video data, which can be modified by the controller 210, from the transmitting client device 110A to the online system 140, as described further above. This sends 415 the captured video data to the receiving client device 110B. Using the display device 115, the receiving client device 110B presents 420 video data from the online system 140 to the viewing user of the online system 140. In various embodiments, the viewing user communicates a request from the receiving client device 110B to the online system 140 to communicate with the transmitting client device 110A. The online system 140 communicates the request to the transmitting client device 110A, which provides a response to the online system 140. If the transmitting client device 110A provides permission to the online system 140 in response to the request, the online system 140 captures 405 by the video capturing device 117 of the transmitting client device 110A. The video data provided to the online system 140 is communicated to the receiving client device 110B, and vice versa, for presentation on the display device 115.

In various embodiments, the user (eg, based on pre-determined privacy settings, and through user actions) allows the user to transmit the client device 110A located in the captured video by the controller 210. It may authorize providing information to the online system 140, which identifies one or more users corresponding to people, which is one or more of the video data transmitted 420 to the receiving client device 110B. Contains information identifying users. Alternatively, the transmitting client device 110A identifies portions of the video data, including those located by the controller 210, to the online system 140, which is the video where the controller 210 locates the people. Portions of the data are compared to stored images that identify online system users. Based on the privacy settings, the online system 140 controls the users identified by the stored images, determined by the online system 140 as having portions of video data of at least a threshold similarity, the controller 210 For the controller 210 of the transmitting client device 110A where people are located, it searches for identifying information. If users opt in to such identification features, this allows online system 140 to identify users of online system 140 included in the video data, received from transmitting client device 110A. To determine whether a person located in a portion of the received video data is an online system user, the online system 140 may use one or more facial recognition processes, or other identification processes, to identify the users. ) Stored by (for example, profile pictures in the user's user profiles, images including the faces of users whose users are tagged with identification information), and where the person is located by the controller 210 , It can be applied to a part of the received video data. For example, the online system 140 augments video data from the transmitting client device 110A, thereby identifying information of the user of the online system (eg, first name and last name, and email address). ) Is overlaid on a portion of the video data from the transmitting client device 110A, including the person identified by the online system 140 as an online system user. The online system 140 transmits 420 the augmented video data to the receiving client device 110B. Alternatively, the online system 140 generates information identifying online system users, corresponding to people located in the video data received from the transmitting client device 110A, and the online system corresponding to people in the video. Information identifying the users is transmitted to the receiving client device 110B along with video data. For example, the online system 140 is connected to video data by the receiving client device 110B and corresponds to people in the video data received from the transmitting client device 110A for presentation to a viewing user. To create a list of user names of online system users or a list of first and last names.

In some embodiments where users permit the use of various user-related data to improve the user experience of the online system 140, online corresponding to people located in video data received from the transmitting client device 110A. When generating information identifying system users, the online system 140 establishes intimacies between the viewing user associated with the receiving client device 110B and the users identified from the video data received from the sending client device 110A. Can handle it. For example, the online system 140 is associated with online system users, identified from video data received from the transmitting client device 110A, having a connection to a viewing user maintained by the online system 140. Generate identification information for; The online system 140 may generate identification information for online system users, identified from video data received from the transmitting client device 110A, which does not have access to the viewing user in some embodiments. none. Alternatively, the online system 140 visually identifies identification information of online system users identified from received video data connected to the viewing user, from identification information of online system users that are not connected to the viewing user. To distinguish. As another example, the online system 140 determines the intimacy of the viewing user for each online system user, identified from the video data received from the transmitting client device 110A, and the online system identified from the video data. Correct the presentation of users' identification information. By way of example, the online system 140 identifies the online system user identified from the received video data, visually distinguishing the identified online system users, where the online system 140 determines that the viewing user has at least a threshold intimacy. Generates information that identifies them. As another example, online system 140 may provide information identifying online system users identified from the received video data, based on intimacies of the user being viewed, for online system users identified from the received video data. Ranking; The online system 140 visualizes information identifying online system users identified from the received video data, having at least a threshold location in ranking, from information identifying other online system users identified from the received video data. It generates information to distinguish. In another embodiment, the online system 140 presents information identifying the online system user identified from the received video data, based on the user's affinity for the online system user identified from the received video data. Generate information.

The receiving client device 110B allows the sending client device 110A to provide video data for presentation to the receiving client device 110B, while from the online system 140 via the display device 115. Present the video data of (420). Information from the online system 140 identifying users of the online system 140 is presented by the receiving client device 110B in connection with video data. From the presented video data, the receiving client device 110B receives a selection of the interested user from the viewing user 425 and transmits information identifying the interested user to the online system 140. The viewing user can select the object of interest in various embodiments. For example, a viewing user selects information identifying a user of interest from information identifying users of the online system 140 presented in connection with video data, and receiving client device 110B and receiving client device 110B. ) Transmits the information identifying the user of interest to the online system 140 (430). The viewing user can select information identifying the user of interest from the information presented by the receiving client device 110B, along with the video data, and describing users included in the video data. Alternatively, the viewing user selects a portion of the presented video data, including a person, and the receiving client device 110B identifies the selected portion of the presented video data to the online system 140, which selects the selected portion of the video data. Compare a portion of the content with stored images of faces or bodies associated with users (eg, images contained in user profiles, images maintained by online system 140 where various users are identified) And identify a user associated with one or more stored images matching content of a selected portion of the video as the user of interest.

The online system 140 sends 435 information identifying the user of interest to the transmitting client device 110A. Based on the characteristics of the objects included in the video data captured 405 by the image capture device 117 and information identifying the user of interest, the controller 210 is a client transmitting to present the user of interest more prominently The parameters of the image capture device 117 of the device 110A or the captured video data are corrected (440). The modification is based on pre-authorization of such changes and / or for permission and review by the user being captured, for a list of pre-authorized persons controlling the receiving client device 110B (eg , Via privacy settings), a notification as to how the captured video can be changed can occur after being displayed on the client device 110A. The modifications include modifying parameters of the image capture device 117 to increase magnification on a portion of the video data that includes the user of interest, or to remove portions of the video data that do not contain the user of interest. It can contain. In various embodiments, the controller 210 of the transmitting client device 110A may modify one or more of the ones described above to modify 440 the captured video data to more prominently present the user of interest. Models are applied to the captured video data. To more prominently present the interested user, the modified video data is transmitted 445 from the sending client device 110A to the online system 140, which makes the receiving client device 110B more noticeable to the interested user. The presenting video data is transmitted 450, which presents the modified video data 455 to the viewing user.

The controller 210, after receiving the information identifying the user of interest, modifies one or more parameters of the image capture device 117 to follow the user of interest around the local area, after receiving the information identifying the user of interest. One or more models can be applied. For example, the controller 210 may match a person's face of interest (identified from video data previously received 405 based on information from the online system 140 or from the online system). In order to identify the face of the camera, one or more face recognition models are applied to people located in the captured video data, and the focus of the image capture device 117 is subsequently applied to a person having a face matching the face of the user of interest. Reposition it. Alternatively, the controller 210 extracts a color map from the captured video data including the user of interest and repositions the focus of the image capture device 117, so that the extracted color map is an image capture device ( 117). In some embodiments, while allowing video data captured by the image capture device 117 to track the movement of the identified person or object, the controller 210 receives information identifying the user of interest, and identifies The image capture device 117 is subsequently modified to track people or objects.

In various embodiments, the viewing user provides additional instructions to the online system 140 to modify the video data presented by the display device 115 of the receiving client device 110B, which sends the instructions. Client device 110A. Based on the instructions from the online system 140, the controller 210 of the transmitting client device 110A may modify one or more parameters of the image capture device 117 or by the image capture device 117. The captured video data is corrected. For example, an instruction provided by the viewing user through the online system 140 causes the controller 210 to configure the image capture device 117 of the transmitting client device 110A, thereby capturing the image. The device 117 is repositioned as the user of interest moves within the field of view of the video capture device 117. As another example, the instructions provided by the viewing user through the online system 140 are captured 405 by the video capture device 117 of the transmitting client device 110A and received through the receiving client device 110B. Identifies objects in the video data that are presented. When the transmitting client device 110A receives an instruction from the online system 140, the controller 210 modifies the captured video data or modifies one or more parameters of the image capture device 117, thereby adding The captured video data provided subsequently to the client device 110 includes the identified object. This is a client sending instructions that the viewing user modifies the video data captured by the video capture device 117 of the sending client device 110A and presented to the viewing user via the receiving client device 110B. Allow to provide to device 110A. In various embodiments, the viewing user is transmitting the client device without identifying the user of interest from video data presented 425 by the receiving client device 110B and captured by the transmitting client device 110A. Instructions for communication to the transmitting client device 110A may be provided to the online system 140 to modify the video data captured by 110A. This allows the viewing user to change the video data captured 405 by the transmitting client device 110A.

conclusion

The foregoing description of embodiments of the present disclosure has been presented for purposes of illustration, and is not intended to be exhaustive or to limit the patent rights to the detailed forms disclosed. Those skilled in the art can recognize that many modified forms and variations are possible in light of the above disclosure.

Several portions of this description describe embodiments of the present disclosure by symbolic representations and algorithms of operations on information. Descriptions and representations of these algorithms are often used by those skilled in data processing techniques to effectively convey the gist of their work to those skilled in the art. It is understood that these operations are performed by computer programs or equivalent electrical circuits, micro code, or the like, although functionally, computationally, or logically described. Furthermore, referring to these arrangements of operations as modules, without loss of generality, has also proven convenient at times. The described operations and their associated modules can be implemented in software, firmware, hardware, or any combinations thereof.

Any of the steps, tasks, or processes described herein may be executed or performed by one or more hardware or software modules alone or in combination with other devices. In one embodiment, a software module is implemented as a computer program product comprising a computer-readable medium having computer program code, which is a computer processor to perform any or all of the described steps, operations, or processes. Can be executed by

Embodiments of the present disclosure may also relate to an apparatus that performs the operations herein. Such devices may be specially configured for the required purposes and / or may include general purpose computing devices that are selectively activated or reconfigured by computer programs stored in the computer. Such a computer program may be stored on a non-transitory and morphological computer readable storage medium or any kind of medium suitable for storing electronic instructions, which may be connected to a computer system bus. Further, any computing systems mentioned in the specification may include a single processor, or may be architectures that employ multiple processor designs for increased computational power.

Embodiments may also relate to products made by the computing process described herein. Such a product may include information arising from a computing process, where the information is stored on a non-transitory and tangible computer readable storage medium, and any embodiment of a computer program product or other data combination described herein. It may include.

Finally, the language used in this specification is easy to read and was mainly selected for educational purposes, and may not be selected to describe or limit patent rights in detail. Therefore, the scope of the patent rights is not limited by this detailed description, but is limited by any claims published in the application based thereon. Accordingly, the present disclosure of the embodiments is intended to be illustrative, and this does not limit the scope of the patent rights set forth in the following claims.

Embodiments according to the present invention are particularly disclosed in the appended claims, which are aimed at methods and computer program products, wherein one claim category, e. For example, it may be claimed in computer program products, systems and storage media. Subsequent dependent or cited claims in the appended claims are selected for formal reasons only. However, any subject arising from an intentional recitation claim after any previous claims (in particular, multiple dependent claims) may also be claimed, so any combination of the claims and their features is disclosed, and in the appended claims It can be claimed regardless of the dependent claims selected. The subject matter that may be claimed may include combinations of features presented in the appended claims, as well as any other combination of features in the claims, with each feature mentioned in the claims being any other in the claims. It can be combined with a feature or a combination of other features. In addition, any of the features and embodiments depicted or described herein may be claimed in a separate claim, and / or in any combination with any embodiments or features depicted or described herein or of the appended claims. Any combination of any of the features can be claimed.

In an embodiment according to the invention, the computer-implemented method comprises:

Providing captured video data of a local area within the field of view of the image capture device included in the client device, the video data comprising captured one or more users of an online system, providing captured video data;

Applying one or more models maintained by the client to the characteristics of the captured video data;

Locating one or more users of the online system included in the captured video data;

Transmitting video data to the online system;

Receiving information from an online system that identifies a user of interest in the captured video data.

One or more parameters of the video capture device to modify video data subsequently captured by the video capture device to crop or zoom portions of the video data that include the user of interest to more significantly include the user of interest. Steps to fix them: and

And transmitting the subsequently captured video data by the image capture device to the online system using the modified one or more parameters.

One or more parameters of the video capture device to modify video data subsequently captured by the video capture device to crop or zoom portions of the video data that include the user of interest to more significantly include the user of interest. The steps to fix them are:

And increasing magnification of a portion of the video data subsequently captured by the image capture device including the user of interest.

One or more parameters of the video capture device to modify video data subsequently captured by the video capture device to crop or zoom portions of the video data that include the user of interest to more significantly include the user of interest. The steps to fix them are:

And modifying the video data subsequently captured by the video capture device to remove one or more portions of the video data subsequently captured by the video capture device that does not include a user of interest.

One or more parameters of the video capture device to modify video data subsequently captured by the video capture device to crop or zoom portions of the video data that include the user of interest to more significantly include the user of interest. The steps to fix them are:

Identifying a user of interest in the video data captured by the image capture device; And

And repositioning the focus of the image capture device to the user of interest.

The steps of identifying a user of interest in the video data captured by the image capture device are:

And applying one or more face recognition models to identify a user of interest in the video data captured by the image capture device.

The steps of identifying a user of interest in the video data captured by the image capture device are:

And extracting a color map from a portion of the captured video data that includes the user of interest.

In an embodiment according to the invention, the computer-implemented method comprises:

Receiving an instruction from an online system to modify additional video data captured by the image capture device; And

And modifying one or more parameters of the image capture device in response to the received instruction.

Modifying one or more parameters of the image capture device in response to the received instruction is:

And repositioning the image capture device according to a movement of a user of interest within the field of view of the image capture device.

One or more parameters of the video capture device to modify video data subsequently captured by the video capture device to crop or zoom portions of the video data that include the user of interest to more significantly include the user of interest. The steps to fix them are:

Applying one or more models to video data subsequently captured by an image capture device that identifies the movement of the user of interest; And

And correcting the field of view of the image capture device based on the identified movement of the user of interest.

In an embodiment according to the invention, the computer-implemented method comprises:

Receiving, in an online system, video data of a local area within the field of view of the image capture device included in the client device, the video data comprising one or more users of the online system, receiving video data in the online system;

Retrieving information identifying one or more users of the online system included in the video data;

Identifying a viewing user of the online system;

Transmitting video data and information identifying one or more users of the online system included in the video data to an additional client device associated with the viewing user;

Receiving information identifying an interested user of one or more users of the online system included in the video data from the additional client device;

Transmitting information identifying the user of interest to the client device;

Receiving additional video data captured by the video capture device of the client device, wherein the additional video data includes a portion comprising a cropped or zoomed interest user to more prominently display the interested user. Receiving basic video data; And

And transmitting additional video data to the additional client device.

The step of retrieving information identifying one or more users of the online system included in the video data is:

Identifying, to the viewing user, users of the online system included in the connected online system through the online system; And

And retrieving, through the online system, information identifying users of the online system included in the connected online system to the viewing user.

The step of retrieving information identifying one or more users of the online system included in the video data is:

Identifying, to the viewing user, users of the online system included in the connected online system through the online system; And

Included in video data that visually discriminates users included in the connected video data, to users being viewed, from users included in unconnected video data, through the online system, and to viewers. And generating information identifying users of the online system.

The step of retrieving information identifying one or more users of the online system included in the video data is:

Determining the intimacy of the user being viewed for each user of the online system included in the video data; And

And generating information identifying each user of the online system included in the video data, based on the intimacy of the user being watched for each user of the online system included in the video data.

Based on the familiarity of the user being viewed for each user of the online system included in the video data, the step of generating information identifying each user of the online system included in the video data is:

And generating information identifying each user of the online system included in the video data, wherein the viewing user has at least a threshold intimacy.

Generating information that identifies each user of the online system included in the video data based on the intimacy of the user being watched for each user of the online system included in the video data includes:

Ranking users of the online system included in the video data, based on intimacies of the user being watched for users of the online system included in the video data;

The ranking step may include generating information identifying each user of the online system included in the video data having at least a threshold position.

In an embodiment according to the invention, the computer-implemented method comprises:

Receiving instructions from the online system to modify video data captured by the image capture device of the client device from the additional client device;

Sending an instruction to the client device;

Receiving additional video data from a client device captured by the image capture device based on the instruction; And

And transmitting additional video data from the online system to the additional client device.

In an embodiment according to the present invention, a computer program product comprising a computer readable storage medium having encoded instructions thereon, which, when executed by a processor, may cause the processor to perform the following steps:

Providing captured video data of a local area within the field of view of the image capture device included in the client device, the video data comprising captured one or more users of an online system, providing captured video data;

Applying one or more models maintained by the client to the characteristics of the captured video data;

Locating one or more users of the online system included in the captured video data;

Transmitting video data to the online system;

Receiving information from an online system that identifies a user of interest in the captured video data;

One or more parameters of the video capture device to modify video data subsequently captured by the video capture device to crop or zoom portions of the video data that include the user of interest to more significantly include the user of interest. Modifying them; And

It is a computer program product capable of performing a step of transmitting subsequently captured video data by an image capture device to an online system using one or more modified parameters.

One or more parameters of the video capture device to modify video data subsequently captured by the video capture device to crop or zoom portions of the video data that include the user of interest to more significantly include the user of interest. The steps to fix them are:

And modifying the video data subsequently captured by the video capture device to remove one or more portions of the video data subsequently captured by the video capture device that does not include a user of interest.

One or more parameters of the video capture device to modify video data subsequently captured by the video capture device to crop or zoom portions of the video data that include the user of interest to more significantly include the user of interest. The steps to fix them are:

Identifying a user of interest in the video data captured by the image capture device; And

And repositioning the focus of the image capture device to the user of interest.

In an embodiment according to the present invention, one or more computer-readable non-transitory storage media are capable of operating software that is operable when executed to perform the method according to the present invention or any of the above-mentioned embodiments. It can contain.

In an embodiment according to the invention, the system comprises:

One or more processors and at least one memory comprising instructions executable by the processors and coupled to the processors, the processors being any of the above-mentioned embodiments or a method according to the invention These are the processors that are operable when executing the instructions to perform.

In an embodiment according to the present invention, a computer program product, preferably comprising a computer-readable non-transitory storage medium, for carrying out the method according to the present invention or according to any of the above-mentioned embodiments May be operable when executed on a data processing system

In an embodiment according to the invention, in particular a computer-implemented method comprises:

Maintaining one or more models at the client device, each model being online, in video data captured by the video capture device of the client device, based on characteristics of the video data captured by the video capture device. Maintaining one or more models, applying one or more rules to determine which part of the user's body of the system is located;

Providing captured video data of a local area within a field of view of the image capture device included in the client device;

Locating a user included in the captured video data;

Applying one or more maintained models to the captured video data;

Modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user, based on the application of the one or more models; And

And transmitting the modified video data to the online system.

Modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user based on the application of the one or more models:

Identifying joints of the user's body; And

And modifying the video data captured by the image capture device to include joints corresponding to different parts of the user's body, based on one or more rules included in the one or more models. have.

Based on the application of one or more models, modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user:

Based on the application of one or more rules included in one or more models, one or more selected from the group consisting of the user's head, the user's torso, the user's entire body, and any combination thereof. And modifying the video data to include.

The one or more rules are one selected from the group consisting of the content of the video data previously captured by the video capture device, the user's movement in the video data previously captured by the video capture device, and any combination thereof. Or more.

Modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user based on the application of the one or more models:

Determining that the user is gesturing towards an object in the local area from application of one or more models; And

And reducing the magnification of the image capture device such that additional video data captured by the image capture device includes objects and users in a direction in which the user is gesturing.

Based on the application of one or more models, modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user:

Determining from the application of one or more models that the user is holding the object in the local area; And

The method may include changing the focus of the image capture device on the object and increasing the magnification of the image capture device.

Based on the application of one or more models, modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user:

And modifying one or more parameters of the image capture device that affect the field of view of the image capture device, based on the application of one or more models.

The parameters of the image capture device may include enlargement of the image capture device.

Based on the application of one or more models, modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user:

And modifying the video data captured by the image capture device to correct an enlargement of a portion of the captured video data that includes the identified user.

Steps to locate the user included in the captured video data are:

Receiving information from an online system that identifies the user; And

And based on the received information, identifying portions of video data captured by the image capture device including the identified user.

In an embodiment according to the present invention, a computer program product comprising a computer readable storage medium thereon having coded instructions that, when executed by a processor, may cause the processor to perform the following steps:

Maintaining one or more models at the client device, each model based on characteristics of the video data captured by the video capture device, to video data captured by the video capture device of the client device, the user Maintaining one or more models, applying one or more rules to determine which part of the body of the body is located;

Providing captured video data of a local area within a field of view of the image capture device included in the client device;

Locating a user included in the captured video data;

Applying one or more maintained models to the captured video data; And

Modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user, based on the application of the one or more models; And

It is a computer program product that allows the user to perform the steps of transferring modified video data to an online system.

Based on the application of one or more models, modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user:

Identifying joints of the user's body; And

And modifying the video data captured by the image capture device to include joints corresponding to different parts of the user's body, based on one or more rules included in the one or more models. have.

Based on the application of one or more models, modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user:

Based on the application of one or more rules included in one or more models, includes one or more selected from the group consisting of the user's head, the user's torso, the user's entire body, and any combination thereof. It may include the step of modifying the video data.

The one or more rules are one selected from the group consisting of the content of the video data previously captured by the video capture device, the user's movement in the video data previously captured by the video capture device, and any combination thereof. Or more.

Based on the application of one or more models, modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user:

Determining from the application of one or more models that the user is gesturing towards the object in the local area; And

And reducing an enlargement of the image capturing device such that the object in the direction in which the user is gesturing and the additional video data captured by the image capturing device are included.

Based on the application of one or more models, modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user:

Determining from the application of one or more models that the user is holding the object in the local area; And

The method may include changing the focus of the image capture device on the object and increasing the magnification of the image capture device.

Based on the application of one or more models, modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user:

And modifying one or more parameters of the image capture device that affect the field of view of the image capture device, based on the application of one or more models.

The parameters of the image capture device may include enlargement of the image capture device.

Based on the application of one or more models, modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user:

And modifying the video data captured by the image capture device to correct an enlargement of a portion of the captured video data including the user.

Steps to locate the user included in the captured video data are:

Receiving information identifying the user from the online system; And

And based on the received information, identifying portions of video data captured by the image capture device including the identified user.

In embodiments according to the present invention, one or more computer-readable non-transitory storage media include software operable when executed to perform the method according to the present invention or any of the above-mentioned embodiments. can do.

In an embodiment according to the invention, the system comprises: one or more processors; And instructions executable by the processors and including at least one memory coupled to the processors, the instructions for performing the method according to the present invention or any of the embodiments mentioned above. When running them, the processors are operational.

In an embodiment according to the present invention, a computer program product, preferably comprising a computer-readable non-transitory storage medium, performs a method according to the present invention or according to any of the above-mentioned embodiments It can be operable when executed on a data processing system.

In an embodiment according to the invention, in particular a computer-implemented method comprises:

Maintaining one or more models at the client device, each model applying one or more rules to locate users in the video data captured by the video capture device of the client device, and captured video data Maintaining one or more models, applying one or more rules to determine one or more located users to show in;

Capturing video data of a local area within a field of view of the image capture device included in the client device;

Locating a plurality of users in the captured video data;

Applying one or more maintained models to the captured video data; And

Modifying the captured video data to show one or more located users based on the application of one or more maintained models; And

And transmitting the modified video data from the client device to the online system.

Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:

Modifying one or more parameters of the image capture device based on application of the one or more maintained models; And

And capturing additional video data of the local area using the modified one or more parameters of the image capture device.

Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:

Determining from the application of one or more maintained models, locations in the captured video data to which the eyes of the located users are directed; And

And correcting the focus of the image capture device with respect to the positioned user person, at least a threshold amount of the positioned users' gazes are directed.

Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:

Determining distances between different users located in the captured video data; And

And modifying the focus of the image capture device so that the additional captured video data includes users within a threshold distance from each other.

Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:

Capturing audio data through an audio capture device included in the client device;

Determining a source of audio data in the local area by applying one or more maintained models to the captured audio data and the captured video data;

Determining that the source of the audio data in the local area is a user; And

And in response to determining that the source of the audio data in the local area is a user, modifying the focus of the image capture device with respect to the source of the audio data.

Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:

Identifying, from the application of one or more maintained models, the movement of the located user; And

And modifying the field of view of the image capture device based on the identified movement of the located user.

From the application of one or more maintained models, the step of identifying the movement of the located user is:

And identifying that the user located toward the object included in the local area is gesturing.

The steps to modify the field of view of the video capture device are:

And reducing the enlargement of the image capture device to include the user in which the additional video data captured by the image capture device is located and the object in which the located user is gesturing.

From the application of one or more maintained models, the step of identifying the movement of the located user is:

And, from the application of one or more maintained models, determining that the located user is holding the object in the local area.

The steps to modify the field of view of the video capture device are:

The method may include changing the focus of the image capture device on the object and increasing the magnification of the image capture device.

Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:

And modifying the captured video data such that each positioned user is presented within the captured video data, at least in the dimensions of the threshold set, for at least a threshold amount of time.

Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:

And modifying the captured video such that each positioned user is presented within the captured video data, at least for a threshold amount of time.

Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:

Identifying a user's face in the captured video data from application of one or more maintained models;

And modifying the captured video data to remove portions of the captured video data that do not include the user's face.

Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:

And modifying the captured video data to include each of the located users.

Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:

Stabilizing the captured video data by application of one or more maintained models.

Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:

Comparing the located user in the captured video data to stored data describing users included in the captured video data previously presented by the captured video data in dimensions of at least a threshold set; And

Modifying the captured video data, in response to a comparison determining that the identified user does not match the user included in the captured video data, which was previously at least by the dimensions of the threshold set by the captured video data. Thus, it may include modifying the captured video data to present the positioned person more prominently.

In an embodiment according to the present invention, a computer program product comprising a computer readable storage medium having encoded instructions thereon, which, when executed by a processor, may cause the processor to perform the following steps,

Maintaining one or more models at the client device, each model applying one or more rules to locate users in the video data captured by the video capture device of the client device, and the characteristics of the video data. Maintaining, from one or more models, applying one or more rules to determine one or more located users to show in the captured video data;

Capturing video data of a local area within a field of view of the image capture device included in the client device;

Locating a plurality of users in the captured video data;

Applying one or more maintained models to the captured video; And

Modifying the captured video data to show one or more located users based on the application of one or more maintained models; And

It is a computer program product capable of performing the steps of transferring modified video data from a client device to an online system.

Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:

Determining from the application of one or more maintained models, locations in the captured video data to which the eyes of the located users are directed;

And correcting the focus of the image capture device for the located person, at least a threshold amount of the identified people's gazes are directed.

Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:

Determining distances between different users located in the captured video data; And

And modifying the focus of the image capture device so that the additional captured video data includes people within a threshold distance from each other.

Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:

Capturing audio data through an audio capture device included in the client device;

Determining a source of audio data in the local area by applying one or more maintained models to the captured audio data and the captured video data;

Determining that a source of audio data in the local area is a located user; And

And in response to determining that the source of the audio data in the local area is a user located, modifying the focus of the image capture device with respect to the source of the audio data.

In an embodiment according to the invention, one or more computer-readable non-transitory storage media are software operable when executed to carry out the method according to the invention or any of the above-mentioned embodiments It may include.

In an embodiment according to the invention, the system comprises: one or more processors; And at least one memory coupled to the processors and including instructions executable by the processors, the processors according to any of the above-mentioned embodiments or a method according to the invention It is operable when executing instructions to carry out.

In an embodiment according to the present invention, a computer program product, preferably comprising a computer-readable non-transitory storage medium, performs a method according to the present invention or according to any of the above-mentioned embodiments It may be operable when executed on a data processing system.

Claims (60)

As a computer-implemented method:
Providing captured video data of a local area within the field of view of the image capture device included in the client device, the video data comprising one or more users of the online system, providing captured video data;
Applying one or more models maintained by the client to the characteristics of the captured video data;
Locating one or more users of the online system included in the captured video data;
Transmitting video data to the online system;
Receiving information from an online system that identifies a user of interest in the captured video data.
One or more parameters of the video capture device to modify the video data subsequently captured by the video capture device to crop or zoom portions of the video data containing the user of interest to more significantly include the user of interest. Steps to fix them: and
And transmitting the subsequently captured video data by the image capture device to the online system using the modified one or more parameters. According to claim 1,
One or more parameters of the video capture device to modify the video data subsequently captured by the video capture device to crop or zoom portions of the video data containing the user of interest to more significantly include the user of interest. The steps to fix them are:
And increasing the magnification of a portion of the video data subsequently captured by the image capture device comprising the user of interest. The method of claim 1 or 2,
One or more parameters of the video capture device to modify the video data subsequently captured by the video capture device to crop or zoom portions of the video data containing the user of interest to more significantly include the user of interest. The steps to fix them are:
Computer-implemented, comprising modifying the video data subsequently captured by the video capture device to remove one or more portions of the video data subsequently captured by the video capture device that does not include a user of interest. Way. The method according to any one of claims 1 to 3,
One or more parameters of the video capture device to modify the video data subsequently captured by the video capture device to crop or zoom portions of the video data containing the user of interest to more significantly include the user of interest. The steps to fix them are:
Identifying a user of interest in the video data captured by the image capture device; And
And repositioning the focus of the image capture device to the user of interest. The method of claim 4,
The steps of identifying a user of interest in the video data captured by the image capture device are:
And applying one or more facial recognition models to identify a user of interest in the video data captured by the image capture device. The method of claim 4,
The steps of identifying a user of interest in the video data captured by the image capture device are:
And extracting a color map from a portion of the captured video data that includes the user of interest. The method according to any one of claims 1 to 6,
Receiving an instruction from an online system to modify additional video data captured by the image capture device; And
And modifying one or more parameters of the image capture device in response to the received instruction. The method of claim 7,
Modifying one or more parameters of the image capture device in response to the received instruction is:
And repositioning the image capture device as a user of interest moves within the field of view of the image capture device. The method of claim 7,
Modifying one or more parameters of the image capture device in response to the received instruction is:
And repositioning the image capture device as a user of interest moves within the field of view of the image capture device. The method according to any one of claims 1 to 9,
One or more parameters of the video capture device to modify the video data subsequently captured by the video capture device to crop or zoom portions of the video data containing the user of interest to more significantly include the user of interest. The steps to fix them are:
Applying one or more models to video data subsequently captured by an image capture device that identifies the movement of the user of interest; And
And modifying the field of view of the image capture device based on the identified movement of the user of interest. As a computer-implemented method:
Receiving, in an online system, video data of a local area within the field of view of the image capture device included in the client device, the video data comprising one or more users of the online system, receiving video data in the online system;
Retrieving information identifying one or more users of the online system included in the video data;
Identifying a viewing user of the online system;
Transmitting video data and information identifying one or more users of the online system included in the video data to an additional client device associated with the viewing user;
Receiving information identifying an interested user of one or more users of the online system included in the video data from the additional client device;
Transmitting information identifying the user of interest to the client device;
Receiving additional video data captured by the image capture device of the client device, wherein the additional video data includes a portion comprising a cropped or zoomed interested user to more prominently display the interested user. Receiving basic video data; And
And sending the additional video data to an additional client device. The method of claim 11,
Searching for information identifying one or more users of the online system included in the video data includes:
Identifying, to a viewing user, users of the online system included in the connected online system through the online system; And
And retrieving to the viewing user, through the online system, information identifying users of the online system included in the connected online system. The method of claim 11 or 12,
Searching for information identifying one or more users of the online system included in the video data includes:
Identifying, to a viewing user, users of the online system included in the connected online system through the online system; And
Included in video data that visually discriminates users included in the connected video data, from the users included in unconnected video data, to the viewing user, through the online system, to the viewing users, through the online system And generating information identifying users of the online system. The method according to any one of claims 11 to 13,
Searching for information identifying one or more users of the online system included in the video data includes:
Determining the intimacy of the user being viewed for each user of the online system included in the video data; And
And generating information identifying each user of the online system included in the video data, based on the intimacy of the user being watched for each user of the online system included in the video data. . The method of claim 14,
Based on the familiarity of the user being watched for each user of the online system included in the video data, the step of generating information identifying each user of the online system included in the video data is:
And generating information identifying each user of the online system included in the video data, wherein the viewing user has at least a threshold intimacy. The method of claim 14,
Generating information that identifies each user of the online system included in the video data based on the intimacy of the user being watched for each user of the online system included in the video data includes:
Ranking users of the online system included in the video data, based on intimacies of the user being viewed with respect to users of the online system included in the video data;
And in the ranking step, generating information identifying each user of the online system included in the video data having at least a threshold location. The method according to any one of claims 11 to 16,
Receiving instructions from the online system to modify video data captured by the image capture device of the client device from the additional client device;
Sending an instruction to the client device;
Receiving additional video data from a client device captured by the image capture device based on the instruction; And
And transmitting additional video data from the online system to the additional client device. A computer program product comprising a computer readable storage medium having coded instructions thereon that, when executed by a processor, causes the processor to perform the following steps:
Providing captured video data of a local area within the field of view of the image capture device included in the client device, the video data comprising captured one or more users of an online system, providing captured video data;
Applying one or more models maintained by the client to the characteristics of the captured video data;
Locating one or more users of the online system included in the captured video data;
Transmitting video data to the online system;
Receiving information from an online system that identifies a user of interest in the captured video data;
One or more parameters of the video capture device to modify the video data subsequently captured by the video capture device to crop or zoom portions of the video data containing the user of interest to more significantly include the user of interest. Modifying them; And
A computer program product that causes a processor to perform the step of transmitting the video data subsequently captured by the image capture device to the online system using the modified one or more parameters. The method of claim 18,
One or more parameters of the video capture device to modify the video data subsequently captured by the video capture device to crop or zoom portions of the video data containing the user of interest to more significantly include the user of interest. The steps to fix them are:
And modifying the video data subsequently captured by the video capture device to remove one or more portions of video data subsequently captured by the video capture device that does not include the user of interest. . The method of claim 18 or 19,
One or more parameters of the video capture device to modify the video data subsequently captured by the video capture device to crop or zoom portions of the video data containing the user of interest to more significantly include the user of interest. The steps to fix them are:
Identifying a user of interest in the video data captured by the image capture device; And
And repositioning the focus of the image capture device to the user of interest. As a computer-implemented method:
Maintaining one or more models at the client device, each model being online, in video data captured by the video capture device of the client device, based on characteristics of the video data captured by the video capture device. Maintaining one or more models, applying one or more rules to determine which part of the user's body of the system is located;
Providing captured video data of a local area within a field of view of the image capture device included in the client device;
Locating a user included in the captured video data;
Applying one or more maintained models to the captured video data;
Based on the application of the one or more models, modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user. , Computer-implemented method. The method of claim 21,
Modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user based on the application of the one or more models:
Identifying joints of the user's body; And
Based on one or more rules included in the one or more models, modifying the video data captured by the image capture device to include joints corresponding to different parts of the user's body, Computer-implemented method. The method of claim 21 or 22,
Based on the application of one or more models, modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user:
Based on the application of one or more rules included in one or more models, one or more selected from the group consisting of the user's head, the user's torso, the user's entire body, and any combination thereof. A computer-implemented method comprising modifying video data to include. The method according to any one of claims 21 to 23,
The one or more rules are one selected from the group consisting of the content of the video data previously captured by the video capture device, the user's movement in the video data previously captured by the video capture device, and any combination thereof. Or more, a computer-implemented method. The method according to any one of claims 21 to 24,
Modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user based on the application of the one or more models:
Determining that the user is gesturing towards an object in the local area from application of one or more models; And
And reducing the magnification of the image capture device such that additional video data captured by the image capture device includes objects and users in a direction in which the user is gesturing. The method according to any one of claims 21 to 25,
Based on the application of one or more models, modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user:
Determining from the application of one or more models that the user is holding the object in the local area; And
And changing the focus of the image capture device on the object and increasing the magnification of the image capture device. The method according to any one of claims 21 to 26,
Based on the application of one or more models, modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user:
And modifying one or more parameters of the image capture device that affect the field of view of the image capture device, based on the application of one or more models. The method of claim 27,
A computer-implemented method, wherein the parameters of the image capture device include an enlargement of the image capture device. The method according to any one of claims 21 to 29,
Based on the application of one or more models, modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user:
And modifying the video data captured by the image capture device to modify the magnification of the portion of the captured video data that includes the identified user. The method according to any one of claims 21 to 30,
Steps to locate the user included in the captured video data are:
Receiving information from an online system that identifies the user; And
And based on the received information, identifying portions of video data captured by the image capture device comprising the identified user. A computer program product comprising a computer readable storage medium thereon having coded instructions that, when executed by a processor, cause the processor to perform the following steps:
Maintaining one or more models at the client device, each model based on characteristics of the video data captured by the video capture device, to video data captured by the video capture device of the client device, the user Maintaining one or more models, applying one or more rules to determine which part of the body of the body is located;
Providing captured video data of a local area within a field of view of the image capture device included in the client device;
Locating a user included in the captured video data;
Applying one or more maintained models to the captured video data; And
Based on the application of one or more models, performing a step of modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user To do, computer program products. The method of claim 31,
Based on the application of one or more models, modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user:
Identifying joints of the user's body; And
Based on one or more rules included in the one or more models, modifying the video data captured by the image capture device to include joints corresponding to different parts of the user's body, Computer program products. The method of claim 31 or 32,
Based on the application of one or more models, modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user:
Based on the application of one or more rules included in one or more models, includes one or more selected from the group consisting of the user's head, the user's torso, the user's entire body, and any combination thereof. A computer program product comprising the step of modifying the video data to be. The method of claim 33,
The one or more rules are one selected from the group consisting of the content of the video data previously captured by the video capture device, the user's movement in the video data previously captured by the video capture device, and any combination thereof. A computer program product based on or more. The method according to any one of claims 31 to 34,
Based on the application of one or more models, modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user:
Determining from the application of one or more models that the user is gesturing towards the object in the local area; And
And reducing the magnification of the image capture device such that the object in the direction in which the user is gesturing and the additional video data captured by the image capture device include the user. The method according to any one of claims 31 to 35,
Based on the application of one or more models, modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user:
Determining from the application of one or more models that the user is holding the object in the local area; And
And changing the focus of the image capture device on the object and increasing the magnification of the image capture device. The method according to any one of claims 31 to 36,
Based on the application of one or more models, modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user:
And modifying one or more parameters of the image capture device that affect the field of view of the image capture device, based on the application of the one or more models. The method of claim 37,
A computer program product, wherein the parameters of the image capture device include an enlargement of the image capture device. The method according to any one of claims 31 to 38,
Based on the application of one or more models, modifying the video data captured by the image capture device to crop or zoom a portion of the video data that includes one or more portions of the user:
And modifying the video data captured by the image capture device to modify an enlargement of a portion of the captured video data comprising the user. The method according to any one of claims 31 to 39,
Steps to locate the user included in the captured video data are:
Receiving information identifying the user from the online system; And
And based on the received information, identifying portions of video data captured by the image capture device comprising the identified user. As a computer-implemented method:
Maintaining one or more models at the client device, each model applying one or more rules to locate users in the video data captured by the video capture device of the client device, and captured video data Maintaining one or more models, applying one or more rules to determine one or more located users to show in;
Providing captured video data of a local area within a field of view of the image capture device included in the client device;
Locating a plurality of users in the captured video data;
Applying one or more maintained models to the captured video data; And
And modifying the captured video data to show one or more located users, based on the application of one or more maintained models. The method of claim 41,
Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:
Modifying one or more parameters of the image capture device based on application of the one or more maintained models; And
And capturing additional video data of the local area using the modified one or more parameters of the image capture device. The method of claim 41 or 42,
Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:
Determining from the application of one or more maintained models, locations in the captured video data to which the eyes of the located users are directed; And
And modifying the focus of the image capture device relative to the positioned user person, at least a threshold amount of positioned users' gazes are directed. The method according to any one of claims 41 to 43,
Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:
Determining distances between different users located in the captured video data; And
And modifying the focus of the image capture device, such that the additional captured video data includes users within a threshold distance from each other. The method according to any one of claims 41 to 44,
Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:
Capturing audio data through an audio capture device included in the client device;
Determining a source of audio data in the local area by applying one or more maintained models to the captured audio data and the captured video data;
Determining that the source of the audio data in the local area is a user; And
And in response to determining that the source of audio data in the local area is a user, modifying the focus of the image capture device relative to the source of audio data. The method according to any one of claims 41 to 45,
Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:
Identifying, from the application of one or more maintained models, the movement of the located user; And
And modifying the field of view of the image capture device based on the identified movement of the located user. The method of claim 46,
From the application of one or more maintained models, the step of identifying the movement of the located user is:
And identifying that the user located towards the object included in the local area is gesturing. The method of claim 47,
The steps to modify the field of view of the video capture device are:
And reducing magnification of the image capture device to include the user in which the additional video data captured by the image capture device is located and the object in which the located user is gesturing. The method of claim 46,
From the application of one or more maintained models, the step of identifying the movement of the located user is:
And, from the application of one or more maintained models, determining that the located user is holding the object in the local area. The method of claim 49,
The steps to modify the field of view of the video capture device are:
And changing the focus of the image capture device on the object and increasing the magnification of the image capture device. The method according to any one of claims 41 to 50,
Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:
And modifying the captured video data such that each positioned user is presented within the captured video data, at least in dimensions of the threshold set, for at least a threshold amount of time. The method according to any one of claims 41 to 51,
Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:
And modifying the captured video such that each positioned user is presented within the captured video data, at least for a threshold amount of time. The method according to any one of claims 41 to 52,
Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:
Identifying a user's face in the captured video data from application of one or more maintained models;
And modifying the captured video data to remove portions of the captured video data that do not include the user's face. The method according to any one of claims 41 to 53,
Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:
And modifying the captured video data to include each of the located users. The method according to any one of claims 41 to 54,
Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:
And stabilizing the captured video data by application of one or more maintained models. The method according to any one of claims 41 to 55,
Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:
Comparing the located user in the captured video data to stored data describing users included in the captured video data previously presented by the captured video data at least in dimensions of a threshold set; And
Modifying the captured video data, in response to a comparison determining that the identified user does not match a user included in the captured video data, which was previously at least by the dimensions of the threshold set by the captured video data. And, modifying the captured video data to present the positioned person more prominently. A computer program product comprising a computer readable storage medium having coded instructions thereon, when executed by a processor, causing the processor to perform the following steps:
Maintaining one or more models at the client device, each model applying one or more rules to locate users in the video data captured by the client device's image capture device, and characteristics of the video data Maintaining, from one or more models, applying one or more rules to determine one or more located users to show in the captured video data;
Providing captured video data of a local area within a field of view of the image capture device included in the client device;
Locating a plurality of users in the captured video data;
Applying one or more maintained models to the captured video; And
A computer program product that, based on the application of one or more maintained models, performs the step of modifying the captured video data to show one or more located users. The method of claim 57,
Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:
Determining from the application of one or more maintained models, locations in the captured video data to which the eyes of the located users are directed;
And modifying the focus of the image capture device relative to the located person, at least a threshold amount of the eyes of the identified people are directed. The method of claim 57 or 58,
Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:
Determining distances between different users located in the captured video data; And
And modifying the focus of the image capture device such that the additional captured video data includes people within a threshold distance from each other. The method according to any one of claims 57 to 59,
Based on the application of one or more maintained models, modifying the captured video data to show one or more located users:
Capturing audio data through an audio capture device included in the client device;
Determining a source of audio data in the local area by applying one or more maintained models to the captured audio data and the captured video data;
Determining that a source of audio data in the local area is a located user; And
And in response to determining that the source of the audio data in the local area is a located user, modifying the focus of the image capture device relative to the source of the audio data.

Images