TW202305661A - Bystander-centric privacy controls for recording devices - Google Patents

Abstract

A recording device provides bystander-centric privacy controls for authorizing the storage of a bystander's identifying information (e.g., video or audio recordings of the bystander). Before a recording device can store identifying information of bystanders, the bystanders may indicate to the recording device whether they authorize the storage. If the bystanders do not authorize the storage, the recording device may modify the identifying information captured by sensors, such as a video camera or a microphone, such that the identity of the non-authorizing bystander is not identifiable through the modified identifying information. Thus, bystanders are given increased agency over whether they want to be recorded. Further, if the bystanders do not want to be recorded, sensor data that may identify them is modified by the recording device to prevent unwanted exposure of their identity in recorded content.

Description

Bystander-Centric Privacy Controls for Recording Devices

The present invention relates generally to sensor data capture, and more specifically, to bystander-centric privacy controls for recording devices. [Cross-references to related applications]

This application claims priority and benefit to U.S. Provisional Application No. 63/218,863, filed July 6, 2021, and U.S. Nonprovisional Patent Application No. 17/739,886, filed May 9, 2022, which etc. applications are incorporated herein by reference in their entirety.

With the adoption of Augmented Reality (AR) devices, sensors that can record sensitive information will become ubiquitous, posing a major challenge to bystander privacy. While current devices often include blinking light emitting diodes (LEDs) to provide notification of a recording, the blinking light may not be sufficient to attract the attention of the public (ie, bystanders) captured in the recording. In addition, notification may not be sufficient to ensure the safety and privacy of bystanders, especially for sensitive groups such as children, individuals with disabilities, etc. who cannot spot and/or understand bystander instructions. Furthermore, it may not be meaningful to provide notifications when the device is recording continuously.

Specific examples of bystander-centric privacy controls for recording devices are described herein. Data capture, such as video or audio recording, is based on bystander privacy controls that determine modification of bystander identifying information captured in the sensor data field. A recording device may also be referred to as a capture device. In one example, the capture device may determine that the bystander did not authorize the recording of its audio and then identify and modify the audio that could have been used to identify the bystander. The capture device may use positioning techniques to determine the location of the bystander relative to the capture device. Using the determined location, the capture device can identify identifying information of the bystander, such as an image of the bystander captured by a video camera of the capture device. The capture device can then modify the identifying information to protect the identity of the bystander from unauthorized recording (eg, blurring the image of the bystander's face in the recorded video). Thus, bystanders are given autonomy over whether they want to be recorded.

In one embodiment, a capture device includes a sensor configured to capture sensor data describing a local area with a bystander. The capture device also includes communication circuitry configured to receive private data from the bystander's device indicating whether the bystander has authorized the capture device to store the bystander's identifying information (eg, an image of the bystander's face). The capture device includes a controller configured to determine the location of the bystander based on the sensor data, determine the permission status of the bystander based on the received private data, and determine whether the bystander has authorized the capture device to store the bystander identifying information. The controller is configured, in response to determining that the bystander is an unauthorized bystander, to use the determined location to determine a region of the sensor data that includes identification information for the bystander and to modify the identification information in the region of the sensor data. Bystanders may not be identifiable by the modified identification information (for example, the face of the bystander is blurred to such an extent that the identity of the bystander cannot be identified based on the blurred image).

In another embodiment, a method includes capturing, by a sensor of a capture device of a user, sensor data describing a local area with a bystander. receiving private data from the bystander's device, wherein the private data indicates whether the bystander has authorized the capture device to store the bystander's identification information. The location of bystanders is determined based on sensor data. The authority status of the bystander is determined based on the received private information. In response to determining that the bystander is an unauthorized bystander, the determined location of the bystander is used to determine a region of the sensor data that includes identifying information for the bystander. The identification information in the zone is modified such that bystanders cannot be identified using the modified identification information.

In yet another embodiment, a non-transitory computer-readable storage medium includes stored instructions that, when executed by a processor of a capture device, cause the capture device to capture a sensor sensor via a sensor of the capture device. sensor data describing the local area including the bystander. When the instructions are executed, the capture device further enables the capture device to receive private information from the device of the bystander, wherein the private data indicates whether the bystander has authorized the capture device to store the identification information of the bystander. When executed, the instructions further cause the capture device to use the sensor data to determine the location of the bystander and use the private data to determine the permission status of the bystander. The instructions, when executed, further cause the capture device to respond to the determination that the bystander is an unauthorized bystander, determine that the sensor data includes identification information of the bystander, and modify the identification information so that the bystander cannot be identified using the modified identification information. .

A recording device provides bystander-centric privacy controls for authorizing storage of bystander-identifying information (eg, video or audio recordings of bystanders). Before the recording device can store the bystander's identifying information, the bystander can use his device to indicate to the recording device whether to authorize storage. As referred to herein, an authorized bystander may refer to a bystander who authorizes the capture device to store its identification information, and an unauthorized bystander may refer to a bystander who does not authorize the capture device to store its identification information. If a bystander indicates through their device a permission status that they do not authorize to store, the recording device may modify the identification information captured by a sensor such as a video camera or microphone so that the modified identification information cannot identify the identity of the unauthorized bystander . As a result, bystanders are given more autonomy over whether they want to be recorded. In addition, if the bystanders do not want to be recorded, the recording device modifies the sensor data that can identify the bystanders to prevent accidental exposure of their identities in the recorded content.

In one embodiment, the recording device includes a camera that captures an image or video of a localized area including a bystander who has the right to determine whether the recording device may store its identifying information. A recording device may also be referred to as a capture device. The capture device may receive private data from a bystander's device (bystander device) communicatively coupled to the capture device. The capture device determines the location of the bystanders based on the image data or additional data captured by the camera. The capture device uses the private data to determine the permission status indicating whether the bystander authorizes the capture device to store the bystander's identifying information. In response to determining that the bystander is an unauthorized bystander based on the permission status of the bystander, the capture device may use the determined location of the bystander to determine a region of interest within the image data that includes identifying information. Additionally, the capture device may modify identifying information within a region of interest of the image data such that the visual representation of a bystander cannot be recognized via the modified region of image data (e.g., disrupting the definition of a region of interest corresponding to an unauthorized bystander). pixels within the bounding box, data within the bounding box of a region of interest that does not represent image data, etc.).

Embodiments of the invention may include or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in a certain way before being presented to the user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (mixed reality; MR), Multiple Reality, or some combination and/or derivative thereof. Artificial reality content may include all generated content or generated content combined with captured (eg, real-world) content. Artificial reality content may include video, audio, haptic feedback, or some combination thereof, any of which may be presented in a single channel or in multiple channels (such as stereoscopic video that creates a three-dimensional effect on the viewer). In addition, in some embodiments, an artificial reality may also be associated with an application, product, accessory, service, or some combination thereof for generating content in an artificial reality and/or otherwise for use in an artificial reality . Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including wearable devices (such as headsets) connected to a host computer system, standalone wearable devices (such as headsets), mobile devices, or computing systems , or any other hardware platform capable of providing artificial reality content to one or more viewers.

FIG. 1 is a perspective view of a head-mounted device 100 implemented as a glasses device according to one or more embodiments. In some embodiments, the eyewear device is a near eye display (NED). In general, the headset 100 may be worn on the user's face such that content (eg, media content) is presented using a display assembly and/or an audio system. However, the headset 100 can also be used to present the media content to the user in different ways. Examples of media content presented by the headset 100 include one or more images, video, audio, or some combination thereof. The headset 100 includes a frame, and may include: a display assembly including one or more display elements 120 , a depth camera assembly (DCA), an audio system and a position sensor 190 , and other components. Although FIG. 1 shows components of the headset 100 in example locations on the headset 100, the components may be positioned elsewhere on the headset 100, on peripheral devices that pair with the headset 100, or some combination thereof. Similarly, there may be more or fewer components on the headset 100 than shown in FIG. 1 .

The frame 110 holds other components of the head-mounted device 100 . Frame 110 includes a front portion that holds one or more display elements 120 and end pieces (eg, temples) that attach to a user's head. The front portion of the frame 110 bridges the top of the user's nose. The length of the end piece may be adjustable (eg, adjustable temple length) to suit different users. End pieces may also include portions that curl behind the user's ears (eg, temple tips, earpieces).

One or more display elements 120 provide light to a user wearing the headset 100 . As shown, the headset includes a display element 120 for each eye of the user. In some embodiments, the display element 120 generates image light that is provided to eye frames of the head-mounted device 100 . The eye frame is the spatial position occupied by the eyes of the user when wearing the head-mounted device 100 . For example, the display element 120 may be a waveguide display. A waveguide display includes a light source (eg, a two-dimensional source, one or more line sources, one or more point sources, etc.) and one or more waveguides. Light from the light source is coupled into one or more waveguides that output the light in such a way that there is pupil replication in the eye socket of the head mounted device 100 . Coupling of light into and/or out of one or more waveguides may be accomplished using one or more diffraction gratings. In some embodiments, waveguide displays include scanning elements (eg, waveguides, mirrors, etc.) that scan light from a light source as it is coupled into one or more waveguides. It should be noted that in some embodiments, one or both of the display elements 120 are opaque and do not transmit light from localized areas around the head mounted device 100 . The local area is an area surrounding the head-mounted device 100 . For example, the partial area can be a space where the user wears the head-mounted device 100 inside, or the user wearing the head-mounted device 100 can be outside and the partial area is the outer area. In this content context, the headset 100 generates VR content. Alternatively, in some embodiments, one or both of display elements 120 are at least partially transparent such that light from a localized area can be combined with light from one or more display elements to produce AR and/or MR content.

In some embodiments, display element 120 does not generate image light, and is instead a lens that transmits light from a localized area to the eye frame. For example, one or both of the display elements 120 may be a non-corrective (no power) lens or a power lens (e.g., single vision, bifocal, and trifocal or progressive) to help correct the user's vision. visual impairment. In some embodiments, the display element 120 may be polarized and/or tinted to protect the user's eyes from sunlight damage.

In some specific examples, the display element 120 may include additional optical element blocks (not shown in the figure). The optics block may include one or more optical elements (eg, lenses, Fresnel lenses, etc.) that direct light from the display element 120 to the eye frame. The optics block may, for example, correct distortion in some or all of the image content, magnify some or all of the image, or some combination thereof.

The DCA determines depth information for a portion of a local area surrounding the headset 100 . The DCA includes one or more imaging devices 130 and a DCA controller (not shown in FIG. 1 ), and may also include an illuminator 140 . In some embodiments, illuminator 140 illuminates a portion of the local area with light. The light may be structured light (eg, dot patterns, bars, etc.), such as in infrared (IR), IR flashes for time-of-flight, and the like. In some embodiments, one or more imaging devices 130 capture images that include portions of the local area of light from illuminator 140 . As shown, FIG. 1 shows a single illuminator 140 and two imaging devices 130 . In an alternative embodiment, illuminator 140 and at least two imaging devices 130 are absent.

The DCA controller calculates depth information for portions of the local area using the captured imagery and one or more depth determination techniques. Depth determination techniques can be, for example, direct time-of-flight (ToF) depth sensing, indirect ToF depth sensing, structured light, passive stereo analysis, active stereo analysis (using light added to the scene from illuminators 140 textures), some other technique for determining the depth of a scene, or some combination thereof.

The audio system provides audio content. The audio system includes a transducer array, a sensor array and an audio controller 150 . However, in other embodiments, the audio system may include different and/or additional components. Similarly, in some cases, functionality described with reference to components of an audio system may be distributed among the components in a manner different than that described herein. For example, some or all of the functions of the controller may be performed by a remote server.

The transducer array presents sound to the user. The transducer array includes a plurality of transducers. The transducer may be a speaker 160 or a tissue transducer 170 (eg, a bone conduction transducer or a cartilage conduction transducer). Although the speaker 160 is shown outside the frame 110 , the speaker 160 may be enclosed in the frame 110 . In some embodiments, instead of individual speakers for each ear, headset 100 includes a speaker array that includes multiple speakers integrated into frame 110 to improve the directionality of presented audio content. Tissue transducer 170 is coupled to the user's head and directly vibrates the user's tissue (eg, bone or cartilage) to generate sound. The number and/or location of transducers may be different than that shown in FIG. 1 .

The sensor array detects sound within a local area of the headset 100 . The sensor array includes a plurality of acoustic sensors 180 . The acoustic sensor 180 captures sound emanating from one or more sound sources in a local area (eg, a room). Each acoustic sensor is configured to detect sound and convert the detected sound to an electronic format (analog or digital). The acoustic sensor 180 may be an acoustic wave sensor, a microphone, an acoustic transducer, or similar sensors suitable for detecting sound.

In some embodiments, one or more acoustic sensors 180 may be placed in the ear canal of each ear (eg, to act as binaural microphones). In some embodiments, acoustic sensor 180 may be placed on an exterior surface of headset 100, on an interior surface of headset 100, and with headset 100 (eg, part of some other device) Separate or some combination thereof. The number and/or location of acoustic sensors 180 may be different than that shown in FIG. 1 . For example, the number of acoustic detection locations can be increased to increase the amount of audio information collected and the sensitivity and/or accuracy of the information. The sound detection locations may be oriented so that the microphones can detect sound in a wide range of directions around the user wearing the headset 100 .

Headset 100 may use a device communicatively coupled to headset 100 to enable onlookers to indicate whether they authorize headset 100 to store images captured using one or more of imaging device 130 or acoustic sensor 180 . its identifying information. For example, imaging device 130 includes a camera that captures video of a local area, and acoustic sensor 180 includes one or more microphones that can capture audio of a local area and enable audio source localization (e.g., with determine the relative position of the source of the bystander's voice relative to the headset 100). The headset 100 may include a controller that enables the headset 100 to determine whether bystanders in the local area can be stored when storing sensor data captured by the imaging device 130, the acoustic sensor 180, or a combination thereof identifying information, and therefore determine whether the identifying information should be modified to protect the privacy of bystanders. The modification of captured data according to the permission status set by the bystander to improve the privacy of the bystander is further described with reference to FIGS. 2 to 5 .

The audio controller 150 processes information from the sensor array describing the sound detected by the sensor array. The audio controller 150 may include a processor and a computer-readable storage medium. The audio controller 150 can be configured to generate a direction of arrival (DOA) estimate, generate an acoustic transfer function (e.g., an array transfer function and/or a head-related transfer function), track the location of a sound source, A beam of light is formed in the direction of the speaker, a sound source is sorted, a sound filter for the speaker 160 is produced, or some combination thereof.

The position sensor 190 generates one or more measurement signals in response to the movement of the head-mounted device 100 . The position sensor 190 can be positioned on a portion of the frame 110 of the headset 100 . The position sensor 190 may include an inertial measurement unit (IMU). Examples of position sensors 190 include: one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor to detect motion, an IMU for error correction A type of sensor, or some combination thereof. The position sensor 190 may be located external to the IMU, internal to the IMU, or some combination thereof.

In some embodiments, the head-mounted device 100 can provide simultaneous localization and mapping (SLAM) for the position of the head-mounted device 100 and update the model of the local area. For example, the head mounted device 100 may include a passive camera assembly (PCA) for generating color image data. The PCA may include one or more RGB cameras capturing images of some or all of the local area. In some embodiments, some or all of the imaging devices 130 of the DCA may also function as a PCA. The image captured by PCA and the depth information determined by DCA can be used to determine the parameters of the local area, generate a model of the local area, update the model of the local area, or some combination thereof. In addition, the position sensor 190 tracks the position (eg, position and pose) of the head-mounted device 100 in the room. Additional details regarding the components of the headset 100 are discussed below in connection with FIG. 6 .

FIG. 2 is a block diagram of a capture device 200 according to an embodiment. The head-mounted device 100 in FIG. 1 may be a specific example of the capture device 200 . The capture device 200 captures information of a local area, and at the same time modifies the identification information of bystanders who have not specified the authorization status for the capture device 200 to store their identification information (for example, their facial images or their voice records). In the specific example of FIG. 2 , capture device 200 includes sensor assembly 210 , communication circuitry 220 , controller 230 , sensor data storage 265 , and capture device trace log 260 . Some embodiments of capture device 200 have different components than those described here. Similarly, in some cases functionality may be distributed among components differently than described herein.

The sensor assembly 210 captures information about a local area that includes one or more bystanders. The captured information may include identifying information of the bystander, such as an image of the bystander's face or audio of the bystander's voice. The sensor assembly 210 may be a camera, a microphone, any suitable device for capturing information of a local area, or a combination thereof. For example, the combination of the acoustic sensor 180 and the imaging device 130 of the head mounted device 100 can be regarded as the sensor assembly 210 . In some embodiments, sensor assembly 210 includes an audio receiver that enables capture device 200 to perform sound localization. For example, sensor assembly 210 includes a software receiver configured to perform adaptive beamforming. Using sound localization, capture device 200 may be configured to determine the location or relative location of bystanders.

Communication circuitry 220 enables communications between capture device 200 and other capture devices, networks, servers, or computing devices. Communication circuitry 220 may include a wireless modem for communicating with communication circuitry of other devices or servers. Such communications may involve the Internet or any other suitable communications network or path (eg, the network environment shown in FIG. 6). Additionally, communication circuitry 200 may include circuitry that enables peer-to-peer communication of devices or communication of devices in locations remote from each other.

Communications circuitry 220 enables capture device 200 to communicate with other devices using an associated communication protocol (e.g., Bluetooth® ^{, ZigBee®} , ultra-wideband (UWB), infrared, ultra-wideband, near-field communication, Wi-Fi ^Direct®, etc.) Communication within a personal area network (PAN). For example, the communication circuitry 220 can implement ^Bluetooth® Low Energy (LE) communication, detect or connect to other Bluetooth® LE devices. Communication circuitry 220 may include a global positioning system (GPS) receiver to use the geographic coordinates of capture device 200 to determine the location of capture device 200 or other devices (eg, as described with respect to positioning module 240 ). In some embodiments, communication circuitry 220 may connect to a local area network (LAN) such as a Wi- ^Fi® network, and recognize other devices connected to the same LAN. Communications circuitry 220 may then enable capture device 200 to couple to an online system (e.g., a social networking system) to determine whether a device connected to the same LAN is related to a user who has a user socially connected to capture device 200 couplet.

Communication circuitry 220 may receive private data from the bystander device. In some embodiments, capture device 200 uses communication circuitry 220 to ensure that a user of capture device 200 is authorized by the bystander to store the bystander's identification information prior to storing the identification information captured by sensor assembly 210 . Communication circuitry 220 may transmit a broadcast message (eg, a ^Bluetooth® LE advertisement) to the communication circuitry of the bystander device, wherein the broadcast message indicates capture device 200's intent to capture information about the local area. In response to receiving the broadcast message, the bystander device may transmit private data to capture device 200 , where the private data is received by communication circuitry 220 . Communication circuitry 220 of capture device 200 may similarly receive broadcast messages from other capture devices (eg, bystander devices that are also capable of capturing and storing identifying information) and transmit private data of the capture device's user to the other capture devices. Privacy data received by communication circuitry 220 may indicate whether the bystander authorizes capture device 200 to capture and store identifying information about the bystander. Privacy information is further described with respect to authorization request module 235 .

The controller 230 controls the operation of the capture device 200 . In the specific example of FIG. 2 , the controller 230 includes an authorization request module 235 , a location module 240 , an information modifier module 245 and a mode selection module 250 . Some specific examples of controller 230 have different components than those described herein. Similarly, functionality may be distributed among components in different ways than described here. For example, some functions of the controller 230 may be performed outside the capture device 200 . An example of an environment for a computing device communicatively coupled to capture device 200 is described with reference to FIG. 6 .

The authorization request module 235 determines the authorization status of the bystander based on the received private information. Permission may refer to authorization for the capture device to store identification information, and permission status may refer to whether the capture device is authorized to store identification information (eg, the status may be authorized or unauthorized). Bystanders can select different permission states for different capture devices or users. Similarly, the authorization request module 235 enables the user of the capture device 200 to also set the permission status for a specific capture device or user. Different permission states may correspond to different degrees of privacy protection. For example, a bystander can choose a permission status between various options for a particular user: a first permission status, which indicates that the user is not allowed to store any identifying information for the bystander; a second permission status, which indicates that the user may be able to requesting or obtaining permission to store identifying information (for example, establishing a social connection on a social networking system prior to storing identifying information); or third permission status, which indicates that the user is allowed to store identifying information. Additionally or alternatively, a bystander may assign different permission states based on the relationship between the user and the bystander. For example, different degrees of privacy protection corresponding to authority states may correspond to degrees of connection (e.g., first-degree connection or second-degree connection on a social networking system), family relationships, or between users and bystanders. any suitable familiar relationship between them. The authorization request module 235 may generate a graphical user interface (graphical user interface; GUI) for display (eg, on a display of the capture device 200 or a device communicatively coupled to the capture device 200) for the user Specify the capture device or user and the corresponding permission status.

In some embodiments, the authorization request module 235 requests temporary permission from the bystander device to store identification information about the bystander. Communication circuitry 220 may be used to transmit the request to the bystander device. Bystanders can respond to requests by granting or denying users temporary permission to store their identifying information. For example, the user of the capture device 200 wants to record video of a local area within half an hour. A user may specify the duration for which he wishes to record information in the local area through the user input interface of the capture device 200 or a device communicatively coupled to the capture device 200 . Authorization request module 235 may receive the requested duration, generate a broadcast message during this duration indicating the intent to record, cause communication circuitry 220 to transmit the generated broadcast message, and receive the broadcast message from bystanders receiving the generated broadcast message. The device receives private data.

In response to determining that the permission status indicates that the bystander device does not grant permission to the capture device 200 to store the identification information according to the received privacy information, the authorization request module 235 may generate a request for the temporary permission. The authorization request module 235 may receive updated privacy data from the bystander device indicating that the bystander has granted temporary permission. In response, authorization request module 235 may update the permission status associated with the bystander to indicate temporary permission (eg, capture device 200 may store the bystander's identifying information for a half-hour period specified by the user). The authorization request module 235 can determine whether the temporary permission to capture the device 200 has expired. In response to determining that the time has elapsed, the authorization request module 235 may update the permission status of the bystander to indicate that the capture device 200 is no longer authorized to store the identification information of the bystander. Additionally, the authorization request module 235 may determine that the captured identification information of the bystander is to be processed to obscure the bystander (eg, the processed identification information cannot be used to identify the bystander). In response to determining that the time has not passed, the authorization request module 235 can maintain the temporary authorization status of the bystander and continue to store the identification information.

The authorization request module 235 can use the social graph to determine whether the capture device 200 is authorized to store the identification information of the bystander. In some embodiments, capture device 200 is communicatively coupled to an on-line system, an example of which is shown in FIG. 6 . The online system may maintain a community graph indicating community connections between users of the online system. Community connections can indicate a level of familiarity that can be used to determine permission status. In some embodiments, the privacy information received by the authorization request module 235 may indicate that the capture device 200's permission to record identifying information about bystanders corresponds to whether or not there is a community graph between the capture device user and the bystanders. There is a community connection. The authorization request module 235 can then access the social graph of the online system to determine whether the user and the bystander have a social connection on the social graph. In response to determining that there is no social connection on the social graph, the authorization request module 235 may determine that the permission status of the bystander for the capture device 200 does not allow the capture device 200 to store the identification information of the bystander.

In some specific examples, the authorization request module 235 can enable the user and the bystander to establish a social connection on the social graph, which can change the authorization status of the capture device 200 for the bystander. The authorization request module 235 can access the social network identifier of the bystander according to the received private information. A social network identifier may belong to the online system and be used to identify a bystander as a particular account holder of the online system. Social network identifiers can be hashed for additional protection of bystander privacy. The authorization request module 235 may query the online system for social connections between the user of the capture device 200 and the bystander using the bystander's social network identifier and the user's social network identifier (e.g., in in the community graph maintained by the online system). In response to determining that no social connection exists, the authorization request module 235 can facilitate the process of establishing a social connection.

In some embodiments, the authorization request module 235 may store the social network identifier in the local storage area of the capture device 200 to determine whether there is or is not a social connection between the user and the bystander. For example, the authorization request module 235 may retrieve the hash social network identifier by which the user has the social connection on the social graph from the online system and store the retrieved identifier. With the identifier stored locally, the capture device 200 can determine whether there is a social connection between the user and the onlooker (for example, to access or query the online system) when the capture device 200 does not have a network connection with the online system. A remote storage copy of the social graph of the network).

The authorization request module 235 can display a reminder to the user to establish a social connection with a spectator on the online system. For example, the authorization request module 235 can cause the display of the capture device 200 or a display of a device coupled to the capture device 200 to display a prompt (eg, "Do you want to add people nearby as friends on your social network? "). In response to receiving a prompt selection from the user or a user input element related to the prompt (eg, a "yes" or "no" button for establishing a social connection), the authorization request module 235 can perform a corresponding action. For example, in response to receiving a user selection indicating that the user wants to establish a social connection, the authorization request module 235 may transmit an instruction to the online system to create a request for a social connection from the user to the spectator. In response to receiving a user selection indicating that the user does not want to establish a social connection, the authorization request module 235 may determine that the permission status of the bystander selection indicates that it does not authorize the user to store the bystander's identification information.

In some specific examples, after the bystander accepts the request to establish a social connection on the social graph of the online system, the authorization request module 235 may receive from the online system a message that the social connection has been established between the user and the bystander. notify. The authorization request module 235 can then update the authorization status of the spectator. The updated permission status may indicate that the bystander is an authorized bystander. That is, the capture device 200 may store the identification information of the bystander (eg, the image, video or audio of the bystander).

The authorization request module 235 can confirm that the authorized bystander authorizes the capture device 200 to store the identification information when operating in the private mode. That is, regardless of the familiarity of the user, the authorization request module 235 can ensure that bystanders still control when the capture device 200 records identification information. For example, when the user has a private conversation with a spectator, the authorization request module 235 confirms the permission status to ensure that the user is authorized by the spectator to record the private conversation. The capture device 200 can determine whether the user may be in a private environment with bystanders. In response to determining that the user may be in a private environment, capture device 200 may operate in a private mode. This is further described with respect to the mode selection module 250 . When operating in private mode, the authorization request module 235 can transmit a broadcast message to a bystander device requesting to record information of a local area. The bystander device may generate a prompt for the bystander (eg, generated at the bystander device by the authorization request module) indicating that it approves or denies the record. This prompt may be generated at the bystander device or at a client device (eg, a smartphone) communicatively coupled to the bystander device. The authorization request module 235 may receive approval or denial of the bystander's request to store identifying information and determine the corresponding permission status (eg, save the audio of the private session if the bystander has approved the request).

The positioning module 240 can determine identification information within the information captured by the sensor assembly 210 of the local area. Location module 240 enables identification of information captured by sensor assembly 210 by distinguishing unauthorized storage of identifying information by capture device 200 from sensor data captured by sensor assembly 210 Local modification (for example, setting limits). For example, sensor assembly 210 may capture video of multiple bystanders, some of which transmit private data indicating that capture device 200 is authorized to store identifying information for the bystanders, while other bystanders transmit private data indicating capture Device 200 unauthorized private data. The location module 240 can determine that the captured video contains identification information of a bystander who is not an authorized bystander. Information modifier module 245 may then modify the identifying information identified by location module 240 to protect the privacy of bystanders who have requested not to be logged. In this way, the location module 240 enables the capture device to capture at least some information of a localized area while protecting the privacy of some bystanders without completely stopping any information capture.

The location module 240 may use one or more of audio or video data to determine identification information of unauthorized bystanders within the information captured by the sensor assembly 210 of the local area. The location module 240 may receive from the authorization request module 235 a list of bystander devices of unauthorized bystanders (eg, a list of bystander devices within a personal area network proximate to the capture device 200 ). The location module 240 may determine the location of a bystander device of an unauthorized bystander. In some embodiments, the location module 240 receives the location of the bystander device from a remote server that maintains the location of the capture device, the bystander device, or a combination thereof. For example, a device may be used to access an online network that requests permission from the device to access its location during use of the online network. The online network can track the location of the device, and the capture device 200 can access the tracked location.

After determining the location of the unauthorized bystander's device, the location module 240 may determine the location of the unauthorized bystander relative to the capture device 200 . The location of the bystander may correspond to the location at which the image of the bystander was captured or the location from which the audio of the bystander emanated. The location module 240 may determine the location of an authorized device in a manner similar to that described with respect to determining the location of an unauthorized device. For example, the positioning module 240 determines the angular offset between the direction in which the optical axis of the camera of the capture device 200 is facing and the line connecting the capture device 200 to the unauthorized bystander's device (e.g., behind the user capturing the video Unauthorized bystander devices can be displaced one hundred and eighty degrees from the optical axis and out of the camera's field of view). The positioning module 240 can use the IMU data of the capture device 200 to determine the orientation (eg, the direction the camera is pointing) of the camera of the capture device 200 . Orientation can be used to determine the direction the camera is facing (eg, the orientation of the optical axis) and the angular offset of the unauthorized bystander device relative to the optical axis of the camera. By using the angular offset between the unauthorized bystander device and the camera of the capture device 200 capturing the sensor data, the localization module 240 can estimate the position of the unauthorized bystander device within the field of view of the camera of the capture device 200 .

In addition to or instead of one or more of the image or audio information captured by sensor assembly 210, location module 240 may also use radio frequency signals to determine the location of the bystander device (e.g., relative to the captured location of device 200). Positioning module 240 may use signal processing techniques such as beamforming, direction of arrival (DOA), time of arrival (TOA), time difference of arrival (TDOA), time of flight (ToF), any suitable Sound localization techniques, or combinations thereof. Location module 240 may use signals received by communication circuitry 220 using a short-range protocol such as UWB to use one or more of the aforementioned signal processing techniques to determine the relative location of the bystander. In some embodiments, location module 240 can use received audio information captured by sensor assembly 210 to identify audio sources within a local area (eg, the distance of a bystander relative to the microphone of sensor assembly 210 direction). The location module 240 can use the current location of the capture device 200 to determine the location of the sound source relative to the capture device 200 .

In some embodiments, location module 240 may use the orientation of capture device 200, the location of capture device 200, and the location of the unauthorized bystander device to determine the relative position of the unauthorized bystander device relative to the microphone array (e.g., including a head-mounted device). The location of the array facing down and facing the front microphone). For example, the positioning module 240 may determine that the headset is being worn on the user's eyes and that the microphones of the headset are oriented in a particular way (eg, downward-facing microphones face downward, and front-facing microphones are oriented north). The location module 240 uses the location of the capture device 200 and the location of the unauthorized bystander device to determine the direction between the two devices. The location module 240 may then use the determined direction between the two devices and the orientation of the microphones to determine the location of the unauthorized bystander device relative to the microphone array. The location module 240 may identify sounds from the direction of the location of the unauthorized bystander device (eg, using beamforming).

Capture device 200 may operate in an environment with a single bystander within its vicinity (eg, near a personal area network). The location module 240 can use sound location technology to determine the relative location of the bystander and the capture device 200 . For example, location module 240 may use beamforming to identify audio signals associated with sounds from bystanders in the local area. The localization module 240 can instruct the software receiver of the sensor assembly 210 to adjust the steering vector parameters, iteratively in the direction of a potential sound source until a relatively large intensity is identified (eg, the decibel level exceeds a threshold value) until the signal. The location module 240 can determine the likelihood that the signal corresponds to a bystander. For example, localization module 240 may apply a machine learning model to an audio signal, where the model is trained on samples of human speech and determine the likelihood that the signal is that of a human speech. After identifying the signal as a human voice signal, the location module 240 may use the identified audio signal to determine the relative location of the bystander. The location module 240 may use the relative location and the GPS coordinates of the capture device to determine the location of the bystander (eg, the geographic area in which the bystander may be located).

The information modifier module 245 modifies identification information in the sensor data captured by the sensor assembly 210 . The information modifier module 245 may modify the identification information in the captured sensor data within a region of interest that coincides with the location of the unauthorized bystander as identified by the location module 240 . The information modifier module 245 can modify the identification information of the bystander so that the bystander cannot be identified based on the processed identification information. For example, information module 245 may blur images of faces of unauthorized bystanders in video captured by sensor assembly 210 . In another example, the information module 245 can change the pitch of the unauthorized bystander's voice in the video captured by the sensor assembly 210 . Other examples of processing identifying information include masking (e.g., covering a bystander's face with a large gray square), sound cancellation (e.g., changing the user's voice to a single-frequency tone), scrambling (e.g., scrambling the pixels within a bounding box or scramble bits of audio spoken by bystanders), any suitable form of de-identifying the user in recorded information, or a combination thereof.

The information modifier module 245 can determine regions of interest within the captured sensor data that have identifying information. Information modifier module 245 may use the bystander location as determined by location module 240 to determine the region of interest. In some embodiments, determining the region of interest includes determining a portion of the captured sensor data, which includes a portion of the bystander's face, a portion of the bystander's body, a portion of the bystander's voice, and a portion of the sensor assembly 210. Any suitable information of the captured bystander that identifies the bystander, or combination thereof.

The information modifier module 245 can modify the identification information by processing image data, audio data or a combination thereof. The information module 245 can identify image data corresponding to the identification information in the region of interest in the captured sensor data. In one example of modifying identifying information in the form of image data, information module 245 modifies images of bystanders. The information module 245 identifies the image of the person as an image of a bystander. In some embodiments, information module 245 may use computer vision, machine learning, or any suitable form of artificial intelligence to perform face recognition on image data as captured by sensor assembly 210 . The information module 245 can use the region of interest to limit, for example, the region of image data to which the machine learning model is applied (e.g., apply the model to a region of interest that includes images of bystanders' faces rather than regions of interest that include vehicles, buildings, and other entire image of the object). Therefore, the information modifier module 245 can reduce the processing resources originally consumed on a large amount of image data. After identifying the presence of a face in the region of interest, the information module 245 can use the reference image of the bystander to identify the bystander. For example, the information modifier module 245 can use the social network identifier received in the private data from the bystander to access the silhouette image of the bystander's face and determine the The degree of similarity between the recognized images.

In another example of modifying identification information in the form of audio data, the information module 245 processes the voice of a bystander. The information modifier module 245 identifies regions of interest in the audio data captured by the sensor assembly 210 . The region of interest may be a sound source that coincides with the location of the bystander as determined by the localization module 240 . The information modifier module 245 may process audio originating from the bystander's location. For example, the information modifier module 245 may use sound localization techniques to identify audio coming from the direction of a bystander's location. The information modifier module 245 can instruct the sensor assembly 210 to increase the signal strength of an audio signal received from the direction of a bystander relative to other bystanders, and the sensor assembly can include components configured to perform functions such as adapting A software receiver for the sound localization technology of sex beamforming. For example, the sensor assembly 210 may use the steering vectors to create a receiver pole pattern that increases the received signal strength of audio from unauthorized bystanders relative to authorized bystanders. The sensor assembly 210 may include an audio receiver (e.g., a microphone) that captures audio in a localized area without beamforming and an audio receiver that may be assigned to perform beamforming and focus on the desired audio signal to be concealed. device. The information modifier module 245 may process audio signals associated with bystanders (e.g., signals received from the direction of the bystanders) to modify captured sensor data such that the bystanders cannot be identified via the processed audio signals ( For example, the voice of a bystander cannot be discerned). For example, the information modifier module 245 can process the audio signal by modulating the frequency of the audio signal received from the direction of bystanders. The information modifier module 245 may then add the FM audio signal to the audio signal captured without beamforming to mask or distort the bystander's audio. In another example, the information modifier module 245 may process by subtracting an audio signal received with beamforming (e.g., a signal from the direction of a bystander) from an audio signal received without beamforming The audio signal within the captured sensor data. In this manner, the message modifier module 245 may lower the volume level of the bystander's voice, effectively muting the bystander.

In some embodiments, the information modifier module 245 may determine not to modify the bystander's identification information when the authorization request module 235 has received a temporary authorization from the bystander to store its identification information. For example, the authorization request module 235 may receive permission from the bystander for a predetermined duration and store its identifying information during that duration, changing the permission status of the bystander to indicate that the bystander is an authorized bystander. In response, the information modifier module 245 does not modify the identification information and the bystander identification information can be passed unmodified to a storage space (e.g., local memory of the capture device 200 or to a remote server) for a or multiple user access. The authorization request module 235 may determine when the predetermined duration expires, and upon expiration, change the permission status of the bystander to indicate that the bystander is an unauthorized bystander. When the bystander is not authorized, the information modifier module 245 can modify the identification information so that the bystander cannot be identified through the processed identification information.

To further improve the privacy of bystanders, the capture device 200 may determine to modify the identification information of bystanders captured in the sensor data that have not yet been determined to be associated with bystander devices. For example, the information modifier module 245 may determine to modify the identification information of the user who does not have a device capable of providing his private information or puts his device in a state where his privacy cannot be provided (eg, power off). In some embodiments, the information modifier module 245 may determine which bystanders may not be associated with a bystander device. For example, the location module 240 determines that the location of the bystander device is in the vicinity of the capture device 200, and the information modifier module 245 may determine that the image or audio data of the sensor data indicates that there are more bystanders than nearby bystander devices the number of Information modifier module 245 may estimate likely areas where no device bystanders are located (e.g., by distinguishing them from areas where information modifier module 245 determines that bystander devices are present), and modify bystanders within these areas identifying information for .

In some embodiments, the authorization request module 235 may receive updated privacy information from the bystander device, the updated privacy information reflecting the bystander's target for the permission status determined by the authorization request module based on the previously received privacy information. Requests for varying degrees of privacy. For example, a bystander may use their bystander device to send updated privacy information that requests additional privacy and is excluded from photos or videos captured by the bystander device. In this example, the authorization request module 235 may use the first set of private data received from the bystander device to determine that the bystander has authorized storage of its identifying information, but then receives an indication from the bystander device that the capture device 200 is not authorized. A second set of private data that stores the updated permission status of the bystander's identifying information. The second set of private data may be received after the capture device 200 has captured and stored the identification information of the bystander according to the previous authorization state. Before transmitting unmodified identification information for others to access (for example, before uploading to an online system for use in a social network), the information modifier module 245 may determine that a predetermined period of time (for example, between The unmodified identification information is stored in the sensor data store 265 for a period in the range of one hour). A user or bystander may provide instructions to authorization request module 235 specifying an amount of time to set a predetermined period of time. By storing identification data unmodified for a predetermined period of time, capture device 200 may take into account the authority status of bystanders who request additional privacy after previously specifying a less stringent authority state (e.g., the right to public access to their identification information). Change. Information modifier module 245 may modify the identification information stored in sensor data store 265 in response to receiving an updated permission state indicating that capture device 200 may not store bystander identification information such that the Information cannot identify bystanders.

In another example of receiving privacy data from a bystander device indicating that the bystander has modified the requested level of privacy, capture device 200 may receive updated privacy data that includes a request to relax a previously specified privacy measure and Included in the photo or video captured by the capture device (for example, where the updated private information may have been concealed by the capture device 200). In this example, the authorization request module 235 may use the first set of private data received from the bystander device to determine that the bystander has not previously authorized storage of its identifying information, but then receive the second set of private data from the bystander device to The updated permission status indicating that capture device 200 is indeed authorized to store the bystander's identifying information. The second set of private data may be received after the capture device 200 has captured and modified the bystander's identifying information according to the previous authorization state. The information modifier module 245 may decide to temporarily store the unmodified identification information on the sensor for a predetermined period of time (eg, a period in the range of one minute to one hour) before deleting the unmodified identification information data store 265, thereby fulfilling the bystander privacy requirement of not storing its identifying information for access by others, including the user of capture device 200. A user or bystander may provide instructions to authorization request module 235 specifying an amount of time to set a predetermined period of time. By storing unmodified identifying information for a predetermined period of time, capture device 200 may consider requesting relaxation of previously established privacy measures after previously specifying a more restrictive permission state (e.g., prohibiting others from storing or accessing bystander identifying information). Bystander permission status changes. In response to receiving an updated permission state indicating that capture device 200 may store the bystander's identification information, information modifier module 245 may replace the bystander identification information with unmodified identification information temporarily stored in sensor data storage 265. The portion of the video or image that modifies the identifying information so that bystanders can be identified based on the identifying information. In this way, bystanders can change their minds to make themselves (e.g., their face or voice invisible) even if the capture device has previously instructed them to anonymize or has processed captured sensor data to anonymize them. ) included in a photo or video.

When operating in private mode, the information modifier module 245 may determine to modify the bystander identification information, as determined by the mode selection module 250 . For example, a bystander may believe that the bystander is having a private conversation with the user and does not want to be recorded, regardless of familiarity with the user. The mode selection module 250 can determine that the user and the bystander are in a private environment and instruct the information modifier module 245 to modify the identification information of the bystander, regardless of the permission status that the bystander has chosen to indicate that the bystander is authorized. In some embodiments, in response to authorization request module 235 receiving confirmation from the bystander that it may continue to record its identification information in the private environment, information modifier module 245 may cease modifying the identification information while operating in the private mode.

The mode selection module 250 determines whether to operate in a specific mode (eg, private mode). The operation mode may correspond to a combination of settings or instructions for the capture device 200 to operate according to the content context of the operation. Examples of modes of operation include private or public modes. The mode selection module 250 can preset to use the public operation mode. In one example of the preset public operation mode, when not operating in the private mode, the mode selection module 250 may determine that the capture device 200 is operating in the public mode. The public mode of operation may correspond to determining the permission status of a bystander and reusing this over a period of time (e.g., during a session capturing sensor data or within a predetermined period of time, such as twenty-four hours since the last determination of the permission status). Instructions on permission status (for example, without confirming whether the permission status has changed).

The private mode of operation may correspond to an instruction to communicate with a bystander device associated with a previously authorized bystander to reconfirm that the bystander is still authorized while in the private environment. In some embodiments, users can interact with onlookers in a private environment. Private environments may include non-public locations, such as a user's home or in a company office. Private environments may include public locations that do not have a critical number of people therein (eg, a park area with no visitors other than the user of capture device 200 and bystanders). By determining whether to operate in a private mode, mode selection module 250 may, regardless of familiarity with the user of capture device 200 , provide a sense of privacy (e.g., confidentiality) to bystanders during interactions with the user. Prevent unwanted storage of bystander identifying information. Thus, the mode selection module 250 further enables the capture device 200 to increase the privacy of the recording of sensitive identifying information about bystanders.

The mode selection module 250 can determine the possibility of the capture device 200 operating in a private environment to determine whether to operate in a private mode. Mode selection module 250 may use sensor assembly 210 to determine information about the environment in which capture device 200 operates. The sensor assembly 210 can capture image or audio data about the environment, which the mode selection module 250 can use to determine factors that help determine whether the capture device 200 is operating in a private environment. Factors may include the number of people depicted in the image data captured by the sensor assembly 210 or the ambient noise level of the audio captured by the sensor assembly 210 . The mode selection module 250 can apply artificial intelligence to identify human features in image data (for example, face recognition), count the number of people in the operating environment based on the identified features, and determine whether the count exceeds the threshold for a private environment value (for example, a maximum of four). The mode selection module 250 can determine the ambient noise level by processing the audio data (e.g., perform peak detection of the received audio signal, remove detected peaks, and determine average magnitude), and compare the ambient noise level with Threshold values for intimate settings (eg, thirty decibels). In addition to determining the number of people in the environment or the noise level of the environment, the mode selection module 250 can also use a model that maps the location of the capture device 200 or bystander device to a private or public property to determine whether the environment is a private environment. In response to determining that the operating environment is a private environment, the mode selection module 250 may determine to operate in a private mode. In response to determining that the operating environment is not a private environment (eg, the ambient noise level exceeds 30 decibels), the mode selection module 250 may determine not to operate in the private mode. When operating in private mode, authorization request module 235 and information modifier module 245 may operate accordingly, as described in the descriptions of the respective modules.

The capture device tracking module 255 tracks a list of capture devices that may have captured the identifying information of the user of the capture device 200 . The capture device tracking module 255 may access messages broadcast by other capture devices requesting permission to store identifying information or broadcasting an intent to record their local area. Capture device tracking module 255 may identify the source of a broadcast message (eg, another capture device that transmitted the broadcast message to capture device 200 ) by a sender identifier included within the broadcast message. In addition, the capture device (eg, the authorization request module of the capture device) may include a social network identifier in the broadcast message indicating the intent to record its local area. The capture device tracking module 255 may record the sender's identifier (eg, by its social network identifier), and thus track which capture devices may have captured the user's identification information of the capture device 200 . The capture device tracking module 255 can display which capture devices may have captured records of identifying information. Capture device tracking module 255 may store a record of which capture devices have broadcast an intent to store the user's identifying information in capture device tracking log 260 . In some embodiments, records may include the date, time, location, duration of bystander devices within the local area with the capture device (e.g., using proximity determined by short-range communication sensors), or a description of the capture device Any suitable information in the context of content interacting with the bystander device for the purpose of capturing sensor data.

The sensor data storage 265 stores identification information captured by the sensor assembly 210 . Sensor data store 265 may additionally store modified identification information as processed by information modifier module 245 that conceals unauthorized bystanders. Information stored in sensor data storage 265 may be accessed by a user of capture device 200, bystanders, online, any suitable recipient of captured sensor data, or a combination thereof. A user of capture device 200 may specify access rights to information stored in sensor data storage 265 . For example, the user can designate that only online network users who have established a social connection with the user on the social graph of the online network can access the stored information. In some embodiments, sensor data store 265 may be located remotely from capture device 200 (eg, a remote server communicatively coupled to capture device 200 ). As referred to herein, the storage or recording of identifying information is persistent in such a way that the stored information can later be accessed by the user. This type of storage can be contrasted with a more temporary storage mechanism, such as random access memory storage of a computing device.

3 depicts a user with a capture device 300 and a bystander with a bystander device 310, according to at least one embodiment. The capture device 300 can be the head-mounted device 100 or the capture device 200 . Capture device 300 may include one or more sensors of the capture environment. Bystander device 310 is depicted as a smart watch, but could alternatively be a headset, smartphone, computer, or any suitable portable computing device. The sensors may include image sensors, audio sensors, or the like. In some embodiments, acquisition device 300 is configured to perform positioning (eg, using ultra-wideband (UWB) or some other short-range radio-based technology). The capture device 300 may further include hardware and/or software integration layers. Capture device 300 may store or be configured to access data about a user's social graph.

A bystander is an individual who is in a local area of the device such that the sensors of the device can capture content (eg, images of the individual and/or utterances of the individual) from it. Bystander device 310 may be configured to perform positioning (eg, using UWB or some other short-range radio-based technology). Bystander device 310 may enable bystanders to select various permissions that indicate whether one or more capture devices, including capture device 300 , may record the bystander's identifying information. In some embodiments, the bystander device 310 may also be a capture instance of the authority state, including authorizing the public to record their identifying information, authorizing specific individuals to record their identifying information (e.g., an individual with whom a bystander has a community connection on an online system) ), and does not authorize the public to record its identifying information. Identification information is information from which the identity of an individual can be determined or inferred, directly or indirectly. Identifying information may include a portion of the individual's face, a portion of the individual's body, a portion of the individual's voice, some other information unique to the individual, or a combination thereof.

In some embodiments, bystander device 310 may generate a log of capture devices that have captured bystanders. This provides additional notification that bystander identifying information has been recorded. The bystander device 310 may provide the bystander with an interface for selecting the permission state of the capture device 300 . The bystander may specify the permission status based on the relationship between the user of the capture device 300 and the bystander. For example, a bystander may specify that the authority status is based on a degree of connection (eg, first degree or second degree connection on an online system), family relationship, or the like.

In one embodiment, the sensors of capture device 300 capture sensor data, such as images or audio, that describe a local area that includes a bystander. Bystander device 310 may transmit to capture device 300 private data associated with the bystander, which may include the bystander's identifying information. The privacy profile may include the permission status set by the bystander for the capture device 300 . Privacy data may include information about social connections between the user of capture device 300 and the bystander (e.g., the social network identifier of the bystander), demographic information about the bystander (e.g., the age range to which the bystander belongs) ) or similar information. Demographic information such as age range may allow the capture device to determine that the capture device's information modifier module should modify identifying information (eg, withhold the identity of children within recorded video to protect their privacy).

The capture device 300 can determine the position of the bystander according to the sensor data captured by the sensors of the capture device 300 . Capture device 300 may use data received from bystander device 310 (e.g., via UWB), sensor data measured by sensors of capture device 300, data received by capture device 300 (e.g., GPS coordinates), or combination to determine the position. The capture device 300 can determine the permission status of the bystander based on the private information associated with the bystander. For example, the privacy profile may specify a permission status based on the presence or absence of a social connection on the social graph (eg, if the social connection exists, the permission status indicates that the capture device 300 is authorized by a bystander). In some specific examples, the individuals included in the user's community graph are each associated with a permission status. Capture device 300 may transmit a request to the bystander to receive explicit authorization for its identifying information to be recorded. Alternatively or additionally, the permission status of a bystander may be based on a community graph associated with the bystander.

In response to determining that the bystander is an unauthorized bystander, capture device 300 may use the bystander's location to determine a region of interest 320 of the captured sensor data that includes identifying information for the bystander. In some embodiments, determining the region of interest 320 of the sensor data including bystander identification information includes determining a portion of the sensor data that represents (eg, depicts or makes a sound for) : at least a portion of a bystander's face, at least a portion of a bystander's body, a bystander's voice, any suitable recordable information identifying a bystander, or a combination thereof. Capture device 300 may modify identification information in region of interest 320 such that bystanders cannot be identified based on the modified identification information. In one example, modifying the identification information includes scrambling the pixels of the image data within the bounding box corresponding to the region of interest 320 . In another example, modifying the identification information includes not presenting the data within a bounding box corresponding to the region of interest 320 . In the specific example where audio is captured by capture device 300, capture device 300 may vary the frequency of audio associated with (e.g., emanating from) region of interest 320, not present audio associated with region of interest 320 , scrambling the bits of the audio associated with the region of interest 320, or the like.

In response to capture device 300 determining that the bystander is an authorized bystander, capture device 300 may modify the sensor data region of interest 320 based on the additional data. Additional data may include, but is not limited to, data in the social graph of the user of the capture device 300, determinations of operations in a private environment, or any suitable contextual information that results in the concealment of bystander identifying information. In some embodiments, capture device 300 may not change the identification information in response to determining that the bystander is an authorized bystander. Capture device 300 may store the identifying information as originally captured within the sensor data. For example, the capture device 300 can store the video of the authorized bystander on the remote server of the social network system for users of the social network system to access.

FIG. 4 shows a workflow of modifying identification information by a capture device 400 according to at least one embodiment. Capture device 400 can identify the permission status of bystander device 410, locate regions of interest within the sensor data that have the bystander's identifying information, and de-identify the identifying information (e.g., make it impossible for bystanders to recognize the bystander from the modified identification information). information to identify). Although two devices are depicted in Figure 4, in alternative or additional embodiments there may be additional capture devices or bystander devices.

Communication circuitry of capture device 400 and bystander device 410 may be used to determine the relative location between the two devices (eg, using short-range wireless communication protocols such as Bluetooth or UWB for positioning). Capture device 400 and bystander device 410 may be in close proximity to each other (eg, within the broadcast range of a short-range wireless communication protocol), and positioning features on both devices may identify each other and their relative physical locations. Capture device 400 includes sensors for capturing recordable information (eg, video, audio, etc.) about a local area.

In some embodiments, after the capture device 400 determines the relative position of the bystander device 410, the capture device 400 may determine whether the bystander has been captured in the fed sensor data based on the relative position of the bystander device 410 identifying information. For example, capture device 400 may determine whether a bystander is within the field of view of an image sensor, within hearing range of a microphone, or a combination thereof. Positioning may be performed using one or more positioning algorithms, machine learning models, heuristics, or the like.

The capture device 400 may determine whether to modify the identification information of the bystander based on the permission status, which is determined based on the private information transmitted from the bystander device 410 to the capture device 400 . The authorization request module of the capture device 400 can determine whether the capture device 400 is authorized to store the identification information. In response to determining that bystander device 410 has not authorized capture device 400 to store identification information, capture device 400 modifies captured sensor data within the determined region of interest, the captured sensor data including the bystander's identification information (e.g., , bystander's raw image feed or raw audio feed). In some embodiments, the capture device 400 may limit or further limit the sharing of bystander information with the user of the capture device 400 . For example, when capturing raw image data, capture device 400 may blur areas of the image containing bystanders (such as parts of the face or body of the bystander), not present data in areas containing bystanders, make any suitable modifications to the image Depictions so that bystanders cannot be identified by modification, or a combination thereof. In another example, when the raw audio data is captured, the integration layer can change the frequency of the audio in the region of interest, not present the audio in the region of interest, scramble the bits of the audio in the region of interest, perform any processing on the audio Appropriate modification such that bystander audio cannot be identified by modification, or a combination thereof. In some embodiments, the identification information modified by capture device 400 may be provided to additional software or hardware components of capture device 400 for further processing or storage. For example, the modified identification information may be provided to remote database 420 for storage.

In some embodiments, the sensor is a camera. In these embodiments, the camera captures a raw image of the local area that includes the bystander. Capture device 400 may receive private data from bystander device 410 communicatively coupled to capture device 400 . The capture device 400 determines the location of the bystanders according to the image data captured by the camera or additional data. The capture device 400 determines the permission status of the bystander based on the private information associated with the bystander. In response to determining that the bystander is an unauthorized bystander based on the permission status of the bystander, the capture device 400 uses the determined location of the bystander to determine a region of interest within the image data that includes identification information. In addition, the information modifier module 445 of the capture device 400 can modify the identification information within the region of interest of the image data such that the visual representation of the bystander cannot be recognized through the modified region of the image data (e.g., scrambling corresponding to unauthorized Pixels within the bounding box of the area of interest of the bystander, data within the bounding box of the area of interest where image data is not presented, etc.). As depicted in FIG. 4, the information modifier module 445 can blur the image of the head of the bystander so that the face of the bystander cannot be recognized by blurring.

In some embodiments, the sensor is a microphone. The microphone of capture device 400 captures audio data describing a local area including a bystander. Capture device 400 receives private data from bystander device 410 communicatively coupled to capture device 400 . The capture device 400 determines the position of the bystander based on the audio data or additional data captured by the microphone. For example, one or more portions of audio data including bystander voices may be determined. The capture device 400 determines the permission status of the bystander based on the private information associated with the bystander. In response to determining that the bystander is an unauthorized bystander based on the permission status of the bystander, capture device 400 may use the determined location of the bystander to determine a region of interest within the audio data that includes identifying information. In addition, the information modifier module 445 of the capture device 400 can modify the identifying information in the region of interest in the audio data (e.g., by changing the frequency, not presenting the audio in the region of interest, scrambling the bits of the audio, etc.) . As depicted in FIG. 4 , the information modifier module 445 may alter the frequency response of the audio signal such that the true pitch of the bystander is not recognizable via the modified audio.

5 is a flowchart of a method 500 for capturing sensor data of unauthorized or authorized users, according to one or more embodiments. The process shown in FIG. 5 may be performed by components of a capture device (eg, capture device 200 ). In other specific examples, other entities may perform some or all of the following steps in FIG. 5 . Embodiments may include different and/or additional steps, or perform steps in a different order.

The capture device captures 510 sensor data describing a local area including a bystander. For example, the video camera of the headset captures video and audio of the park including park visitors.

The capture device receives 520 private data associated with the bystander from the bystander's device. The bystander's device is communicatively coupled to the capture device. According to a previous example, a headset may receive private data from a bystander's smartphone indicating a capture belonging to a user connected to the bystander on a social graph of an online system (eg, a social networking system) Devices may be authorized to store bystander identification information, and capture devices belonging to those users who are not connected to bystanders are not authorized to store bystander identification information.

The capture device determines 530 the location of the bystander based on the sensor data. For example, the location module of the headset of the previous example uses a combination of beamforming and proximity detection to determine the location of bystanders via a short-range communication protocol (eg, UWB).

The capture device determines 540 the permission status of the bystander based on the private data associated with the bystander. According to the previous example, the authorization request module of the headset can use the social graph of the social network and the social network identifier of the bystander received in the private profile to determine the use of the bystander and the capture device There is no community connection between them.

The capture device determines 550 whether the bystander is an authorized bystander or an unauthorized bystander. For example, the headset of the previous examples can take advantage of the lack of social connection between the user and the bystander to determine that the bystander is an unauthorized bystander. In response to determining that the bystander is an unauthorized bystander, the capture device may use the determined location of the bystander to determine 560 a region of the sensor data that includes identifying information for the bystander. Continuing with the previous example, a region of the captured video data depicting the face of the unauthorized bystander and a portion of the audio data including the unauthorized bystander's voice are determined. Capture device 570 modifies the identification information in the field of sensor data such that bystanders cannot be identified. For example, the headset of the previous examples blurs the face of an unauthorized bystander and changes the frequency of the audio signal corresponding to the bystander's voice such that the bystander cannot be identified based on the blurred face of the bystander or its modified voice By.

The capture device stores 580 the sensor data. The capture device may store sensor data including modified identification information in response to determining that the bystander is an unauthorized bystander. Alternatively, the capture device may store sensor data including identification information of the bystander in response to determining that the bystander is an authorized bystander.

The capture device provides 590 the sensor data to a display device. Sensor data can be distributed to or accessed by other devices. For example, the headset of the previous example provides video data in which unauthorized bystanders have blurred faces and distorted speech to an online system for storage, where the video data can be read by the user of the headset or by additional users of the online system access. However, the identity of the bystander is protected within the provided video data because the headset has made the modified identification information unidentifiable to the bystander.

FIG. 6 is a system 600 including a headset 605 according to one or more embodiments. In some specific examples, the head-mounted device 605 can be the head-mounted device 100 of FIG. 1 . System 600 may operate in an artificial reality environment (eg, a virtual reality environment, an augmented reality environment, a mixed reality environment, or some combination thereof). The system 600 shown in FIG. 6 includes a head-mounted device 605 , an input/output (input/output; I/O) interface 610 , a bystander device 615 , a network 620 and an online system 625 . Although FIG. 6 shows an example system 600 including a headset 605 and an I/O interface 610, in other embodiments any number of these components may be included in the system 600. For example, there may be multiple headsets each having an associated I/O interface 610 , where each headset communicates with a bystander device 615 . In alternative configurations, different and/or additional components may be included in system 600 . Additionally, in some embodiments, the functionality described in connection with one or more of the components shown in FIG. 6 may be distributed among the components differently than described in connection with FIG. 6 .

Headset 605 includes display assembly 630 , optics block 635 , one or more position sensors 640 , DCA 645 , audio system 650 , communication circuitry 655 and controller 660 . Some embodiments of headset 605 have different components than those described in connection with FIG. 6 . For example, the headset 605 may include sensors such as a microphone. Additionally, in other embodiments, the functionality provided by the various components described in connection with FIG. 6 may be distributed among the components of the headset 605 in different ways, or captured in a separate assembly remote from the headset 605 .

The display assembly 630 can display content to the user according to the data received from the console. The display assembly 630 uses one or more display elements (eg, the display element 120 ) to display content. The display element may be, for example, an electronic display. In various embodiments, display assembly 630 includes a single display element or multiple display elements (eg, one display for each eye of the user). Examples of electronic displays include: liquid crystal display (liquid crystal display; LCD), organic light emitting diode (organic light emitting diode; OLED) display, active-matrix organic light-emitting diode display (active-matrix organic light-emitting diode display; AMOLED), a waveguide display, some other display, or some combination thereof. It should be noted that in some embodiments, the display element 120 may also include some or all of the functionality of the optics block 635 .

Optics block 635 may amplify image light received from the electronic display, correct optical errors associated with the image light, and present the corrected image light to one or both eye sockets of head mounted device 605 . In various embodiments, optics block 635 includes one or more optical elements. Example optical elements included in optics block 635 include apertures, Fresnel lenses, convex lenses, concave lenses, filters, reflective surfaces, or any other suitable optical elements that affect image light. Additionally, the optics block 635 may include a combination of different optical elements. In some embodiments, one or more of the optical elements in optics block 635 may have one or more coatings, such as partially reflective or antireflective coatings.

Magnification and focusing of image light by optics block 635 allows electronic displays to be physically smaller, weigh less and consume less power than larger displays. In addition, magnification can increase the field of view of content presented by the electronic display. For example, the field of view of the displayed content is such that nearly all of the user's field of view (eg, about a 110 degree diagonal), and in some cases all of the user's field of view is used to present the displayed content. Additionally, in some embodiments, the amount of magnification can be adjusted by adding or removing optical elements.

In some embodiments, optics block 635 can be designed to correct for one or more types of optical errors. Examples of optical errors include barrel or pincushion distortion, longitudinal chromatic aberration, or lateral chromatic aberration. Other types of optical errors may further include spherical aberration, chromatic aberration, or errors due to lens field curvature, astigmatism, or any other type of optical error. In some embodiments, the content provided to the electronic display for display is pre-distorted, and the optics block 635 corrects the distortion as it receives image light generated based on the content from the electronic display.

Position sensor 640 is an electronic device that generates data indicative of the position of headset 605 . The position sensor 640 generates one or more measurement signals in response to the movement of the headset 605 . The position sensor 190 is a specific example of the position sensor 640 . Examples of position sensors 640 include: one or more IMUs, one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects motion, or one of its combinations. Position sensors 640 may include multiple accelerometers to measure translational motion (forward/backward, up/down, left/right), and to measure rotational motion (e.g., pitch, yaw, roll) multiple gyroscopes. In some embodiments, the IMU rapidly samples the measurement signal and calculates the estimated position of the headset 605 from the sampled data. For example, the IMU integrates measurements received from the accelerometer over time to estimate a velocity vector, and integrates the velocity vector over time to determine the estimated position of a reference point on the headset 605 . A reference point is a point that can be used to describe the position of the headset 605 . While a reference point may generally be defined as a point in space, in practice, a reference point is defined as a point within the headset 605 .

DCA 645 generates depth information for a portion of the local area. A DCA includes one or more imaging devices and a DCA controller. DCA 645 may also include illuminators. The operation and structure of DCA 645 are described above with respect to FIG. 1A .

The audio system 650 provides audio content to the user of the headset 605 . The audio system 650 is substantially the same as the audio system 200 described above. Audio system 650 may include one or more acoustic sensors, one or more transducers, and an audio controller. The audio system 650 can provide spatialized audio content to the user. In some embodiments, the audio system 650 can request the acoustic parameters from the mapping server via the network 620 . Acoustic parameters describe one or more acoustic properties of a local area (eg, room impulse response, reverberation time, reverberation level, etc.). Audio system 650 may provide information describing at least a portion of a local area from, for example, DCA 645 and/or position information of headset 605 from position sensor 640 . The audio system 650 may use one or more of the acoustic parameters received from the mapping server to generate one or more audio filters, and provide audio content to the user using the audio filters.

The functions performed by the communication circuitry 655 and the controller 660 of the headset 605 are similar to the functions respectively performed by the controller 230 and the communication circuitry 220 of FIG. 2 . Accordingly, the headset 605 is configured to capture sensor data and modify identification information as necessary based on the authorization status specified by the bystander to ensure that the privacy of the bystander is ensured.

I/O interface 610 is a device that allows a user to send action requests and receive responses from the console or other suitable controller of headset 605 (eg, smartphone). An action request is a request to perform a specific action. For example, the action request can be an instruction to start or end the capture of image or video data, to request permission from the bystander device 615 to record identification information of a bystander, or to execute a specific action within an application. The I/O interface 610 may include one or more input devices. Example input devices include a keyboard, mouse, game controller, or any other suitable device for receiving and communicating action requests to the console. Action requests received by the I/O interface 610 are communicated to the console, which executes the action corresponding to the action request. In some embodiments, I/O interface 610 includes an IMU that captures calibration data indicating an estimated position of I/O interface 610 relative to an initial position of I/O interface 610 . In some embodiments, I/O interface 610 can provide tactile feedback to the user based on commands received from the console. For example, tactile feedback is provided when an action request is received, or the console transmits an instruction to the I/O interface 610, so that the I/O interface 610 generates tactile feedback when the console performs an action.

The bystander device 615 provides the privacy information to the headset 605 for determining whether the bystander of the bystander device 615 authorizes the headset to record the identification information of the bystander. In the example shown in FIG. 6 , bystander device 615 includes communication circuitry 665 and controller 670 . Some embodiments of bystander device 615 have different modules or components than those described in connection with FIG. 6 . For example, a bystander device may include one or more sensors. Communications circuitry 665 may perform functions similar to those performed by communications circuitry 220 of FIG. 2 . Similarly, controller 670 may perform functions similar to those performed by controller 230 of FIG. 2 .

Network 620 couples headset 605 and/or spectator device 615 to online system 625 . The online system 625 may be a social networking system that maintains a social graph including social connections between users of the social networking system. Network 620 may include any combination of local area networks and/or wide area networks using both wireless and/or wired communication systems. For example, the network 620 may include the Internet, and a mobile phone network. In one embodiment, network 620 uses standard communication techniques and/or protocols. Therefore, the network 620 may include using such as Ethernet 802.11, worldwide interoperability for microwave access (WiMAX), 2G/3G/4G mobile communication protocols, digital subscriber line (digital subscriber line; DSL), Links of technologies such as asynchronous transfer mode (asynchronous transfer mode; ATM), wireless bandwidth, and fast PCT advanced switching. Similarly, network connection protocols used on network 620 may include multiprotocol label switching (MPLS), transmission control protocol/Internet protocol (TCP/IP), using User Datagram Protocol (UDP), Hypertext Transport Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), File Transfer Protocol (FTP), etc. Data exchanged via the network 620 may use images including binary forms (e.g., Portable Network Graphics (PNG)), hypertext markup language (HTML), extensible markup language (extensible markup language; XML) and other technologies and/or formats. Additionally, some or all of the links may be encrypted using known encryption techniques, such as secure sockets layer (SSL), transport layer security (TLS), virtual private network; VPN), Internet Protocol security (Internet Protocol security; IPsec), etc.

One or more components of system 600 may include a privacy module that stores one or more privacy settings for user data elements. The user data element describes the user or headset 605 . For example, user data elements may describe the user's physical characteristics, actions performed by the user, the user's location of the headset 605, the location of the headset 605, the user's HRTF, and the like. Privacy settings (or "access settings") for User Data Elements may be stored in any suitable manner, such as associated with the User Data Element, in an index on an authorized server, in another suitable manner, or in any other Suitable combination storage. One example of a privacy setting is for a bystander to select the permission state for the capture device.

Privacy settings for User Data Elements specify how User Data Elements (or certain information associated with User Data Elements) may be accessed, stored, or otherwise used (for example, viewed, shared, modified, copied, executed, surface treatment or identification). In some embodiments, the privacy settings of a user data element may specify a "denied list" of entities that may not have access to certain information associated with the user data element. A privacy setting associated with a user data element may specify any suitable detail with which access is permitted or denied. For example, some entities may have permissions to view the existence of certain user data elements, some entities may have permissions to view the content of certain user data elements, and some entities may have permissions to modify certain user data elements. Privacy settings may allow a user to allow other entities to access or store user data elements for a limited period of time.

Privacy settings may allow a user to specify one or more geographic locations from which user data elements can be accessed. Access or denial of access to a user data element may depend on the geographic location of the entity attempting to access the user data element. For example, a user may allow access to user data elements and specify that user data elements are only accessible by the entity when the user is in a particular location. If the user leaves a particular location, the user data element may no longer be accessible by the entity. As another example, a user may specify that user data elements are only accessible by entities within a threshold distance from the user, such as another user of the headset within the same local area as the user. If the user subsequently changes location, entities that have access to the user's data elements may lose access, while entities of the new group may gain access when they appear within a threshold distance of the user.

System 600 may include one or more authorization/privacy servers for enforcing privacy settings. A request from an entity for a particular User Data Element may identify the entity associated with the request if the Authorization Server determines that the entity is authorized to access the User Data Element based on the Privacy Settings associated with the User Data Element, and Only user data elements can be sent to the entity. If the requesting entity is unauthorized to access the user data element, the authorization server may prevent the requested user data element from being retrieved or may prevent the requested user data element from being sent to the entity. Although this disclosure describes enforcing privacy settings in a particular manner, this disclosure contemplates enforcing privacy settings in any suitable manner. Additional Configuration Information

The foregoing descriptions of specific examples have been presented for purposes of illustration; they are not intended to be exhaustive or to limit patent rights to the precise forms disclosed. Those skilled in the related art can understand that many modifications and changes can be made in consideration of the above invention.

Portions of this specification describe specific examples in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in data processing to effectively convey the substance of their work to others skilled in the art. While such operations are described functionally, computationally or logically, such operations should be understood to be implemented by computer programs or equivalent circuits, microcode or the like. Furthermore, it has also proven convenient, without loss of generality, to refer to such configurations of operation as modular. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

Any of the steps, operations or processes described herein may be performed or implemented by one or more hardware or software modules alone or in combination with other devices. In one embodiment, a software module is implemented by a computer program product comprising a computer readable medium containing computer code executable by a computer processor to perform any or all of the steps described , operation or process.

Specific examples may also relate to apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes and/or it may comprise a general purpose computing device selectively enabled or reconfigured by a computer program stored in the computer. The computer program can be stored on a non-transitory tangible computer readable storage medium that can be coupled to a computer system bus or any type of medium suitable for storing electronic instructions. Additionally, any computing system referred to in this specification may include a single processor, or may be an architecture designed to increase computing power using multiple processors.

Particular examples may also relate to products resulting from the computational processes described herein. This product may contain information generated by a computing process, where the information is stored on a non-transitory tangible computer-readable storage medium, and may include any specific instance of a computer program product or other combination of data described herein.

Finally, the language used in this specification has been chosen primarily for readability and instructional purposes, and it may not have been chosen to delineate or limit patent rights. Accordingly, it is intended that the scope of patent rights be limited not by this detailed description, but rather by the scope of any claims that issue based on applications herein. Accordingly, the disclosure of specific examples is intended to illustrate but not limit the scope of the patent rights set forth in the following claims.

100,605:Headsets 110: frame 120: display components 130: Imaging device/image device 140: illuminator 150:Audio controller 160: speaker 170: Tissue transducer 180: Acoustic sensor 190,640: position sensor 200,300,400: capture device 210: Sensor assembly 220,655,665: Communication circuit systems 230,660,670: Controller 235: Authorization request module 240: Positioning module 245,445: Info modifier mod 250: Mode selection module 255: Capture device tracking module 260: Capture Device Tracking Log 265: Sensor data storage 310,410,615: Bystander Devices 320: area of interest 420: Remote database 500: method 510,520,530,540,550,560,570,580,590: steps 600: system 610: input/output interface 620: network 625: Online system 630: display assembly 635: Optics block 645: Depth camera assembly 650: Audio system

[ FIG. 1 ] is a perspective view of a head-mounted device implemented as a glasses device according to one or more embodiments.

[ Fig. 2 ] is a block diagram of a capture device according to one or more embodiments.

[ FIG. 3 ] Depicts a user with a capture device and a bystander with a bystander device, according to one or more embodiments.

[ FIG. 4 ] shows a workflow of modifying identification information by a capture device according to one or more embodiments.

[ FIG. 5 ] is a flowchart of a method for capturing sensor data of unauthorized or authorized users according to one or more embodiments.

[ Fig. 6 ] is a system including a head-mounted device according to one or more embodiments.

The figures depict various specific examples for purposes of illustration only. Those of ordinary skill in the art will readily recognize from the discussion below that alternative embodiments of the structures and methods illustrated herein may be used without departing from the principles described herein.

500: method

510,520,530,540,550,560,570,580,590: steps

Claims (20)

A capture device comprising: A sensor configured to: Capture sensor data describing a localized area including bystanders; Communications circuitry configured to: receiving private data associated with the bystander from a device of the bystander that is communicatively coupled to the capture device; and A controller configured to: Determine the position of the bystander based on the sensor data, Based on the private information associated with the bystander, determine the authority status of the bystander, and In response to a determination that the bystander is an unauthorized bystander based on the authority status of the bystander: using the determined location to determine a region of the sensor data that includes identifying information for the bystander, and modifying the identification information in the area of the sensor data such that the bystander cannot be identified using the modified identification information. The capture device of claim 1, wherein the controller is further configured to: In response to a determination that the bystander is a temporarily authorized bystander based on the authority status of the bystander: transmit a request to the device requesting permission to store the identifying information for a predetermined duration; receiving from the bystander the authorization to store the identifying information for the predetermined duration; and In response to a determination that the predetermined duration has expired, the identification information in the field of the sensor data is modified. The capture device of claim 1, wherein the controller is further configured to: receiving a first broadcast message from a nearby capture device of a nearby user, the first broadcast message indicating an intent to capture the sensor data, the first broadcast message including an identifier of the nearby capture device or a hash of the nearby user at least one of the group network identifiers; and In response to receiving the first broadcast message: generating a second broadcast message including private data associated with the neighboring user, and transmit the second broadcast message. The capture device of claim 1, wherein the controller is further configured to: using beamforming to identify an audio signal associated with the sound from the bystander in the local area; use the identified audio signal to determine the relative location of the bystander; and The location of the bystander is determined using the relative location and Global Positioning System (GPS) coordinates of the capture device. The capture device of claim 1, wherein the controller is further configured to: Determine to operate in private mode; In response to operating in this private mode: request permission from a nearby device to store audio data associated with the user of the nearby device, the nearby device is within the local area and the personal area network of the capture device, storing the audio data associated with the user of the neighboring device in response to receiving approval of the request from the neighboring device, and In response to receiving a denial of the request from at least one of the neighboring devices: determining a plurality of fields in the sensor data that include identification information for the user of the at least one of the neighboring devices; and The identification information in the plurality of fields of the sensor data is modified such that the user of the at least one of the neighboring devices cannot be identified using the modified identification information in the plurality of fields. The capturing device of claim 5, wherein the controller is further configured to: Determine at least one of the ambient background volume level or the number of people in the local area; and It is determined to operate in the private mode in response to at least one of the following situations: the ambient background volume level drops below a threshold volume level, or the number of people drops below a threshold number. The capture device of claim 1, wherein the controller is further configured to: identifying image data corresponding to the identifying information in the region of the sensor data; and Processing the image data, the processed image data represents at least one of blurred or restricted images of the bystander's face. The capture device of claim 1, wherein the controller is further configured to: identifying audio data corresponding to the identifying information in the field of the sensor data; and processing the audio data, the processed audio data representing at least one of the bystander's FM voice. The capture device of claim 1, wherein the controller is further configured to: accessing a hash social network identifier associated with the online system on which the user holds an account, from the private data associated with the bystander; displaying a prompt to the user to establish a social connection with the bystander on the online system; Responsive to selection of the prompt, receive notification from the online system that the social connection has been established between the user and the bystander; and The permission status of the bystander is updated, the updated permission status indicating that the bystander is an authorized bystander. The capturing device of claim 1, wherein the received private data includes a hash social network identifier of the bystander, the hash social network identifier is associated with an online system, and wherein the controller is further configured state to: using the hash social network identifier to access a social graph representing social connections between users of the online system; identifying a lack of social connections between the user and the bystander in the social graph; and Determining the absence of the social link corresponds to the permission status of the unauthorized bystander indicating that the bystander is refusing to store the identifying information. A method comprising: capture sensor data by the sensors of the user's capture device, the sensor data describing the local area including the bystander; receiving private data associated with the bystander from a device of the bystander that is communicatively coupled to the capture device; Determine the location of the bystander based on the sensor data; Based on the private data associated with the bystander, determine the authority status of the bystander; and In response to determining that the bystander is an unauthorized bystander based on the permission status of the bystander: using the determined location to determine a region of the sensor data that includes identifying information for the bystander, and modifying the identification information in the area of the sensor data such that the bystander cannot be identified using the modified identification information. The method as claimed in item 11, further comprising: In response to determining that the bystander is a temporarily authorized bystander based on the authority status of the bystander: transmit a request to the device requesting permission to store the identifying information for a predetermined duration; receiving authorization from the bystander to store the identifying information for the predetermined duration; and In response to determining that the predetermined duration has expired, modifying the identification information in the field of the sensor data. The method as claimed in item 11, further comprising: receiving a first broadcast message from a nearby capture device of a nearby user, the first broadcast message indicating an intent to capture the sensor data, the first broadcast message including an identifier of the nearby capture device or a hash of the nearby user at least one of the group network identifiers; and In response to receiving the first broadcast message: generating a second broadcast message including private data associated with the neighboring user, and transmit the second broadcast message. The method as claimed in item 11, further comprising: using beamforming to identify an audio signal associated with the sound from the bystander in the local area; use independent audio signals to determine the relative location of the bystander; and The location of the bystander is determined using the relative location and Global Positioning System (GPS) coordinates of the capture device. The method as claimed in item 11, further comprising: Determine to operate in private mode; In response to operating in this private mode: requesting permission from a nearby device to store audio data associated with a user of the nearby device within the local area and personal area network of the capture device, storing the audio data associated with the user of the neighboring device in response to receiving approval of the request from the neighboring device, and In response to receiving a denial of the request from at least one of the neighboring devices: determining a plurality of fields in the sensor data that include identification information for the user of the at least one of the neighboring devices; and The identification information in the plurality of fields of the sensor data is modified such that the user of the at least one of the neighboring devices cannot be identified using the modified identification information in the plurality of fields. The method as claimed in item 15, further comprising: Determine at least one of the ambient background volume level or the number of people in the local area; and It is determined to operate in the private mode in response to at least one of the following situations: the ambient background volume level drops below a threshold volume level, or the number of people drops below a threshold number. The method as claimed in item 11, further comprising: identifying image data corresponding to the identifying information in the region of the sensor data; and Processing the image data, the processed image data represents at least one of blurred or restricted images of the bystander's face. The method of claim 17, further comprising: accessing a hash social network identifier associated with the online system on which the user holds an account, from the private data associated with the bystander; displaying a prompt to the user to establish a social connection with the bystander on the online system; in response to selecting the prompt, receive notification from the online system that the social connection has been established between the user and the bystander; and The permission status of the bystander is updated, the updated permission status indicating that the bystander is an authorized bystander. The method as claimed in item 11, further comprising: identifying audio data corresponding to the identifying information in the field of the sensor data; and processing the audio data, the processed audio data representing at least one of the bystander's FM voice. A non-transitory computer-readable storage medium comprising stored instructions that, when executed by a processor of a capture device, cause the capture device to: capturing sensor data by a sensor of the capture device of the user, the sensor data describing a local area including bystanders; receiving private data associated with the bystander from a device of the bystander that is communicatively coupled to the capture device; Determine the location of the bystander based on the sensor data; Based on the private data associated with the bystander, determine the authority status of the bystander; and In response to determining that the bystander is an unauthorized bystander based on the authority status of the bystander: using the determined location to determine the region of the sensor data that includes identifying information for the bystander, and modifying the identification information in the area of the sensor data such that the bystander cannot be identified using the modified identification information.

Images