KR20240040807A - Low-power machine learning using regions of interest captured in real time - Google Patents

Abstract

Systems and methods for generating image content are described. Systems and methods include, in response to receiving a request to cause a sensor of a computing device to identify image content associated with optical data captured by the sensor, detecting a first sensor data stream having a first image resolution, and 2 and detecting a second sensor data stream having an image resolution. Systems and methods also include identifying, by processing circuitry of a computing device, at least one region of interest in a first sensor data stream, extracting coordinates defining a plurality of first pixels in the at least one region of interest in the first sensor data stream. It may include determining. and generating a first sensor data stream and an excerpt image representing at least one region of interest.

Description

Low-power machine learning using regions of interest captured in real time

This application claims priority to the continuation-in-application of U.S. Application No. 17/819,170, and claims priority to the continuation-in-application of U.S. Application No. 17/819,154, filed on August 11, 2022, both of which claim priority thereto. The benefit of U.S. Provisional Application No. 63/260,206, filed August 12, 2021, and U.S. Provisional Application No. 63/260,207 are claimed. The disclosures of both of these applications are incorporated herein by reference in their entirety.

This application also claims priority to U.S. Provisional Application No. 63/260,206, filed Aug. 12, 2021, and U.S. Provisional Application No. 63/260,207, filed Aug. 12, 2021, the disclosures of which are incorporated by reference in their entirety. It is included in this specification as.

This description generally relates to methods, devices and algorithms used to process image content.

Computer vision technology allows computers to analyze and extract information from images. These computer vision technologies can be expensive in terms of processing and power consumption. As the need to balance performance and power consumption in mobile computing devices continues to grow, device manufacturers are tasked with configuring their devices to balance device performance and image degradation to avoid exceeding the limits of mobile computing devices. I'm in charge of the rules.

A system comprised of one or more computers may be configured to perform specific operations or actions that cause or cause the system to perform actions when software, firmware, hardware, or a combination thereof is installed and operated on the system. One or more computer programs may be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause the device to perform the actions.

In a first general aspect, an image processing method is described. The method is responsive to receiving a request to cause a sensor of a wearable or non-wearable computing device to identify image content associated with optical data captured by the sensor, detecting a first sensor data stream having a first image resolution. and detecting a second sensor data stream having a second image resolution, wherein the first sensor data stream is based on optical data. The method includes identifying, by processing circuitry of a wearable or non-wearable computing device, at least one region of interest in a first sensor data stream, and, by processing circuitry, identifying a plurality of regions of interest in the at least one region of interest in the first sensor data stream. determining cropping coordinates defining first pixels of the pixels, and generating, by the processing circuitry, a cropped image representing at least one region of interest. You can. The generating step includes identifying a second plurality of pixels in a second sensor data stream using coordinates to extract that define a first plurality of pixels in at least one region of interest in the first sensor data stream, and Extracting the data stream into a plurality of second pixels may be included.

Implementations may include any of the following features, alone or in combination:

The method also includes, by processing circuitry, performing optical character resolution on the excerpt image to generate a machine-readable version of the excerpt image, and performing, by the processing circuitry, a search query using the machine-readable version of the excerpt image. This may include generating a plurality of search results and displaying the search results on a display of the computing device.

In some implementations, the second sensor data stream is stored in a memory of a wearable or non-wearable computing device, and the method, in response to identifying at least one region of interest in the first sensor data stream, comprises: generating the second sensor data stored in the memory; and restricting access to the first sensor data stream while continuing to sense and access the second sensor data stream.

In some implementations, a method includes, in response to generating a snippet image representative of at least one region of interest, transmitting the generated snippet image to a mobile device in communication with a wearable or non-wearable computing device, from the mobile device, at least one The method may further include receiving information regarding the region of interest, and displaying the information on a display of the wearable or non-wearable computing device.

In some implementations, the computing device is a battery-powered computing device, and the first image resolution has a low image resolution and the second image resolution has a high image resolution. In some implementations, identifying at least one region of interest includes using the first sensor data as input by a machine learning algorithm running on a wearable or non-wearable computing device to detect at least one or more text displayed in the first sensor data. It further includes the step of identifying the object.

In some implementations, the processing circuitry includes at least a first image processor configured to perform image signal processing for a first sensor data stream, and a second image processor configured to perform image signal processing for a second sensor data stream; The first image resolution of the first sensor data stream is lower than the second image resolution of the second data sensor stream.

In some implementations, generating an excerpt image representing the at least one region of interest is performed in response to detecting that the at least one region of interest meets a threshold condition, wherein the threshold condition is a plurality of first regions of the second sensor data stream. 2. Detecting that the pixel has low blur.

In a second general aspect, a wearable computing device is described. The wearable computing device includes at least one processing device, at least one image sensor configured to capture optical data, and memory storing instructions that, when executed, cause the wearable computing device to perform operations, the operations comprising: In response to receiving a request to cause a sensor to identify image content associated with the optical data, detecting a first sensor data stream having a first image resolution, the first sensor data stream being based on the optical data; and and detecting a second sensor data stream having a second image resolution, wherein the second sensor data stream is based on optical data.

The operations include identifying at least one region of interest in the first sensor data stream, determining excerpt coordinates defining a plurality of first pixels in the at least one region of interest in the first sensor data stream, and at least Generating an excerpt image representing a region of interest, wherein generating the second sensor data using the excerpt coordinates defining a plurality of first pixels in the at least one region of interest in the first sensor data stream. Identifying a second plurality of pixels in the stream, and extracting the second sensor data stream into the plurality of second pixels.

Implementations may include any of the following features alone or in combination: In some implementations, at least one image sensor is a dual stream image sensor configured to operate in a low image resolution mode until triggered to switch to operation in a high image resolution mode. In some implementations, the operations include performing optical character resolution on the excerpt image to generate a machine-readable version of the excerpt image, and performing a search query using the machine-readable version of the excerpt image to produce a plurality of search results. It further includes an operation of generating, and an operation of displaying the search results on the display of the wearable computing device. In some implementations, a wearable computing device can trigger audio output of the achieved optical character resolution from speakers of the wearable computing device.

In some implementations, the second sensor data stream is stored in a memory of the wearable computing device, and the operations are responsive to identifying at least one region of interest in the first sensor data stream and corresponding in the second sensor data stored in the memory. searching for at least one region of interest, and restricting access to the first sensor data stream while continuing to sense and access the second sensor data stream.

In some implementations, the operations include transmitting the generated snippet image to a mobile device in communication with the wearable computing device in response to generating the snippet image representing at least one region of interest, and transmitting the generated snippet image from the mobile device to the at least one region of interest. It further includes an operation of receiving information about the device, and an operation of displaying the information on the display of the wearable computing device. In some implementations, the operations include causing information to be output (eg, auditory, visual, tactile, etc.) from the wearable computing device.

In some implementations, the first image resolution has a low image resolution, the second image resolution has a high image resolution, and the operations may further include identifying at least one region of interest, the operations comprising identifying the wearable The method may further include identifying text or at least one object displayed in the first sensor data using the first sensor data as input by a machine learning algorithm running on the computing device.

In some implementations, the at least one processing device includes a first image processor configured to perform image signal processing for the first sensor data stream, and a second image processor configured to perform image signal processing for the second sensor data stream. wherein the first image resolution of the first sensor data stream is lower than the second image resolution of the second data sensor stream.

Implementations of the described techniques may include hardware, methods or processes, or computer software on a computer-accessible medium. Details of one or more implementations are set forth in the accompanying drawings and description below. Other features will become apparent from the description, drawings, and claims.

1 is an example of a wearable computing device for generating and processing image content using limited computational and/or power resources in accordance with implementations described throughout this disclosure.
2 illustrates a system for performing image processing in a wearable computing device according to implementations described throughout this disclosure.
3A and 3B illustrate examples of wearable computing devices according to implementations described throughout this disclosure.
4A-4C illustrate example flow diagrams for performing image processing tasks in a wearable computing device according to implementations described throughout this disclosure.
5A and 5B illustrate example flow diagrams for performing image processing tasks in a wearable computing device according to implementations described throughout this disclosure.
6A-6C illustrate example flow diagrams for performing image processing tasks in a wearable computing device according to implementations described throughout this disclosure.
7A-7C illustrate example flow diagrams for performing image processing tasks in a wearable computing device according to implementations described throughout this disclosure.
8A and 8B illustrate example flow diagrams for emulating a dual resolution image sensor with a single image sensor to perform image processing tasks according to implementations described throughout this disclosure.
9 is a flow diagram illustrating an example of a process for performing an image processing task in a wearable computing device according to implementations described throughout this disclosure.
10 shows examples of computer devices and mobile computer devices that can be used with the techniques described herein.
Similar reference symbols in the various drawings represent similar elements.

This disclosure describes systems and methods for performing image processing that can enable wearable computing devices to efficiently perform computer vision tasks. For example, the systems and methods described herein may utilize processors, sensors, neural networks, and/or image analysis algorithms to recognize text, symbols, and/or objects within an image and/or extract information from the image. In some implementations, the systems and methods described herein can perform these tasks while operating in a reduced compute mode and/or reduced power mode. For example, the systems and methods described herein can be used to minimize the amount of image data used when performing computer vision and image analysis tasks, thereby reducing the use of certain hardware and/or device resources (e.g., memory, processor, network bandwidth, etc.). ) can be reduced.

The advantage of configuring wearable computing devices to reduce resource usage is that these devices can perform relatively low-power computational tasks within the device without transmitting the work to other devices (e.g., servers, mobile devices, other computers, etc.). can be provided. For example, instead of analyzing the entire image, the systems and methods described herein can identify regions of interest (ROIs) in the image data and one or more of these regions can be analyzed onboard in a wearable computing device. Reducing the amount of information to analyze while maintaining accurate results allows complex image processing tasks to be performed on wearable computing devices without the resource burden of analyzing the entire image.

In some implementations, the systems and methods described herein can be configured to run on a device to ensure that the device can perform efficient image processing tasks while reducing computational load and/or power. For example, the systems and methods described herein allow wearable devices to perform object detection tasks, optical character recognition (OCR) tasks, and/or other image processing while utilizing certain technologies to reduce power, memory, and/or processing consumption. You can enable it to get the job done.

In some implementations, the systems and methods described herein can ensure that complex image processing calculations can be performed on a wearable device without the support of other resources and/or devices. For example, an existing system may request assistance from other communicatively connected mobile devices, servers, and/or offboard systems to perform computationally intensive image processing tasks. The systems and methods described herein provide, for example, image portions (e.g., ROIs, objects, excerpted image content, etc.) that can be operated by a device with less computational power than a server while providing complete and accurate image processing capabilities. It provides the advantage of creating .

The systems and methods described herein allow wearable computing devices to utilize machine learning intelligence (e.g., low power consumption and/or low processing consumption) to perform computer vision tasks such as object detection, motion tracking, facial recognition, OCR tasks, etc. For example, neural networks, algorithms, etc.) can be used.

In some implementations, the systems and methods described herein may use scalable levels of image processing to utilize fewer processing and/or power resources. For example, a low-resolution image signal processor may be used to perform low-level image processing operations on a low-resolution image stream, while a high-resolution image signal processor may be used to perform high-level image processing operations on a high-resolution image stream. In some implementations, the output of the low-resolution image signal processor may be provided as input to additional sensors, detectors, machine learning networks, and/or processors. This output may be a portion of the image (e.g., a pixel, region of interest, recognized image content, etc.).

In some implementations, the systems and methods described herein may utilize one or more sensors onboard a wearable computing device to partition the sensing and/or processing tasks. For example, a wearable computing device may include a dual stream sensor (e.g., a camera) capable of retrieving both high-resolution and low-resolution image streams from a plurality of images. For example, a low-resolution image stream may be captured by a sensor acting as a scene camera. A high-resolution image stream can be captured by the same sensor that acts as a detail camera. One or two streams may be provided to different processors to perform image processing tasks. In some implementations, each stream may be used to generate different information for use in performing computer vision tasks, machine learning tasks, and/or other image processing tasks.

1 is an example of a wearable computing device 100 for generating and processing image content using limited computational and/or power resources according to implementations described throughout this disclosure. In this example, wearable computing device 100 is depicted in the form of AR smart glasses. However, all form factors of battery-powered devices can be replaced and combined with the systems and methods described herein. In some implementations, wearable computing device 100 includes a system-on-chip (SOC) architecture (not shown) coupled with some or all of one or more sensors, low-power island processors, high-power island processors, core processors, encoders, etc.

When operating, the wearable computing device 100 may capture the scene 102 through a camera, an image sensor, etc. Wearable computing device 100 may be worn and operated by user 104. Scene 102 may include augmented reality (AR) content as well as physical content. In some implementations, wearable computing device 100 may be communicatively coupled with another device, such as mobile computing device 106.

In operation, mobile computing device 106 may include a dual stream image sensor 108 (e.g., a dual resolution image sensor). Dual stream image sensor 108 may provide a low-resolution image stream 110 to a low-resolution (e.g., lightweight) image signal processor 112. The dual stream image sensor 108 can simultaneously provide high-resolution image streams 114 to the buffer 116. High-resolution image signal processor 118 may obtain high-resolution image stream 114 from buffer 116 to perform further processing. Low-resolution image stream 110 may be provided to a ML model that identifies one or more ROIs (e.g., text, object, hand, paragraph, etc.), for example, via ROI detector 120. Processor 112 may determine ROI coordinates 122 (e.g., excerpt coordinates) associated with one or more ROIs.

High-resolution image signal processor 112 may receive (or request) ROI coordinates 122 and use ROI coordinates 122 to crop one or more high-resolution images from high-resolution image stream 114 . For example, processor 112 may coordinate ROI associated with low-resolution image stream 110 to determine which frames and regions within high-resolution image stream 114 coincide with the same capture time as the associated low-resolution image frame for which the ROI has been identified. You can search (122). The high-resolution image signal processor 118 can then extract the image to the same extracted ROI coordinates 122. The excerpted image 124 is provided to an ML engine (e.g., ML processor 126) to perform functions including OCR, object recognition, deep paragraphing, hand detection, face detection, symbol detection, etc. Can perform, but is not limited to, computer vision tasks.

In a non-limiting example, wearable computing device 100 may be triggered to begin real-time image processing in response to a request to identify image content associated with optical data captured by dual stream image sensor 108. The request may be from a user wearing wearable computing device 100. Real-time image processing may begin with detecting a first sensor data stream (e.g., low-resolution image stream 110) and a second sensor data stream (e.g., high-resolution image stream 114). Wearable computing device 100 can then identify at least one region of interest (e.g., ROI 128 and ROI 130). In this example, ROI 128 contains an analog clock and ROI 130 contains the text and images of the presentation. Device 100 may analyze the first stream (e.g., low-resolution image stream 110) to determine excerpt coordinates for ROIs 128 and 130.

Wearable computing device 100 extracts high-resolution image stream 114 from buffer 116 to extract one or more image frames of high-resolution image stream 114 corresponding to ROIs 128 and 130 from low-resolution image stream 110. Acquisition may identify ROIs 128 and 130 with identical correspondence (e.g., same capture time, same excerpt coordinates within frame) within the high-resolution image stream 114. One or more extracted image frames may be provided to an onboard ML processor 126 to perform further processing on the extracted images. Excerpting an image to provide a smaller amount of image content (e.g., less visual sensor data, portions of an image, etc.) for image analysis requires less power and latency than traditional ML processors that are asked to analyze the entire image. ML processing can be performed using (latency). Accordingly, the wearable computing device 100 can perform ML processing for tasks such as OCR, object recognition, deep paragraphing, hand detection, face detection, symbol detection, etc. in real time.

For example, wearable computing device 100 may identify an ROI 130, perform image extraction, and use ML processing to indicate that the user's 104 hand is pointing to the image and text in ROI 130. The point can be decided. Processing may perform hand detection and OCR for ROI 130. Similarly, if audio content is available during the presentation, wearable computing device 100 may evaluate ROI 128 relative to the timestamp when user 104 pointed to ROI 130. A timestamp (e.g., 9:05) may be used to retrieve and analyze audio data (e.g., transcription) associated with ROI 130. This audio data may be converted to visual text for presentation on the display of wearable computing device 100. Accordingly, ML processor 126 may select the ROIs, but may also correlate the ROIs as shown in output 132. Output 132 may be presented on a display of wearable computing device 100.

Although multiple processor blocks are shown (e.g., processor 112, processor 118, processor 126, etc.), a single processor may be utilized to perform all processing tasks on wearable computing device 100. . That is, each processor block 112, 118, and 126 may represent a different algorithm or code snippet (piece) that can be executed on a single processor mounted on the wearable computing device 100.

In some implementations, objects and/or ROIs may be extracted from dual stream image sensor 108 rather than by processor 112 or 118. In some implementations, objects and/or ROIs may be extracted from image signal processor 112 or image signal processor 118, as detailed above. In some implementations, the ROI may instead be prepared and transmitted to a mobile device that communicates with wearable computing device 100. The mobile device may extract the ROI and provide the extracted image back to the wearable computing device 100.

2 illustrates a system 200 for performing image processing in a wearable computing device 100 according to implementations described throughout this disclosure. In some implementations, image processing is performed in wearable computing device 100. In some implementations, image processing is shared between one or more devices. For example, image processing may be completed partially at wearable computing device 100 and partially at mobile computing device 202 (e.g., mobile computing device 106) and/or server computing device 204. It can be completed with In some implementations, image processing is performed at wearable computing device 100 and output from such processing is provided to mobile computing device 202 and/or server computing device 204.

In some implementations, wearable computing device 100 includes one or more computing devices, where at least one of the devices is a display device that can be worn on or near a person's skin. In some examples, wearable computing device 100 is or includes one or more wearable computing device components. In some implementations, wearable computing device 100 is a head-mounted display (HMD), such as an optical head-mounted display (OHMD) device, a transparent head-up display (HUD) device, a virtual reality (VR) device, or an augmented reality (AR) device. ) device, or other devices such as goggles or headsets with sensors, displays, and computing capabilities. In some implementations, wearable computing device 100 includes AR glasses (eg, smart glasses). AR glasses represent an optical head-mounted display device designed to resemble a pair of glasses. In some implementations, wearable computing device 100 is or includes a smart watch. In some implementations, wearable computing device 100 is jewelry or includes jewelry. In some implementations, wearable computing device 100 is or includes a ring controller device or other wearable controller. In some implementations, wearable computing device 100 is or includes earbuds/headphones or smart earbuds/headphones.

As shown in FIG. 2 , system 200 includes a wearable computing device 100 communicatively coupled to a mobile computing device 202 and, optionally, a server computing device 204 . In some implementations, communicable coupling may occur over network 206. In some implementations, communicable coupling may occur directly between wearable computing device 100, mobile computing device 202, and/or server computing device 204.

Wearable computing device 100 includes one or more processors 208 that may be formed on a substrate configured to execute one or more machine-executable instructions or software, firmware, or a combination thereof. Processor 208 may be semiconductor-based and may include semiconductor materials capable of performing digital logic. Processor 208 may include a CPU, GPU, and/or DSP, to name a few.

Wearable computing device 100 may also include one or more memory devices 210. Memory device 210 may include any type of storage device that stores information in a format that can be read and/or executed by processor(s) 208. Memory device 210 may store applications and modules that perform certain operations when executed by processor(s) 208 . In some examples, applications and modules may be stored on an external storage device and loaded into memory device 210. Memory 210 may include or access buffer 212, for example, to store and retrieve image content and/or audio content for wearable computing device 100.

Wearable computing device 100 includes sensor system 214. Sensor system 214 includes one or more image sensors 216 configured to detect and/or acquire image data. In some implementations, sensor system 214 includes multiple image sensors 216. As shown, sensor system 214 includes one or more image sensors 216. Image sensor 216 may capture and record images (e.g., pixels, frames, and/or portions of images) and video.

In some implementations, image sensor 216 is a Red Green Blue (RGB) camera. In some examples, image sensor 216 includes a pulsed laser sensor (eg, LiDAR sensor) and/or a depth camera. For example, image sensor 216 may be a camera configured to detect and convey information used to create an image represented by image frame 226. Image sensor 216 is capable of capturing and recording both images and video.

In operation, image sensor 216 is configured to acquire (e.g., capture) image data (e.g., optical sensor data) continuously or periodically while wearable computing device 100 is activated. In some implementations, image sensor 216 is configured to operate as an always-on (always on) sensor. In some implementations, imaging sensor 216 may be activated in response to detection of an object of interest or region of interest.

In some implementations, image sensor 216 is a single sensor configured to function in dual resolution streaming mode. For example, image sensor 216 may operate in a low-resolution streaming mode to capture (e.g., receive, acquire, etc.) multiple images with low image resolution. Sensor 216 may operate simultaneously in a high-resolution streaming mode, capturing (e.g., receiving, acquiring, etc.) multiple images of the same at a high image resolution. Dual mode can capture or retrieve multiple images as a first image stream with a low image resolution and a second image stream with a high image resolution.

In some implementations, wearable computing device 100 may employ image sensor 216 to use scalable levels of image processing to utilize fewer processing and/or power resources. For example, the systems and methods described herein may provide low-level image processing, mid-level image processing, high-level image processing, and/or level image processing by using two or more different types of image signal processors. Any combination of levels and/or levels may be used. In some implementations, the systems and methods described herein may change the resolution of an image when performing image processing techniques.

As used herein, high level processing includes image processing for high resolution image content. For example, computer vision techniques to perform content (e.g., object) recognition may be considered high-level processing. Mid-level image processing includes processing operations that extract scene descriptions from image metadata, which may indicate, for example, the location and/or shape of parts of the scene. Low-level image processing involves extracting descriptions from images while ignoring details of scene objects and observers.

Sensor system 214 may also include an inertial motion unit (IMU) sensor 218. IMU sensor 218 may detect motion, movement, and/or acceleration of the wearable computing device 100. IMU sensors 218 may include various other types of sensors, such as accelerometers, gyroscopes, magnetometers, and other sensors.

In some implementations, sensor system 214 may also include an audio sensor 220 configured to detect audio received by wearable computing device 100. Sensor system 214 may include other types of sensors, such as optical sensors, distance and/or proximity sensors, contact sensors such as capacitive sensors, timers, and/or other sensors, and/or combination(s) of different sensors. there is. Sensor system 214 may be used to obtain information related to the location and/or orientation of wearable computing device 100.

Wearable computing device 100 may also include a low-resolution image signal processor 222 to process images from a low-resolution image stream (e.g., low-resolution image stream 110). The low-resolution image signal processor 222 may also be referred to as a low-power image signal processor. Wearable computing device 100 may also include a high-resolution image signal processor 224 to process images from a high-resolution image stream (e.g., high-resolution image stream 114). High-resolution image signal processor 224 may also be referred to as a high-power image signal processor (with respect to low-resolution image signal processor 222). The image stream may include image frames 226 each having a specific image resolution 228. In operation, the image sensor 216 may be dual-functional, operating in one or both low power, low resolution (LPLR) mode and/or high power, high resolution (HPLR) mode.

Wearable computing device 100 may include a region of interest (ROI) detector 230 configured to detect ROIs 232 and/or objects 234 . Additionally, wearable computing device 100 may include a cropper 236 configured to extract image content (e.g., image frame 226) and ROI 232 according to cropping coordinates 238. there is. In some implementations, wearable computing device 100 includes an encoder 240 configured to compress specific pixel blocks, image content, etc. The encoder 240 may receive image content for compression from the buffer 212, for example.

Low-resolution image signal processor 222 may perform low-power calculations to analyze the image stream generated from sensor 216 to detect objects and/or regions of interest within the image(s). Detection may include hand detection, object detection, paragraph detection, etc. If an object of interest and/or ROI is detected and detection of such region or object meets threshold condition 256, bounding boxes and/or other coordinates associated with the object and/or region may be identified. A threshold condition may indicate a specific high quality level for an object or region in an image frame of an image stream. For example, a threshold condition determines whether an object or area is in focus, still has low blur (e.g., low motion blur), and/or has correct exposure, and/or whether a particular object is detected in the computing device's view. It may relate to any or all of the following:

Wearable computing device 100 may also include one or more antennas 242 configured to communicate with other computing devices via wireless signals. For example, wearable computing device 100 may receive one or more wireless signals and use the wireless signals to communicate with other devices, such as mobile computing device 202 and/or server computing device 204, or within range of antenna 242. You can communicate with other devices within it. The wireless signal may be triggered via a wireless connection, such as a short-range connection (e.g., a Bluetooth connection, a near-field communication (NFC) connection) or an Internet connection (e.g., Wi-Fi, a mobile network).

Wearable computing device 100 includes a display 244 . The display 244 may include an LCD, an LED display, an OLED, an electro-poretic display (EPD), or a micro-projection display employing an LED light source. In some examples, display 244 is projected into the user's field of view. In some examples, in the case of AR glasses, display 244 allows a user wearing AR glasses to view images presented by display 244 as well as information located in the AR glasses field of view behind the projected image. Transparent or translucent displays can be provided.

Wearable computing device 100 also includes a control system 246 that includes various control system devices to facilitate operation of wearable computing device 100. Control system 246 may include processor 208, sensor system 214, and/or processor 248 (e.g., CPU, GPUS, DSP, etc.) operably connected to components of wearable computing device 100. You can use it.

Wearable computing device 100 also includes UI renderer 250. UI renderer 250 may function in conjunction with display 244 to depict user interface objects or other content to a user of wearable computing device 100. For example, UI renderer 250 may receive images captured by wearable computing device 100 to generate and render additional user interface content on display 244 .

Wearable computing device 100 also includes communication module 252. Communication module 252 may enable wearable computing device 100 to communicate to exchange information with other computing devices within range of wearable computing device 100. For example, wearable computing device 100 may be operable with other computing devices to facilitate communication, for example, through a wired connection, wirelessly, for example, via Wi-Fi or Bluetooth, or another type of connection. can be combined

In some implementations, wearable computing device 100 is configured to communicate with server computing device 204 and/or mobile computing device 202 over network 206. Server computing device 204 may represent a number of different devices, such as one or more computing devices that take the form of a standard server, a group of such servers, or a rack server system. In some implementations, server computing device 204 is a single system that shares components such as processors and memory. Network 206 may include the Internet and/or other types of data networks, such as a local area network (LAN), wide area network (WAN), cellular network, satellite network, or other type of data network. Network 206 may also include any number of computing devices (e.g., computers, servers, routers, network switches, etc.) configured to receive and/or transmit data within network 206.

In some implementations, wearable computing device 100 also includes neural network NNs 254 for performing machine learning tasks mounted on wearable computing device 100. Neural networks can be used in conjunction with machine learning models and/or operations to create, modify, predict, and/or detect specific image content. In some implementations, image processing performed on sensor data acquired by sensor(s) 216 is referred to as a machine learning (ML) inference operation. An inference operation may refer to an image processing operation, step, or substep associated with an ML model that makes (or leads to) one or more predictions. Certain types of image processing performed by wearable computing device 100 may use ML models for prediction. For example, machine learning can use statistical algorithms that learn from existing data to make decisions about new data, called inference. In other words, inference refers to the process of taking an already trained model and making predictions using that trained model. Some examples of inference include image recognition (e.g., OCR text recognition, face recognition, face, body or object tracking, etc.) and/or perception (e.g., always-on audio detection, voice input request audio detection, etc.) Includes.

In some implementations, the ML model includes one or more NNs 254. NN 254 transforms the input received by the input layer through a series of hidden layers and produces output through the output layer. Each layer consists of a subset of the node set. Nodes in the hidden layer are fully connected to all nodes in the previous layer and provide individual(s) output to all nodes in the next layer. Nodes in a single layer function independently of each other (i.e., do not share connections). Nodes in the output layer provide converted input to the requesting process. In some implementations, NN 254 utilized by wearable computing device 100 is a convolutional neural network that is a fully disconnected NN.

In general, the wearable computing device 100 may use a NN and a classifier (not shown) to generate train data and perform ML operations. For example, processor 208 may be configured to execute a classifier that detects, for example, whether ROI 232 is included within image frame 226 captured by image sensor 216. ROI 232 may also be referred to as an object of interest. A classifier may contain an ML model or be defined by an ML model. The ML model may define a number of parameters that are used by the ML model to make predictions (e.g., whether ROI 232 is contained within image frame 226). ML models can be relatively small and configured to function on ROIs rather than the entire image. Accordingly, the classifier can be configured to function to save power and latency through relatively small ML models.

In some implementations, wearable computing device 100 may use an ROI classifier that executes an ML model to compute an ROI dataset (e.g., ROI 232). In some examples, ROI 232 represents an example of object location data and/or bounding box dataset. ROI 232 may be data defining where the ROI is located within image frame(s) 226.

In operation, wearable computing device 100 may receive or capture image content (e.g., multiple images/frames) from a dual resolution image sensor (e.g., image sensor 216). Image content may be a low-resolution image transmitted to the low-resolution image signal processor 222 while a high-resolution image among a plurality of images/frames is transmitted to the high-resolution image signal processor 224 at the same time. The low-resolution images can then be sent to an on-board machine learning model, such as a neural network 254. NN 254 may utilize ROI detector 230 to identify one or more ROIs (e.g., text, object, image, frame, pixel, etc.). The identified ROIs may each be associated with the determined excerpt coordinates 238. The wearable computing device 100 may transmit these excerpt coordinates 238 to the high-resolution image signal processor 224 to cause extraction of a high-resolution image based on the excerpt coordinates of the low-resolution image corresponding to the detected ROI. The extracted image is smaller than the original high-resolution image, and therefore wearable computing device 100 can perform machine learning activities (e.g., OCR, object detection, etc.) on the extracted image using NN 254. .

In some implementations, wearable computing device 100 may perform other vision-based algorithms on the extracted area. For example, object recognition may be performed on an excerpted area to identify and utilize information related to the recognized object(s). For example, specific products, landmarks, storefronts, plants, animals, and/or various other common objects and/or object classifiers may be recognized and used as input to determine additional information for the user. In some implementations, barcode recognition and reading may be performed on the excerpted areas to identify and provide additional information about wearable computing device 100 that may be displayed, presented audibly, or otherwise output. . In some implementations, facial recognition may be performed on the excerpted areas to place AR and/or VR content on the recognized face, capture specific features of the recognized face, and calculate specific features of the face. In some implementations, feature tracking (e.g., feature point detection) may be performed on the extracted region to track object movement and/or user movement.

3A and 3B show various diagrams of examples of AR wearable computing devices according to implementations described throughout this disclosure. 3A is a front view of an example of a wearable computing device according to implementations described throughout this disclosure. In this example, the wearable computing device is a pair of AR glasses 300A (e.g., wearable computing device 100 of FIG. 1). In general, AR glasses 300A may include any or all components of system 200. AR glasses 300A may also be referred to as smart glasses, which represent optical head-mounted display devices designed in the form of glasses. For example, smart glasses are glasses that add information (e.g., projected displays) along with what the wearer sees through the glasses.

Although AR glasses 300A are shown as the wearable computing device described herein, other types of wearable computing devices are also possible. For example, wearable computing devices can be head-mounted display (HMD) devices such as optical head-mounted display (OHMD) devices, transparent heads-up displays (HUD), augmented reality (AR) devices, or devices with sensors, displays, and computing capabilities. Can include, but is not limited to, any battery-powered device, including other devices such as goggles or headsets. In some examples, wearable computing device 300A may be a watch, mobile device, jewelry, ring controller, or other wearable controller.

As shown in FIG. 3A, AR glasses 300A have a frame 302 and a display device 304 coupled to the frame 302 (or a glass portion of the frame 302). AR glasses 300A also include an audio output device 306, a lighting device 308, a sensing system 310, a control system 312, at least one processor 314, and a camera 316.

Display device 304 may include a see-through near-eye display, such as using birdbath or waveguide optics. For example, this optical design could project light from a display source onto a piece of teleprompter glass that acts as a beam splitter positioned at a 45-degree angle. The beam splitter can allow reflection and transmission values such that light from the display source is partially reflected and the remaining light is transmitted. This optical design allows the user to see all physical (real-world) items in the world next to the digital images (e.g., UI elements, virtual content, etc.) generated by the display. In some implementations, waveguide optics may be used to depict content on the display device 304 of AR glasses 300A.

Audio output devices 306 (e.g., one or more speakers) may be coupled to frame 302. Sensing system 310 may include various sensing devices and a control system 312 including various control system devices to facilitate operation of AR glasses 300A. Control system 312 may include a processor 314 operably coupled to components of control system 312 .

Camera 316 may capture still images and/or video. In some implementations, camera 316 may be a depth camera that can collect data related to the distance of external objects from camera 316. In some implementations, camera 316 may be a point tracking camera capable of detecting and tracking, for example, an optical marker on an input device or one or more optical markers on an external device, such as a finger on a screen. In some implementations, AR glasses 300A may be configured with an illumination device 308 that can selectively operate, for example, with camera 316 to detect objects (e.g., virtual and physical) in the field of view of camera 316. ) may include. Illumination device 308 may optionally operate with camera 316, for example, to detect objects in the field of view of camera 316.

AR glasses 300A may include a communication module (e.g., communication module 252) that communicates with processor 314 and control system 312. The communication module may provide communication between devices housed within the AR glasses 300A as well as communication with external devices, such as controllers, mobile devices, servers, and/or other computing devices. The communication module may enable AR glasses 300A to communicate and exchange information with other computing devices and authenticate other devices within range of AR glasses 300A or other identifiable elements in the environment. For example, AR glasses 300A may be operably connected to another computing device to facilitate communication, for example, via a wired connection, a wireless connection (e.g., Wi-Fi or Bluetooth, or another type of connection). You can.

3B is a back view 300B of AR glasses 300A according to an implementation described throughout this disclosure. AR glasses 300B may be an example of the wearable computing device 100 of FIG. 1 . AR glasses 300B are glasses that add information (e.g., project a display 320) along with what the wearer sees through the glasses. In some implementations, instead of projecting information, display 320 is an in-lens micro display. In some implementations, AR glasses 300B (e.g., eyeglasses or spectacles) include a frame 302 that utilizes a bridge 324 over the nose to secure the lenses in front of a person's eyes. A vision aid comprising a lens 322 (e.g., a glass or hard plastic lens) mounted on the eye and bows 326 (e.g., a temple or bridge portion) that rest over the ear. aids).

Figures 4A-4C depict example flow diagrams 400A, 400B, and 400C for performing image processing tasks in a wearable computing device according to implementations described throughout this disclosure. For example, flow diagrams 400A-400C may be performed by wearable computing device 100 using one or more image sensors 216. In the example of flow diagrams 400A-400C, sensor 402 operates in a dual stream mode in which sensor 402 processes both an incoming (or captured) low-resolution image stream and an incoming (or captured) high-resolution image stream continuously and simultaneously. It can function as

FIG. 4A is a flow diagram 400A illustrating the use of an image sensor 402 installed in a wearable computing device and configured to capture or receive a stream of image data. Sensor 402 can capture and output two streams of image content. For example, sensor 402 may acquire multiple high-resolution image frames captured with a full field of view (e.g., the full resolution of camera/sensor 402). Multiple high-resolution image frames may be stored in buffer 404. In general, high-output image signal processor 406 may optionally be used to perform analysis of high-resolution image frames.

Additionally, sensor 402 may acquire (simultaneously) multiple low-resolution image frames captured with the entire field of view. Low-resolution image frames may be provided to a low-power image signal processor 408. Processor 408 processes the low-resolution image frames to produce image streams 410 (e.g., one or more images) that are debayered, color-corrected, and shade-corrected images (e.g., YUV format image streams). ) is created.

The low-power image signal processor 408 performs low-power compute 412 to analyze the generated image stream 410 to detect objects and/or regions of interest 414 within the image(s) 410. You can. Detection may include hand detection, object detection, paragraph detection, etc. If an object of interest and/or ROI is detected and detection of such region or object meets a threshold condition, the bounding box of that object and/or region is identified. A threshold condition may indicate a specific high quality level for an object or region in an image frame of an image stream. For example, a threshold condition may relate to any or all of the following: determining whether an object or area is in focus, still has low blur, and/or has correct exposure. The bounding box can represent the extracted coordinates of each of the four corners of the box. In some implementations, the bounding box may not be box-shaped, but instead may be triangular, circular, oval, or other shape.

FIG. 4B is a flow diagram 400B illustrating the use of an image sensor 402 installed in a wearable computing device and configured to capture or receive a stream of image data. Similar to Figure 4A, sensor 402 can capture and output two streams of image content: a plurality of high-resolution image frames and a plurality of low-resolution image frames. Multiple high-resolution image frames may be provided to buffer 404.

Low-resolution image frames may be provided to a low-power image signal processor 408. Processor 408 processes the low-resolution image frames to generate image streams 410 (e.g., one or more images) that are debayered, color-corrected, and shade-corrected images (e.g., YUV format image streams). do.

The low-power image signal processor 408 may perform low-power calculations 412 to analyze the generated image stream 410 to detect objects and/or regions of interest 414 within the image(s) 410 . If an object of interest and/or ROI is detected and detection of such region or object meets a threshold condition, the bounding box of that object and/or region is identified. A threshold condition may indicate a specific high quality level for an object or region in an image frame of an image stream. For example, a threshold condition may relate to any or all of the following: determining whether an object or area is in focus, still has low blur, and/or has correct exposure. The bounding box may represent a region of interest or excerpt coordinates representing an object 414. Once the object or region of interest 414 is determined and the threshold condition(s) are met, the high power image signal processor 406 outputs a high-resolution image frame from the buffer 404 (e.g., a low-resolution image frame with the object or region of interest identified). Retrieves (or receives) a raw frame (that matches the same capture time as the image frame).

For example, the excerpt coordinates and retrieved high-resolution image frames may be provided as input to the high-power image signal processor 406 (e.g., arrow 416). The high-power image signal processor 406 may output a processed (e.g., converted, color-corrected, shade-corrected) image 418 extracted for objects of interest and/or regions of interest determined from low-resolution image frames. there is. This extracted full resolution image 418 may be provided to additional onboard or offboard devices for further processing.

In some implementations, instead of multiple image frames, a single high-resolution image may be processed through a high-power image signal processor to save computational and/or device power. Additional computation and/or device power may also result from additional downstream processing because such downstream processing may operate on excerpted images 418 rather than the full high-resolution image.

In general, low-power image signal processor 408 may not continue processing the stream of low-resolution image frames after objects and/or regions of interest have been identified. In some implementations, low-power image signal processor 408 may be in standby mode to await additional quality metrics for previously provided/identified image frames. In some implementations, sensor 402 may continue to trigger performance of perceptual (recognition) functions (i.e., ML processing) on low-resolution image streams, which may further affect output to downstream functions.

FIG. 4C is a flow diagram 400C illustrating an example of additional downstream processing. As shown, excerpted image 418 represents a region of interest. The excerpted image 418 is a high-resolution image that can be sent to the encoder (e.g., compression) block without buffering the image 418 (or images) in memory. The compressed output of the encoder is buffered in buffer 422 and encrypted by encryption block 424. Encrypted image 418 may be buffered in buffer 426 and transmitted to companion mobile computing device 202, for example, via Wi-Fi (e.g., network 206). Mobile computing device 202 may be configured to include computing power, memory, and battery capacity to support more complex processing than that supported in wearable computing device 100, for example. Because further downstream processing utilizes excerpted versions of the image content rather than the entire image, downstream processing also benefits from global savings in computation, memory, and battery. For example, a larger amount of power is used to transmit additional packets over Wi-Fi for larger images rather than just the excerpted image 418.

5A and 5B illustrate example flow diagrams 500A and 500B for performing image processing tasks in a wearable computing device according to implementations described throughout this disclosure. For example, flowcharts 500A and 500B may be executed by wearable computing device 100 using one or more image sensors 216. In the example of flow diagrams 500A and 500B, sensor 402 is in a dual stream mode in which sensor 402 processes both an incoming (or captured) low-resolution image stream and an incoming (or captured) high-resolution image stream continuously and simultaneously. can function in

FIG. 5A is a flow diagram 500A illustrating the use of an image sensor 402 installed in, for example, a wearable computing device 100 and configured to capture or receive a stream of image data. Sensor 402 can capture and output two streams of image content. For example, sensor 402 may acquire multiple high-resolution image frames captured with a full field of view (e.g., the full resolution of camera/sensor 402). In this example, multiple high-resolution image frames may be processed by high-power image signal processor 406. Processed high-resolution image frames may be encoded (e.g., compressed) by encoder 502 and stored in buffer 404. This offers the advantage of saving buffer memory and memory bandwidth for storing high-resolution image frames before compression. Instead, this example compresses and stores a compressed version of the high-resolution image frame instead of the original, uncompressed image frame.

Additionally, sensor 402 may acquire (simultaneously) multiple low-resolution image frames captured with the entire field of view. Low-resolution image frames may be provided to a low-power image signal processor 408. Processor 408 processes the low-resolution image frames to generate image streams 410 (e.g., one or more images) that are debayered, color-corrected, and shade-corrected images (e.g., YUV format image streams). do.

The low-power image signal processor 408 may perform low-power calculations 412 to analyze the generated image stream 410 to detect objects and/or regions of interest 414 within the image(s) 410 . Detection may include hand detection, object detection, paragraph detection, etc. If an object of interest and/or ROI is detected and detection of such region or object meets a threshold condition, the bounding box of that object and/or region is identified. A threshold condition may indicate a specific high quality level for an object or region in an image frame of an image stream. For example, a threshold condition may relate to any or all of the following: determining whether an object or area is in focus, still has low blur, and/or has correct exposure. The bounding box may represent, for example, the extracted coordinates of each of the four corners of the box.

FIG. 5B is a flow diagram 500B illustrating a process for generating partial image content from, for example, one or more image frames 410. Similar to Figure 5A, sensor 402 may capture and output two streams of image content including at least one high-resolution image frame and at least one low-resolution image frame. In this example, at least one high-resolution image frame may be processed by high-power image signal processor 406. Processed high-resolution image frames may be encoded (e.g., compressed) by encoder 502 and stored in buffer 404.

Additionally, sensor 402 may (simultaneously) acquire at least one low-resolution image frame that may be provided to low-power image signal processor 408. Processor 408 processes at least one low-resolution image frame to generate at least one image 410 that is a debayered, color-corrected, shadow-corrected image (e.g., a YUV format image).

Low-power image signal processor 408 may perform low-power calculations 412 to analyze at least one image 410 to, for example, detect objects and/or regions of interest 414 within at least one image 410. there is. Detection may include hand detection, object detection, paragraph detection, etc. If an object of interest and/or ROI is detected and detection of such region or object meets a threshold condition, the bounding box of that object and/or region is identified. A threshold condition may indicate a specific high quality level for an object or region in an image frame of an image stream. For example, a threshold condition may relate to any or all of the following: determining whether an object or area is in focus, still has low blur, and/or has correct exposure. The bounding box may represent, for example, the extracted coordinates of each of the four corners of the box.

Once the ROI or object of interest is determined and the threshold conditions are met, the coordinates to extract (e.g., extract coordinates 238) may be transmitted to or obtained by an extractor (e.g., extractor 236). . Flow diagram 500B may then retrieve at least one high-resolution image from buffer 404. The retrieved high-resolution image is then extracted for the region or object of interest according to the extracted coordinates determined for the low-resolution image (504). High-resolution images typically have the same capture time as lower-resolution images with the area or object of interest identified. The output of the excerpt may include an excerpt 506 of a high-resolution image. Additional downstream processing may be performed on-board or off-board the wearable computing device 100.

Unlike traditional image analysis, which occurs on devices with higher computational and power resources, the system described herein avoids continuous processing of high-resolution images. Additionally, because scaling is not performed by the wearable computing device 100, the system described herein can avoid computational resources for scaling. Additionally, typical image analysis systems that utilize both low-resolution and high-resolution image streams use separate buffered storage for each stream. The systems described herein utilize buffers for high-resolution images and avoid the use of buffers for low-resolution images, which provides the advantage of less memory traffic and/or memory.

6A-6C depict example flow diagrams 600A, 600B, and 600C for performing image processing tasks in wearable computing device 100 according to implementations described throughout this disclosure. In the examples of flow diagrams 600A-600C, sensor 402 processes a single mode in which sensor 402 analyzes an incoming (or captured) low-resolution image stream or an incoming (or captured) high-resolution image stream. It can function as Sensor 402 may then be triggered to function in dual stream mode to process both the incoming (or captured) low resolution image stream and the incoming (or captured) high resolution image stream continuously and simultaneously.

FIG. 6A is a flow diagram 600A illustrating the use of an image sensor 402 installed, for example, in wearable computing device 100. Sensor 402 may be configured to switch between dual mode processing and single mode processing. For example, sensor 402 may function in a mode that first streams/captures a low-resolution image stream (e.g., multiple consecutive image frames). This mode may save computational power and battery power for the wearable computing device 100 because high-resolution image analysis is avoided in the first capture and/or image retrieval step. Buffer 404 and high power image signal processor 406 may be available, but are disabled in this example.

In operation, low-resolution image frames may be provided to low-power image signal processor 408. Processor 408 processes the low-resolution image frames to generate image streams 410 (e.g., one or more images) that are debayered, color-corrected, and shade-corrected images (e.g., YUV format image streams). do. The low-power image signal processor 408 may perform low-power calculations 412 to analyze the generated image stream 410 to detect objects and/or regions of interest 414 within the image(s) 410 . Detection may include hand detection, object detection, paragraph detection, etc.

If an object of interest and/or ROI is detected and detection of such region or object meets a threshold condition, the bounding box of that object and/or region is identified. A threshold condition may indicate a specific high quality level for an object or region in an image frame of an image stream. For example, a threshold condition may relate to any or all of the following: determining whether an object or area is in focus, still has low blur, and/or has correct exposure. The bounding box may represent, for example, the extracted coordinates of each of the four corners of the box. Identification and extraction coordinates of a region of interest or object may be a trigger that allows sensor 402 to begin operating in dual stream mode.

In some implementations, the trigger to initiate dual stream mode may occur when several, but not all, threshold conditions are met. For example, the wearable computing device 100 may recognize an object of interest that satisfies a threshold condition for camera focus and a threshold condition for exposure to the camera. However, because the object may be moving (e.g., there may be motion blur), wearable computing device 100 uses processor 118 to prepare the high-resolution image capture when the final conditions are met. can trigger the use of . This can be useful when there is a delay of several frames in switching to dual stream mode. That is, the period in which the dual stream mode is executed may be a transition period that enters when almost all threshold conditions are met and the final threshold condition for detection and decision is expected to occur in the upcoming video frame.

FIG. 6B is a flow diagram 600B illustrating the use of an image sensor 402 installed in a wearable computing device 100 where the sensor 402 is switched to dual stream mode. A trigger for switching to dual stream mode may include detection of one or more objects or regions of interest. Dual stream mode triggers sensor 402 to begin capturing (or acquiring) high-resolution image frames in addition to the low-resolution image frames that are already streaming. In some implementations, multiple image frames of a high-resolution image stream may be used to configure automatic gain measurements, exposure settings, and focus settings. Once configuration for the high-resolution image stream is complete, the streams can be synchronized and the low-resolution image frames can continue through the same flow as in Figure 6A, as shown by arrow 602. However, sensor 402 may begin buffering (using buffer 404) high-resolution image frames using the processor. The buffer 404 may be smaller than the buffer used when the sensor operates in continuous dual stream mode from the beginning of video streaming.

FIG. 6C is a flow diagram 600C illustrating the use of an image sensor 402 installed in a wearable computing device 100 and configured to capture or receive a stream of image data after switching from single-mode streaming to dual-mode streaming. In operation, sensor 402 may obtain multiple high-resolution image frames captured with a full field of view (e.g., the full resolution of camera/sensor 402). Multiple high-resolution image frames may be stored in buffer 404.

Low-resolution image frames may continue to be captured and/or acquired by sensor 402 and provided to low-power image signal processor 408. Processor 408 processes the low-resolution image frames to generate image streams 410 (e.g., one or more images) that are debayered, color-corrected, and shade-corrected images (e.g., YUV format image streams). do. The low-power image signal processor 408 may perform low-power calculations 412 to analyze the generated image stream 410 to detect objects and/or regions of interest 414 within the image(s) 410 . If an object of interest and/or ROI is detected and detection of such region or object meets a threshold condition, the bounding box of the object and/or region is identified.

A threshold condition may indicate a specific high quality level for an object or region in an image frame of an image stream. For example, a threshold condition may relate to any or all of the following: determining whether an object or area is in focus, still has low blur, and/or has correct exposure. The bounding box may represent a region of interest or excerpt coordinates representing an object 414. Once the object or region of interest 414 is determined and the threshold condition(s) are met, the high-power image signal processor 406 extracts a high-resolution image frame from buffer 404 (e.g., a low-resolution image with the object or region of interest identified). Retrieves (or receives) a raw frame (that matches the same capture time as the frame).

The excerpt coordinates and retrieved high-resolution image frames may be provided as input to the high-power image signal processor 406 (e.g., arrow 604). The high-power image signal processor 406 may output a processed (e.g., converted, color-corrected, shade-corrected) image 606 extracted for the determined object and/or region of interest from the low-resolution image frame. . This excerpted full resolution image 606 may be provided to additional onboard or offboard devices for further processing.

7A-7C depict example flow diagrams 700A, 700B, and 700C for performing image processing tasks in wearable computing device 100 according to implementations described throughout this disclosure. In the example of flowcharts 700A-700C, sensor 402 may function in a first mode and then be triggered to function in a second mode. For example, sensor 402 may be triggered to begin streaming and/or capturing content in a low-resolution streaming mode in which low-resolution images are captured and utilized. Upon detecting one or more objects and/or areas of interest, sensor 402 may switch to a high-resolution streaming mode in which high-resolution images are utilized.

FIG. 7A is a flow diagram 700A illustrating the use of an image sensor 402 installed, for example, in wearable computing device 100. In this example, sensor 402 may function in a low-resolution streaming mode, first streaming and/or capturing (or acquiring) a low-resolution image stream (e.g., a plurality of consecutive image frames).

In operation, low-resolution image frames may be provided to low-power image signal processor 408. Processor 408 processes the low-resolution image frames to generate image streams 410 (e.g., one or more images) that are debayered, color-corrected, and shade-corrected images (e.g., YUV format image streams). do. The low-power image signal processor 408 may perform low-power calculations 412 to analyze the generated image stream 410 to detect objects and/or regions of interest 414 within the image(s) 410 . Detection may include hand detection, object detection, paragraph detection, etc.

If an object of interest and/or ROI is detected and detection of such region or object meets a threshold condition, the bounding box of the object and/or region is identified. A threshold condition may indicate a specific high quality level for an object or region in an image frame of an image stream. For example, a threshold condition may relate to any or all of the following: determining whether an object or area is in focus, still has low blur, and/or has correct exposure. The bounding box may represent, for example, the extracted coordinates of each of the four corners of the box. Identification and extraction coordinates of a region of interest or object may be a trigger that allows sensor 402 to switch from a low-resolution streaming mode to a high-resolution streaming mode.

FIG. 7B is a flow diagram 700B illustrating the use of an image sensor 402 installed on a wearable computing device 100 to trigger the sensor 402 to transition from a low-resolution streaming mode to a high-resolution streaming mode. Triggers to switch these modes may include detection of one or more objects or regions of interest. The transition from low-resolution streaming mode to high-resolution streaming mode triggers sensor 402 to stop capturing low-resolution image frames and begin capturing (or acquiring) high-resolution image frames.

In this example, at least one high-resolution image frame may be processed by high-power image signal processor 406. One or more of the processed high-resolution image frames may be provided to a scaler 702 to be scaled (e.g., reduced resolution) to produce a lower-resolution image 704 (or image stream). High-power image signal processor 406 may perform low-power calculations 412 on image 704 (or image stream). Low-power computation may include analyzing the at least one image 704 to, for example, detect objects and/or regions of interest 706 within the at least one image 704. Detection may include hand detection, object detection, paragraph detection, etc. If an object of interest and/or ROI is detected and detection of such region or object meets a threshold condition, the bounding box of the object and/or region is identified. A threshold condition may indicate a specific high quality level for an object or region in an image frame of an image stream. For example, a threshold condition may relate to any or all of the following: determining whether an object or area is in focus, still has low blur, and/or has correct exposure. The bounding box may represent, for example, the extracted coordinates of each of the four corners of the box. High-power image signal processor 406 may also buffer the entire high-resolution image 708 in buffer 710. Image 708 may be used for operations that would benefit from a high-resolution image.

FIG. 7C is a flow diagram 700C illustrating the use of the image sensor 402 installed on the wearable computing device 100. In this example, sensor 402 transitions to high-resolution streaming mode in response to determining the excerpt coordinates of ROI 706. For example, once an ROI 706 (or object of interest) is determined and a threshold condition is met, the excerpt coordinates (e.g., excerpt coordinates 238) are sent to an extractor (e.g., extractor 236). It can be transmitted or obtained by an extractor. High-resolution images 708 may be retrieved from buffer 710. The retrieved high-resolution image 708 is then extracted 712 as a region or object of interest according to the extracted coordinates (e.g., ROI 706) determined for the low-resolution image 704. The high-resolution image 708 generally has the same capture time as the lower-resolution image 704 in which the region or object of interest 706 was identified. The output of the excerpt may include an excerpt 714 of the high-resolution image. Additional downstream processing may be performed on-board or off-board the wearable computing device 100.

In some implementations, sensor 402 may determine when to perform a fast dynamic transition between using a low-resolution streaming mode and a high-resolution streaming mode. For example, sensor 402 may determine when to provide sensor data (e.g., image data, pixels, optical data, etc.) to high power image signal processor 406 or low power image signal processor 408. This decision may be based on detected events occurring in connection with the image stream, for example, for the purpose of reducing and/or minimizing power usage and system delay. Examples of detected events may include detecting one or more of the following: an object, an ROI, moving or stopping moving, a lighting change, a new user, another user, another object, etc.

In this example, wearable computing device 100 may be triggered to operate with content having a low resolution and switch to content having a high resolution. Because both low and high resolution image streams can be used, the wearable computing device 100 can select from the two streams without retrieving content from buffered data. This saves computational conversion costs, memory costs, and content retrieval costs from memory.

8A and 8B illustrate example flow diagrams 800A and 800B for emulating a dual resolution image sensor with a single image sensor to perform image processing tasks according to implementations described throughout this disclosure. In this example, a dual stream output sensor (e.g., sensor 402) may be configured to output a first image stream with a low resolution, wide field of view to represent the scene camera 802. Likewise, the sensor may be configured to output a second stream with a high resolution, narrow field of view to represent the detail camera 804.

FIG. 8A is a flow diagram 800A illustrating the use of image sensors (e.g., scene camera 802 and detail camera 804) installed in a wearable computing device and configured to capture or receive a stream of image data. Detail camera 804 may acquire (e.g., capture) and output a high-resolution image stream (e.g., multiple image frames). Multiple high-resolution image frames may be stored in buffer 806. In general, a high-output image signal processor 808 is optionally available to perform analysis of high-resolution image frames.

Scene camera 802 may acquire (e.g., capture) and output (simultaneously) multiple low-resolution image frames. Low-resolution image frames may be provided to the low-power image signal processor 810. Processor 810 processes the low-resolution image frames to generate image streams 812 (e.g., one or more images) that are debayered, color-corrected, and shade-corrected images (e.g., YUV format image streams). do.

The low-power image signal processor 810 may perform low-power calculations 814 to analyze the generated image stream 812 to detect objects and/or regions of interest 816 within the image(s) 812. Detection may include hand detection, object detection, paragraph detection, etc. If an object of interest and/or ROI is detected and detection of such region or object meets a threshold condition, the bounding box of the object and/or region is identified. The bounding box can represent the extracted coordinates of each of the four corners of the box. A threshold condition may indicate, for example, a trigger event in the scene camera view that causes a switch to the detail camera for a single capture (or simple capture burst) based on the detected trigger event. A threshold condition (e.g., a trigger event) may occur, for example, when a condition and/or item is recognized in a view of an image.

FIG. 8B is a flow diagram 800B illustrating the use of image sensors (e.g., scene camera 802 and detail camera 804) installed in a wearable computing device and configured to capture or receive a stream of image data. Similar to Figure 8A, scene camera 802 captures and outputs one or more low-resolution images (e.g., image frames) and processes the images with low-power image signal processor 810 and/or low-power computing 814. By performing calculations, a low-resolution image 812 can be generated and the object and/or region of interest 816 can be determined.

Once the object and/or area of interest 816 is identified and the threshold conditions are determined to be met, a high-resolution image may be retrieved from the buffer 806. For example, the high-power image signal processor 808 may use the low-resolution image 812 and the excerpted coordinates of the extracted ROI 816 to capture the high-resolution, narrower image signal processor 808 stored in the buffer 806 and captured simultaneously with the image 812. The same ROI 816 can be identified within the field of view image.

For example, excerpt coordinates (e.g., arrows 818) and retrieved high-resolution image frames may be provided as input to a high-power image signal processor 808. High-power image signal processor 808 may output a processed (e.g., converted, color-corrected, shade-corrected) image 820 extracted for regions of interest and/or objects determined from low-resolution image frames. . This extracted full resolution image 820 may be provided to additional onboard or offboard devices for further processing.

In some implementations, rather than multiple image frames, a single high-resolution image may be processed through a high-power image signal processor to save computational and/or device power. Additional computation and/or device power may also result from additional downstream processing because such downstream processing may operate on excerpted images 820 rather than the full high-resolution image.

Figures 4A-8B illustrate flow diagrams utilizing a single sensor to provide the functionality of dual sensors (e.g., a scene camera and a detail camera). In some implementations, the output of a single sensor may be configured to operate in dual mode without a separate sensor because that sensor may output at least two streams at different resolutions and/or different fields of view. For example, the first stream may be configured to output a wide field of view over a low-resolution stream of images to represent a scene camera. Similarly, the second stream may be configured to output a narrow field of view over a high-resolution stream of images to represent camera details.

FIG. 9 is a flow diagram illustrating an example of a process 900 for performing an image processing task at a computing device in accordance with implementations described throughout this disclosure. In some implementations, the computing device is a battery-powered, wearable computing device. In some implementations, the computing device is a battery-powered, non-wearable computing device.

Process 900 includes image processing of a computing device having at least one processing device, a speaker, an optional display function, and a memory that stores instructions that, when executed, cause the processing device to perform a plurality of operations and computer implemented steps described in the claims. You can use the system. In general, wearable computing device 100, system 200 and/or 1000 may be used to describe and perform process 900. A combination of wearable computing device 100 and systems 200 and/or 1000 may represent a single system in some implementations. Generally, process 900 utilizes the systems and algorithms described herein to detect image data to be extracted (cropped) to reduce the amount of data used to perform image processing in wearable computing device 100. Identify one or more regions of interest in real time (e.g. at capture time).

At block 902, process 900 may include waiting for receipt of image content (or a request for image content identification). For example, image sensor 216 on wearable computing device 100 may receive, capture, or detect image content. In response to receiving a request to cause content or sensor 216 to identify image content associated with optical data captured by the sensor, sensor 216 and/or processor 208, 222 and/or 224 may execute a block ( At 904), a first sensor data stream having a first image resolution may be detected, and the first sensor data stream may be based on optical data. For example, image sensor 216 may detect a low-resolution data stream based on optical data received at image sensor 216 and provide the low-resolution image data stream to low-resolution image signal processor 222. Additionally, at block 906, sensor 216 and/or processor 208, 222, and/or 224 generate a second sensor data stream wherein the second sensor data stream also has a second image resolution based on optical data. It can be detected. For example, image sensor 216 may detect a high-resolution data stream based on optical data received at image sensor 216 and provide the high-resolution data stream to high-resolution image signal processor 224.

In operation, the first sensor data stream and the second data stream are simultaneously and/or cross-referenced to obtain a low-resolution image from the first sensor data stream that matches the timestamp of the high-resolution image from the second sensor data stream. Capture and/or reception can be achieved with a possible capture time and correlation.

At block 908 , process 900 includes identifying, by processing circuitry of wearable computing device 100 , at least one region of interest in the first sensor data stream. For example, one or more ROIs 232 may be identified within image frame 226 acquired by image sensor 216. Image frame 226 may have been analyzed by low-resolution image signal processor 222 and/or ROI detector 230. ROI may be defined by pixels, edge detection coordinates, object shape or detection, or other area or object identification metrics (metrics). In some implementations, the ROI may be an object or other image content in the image stream.

In some implementations, processing circuitry includes a first image processor (e.g., low-resolution processor 222) configured to perform image signal processing on a first sensor data stream (e.g., low-resolution stream) and a second sensor data stream. It may include a second image processor (e.g., high-resolution image signal processor 224) configured to perform image signal processing on (e.g., high-resolution stream). Typically, the first image resolution of the first sensor data stream is lower than the second image resolution of the second data sensor stream.

At block 910, the process 900 includes determining, by the processing circuitry, excerpt coordinates 238 that define a first plurality of pixels in at least one region of interest in the first sensor data stream. For example, the low-resolution image signal processor 222 may use an image frame 226, an analyzed resolution 228, and an ROI detector to identify excerpt coordinates that define an object 234 or a region of interest (ROI) 232. 230) can be used.

At block 912, process 900 includes generating, by processing circuitry, an excerpted image representing at least one region of interest. Generating the excerpted image may include the extractor 236, at block 914, using the excerpt coordinates 238 to define a first plurality of pixels of at least one region of interest in the first sensor data stream. Triggering identification of a second plurality of pixels in the data stream and extracting the second sensor data stream with the second plurality of pixels at block 916. The resulting excerpted image may be a high-resolution version of the originally identified region of interest or object.

In some implementations, generating an excerpted image representing at least one region of interest (or object) is performed in response to detecting that at least one region of interest meets threshold condition 256. In general, threshold condition 256 may indicate a particular high quality level for an object or region in an image frame of an image stream. In some implementations, threshold condition 256 may include detecting whether a plurality of second pixels of the second sensor data stream have low blur. In some implementations, threshold condition 256 may include detecting whether a plurality of second pixels have a particular image exposure measurement. In some implementations, threshold condition 256 may include detecting whether the plurality of second pixels are in focus.

For example, a threshold condition may relate to any or all of the following: determining whether an object or area is in focus, still has minimal blur, and/or has correct exposure. The bounding box may represent a region of interest or excerpt coordinates representing an object 414. Once an object or region of interest (e.g., a second plurality of pixels) is determined and the threshold condition(s) are met, the high-power image signal processor may output a high-resolution image frame (e.g., a low-resolution image with the object or region of interest identified). You can retrieve (or receive) a raw frame (that matches the same capture time as the frame).

Extracted high-resolution images can be used for further image analysis. For example, processing circuitry 224 may perform optical character resolution (OCR) on the excerpted image to generate a machine-readable version of the excerpted image (e.g., output 132). It may include: Processing circuitry 224 then performs a search query using a machine-readable version of the excerpted image (e.g., image 418) and triggers it for display on display 244 of wearable computing device 100. A plurality of search results (e.g., displayed in output 132) may be generated. In some implementations, instead of performing a search query, OCR may be performed on the extracted images to generate output 132, which may be displayed to a user via wearable computing device 100, and the user may be read aloud to a user, and/or may be provided as output for consumption by a user. For example, audio output of the performed OCR may be provided from a speaker of the wearable computing device 100.

In some implementations, the second sensor data stream is stored in memory of wearable computing device 100. For example, a high-resolution image stream may be stored in buffer 404 for later access to image frames and metadata associated with the high-resolution image stream. In response to identifying at least one area of interest (or object) in the first sensor data stream, processing circuitry 224 may retrieve the corresponding at least one area of interest (or object) in the second sensor data stored in memory. You can. For example, buffered data may be accessed to obtain a high-resolution version of image content identified in a region (or object) of interest in a low-resolution image stream. Additionally, wearable computing device 100 may trigger access restrictions to a first sensor data stream (e.g., a low-resolution image stream) while continuing to detect and access a second sensor data stream (e.g., a high-resolution image stream). You can. That is, once an object of interest or region of interest is determined, wearable computing device 100 may no longer wish to use the power or resources associated with the low-resolution image stream, and may no longer wish to use the power or resources associated with streaming, storing, and/or accessing the low-resolution image stream. thus limiting access to the stream to avoid abuse of resources and/or power.

In some implementations, process 900 may also include transmitting the generated excerpted image to a mobile device in communication with the computing device in response to generating the excerpted image representing at least one region of interest. For example, upon identification and extraction of high-resolution images performed on wearable computing device 100, wearable computing device 100 may transmit the excerpted images to mobile device 202 for further processing. For example, mobile device 202 may generate data or information regarding an area or object of interest. Mobile device 202 may have higher power and/or computational resources than wearable computing device 100 so that the generated data and/or information may be retrieved and/or computed on mobile device 202 and stored in wearable computing device 100. ) can be sent again. The wearable computing device 100 may receive information about at least one region of interest from the mobile device and display the corresponding information on the display 244 of the wearable computing device 100.

In some implementations, the first image resolution has a low image resolution and the second image resolution has a high image resolution. Identifying at least one area of interest or object of interest may further include the use of a machine learning algorithm (e.g., via NN 254) executing on wearable computing device 100. The machine learning algorithm may use the first sensor data as an input to identify text or at least one object displayed in the first sensor data. For example, even if the first data sensor stream has a low resolution, a machine learning algorithm can determine whether text or an object appears in the image stream.

For example, wearable computing device 100 may use a machine learning algorithm to determine whether specific image content (e.g., object, ROI) is present (e.g., depicted or displayed) in a low-resolution image. The extractor 236 may extract (crop) the determined unknown image content to the extract coordinates 238 and provide the extracted image to a trained machine learning algorithm to determine whether the specific image content is related to the object(s) of interest and/or Alternatively, it may be determined whether or not it represents a region(s) of interest. For example, a machine learning algorithm that determines image content to be extracted may include identifying one or more details of the captured image. The identified detailed information may be analyzed to determine whether the detailed information includes recognizable objects, symbols, object parts, etc.

For example, if detailed information is identified as a recognizable object, wearable computing device 100 may attempt to recognize certain characteristics of the object. For example, if the object is a soda can, the wearable computing device 100 can evaluate what brand of soda it is. Machine learning algorithms can run lightweight detector architectures on low-resolution images on the device to identify specific patterns of ROI (or object of interest) coordinates. For example, a machine learning algorithm may determine that, as output, the brand of a soda can is a clustered text region that can be identified by a pattern, and that pattern can be extracted and used for further analysis. For example, a pattern may be extracted based on a machine learning algorithm output indicating the extraction coordinates. These coordinates can then be used to select high-resolution image portions of the same scene (e.g., clustered text regions). This section can be further analyzed to determine the brand. Because a portion of the high resolution is used based on input retrieved from low resolution images, machine learning algorithms may provide the benefit of optimizing energy and latency in the wearable computing device 100 since analysis of the entire high resolution image is avoided.

Examples described throughout this disclosure may refer to computers and/or computing systems. As used herein, a computer (and/or computing) system includes any suitable combination of one or more devices comprised of hardware, firmware, and software to perform one or more of the computerized techniques described herein. However, it is not limited to this. As used herein, a computer (and/or computing) system may be a single computing device or multiple computing devices operating collectively, with data storage and function execution distributed among the various computing devices.

Examples described throughout this disclosure may refer to augmented reality (AR). As used herein, AR refers to a user experience in which a computer system promotes sensory perception that includes at least one virtual modality and at least one real modality. AR experiences may be provided through one of various types of computer systems, including, but not limited to, battery-powered wearable computing devices or battery-powered non-wearable computing devices. In some implementations, wearable computing devices may include AR headsets, which may include, but are not limited to, AR glasses, other wearable AR devices, tablets, watches, or laptop computers.

In some types of AR experiences, users can perceive aspects of reality directly with their senses without intermediation by a computer system. For example, some AR glasses, such as wearable computing device 100, transmit an image (e.g., a virtual aspect to be perceived) to the user's retina while simultaneously directing the eye to other light not generated by the AR glasses. Designed for registration. As another example, an in-lens micro display may be embedded in a perspective lens or a projected display may be superimposed on a perspective lens. In other types of AR experiences, the computer system may enhance, supplement, change, and/or activate the user's impression of reality (e.g., the aspect of reality to be perceived) in one or more ways. In some implementations, the AR experience is perceived on a display device screen of a computer system. For example, some AR headsets and/or AR glasses are designed with a camera feedthrough to present camera images of the user's surroundings on a display device positioned in front of the user's eyes.

10 illustrates a method that can be used with the techniques described herein (e.g., to implement wearable computing device 100, such as client computing device, server computing device 204, and/or mobile device 202). Examples of computer device 1000 and mobile computer device 1050 are shown. Computing device 1000 includes a processor 1002, memory 1004, a storage device 1006, a high-speed interface 908 coupled to memory 1004 and a high-speed expansion port 1010, and a low-speed bus 1014 and storage. and a low-speed interface 1012 coupled to device 1006. Each component 1002, 1004, 1006, 1008, 1010, 1012 is interconnected using various buses and may be mounted on a common motherboard or in any other suitable manner. Processor 1002 may perform computing, including instructions stored in memory 1004 or storage device 1006 to display graphical information for the GUI on an external input/output device, such as display 1016 coupled to high-speed interface 1008. Commands for execution within the device 1000 can be processed. In other implementations, multiple processors and/or multiple buses along with multiple memories and memory types may be used as appropriate. Additionally, multiple computing devices 1000 may be connected, with each device providing some of the required operations (e.g., a server bank, a blade server group, or a multiprocessor system).

Memory 1004 stores information within computing device 1000. In one implementation, memory 1004 is a volatile memory unit or units. In another implementation, memory 1004 is a non-volatile memory unit or units. Memory 1004 may also be another form of computer-readable medium, such as a magnetic or optical disk.

Storage device 1006 can provide computing device 1000 with a mass storage device. In one implementation, storage device 1006 is a computer-readable medium such as a floppy disk device, hard disk device, optical disk device, or tape device, flash memory, or other similar solid-state memory device, or a storage area network or other configuration. It may be or include a device array including devices. A computer program product can be tangibly embodied in an information medium. A computer program product may also include instructions that, when executed, perform one or more methods such as those described above. The information medium is a computer or machine-readable medium, such as memory 1004, storage device 12006, or memory of processor 1002.

High-speed controller 1008 manages bandwidth-intensive operations for computing device 1000, while low-speed controller 1012 manages low-bandwidth-intensive operations. These functional assignments are just examples. In one implementation, the high-speed controller 1008 includes memory 1004, a display 1016 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 1010 that can accommodate various expansion cards (not shown). is combined with In an implementation, low speed controller 1012 is coupled to storage device 1006 and low speed expansion port 1014. Low-speed expansion ports, which may include a variety of communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), for example, one or more input/output devices such as a keyboard, pointing device, scanner, or switch or through a network adapter. It can be connected to networking devices such as routers.

Computing device 1000 may be implemented in a number of different forms as shown in the drawing. For example, this could be implemented as a standard server 1020 or multiple times across groups of such servers. This may be implemented as part of a rack server system 1024. Additionally, this may also be implemented in a personal computer such as a laptop computer 1022. Alternatively, components of computing device 1000 may be combined with other components of a mobile device (not shown), such as device 1050. Each of these devices may include one or more of computing devices 1000, 1050, and the overall system may be comprised of multiple computing devices 1000, 1050 in communication with each other.

Computing device 1050 includes, among other things, a processor 1052, memory 1064, input/output devices such as display 1054, communication interface 1066, and transceiver 1068. Device 1050 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components 1050, 1052, 1064, 1054, 1066, and 1068 is interconnected using various buses, and some components may be mounted on a common motherboard or in some other manner as appropriate.

Processor 1052 may execute instructions within computing device 1050, including instructions stored in memory 1064. The processor may be implemented as a chipset of chips containing multiple separate analog and digital processors. The processor may provide coordination of other components of device 1050, such as, for example, control of the user interface, applications executed by device 1050, and wireless communications by device 1050.

Processor 1052 may communicate with the user through control interface 1058 and display interface 1056 coupled to display 1054. Display 1054 may be, for example, a TFT LCD (thin film transistor liquid crystal display), an LED (light emitting diode) or OLED (organic light emitting diode) display, or other suitable display technology. Display interface 1056 may include suitable circuitry to drive display 1054 to present graphics and other information to a user. Control interface 1058 may receive commands from a user and transform them for submission to processor 1052. Additionally, the external interface 1062 may be provided to communicate with the processor 1052 to enable short-range communication between the device 1050 and other devices. External interface 1062 may provide wired communications in some implementations and wireless communications in others, for example, and multiple interfaces may also be used.

Memory 1064 stores information within computing device 1050. Memory 1064 may be implemented as one or more of computer-readable medium(s), volatile memory unit(s), or non-volatile memory unit(s). Expansion memory 1074 may also be provided and connected to device 1050 via expansion interface 1072, which may include, for example, a Single In-Line Memory Module (SIMM) card interface. This expansion memory 1074 may provide additional storage space for device 1050 or may store applications or other information for device 1050. Specifically, the expansion memory 1074 may include instructions for performing or supplementing the above-described process, and may also include security information. Thus, for example, expansion memory 1074 may serve as a security module for device 1050 and may be programmed with instructions to allow secure use of device 1050. Additionally, security applications may be provided via the SIMM card with additional information, such as placing identifying information on the SIMM card in an unhackable manner.

The memory may include, for example, flash memory and/or NVRAM memory, as described below. In one implementation, the computer program product is explicitly embodied in an information carrier. A computer program product contains instructions that, when executed, perform one or more methods such as those described above. The information carrier may be a computer-readable or machine-readable medium, such as, for example, memory of memory 1064, expansion memory 1074, or processor 1052, which may be received via transceiver 1068 or external interface 1062. am.

Device 1050 may communicate wirelessly via communication interface 1066, which may include digital signal processing circuitry, if desired. Communication interface 1066 may provide communication under various modes or protocols, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, among others. Such communication may occur, for example, via radio frequency transceiver 1068. Additionally, short-range communication may occur, such as using Bluetooth, Wi-Fi, or other transceivers (not shown). Additionally, the Global Positioning System (GPS) receiver module 1070 may provide the device 1050 with additional navigation-related wireless data and location-related wireless data that can be appropriately used by applications running on the device 1050.

Device 1050 may also communicate aurally using audio codec 1060, which can receive audio information from a user and convert it into usable digital information. Audio codec 1060 may likewise generate audible sound for a user, such as through a speaker in a handset of device 1050. These sounds may include sounds from voice phone calls, may include recorded sounds (e.g., voice messages, music files, etc.), and may also include sounds generated by applications running on device 1050. .

Computing device 1050 may be implemented in a number of different forms as shown in the figure. For example, this could be implemented as a mobile phone 1080. This may be implemented as part of a smartphone 1082, a personal digital assistant (PDA), or other similar mobile device.

Various implementations of the systems and techniques described herein may be realized in digital electronic circuits, integrated circuits, application specific integrated circuits (ASICs), computer hardware, firmware, software, and/or combinations thereof. These various implementations include at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device. It may include implementation in one or more computer programs executable and/or interpretable on a programmable system.

Such computer programs (also called programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in a high-level procedural and/or object-oriented programming language and/or assembly/machine. As used herein, the terms “machine-readable medium” and “computer-readable medium” include a machine-readable medium that receives machine instructions as machine-readable signals and/or transmits machine instructions and/or instructions to a programmable processor. Refers to any computer program product, apparatus, and/or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide data. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide interaction with a user, the systems and techniques described herein may include a display device (LED, OLED, or LCD monitor/liquid crystal display) for displaying information to the user and for allowing the user to provide input to the computer. It may be implemented on a computer equipped with a keyboard and a pointing device (e.g., a mouse or trackball). Other types of devices may also be used to provide interaction with the user, for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); , input from the user may be received in any form, including acoustic, voice, or tactile input.

The systems and technologies described herein may include a back-end component (e.g., a data server), a middleware component (e.g., an application server), or a front-end component (e.g., a user implemented on a computing system that includes a client computer equipped with a graphical user interface, web browser, or app capable of interacting with an implementation of the subject matter described herein, or that includes a combination of one or more of backend, middleware, or frontend components. It can be. The components of a system may be interconnected through any form or medium of digital data communication, such as a telecommunications network. Examples of communications networks include local area networks (“LANs”), wide area networks (“WANs”), and the Internet.

A computing system may include clients and servers. Clients and servers are usually remote from each other and typically interact through a communications network. The relationship between client and server arises due to computer programs running on each computer and having a client-server relationship with each other.

In some implementations, the computing device shown in the figures may include sensors that interface with AR headset/HMD device 1090 to create an augmented environment for viewing content embedded within the physical space. For example, one or more sensors included in computing device 1050 or another computing device shown in the figures may provide input to AR headset 1090 or provide input to the AR space generally. Sensors may include, but are not limited to, touchscreens, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. Computing device 1050 may use sensors to determine an absolute position and/or detected rotation of the computing device in AR space, which can be used as an input to the AR space. For example, computing device 1050 may be integrated into the AR space as a virtual object such as a controller, laser pointer, keyboard, weapon, etc. Positioning of the computing device/virtual object by the user when integrated into the AR space may allow the user to position the computing device to view the virtual object in a particular way in the AR space.

In some implementations, one or more input devices included in or connected to computing device 1050 may be used as inputs to the AR space. Input devices may include a touchscreen, keyboard, one or more buttons, trackpad, touchpad, pointing device, mouse, trackball, joystick, camera, microphone, earphones or buds with input capabilities, game controller, or other connectable input device. It may be possible, but it is not limited to this. When a computing device is integrated into an AR space, a user interacting with an input device included in computing device 1050 can cause a specific action to occur in the AR space.

In some implementations, the touchscreen of computing device 1050 may be rendered as a touchpad in the AR space. A user may interact with the touchscreen of computing device 1050. For example, in AR headset 1090, interactions are rendered as movements on a rendered touchpad in the AR space. Rendered movements can control virtual objects in the AR space.

In some implementations, one or more output devices included in computing device 1050 may provide output and/or feedback to a user of AR headset 1090 in the AR space. Output and feedback can be visual, tactile, or audio. Outputs and/or feedback may include, but are not limited to, vibration, turning one or more lights or strobes on and off or blinking and/or flashing, sounding an alarm, playing a chime, playing a song, and playing an audio file. Output devices may include, but are not limited to, vibrating motors, vibrating coils, piezoelectric devices, electrostatic devices, light emitting diodes (LEDs), strobes, and speakers.

In some implementations, computing device 1050 may appear as other objects in a computer-generated 3D environment. The user's interaction with the computing device 1050 (e.g., rotating, shaking, touching the touch screen, swiping a finger on the touch screen) may be interpreted as interaction with an object in the AR space. In the example of a laser pointer in AR space, computing device 1050 appears as a virtual laser pointer in a computer-generated 3D environment. As the user manipulates the computing device 1050, the user in the AR space sees the movement of the laser pointer. The user receives feedback from interactions with computing device 1050 in an AR environment on computing device 1050 or AR headset 1090. A user's interaction with a computing device can be translated into interaction with a user interface created in an AR environment for a controllable device.

In some implementations, computing device 1050 may include a touchscreen. For example, a user may interact with a touchscreen to interact with the user interface of a controllable device. For example, a touchscreen may include user interface elements such as sliders that can control properties of a controllable device.

Computing device 1000 is intended to represent various types of digital computers and devices, including but not limited to laptops, desktops, workstations, PDAs, servers, blade servers, mainframes, and other suitable computers. Computing device 1050 is intended to represent various types of mobile devices, such as PDAs, cell phones, smartphones, and other similar computing devices. The components, their connections and relationships, and their functions shown herein are examples only and are not meant to limit the implementation of the invention described and/or claimed in this document.

A number of embodiments are described. Nonetheless, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure.

Additionally, the logic flow depicted in the figures does not require the specific order or sequential order shown to achieve the desired results. Additionally, other steps may be provided or steps may be removed from the described flow, and other components may be added or removed from the described system. Accordingly, other embodiments are within the scope of the following claims.

In addition to the description above, User acknowledges that any system, program, or feature described herein may collect User information (e.g., information about User's social networks, social activities, activities, occupation, User preferences, or User's current location). You may be provided with controls that allow you to choose whether to enable the collection of content and whether you can transmit content or communications to the server. Additionally, certain data may be processed in one or more ways before being stored or used so that personally identifiable information can be removed. For example, the user's identity may be processed so that no personally identifiable information can be determined about the user, or the user's geographic location may be generalized from where the location information was obtained (e.g. , city, zip code, or state level). Accordingly, users can have control over what information is collected about them, how that information is used, and what information is provided to them.

Although certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. Accordingly, it is to be understood that the appended claims are intended to cover all modifications and changes that fall within the scope of implementation. It should be understood that this is provided as an example and is not limited thereto, and that the form or details may vary. Any portion of the devices and/or methods described herein may be combined in any combination except mutually exclusive combinations. Implementations described herein may include various combinations and/or sub-combinations of the functions, components, and/or features of the various implementations described.

Claims (20)

As an image processing method,
In response to receiving a request to cause a sensor of a computing device to identify image content associated with optical data captured by the sensor:
detecting a first sensor data stream having a first image resolution, the first sensor data stream being based on optical data; and
detecting a second sensor data stream having a second image resolution, the second sensor data stream being based on optical data; and
identifying, by processing circuitry of the computing device, at least one region of interest in the first sensor data stream;
determining, by the processing circuitry, excerpt coordinates defining a plurality of first pixels in at least one region of interest in the first sensor data stream;
generating, by processing circuitry, a cropped image representing at least one region of interest, wherein generating:
identifying a second plurality of pixels in a second sensor data stream using excerpt coordinates that define the first plurality of pixels in at least one region of interest in the first sensor data stream, and
An image processing method comprising extracting a second sensor data stream into a plurality of second pixels. According to paragraph 1,
performing, by processing circuitry, optical character resolution on the excerpt image to generate a machine-readable version of the excerpt image;
performing, by processing circuitry, a search query using the machine-readable version of the excerpt image to generate a plurality of search results; and
An image processing method further comprising displaying search results on a display of a computing device. According to claim 1 or 2,
the second sensor data stream is stored in a memory of the computing device,
The image processing method is,
In response to identifying at least one region of interest in the first sensor data stream, retrieving a corresponding at least one region of interest in second sensor data stored in memory; and
An image processing method further comprising restricting access to the first sensor data stream while continuing to sense and access the second sensor data stream. In any of the preceding clauses,
In response to generating a snippet of an image representing at least one region of interest, transmitting the generated snippet of the image to a mobile device in communication with the computing device;
Receiving, from a mobile device, information regarding at least one region of interest; and
An image processing method further comprising displaying information on a display of a computing device. In any of the preceding clauses,
the computing device is a battery-powered computing device, and
the first image resolution has a low image resolution and the second image resolution has a high image resolution; and
Identifying the at least one region of interest includes:
An image processing method further comprising identifying text or at least one object displayed in the first sensor data using the first sensor data as input by a machine learning algorithm executing on a computing device. In any of the preceding clauses,
The processing circuit is at least:
a first image processor configured to perform image signal processing on the first sensor data stream; and
a second image processor configured to perform image signal processing on a second sensor data stream, wherein a first image resolution of the first sensor data stream is lower than a second image resolution of the second data sensor stream. Image processing method. In any of the preceding clauses,
The step of generating an excerpt image representing the at least one region of interest includes:
performed in response to detecting that at least one region of interest meets a threshold condition, wherein the threshold condition includes detecting whether the plurality of second pixels in the second sensor data stream have low blur. Image processing method. As a wearable computing device,
at least one processing device;
at least one image sensor configured to capture optical data;
a memory storing instructions that, when executed, cause the wearable computing device to perform operations, the operations comprising:
In response to receiving a request to cause at least one image sensor to identify image content associated with optical data:
detecting a first sensor data stream having a first image resolution, the first sensor data stream being based on optical data; and
detecting a second sensor data stream having a second image resolution, the second sensor data stream being based on optical data; and
identifying at least one region of interest in the first sensor data stream;
determining excerpt coordinates defining a first plurality of pixels in at least one region of interest in the first sensor data stream; and
Generating an excerpt image representing at least one region of interest, wherein the generating operation comprises generating an excerpt image representing at least one region of interest in the first sensor data stream using the excerpt coordinates defining a plurality of first pixels in the region of interest of the second sensor. A wearable computing device comprising: identifying a second plurality of pixels of a data stream; and extracting a second sensor data stream into the plurality of second pixels. According to clause 8,
The at least one image sensor is:
A wearable computing device comprising a dual stream image sensor configured to operate in a low image resolution mode until triggered to switch to operation in the high image resolution mode. According to clause 8 or 9,
The above operations are:
performing optical character resolution on the excerpt image to generate a machine-readable version of the excerpt image;
performing a search query using the machine-readable version of the excerpt image to generate a plurality of search results; and
A wearable computing device further comprising causing audio output at the optical character resolution from a speaker of the wearable computing device. In any of the preceding clauses,
The second sensor data stream is stored in a memory of the wearable computing device, and the operations include:
In response to identifying at least one area of interest in the first sensor data stream, searching for at least one corresponding area of interest in second sensor data stored in a memory; and
The wearable computing device further comprising limiting access to the first sensor data stream while continuing to sense and access the second sensor data stream. In any of the preceding clauses,
The above operations are:
In response to generating an image excerpt representing at least one region of interest, transmitting the generated image excerpt to a mobile device in communication with the wearable computing device;
Receiving information about at least one region of interest from a mobile device; and
A wearable computing device further comprising an operation of outputting information from the wearable computing device. In any of the preceding clauses,
The first image resolution has a low image resolution, the second image resolution has a high image resolution; and
The operation of identifying the at least one region of interest includes:
A wearable computing device further comprising identifying text or at least one object displayed in the first sensor data using the first sensor data as input by a machine learning algorithm executing on the wearable computing device. In any of the preceding clauses,
The at least one processing device includes:
a first image processor configured to perform image signal processing on the first sensor data stream; and
a second image processor configured to perform image signal processing on a second sensor data stream, wherein a first image resolution of the first sensor data stream is lower than a second image resolution of the second data sensor stream. Wearable computing device. A non-transitory computer-readable medium having stored instructions that, when executed by processing circuitry, cause a wearable computing device to:
detect a first sensor data stream having a first image resolution, the first sensor data stream being based on optical data acquired by a sensor of the wearable computing device; and
detect a second sensor data stream having a second image resolution, the second sensor data stream being based on the optical data;
identify at least one region of interest in the first sensor data stream;
determine coordinates to extract that define a plurality of first pixels in at least one region of interest in the first sensor data stream;
generate an excerpt image representing at least one region of interest;
The above generates,
identify a second plurality of pixels in the second sensor data stream using excerpt coordinates that define the first plurality of pixels in at least one region of interest in the first sensor data stream, and
A non-transitory computer-readable medium comprising extracting a second sensor data stream into a second plurality of pixels. According to clause 15,
The processing circuit is,
Wearable computing devices allow:
performing optical character resolution on the excerpt image to generate a machine-readable version of the excerpt image;
perform a search query using the machine-readable version of the excerpted image to generate a plurality of search results; and
A non-transitory computer readable medium further configured to display search results on a display of a wearable computing device. According to claim 15 or 16,
The second sensor data stream is stored in a memory of the wearable computing device, and
The processing circuit is,
Wearable computing devices allow:
In response to identifying at least one area of interest in the first sensor data stream, retrieve the corresponding at least one area of interest in the second sensor data stored in the memory; and
A non-transitory computer readable medium further configured to limit access to the first sensor data stream while continuing to sense and access the second sensor data stream. In any of the preceding clauses,
The processing circuit is,
Wearable computing devices allow:
In response to generating a snippet of an image representing at least one region of interest, transmit the generated snippet of the image to a mobile device in communication with the wearable computing device;
Receive, from a mobile device, information regarding at least one region of interest; and
A non-transitory computer readable medium configured to display information on a display of a wearable computing device. In any of the preceding clauses,
the first image resolution has a low image resolution and the second image resolution has a high image resolution; and
Identifying the at least one area of interest includes:
Non-transitory computer reading, further comprising identifying text or at least one object represented in the first sensor data by a machine learning algorithm running on the wearable computing device and using the first sensor data as input. Available medium. In any of the preceding clauses,
The processing circuit is at least:
a first image processor configured to perform image signal processing on the first sensor data stream; and
a second image processor configured to perform image signal processing on the second sensor data stream, wherein the first image resolution of the first sensor data stream is lower than the second image resolution of the second data sensor stream. -Transient computer-readable media.