CN117616773A

CN117616773A - Auto Focus (AF) and Auto Exposure Control (AEC) coordination

Info

Publication number: CN117616773A
Application number: CN202280045707.7A
Authority: CN
Inventors: 冯文俊; H-M·刘; H·S·高; H-W·廖; Y·迟
Original assignee: Qualcomm Inc
Current assignee: Qualcomm Inc
Priority date: 2021-07-06
Filing date: 2022-07-01
Publication date: 2024-02-27

Abstract

The present disclosure provides systems, apparatuses, methods, and computer-readable media that support improved image signal processing, particularly in low light or High Dynamic Range (HDR) scenarios. The image processing technique may include performing two Automatic Exposure Control (AEC) operations, wherein a first AEC operation is aimed at obtaining a good condition for an Autofocus (AF) operation, and a second AEC operation follows the AF operation and is aimed at obtaining a good condition for image capturing. The image processing technique may also include communication between AEC operation and AF operation to coordinate operation by locking and releasing exposure levels and focus positions as part of image signal processing. In one aspect, processing techniques may include AF measurement statistics from targets, coordinating with AECs, and performing interleaving of lock-in and recovery state machines to achieve better AF results in challenging low-light or HDR scenes while maintaining overall image quality.

Description

Auto Focus (AF) and Auto Exposure Control (AEC) coordination

Cross Reference to Related Applications

The present application claims the benefit of U.S. patent application Ser. No. 17/647,150 entitled "AUTOFOCUS (AF) AND AUTO EXPOSURE CONTROL (AEC) COORDINATION" filed on 1 month 5 of 2022 and U.S. provisional patent application Ser. No. 63/218,828 entitled "AUTOFOCUS (AF) AND AUTO EXPOSURE CONTROL (AEC) COORDINATION", filed on 7 month 6 of 2021, the entire contents of both of which are expressly incorporated herein by reference.

Technical Field

Aspects of the present disclosure relate generally to image signal processing. Some features may be implemented and provide improved image signal processing, including improved Autofocus (AF).

Background

An image capture device is a device that can capture one or more digital images, whether for still images of photographs or for image sequences of videos. The capture device may be incorporated into a variety of devices. For example, the image capture device may include a stand-alone digital camera or digital video camera, a camera-equipped wireless communication device handset (e.g., a mobile phone, cellular or satellite radiotelephone), a Personal Digital Assistant (PDA), a panel or tablet computer, a gaming device, a computer device (e.g., a webcam), a video surveillance camera, or other devices with digital imaging or video capabilities.

Image capturing devices have inherent limitations. Image quality is related to the sensitivity of the image sensor that captures the image and the brightness of the scene captured in the image. Limitations on image sensor capabilities and scene illumination can inhibit the capture of high quality images.

Disclosure of Invention

The following summarizes some aspects of the present disclosure to provide a basic understanding of the techniques discussed. This summary is not an extensive overview of all contemplated features of the disclosure, and is intended to neither identify key or critical elements of all aspects of the disclosure nor delineate the scope of any or all aspects of the disclosure. Its sole purpose is to present some concepts of one or more aspects of the disclosure in a simplified form as a prelude to the more detailed description that is presented later.

In one aspect of the present disclosure, a method for image processing and/or image capturing includes: performing a first automatic exposure control operation using a first set of target parameters to determine a first exposure level; performing a first autofocus operation at a first exposure level to determine a first focus position; performing a second auto-exposure control operation to determine a second exposure level using a second set of target parameters after performing the first auto-focus operation; and/or capturing the first image frame at a second exposure level.

In an additional aspect of the disclosure, an apparatus includes at least one processor and a memory coupled to the at least one processor. The at least one processor is configured to perform operations comprising: performing a first automatic exposure control operation using a first set of target parameters to determine a first exposure level; performing a first autofocus operation at a first exposure level to determine a first focus position; performing a second auto-exposure control operation to determine a second exposure level using a second set of target parameters after performing the first auto-focus operation; and/or capturing the first image frame at a second exposure level.

In an additional aspect of the disclosure, an apparatus includes: a unit for performing a first automatic exposure control operation using a first set of target parameters to determine a first exposure level; a unit for performing a first autofocus operation at a first exposure level to determine a first focus position; means for performing a second auto-exposure control operation to determine a second exposure level using a second set of target parameters after performing the first auto-focus operation; and/or a unit for capturing the first image frame at a second exposure level.

In additional aspects of the disclosure, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform operations. These operations include: performing a first automatic exposure control operation using a first set of target parameters to determine a first exposure level; performing a first autofocus operation at a first exposure level to determine a first focus position; performing a second auto-exposure control operation to determine a second exposure level using a second set of target parameters after performing the first auto-focus operation; and/or capturing the first image frame at a second exposure level.

Image capture devices (devices that can capture one or more digital images, whether still image photographs or image sequences for video) can be incorporated into a wide variety of devices. For example, the image capture device may include a stand-alone digital camera or digital video camera, a camera-equipped wireless communication device handset (e.g., a mobile phone, cellular or satellite radiotelephone), a Personal Digital Assistant (PDA), a panel or tablet computer, a gaming device, a computer device (e.g., a webcam), a video surveillance camera, or other devices with digital imaging or video capabilities.

In general, this disclosure describes image processing techniques involving an image capture device having an image sensor and an Image Signal Processor (ISP). The image signal processor may be configured to control capture of image frames from the one or more image sensors and process the one or more image frames from the one or more image sensors to produce a view of the scene in the corrected image frames. The corrected image frames may be part of a sequence of image frames forming a video sequence. The video sequence may include other image frames received from an image sensor or other image sensor and/or other corrected image frames based on input from the image sensor or another image sensor.

In an example, an image signal processor may receive instructions for capturing a sequence of image frames to generate a preview display from an image capture device in response to loading software (e.g., a camera application). The image signal processor may be configured to generate a single stream of output frames based on image frames received from the one or more image sensors. The single stream of output frames may include: raw image data from an image sensor or corrected image frames processed by one or more algorithms within an image signal processor. For example, image frames obtained from an image sensor may be processed in an image signal processor by processing the image frames via an image post-processing engine (IPE) and/or other image processing circuitry for performing one or more of tone mapping, portrait lighting, contrast enhancement, gamma correction, etc.

After the output frames representing the scene are determined by the image signal processor using the image correction described in the various embodiments herein, the output frames may be displayed on a device display as a single still image and/or as part of a video sequence, saved to a storage device as a picture or video sequence, sent over a network and/or printed to an output medium. For example, the image signal processor may be configured to obtain input frames (e.g., pixel values) of image data from different image sensors, and then generate corresponding output frames (e.g., preview display frames, still image capture, frames for video, etc.) of image data. In other examples, the image signal processor may output frames of image data to various output devices and/or camera modules for further processing, such as for 3A parameter synchronization (e.g., auto Focus (AF), auto White Balance (AWB), and Auto Exposure Control (AEC)), generating video files via the output frames, configuring the frames for display, configuring the frames for storage, transmitting the frames over a network connection, and so forth. That is, the image signal processor may obtain input frames from one or more image sensors, each coupled to one or more camera shots, and may in turn generate and output streams of output frames to various output destinations. In such an example, the image signal processor may be configured to generate a stream of output frames that may have improved performance in low light conditions.

In some aspects, corrected image frames may be generated by combining aspects of the image correction of the present disclosure with other computational photography techniques, such as High Dynamic Range (HDR) photography or multi-frame noise reduction (MFNR). With HDR photography, the first image frame and the second image frame are captured using different exposure times, different apertures, different lenses, and/or other characteristics, which may result in an improved dynamic range of the fused image when the two image frames are combined. In some aspects, a method may be performed for MFNR photography, in which a first image frame and a second image frame are captured using the same or different exposure times, and the first image frame and the second image frame are fused to produce a corrected first image frame having reduced noise compared to the captured first image frame.

In some aspects, an apparatus may include an image signal processor or processor (e.g., an application processor) that includes specific functions for camera control and/or processing, such as enabling or disabling aspects of image correction or otherwise controlling image correction, such as by specifying an Autofocus (AF) region of interest and/or target parameters for AEC convergence. The methods and techniques described herein may be performed entirely by an image signal processor or processors, or various operations may be split between an image signal processor and a processor and in some aspects across additional processors.

The device may comprise one, two or more image sensors, for example comprising a first image sensor. When there are multiple image sensors, the first image sensor may have a larger field of view (FOV) than the second image sensor, or the first image sensor may have a different sensitivity or a different dynamic range than the second image sensor. In one example, the first image sensor may be a wide angle image sensor and the second image sensor may be a tele image sensor. In another example, the first sensor is configured to obtain an image through a first lens having a first optical axis, and the second sensor is configured to obtain an image through a second lens having a second optical axis different from the first optical axis. Additionally or alternatively, the first lens may have a first magnification and the second lens may have a second magnification different from the first magnification. This configuration may occur with a lens cluster on the mobile device, for example, where multiple image sensors and associated lenses are located in offset positions on the front or back side of the mobile device. Additional image sensors with larger, smaller, or the same field of view may be included. The image correction techniques described herein may be applied to image frames captured from any image sensor in a multi-sensor device.

In an additional aspect of the disclosure, a device configured for image processing and/or image capturing is disclosed. The apparatus includes means for capturing an image frame. The apparatus also includes one or more units for capturing data representing a scene, such as an image sensor (including a Charge Coupled Device (CCD), a bayer filter sensor, an Infrared (IR) detector, an Ultraviolet (UV) detector, a Complementary Metal Oxide Semiconductor (CMOS) sensor), a time-of-flight detector. The apparatus may further comprise one or more units for accumulating and/or focusing light into one or more image sensors, including simple lenses, compound lenses, spherical lenses and aspherical lenses. These components may be controlled to capture a first image frame and/or a second image frame input to the image processing techniques described herein.

Other aspects, features and implementations will become apparent to those of ordinary skill in the art upon review of the following description of specific exemplary aspects in conjunction with the accompanying drawings. While features may be discussed below with respect to certain aspects and figures, various aspects may include one or more of the advantageous features discussed herein. In other words, while one or more aspects may be discussed as having certain advantageous features, one or more of such features may also be used in accordance with various aspects. In a similar manner, although exemplary aspects may be discussed below as device, system, or method aspects, the exemplary aspects may be implemented in a variety of devices, systems, and methods.

The method may be embodied in a computer readable medium as computer program code comprising instructions for causing a processor to perform the steps of the method. In some embodiments, the processor may be part of a mobile device comprising: a first network adapter configured to transmit data, such as images or video as recorded or as streaming data, over a first network connection of a plurality of network connections; and a processor coupled to the first network adapter and the memory. The processor may cause transmission of the correction image frames described herein over a wireless communication network, such as a 5G NR communication network.

The foregoing has outlined rather broadly the features and technical advantages of examples in accordance with the present disclosure in order that the detailed description that follows may be better understood. Additional features and advantages will be described hereinafter. The disclosed concepts and specific examples may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. Such equivalent constructions do not depart from the scope of the appended claims. The features of the concepts disclosed herein, both as to their organization and method of operation, together with the associated advantages will be better understood from the following description when considered in connection with the accompanying figures. Each of the figures is provided for the purpose of illustration and description, and is not intended as a definition of the limits of the claims.

Although aspects and implementations are described in this application by way of illustration of some examples, those skilled in the art will appreciate that additional implementations and use cases may be generated in many different arrangements and scenarios. The innovations described herein may be implemented across many different platform types, devices, systems, shapes, sizes, package arrangements. For example, aspects and/or uses may be generated via integrated die implementations and other non-module component-based devices (e.g., end user devices, vehicles, communication devices, computing devices, industrial devices, retail/purchase devices, medical devices, artificial Intelligence (AI) enabled devices, etc.). While some examples may or may not be specifically directed to use cases or applications, applicability of the various types of innovations described may occur. Implementations may range from die-level or modular components to non-modular, non-die-level implementations, and further to aggregate, distributed, or Original Equipment Manufacturer (OEM) devices or systems that incorporate one or more aspects of the described innovations. In some practical arrangements, devices incorporating the described aspects and features may also necessarily include additional components and features for implementation and implementation of the claimed and described aspects. For example, transmission and reception of wireless signals necessarily include: a number of components for analog and digital purposes (e.g., hardware components including antennas, radio Frequency (RF) chains, power amplifiers, modulators, buffers, processors, interleavers, adders/adders, etc.). The innovations described herein are intended to be implemented in a variety of devices, chip-scale components, systems, distributed arrangements, or end-user apparatuses having different sizes, shapes, and configurations.

Drawings

A further understanding of the nature and advantages of the present disclosure may be realized by reference to the following drawings. In the drawings, similar components or features may have the same reference numerals. Furthermore, various components of the same type may be distinguished by following the reference label by a dash and a second label that distinguishes among the similar components. If only the first reference label is used in the specification, the description applies to any one of the similar components having the same first reference label, regardless of the second reference label.

FIG. 1 illustrates a block diagram of an example device 100 for performing image capture from one or more image sensors.

FIG. 2 is a block diagram illustrating operation of an AF and AEC in accordance with one or more aspects.

Fig. 3 is a block diagram illustrating one example of coordination between Automatic Exposure Control (AEC) and auto-focus (AF), in accordance with one or more aspects.

Fig. 4A is a flow diagram illustrating a method of image signal processing using two AEC operations, in accordance with one or more aspects.

Fig. 4B is a flow diagram illustrating a method of image signal processing using two AEC operations, in accordance with one or more aspects.

Fig. 5 is a block diagram illustrating one example of coordination between Automatic Exposure Control (AEC) and auto-focus (AF) in accordance with one or more aspects.

Like reference numbers and designations in the various drawings indicate like elements.

Detailed Description

The detailed description set forth below in connection with the appended drawings is intended as a description of various configurations and is not intended to limit the scope of the present disclosure. Conversely, the "detailed description" includes: for providing a thorough understanding of the present subject matter. It will be apparent to one skilled in the art that these specific details are not required in every case and that, in some instances, well-known structures and components are shown in block diagram form for clarity of presentation.

Inherent limitations of image sensors in image capture devices can lead to low quality images. The low quality image produced by the image sensor may also create problems for other aspects of the image signal processing. For example, low quality images produced under low light conditions challenge the camera control algorithm, which can further reduce image quality. For example, an Auto Focus (AF) operation based on detecting contrast in an image frame may be difficult to accomplish in low light conditions where little contrast is available. As another example, an AF operation based on detecting contrast in a low light portion of a high dynamic range image may be difficult to accomplish with little contrast available. The disadvantages noted herein are merely representative and are included to highlight problems that the inventors have identified and sought improvement with respect to existing devices. Aspects of the devices described below may address some or all of the shortcomings and others known in the art. Aspects of the improved devices described below may exhibit other benefits in addition to those described above and be used in other applications in addition to those described above.

The present disclosure provides systems, apparatuses, methods, and computer-readable media that support improved image signal processing, particularly in low light or High Dynamic Range (HDR) scenarios. The image processing technique may include performing two Automatic Exposure Control (AEC) operations, where a first AEC operation is aimed at obtaining a condition that improves convergence and confidence for an Autofocus (AF) operation, and a second AEC operation follows the AF operation and is aimed at obtaining a good condition for image capture. The image processing technique may also include communication between AEC operation and AF operation to coordinate operation by locking and releasing exposure levels and focus positions as part of image signal processing.

In one aspect, the processing technique may include an AF algorithm that measures statistics from a target region of interest (which may be a detected object of interest, such as a face), coordinates with the AEC algorithm, and performs interleaving of lock-in and recovery state machines to achieve better AF results in challenging low-light or HDR scenes while maintaining the overall image quality preferred by AEC operation. Coordination may include the AF locking in the first in-focus position waiting for the first AEC operation to complete. The AEC operation may converge to a first exposure level revealing details within the region of interest, after which the exposure level locks and the AF algorithm unlocks and determines the in-focus position of the region of interest when the exposure level is locked. After the AF operation is completed, the AF algorithm locks the focus, and the AEC algorithm unlocks the exposure level and performs a second AEC operation to adjust the exposure level to obtain a desired image result. In some embodiments, the AF may unlock the focus position during the second AEC operation during adjustment of the exposure level.

Particular implementations of the subject matter described in this disclosure can be implemented to realize one or more of the following potential advantages or benefits. In some aspects, the present disclosure provides techniques for image signal processing methods in which AF may actively cooperate with AEC to obtain better AF results in scenes with low light or HDR without changing the original AEC preferences. This may result in faster and/or more accurate focus operations, which results in a photograph with more in-focus areas and thus improved image quality (as measured by preserving details of the scene) while still having the desired illumination quality throughout the photograph, and particularly in High Dynamic Range (HDR) photography.

Example devices for capturing image frames using one or more image sensors (e.g., a smart phone) may include configurations of two, three, four, or more cameras on a back side (e.g., a side opposite a user display) or a front side (e.g., a side same as the user display) of the device. A device having multiple image sensors includes one or more Image Signal Processors (ISPs), computer Vision Processors (CVPs) (e.g., AI engines), or other suitable circuitry for processing images captured by the image sensors. The one or more image signal processors may provide the processed image frames to a memory and/or processor (e.g., an application processor, an Image Front End (IFE), an image post-processing engine (IPE), or other suitable processing circuitry) for further processing, e.g., for encoding, storage, transmission, or other manipulation.

As used herein, an image sensor may refer to the image sensor itself and any particular other component coupled to the image sensor for generating image frames for processing by a storage unit in an image signal processor or other logic or memory, whether short term buffer or long term non-volatile memory. For example, the image sensor may include other components of the camera, including shutters, buffers, or other readout circuitry for accessing individual pixels of the image sensor. The image sensor may also refer to an analog front end or other circuitry for converting analog signals for an image frame into a digital representation that is provided to digital circuitry coupled to the image sensor.

In the following description, numerous specific details are set forth, such as examples of specific components, circuits, and processes, in order to provide a thorough understanding of the present disclosure. The term "coupled," as used herein, means directly connected or connected through one or more intermediate components or circuits. In addition, in the following description and for purposes of explanation, specific nomenclature is set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that: no such specific details may be required to practice the teachings disclosed herein. In other instances, well-known circuits and devices are shown in block diagram form in order not to obscure the teachings of the present disclosure.

Some portions of the detailed descriptions which follow are presented in terms of procedures, logic blocks, processing, and other symbolic representations of operations on data bits within a computer memory. In this disclosure, a procedure, logic block, process, etc., is conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system.

In the figures, a single block may be described as performing one or more functions. One or more functions performed by the block may be performed in a single component or across multiple components, and/or may be performed using hardware, software, or a combination of hardware and software. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described below generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. Moreover, example devices may include components other than those shown, including well-known components such as processors, memory, and the like.

Aspects of the present disclosure are applicable to any suitable electronic device that includes or is coupled to two or more image sensors capable of capturing image frames (or "frames"). Further, aspects of the present disclosure may be implemented in image sensors or devices coupled to image sensors having the same or different capabilities and characteristics (e.g., resolution, shutter speed, sensor type, etc.). Further, aspects of the present disclosure may be implemented in a device for processing image frames, whether the device includes or is coupled to an image sensor, such as a processing device that may acquire stored images for processing, including processing devices that are present in a cloud computing system.

Unless specifically stated otherwise, it should be apparent from the following discussion that: throughout this application, discussions utilizing terms such as "accessing," "receiving," "transmitting," "using," "selecting," "determining," "normalizing," "multiplying," "averaging," "monitoring," "comparing," "applying," "updating," "measuring," "deriving," "setting," "generating," or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's registers, memories, or other such information storage, transmission or display devices.

The terms "device" and "apparatus" are not limited to one or a particular number of physical objects (e.g., a smart phone, a camera controller, a processing system, etc.). As used herein, a device may be any electronic device having one or more portions that may implement at least some portions of the present disclosure. Although the following description and examples use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a particular configuration, type, or number of objects. As used herein, an apparatus may comprise a device or portion of a device for performing the described operations.

FIG. 1 illustrates a block diagram of an example device 100 for performing image capture from one or more image sensors. The device 100 may include or otherwise be coupled to an image signal processor 112 for processing image frames from one or more image sensors (e.g., the first image sensor 101, the second image sensor 102, and the depth sensor 140). In some implementations, the device 100 also includes or is coupled to the processor 104 and the memory 106 that stores instructions 108. The device 100 may also include or be coupled to a display 114 and an input/output (I/O) component 116. The I/O component 116 can be used to interact with a user, such as a touch screen interface and/or a physical button interface. The I/O component 116 may also include a network interface for communicating with other devices comprising a Wide Area Network (WAN) adapter 152, a Local Area Network (LAN) adapter 153, and/or a Personal Area Network (PAN) adapter 154. Examples of WAN adapter 152 include a 4G LTE or 5G NR wireless network adapter. An example LAN adapter 153 is an IEEE 802.11WiFi wireless network adapter. An example PAN adapter 154 is a bluetooth wireless network adapter. Each of adapters 152, 153, and/or 154 may be coupled to one antenna and may be coupled to multiple antennas configured for primary reception and diversity reception and/or configured for receiving a particular frequency band. The device 100 may also include or be coupled to a power source 118 for the device 100, such as a battery or a component that couples the device 100 to an energy source. The device 100 may also include or be coupled to additional features or components not shown in fig. 1. In one example, one or more transceivers and baseband processors may be coupled to or included in WAN adapter 152 for a wireless communication device. In another example, an Analog Front End (AFE) for converting analog image frame data to digital image frame data may be coupled between the image sensors 101 and 102 and the image signal processor 112.

The device may include or be coupled to a sensor hub 150 for interfacing with sensors to receive data regarding movement of the device 100, data regarding the environment surrounding the device 100, and/or other non-camera sensor data. One example non-camera sensor is a gyroscope, a device configured for measuring rotation, orientation, and/or angular velocity to generate motion data. Another example non-camera sensor is an accelerometer, a device configured to measure acceleration, which may also be used to determine speed and distance traveled by appropriately integrating the measured accelerations, and one or more of the accelerations, speeds, and/or distances may be included in the generated motion data. In some aspects, a gyroscope in an electronic image stabilization system (EIS) may be coupled to the sensor hub or directly to the image signal processor 112. In another example, the non-camera sensor may be a Global Positioning System (GPS) receiver. Data from sensor hub 150 may be used by image signal processor 112 to generate corrected image frames, such as by applying Electronic Image Stabilization (EIS) and/or Digital Image Stabilization (DIS).

The image signal processor 112 may receive image data from one or more cameras in the form of image frames. In one embodiment, a local bus connection couples the image signal processor 112 to the image sensors 101 and 102 of the first and second cameras, respectively. In another embodiment, the line interface couples the image signal processor 112 to an external image sensor. In another embodiment, a wireless interface couples the image signal processor 112 to the image sensors 101, 102.

The first camera may include a first image sensor 101 and a corresponding first lens 131. The second camera may include a second image sensor 102 and a corresponding second lens 132. Each of lenses 131 and 132 may be controlled by an associated Autofocus (AF) algorithm 133 executing in ISP 112 that adjusts lenses 131 and 132 to focus on a particular focal plane corresponding to a certain focus position. The AF algorithm 133 may be aided by the depth sensor 140 by approximating the in-focus position using the depth data.

The first image sensor 101 and the second image sensor 102 are configured to capture one or more image frames. Lenses 131 and 132 focus light at image sensors 101 and 102, respectively, through one or more apertures for receiving light, one or more shutters for blocking light when outside of an exposure window, one or more Color Filter Arrays (CFAs) for filtering light outside of a particular frequency range, one or more analog front ends for converting analog measurements to digital information, and/or other suitable components for imaging. The first lens 131 and the second lens 132 may have different fields of view to capture different representations of the scene. For example, the first lens 131 may be an Ultra Wide (UW) lens, and the second lens 132 may be a wide (W) lens. The plurality of image sensors may include a combination of ultra-wide (high field of view (FOV)), wide, tele, and ultra-tele (low FOV) sensors. That is, each image sensor may be configured by hardware configuration and/or software settings to obtain different but overlapping fields of view. In one configuration, the image sensor is configured with different lenses having different magnifications forming different fields of view. The sensor may be configured such that the UW sensor has a larger FOV than the W sensor, which has a larger FOV than the T sensor, which has a larger FOV than the UT sensor. For example, a sensor configured for a wide FOV may capture a field of view in the range of 64-84 degrees, a sensor configured for a super-sided FOV may capture a field of view in the range of 100-140 degrees, a sensor configured for a tele FOV may capture a field of view in the range of 10-30 degrees, and a sensor configured for a super-tele FOV may capture a field of view in the range of 1-8 degrees.

The image signal processor 112 processes image frames captured by the image sensors 101 and 102. Although fig. 1 shows device 100 as including two image sensors 101 and 102 coupled to image signal processor 112, any number (e.g., 1, 2, 3, 4, 5, 6, etc.) of image sensors may be coupled to image signal processor 112. In some aspects, a depth sensor (e.g., depth sensor 140) may be coupled to image signal processor 112 and output from the depth sensor processed in a manner similar to that of image sensors 101 and 102 to generate corrected image frames based on image frames captured by depth sensor 140. The depth sensor 140 may also be used to apply a correction to a first image frame captured from one of the image sensors 101 and 102, such as by using the depth data to segment the image frame from the sensor 101 or 102 into foreground and background regions, and processing the foreground and background regions, respectively, in determining the corrected first image frame. While the apparatus shown in fig. 1 may reflect a configuration for some embodiments of the disclosed image signal processing techniques and methods, any number of additional image sensors or image signal processors may be included in other embodiments of the device 100 while still implementing aspects of the disclosed image signal processing techniques and methods.

In some aspects, the image signal processor 112 may execute instructions from a memory, such as instructions 108 from the memory 106, instructions stored in a separate memory coupled to or included in the image signal processor 112, or instructions provided by the processor 104. Additionally or alternatively, the image signal processor 112 may include specific hardware (e.g., one or more Integrated Circuits (ICs)) configured to perform one or more operations described in this disclosure. For example, the image signal processor 112 may include one or more Image Front Ends (IFEs) 135, one or more image post-processing engines 136 (IPEs), one or more Automatic Exposure Control (AEC) 134 engines, and/or one or more Autofocus (AF) 133 engines. AF 133, AEC 134, IFE 135, IPE 136 may each comprise dedicated circuitry embodied as software code executed by ISP 112, and/or a combination of hardware within ISP 112 and software code executing on ISP 112.

In some implementations, the memory 106 may include a non-transitory or non-transitory computer-readable medium storing computer-executable instructions 108 to perform all or part of one or more operations described in this disclosure. In some implementations, the instructions 108 include a camera application (or other suitable application) to be executed by the device 100 for generating images or video. The instructions 108 may also include other applications or programs executed by the device 100, such as an operating system and specific applications other than for image or video generation. Execution of the camera application (e.g., by the processor 104) may cause the device 100 to generate an image using the image sensors 101 and 102 and the image signal processor 112. The memory 106 may also be accessed by the image signal processor 112 to store processed frames or may be accessed by the processor 104 to obtain processed frames. In some embodiments, the device 100 does not include the memory 106. For example, the device 100 may be a circuit including the image signal processor 112, and the memory may be located external to the device 100. The apparatus 100 may be coupled to an external memory and configured to access the memory to write the output frame for display or long term storage. In some embodiments, device 100 is a system on a chip (SoC) that incorporates image signal processor 112, processor 104, sensor hub 150, memory 106, and input/output component 116 into a single package.

In some embodiments, at least one of the image signal processor 112 or the processor 104 executes instructions to perform various operations described herein, including AEC and AF operations and coordination of AEC and AF operations. For example, execution of the instructions may instruct the image signal processor 112 to begin or end capturing an image frame or sequence of image frames, where the capturing includes AF and AEC operations as described in embodiments herein. In some embodiments, the processor 104 may include one or more general-purpose processor cores 104A capable of executing scripts or instructions (e.g., instructions 108 stored in the memory 106) of one or more software programs. For example, the processor 104 may include one or more application processors configured to execute a camera application (or other suitable application for generating images or video) stored in the memory 106.

In executing a camera application, the processor 104 may be configured to instruct the image signal processor 112 to perform one or more operations with reference to the image sensor 101 or 102. For example, the camera application may receive a command to start a video preview display on which video comprising a sequence of image frames is captured and processed from one or more image sensors 101 or 102. The image correction may be applied to one or more image frames in the sequence. Execution of the instructions 108 by the processor 104 external to the camera application may also cause the device 100 to perform any number of functions or operations. In some embodiments, the processor 104 may include an IC or other hardware (e.g., an Artificial Intelligence (AI) engine 124) in addition to the ability to execute software to cause the device 100 to perform a plurality of functions or operations, such as those described herein. In some other embodiments, such as when all of the described functions are configured in the image signal processor 112, the device 100 does not include the processor 104.

In some embodiments, the display 114 may include one or more suitable displays or screens that allow user interaction and/or presentation of items to a user, such as previews of image frames captured by the image sensors 101 and 102. In some aspects, the display 114 is a touch sensitive display. The I/O component 116 may be or include any suitable mechanism, interface, or device for receiving input (e.g., commands) from a user and providing output to the user via the display 114. For example, the I/O component 116 may include, but is not limited to, a Graphical User Interface (GUI), a keyboard, a mouse, a microphone, a speaker, a compressible bezel, one or more buttons (e.g., power buttons), sliders, switches, and the like.

Although shown coupled to each other via the processor 104, components (such as the processor 104, memory 106, image signal processor 112, display 114, and I/O component 116) may be coupled to each other in other various arrangements, such as via one or more local buses, which are not shown for simplicity. Although the image signal processor 112 is shown as being separate from the processor 104, the image signal processor 112 may be the core of the processor 104, the processor 104 being an Application Processor Unit (APU), contained in a system on a chip (SoC), or otherwise contained in the processor 104. Although the device 100 is referred to in the examples herein as performing aspects of the present disclosure, some device components may not be shown in fig. 1 to prevent aspects of the present disclosure from obscuring. Additionally, other components, numbers of components, or combinations of components may be included in a suitable device for performing aspects of the present disclosure. Accordingly, the present disclosure is not limited to the configuration of a particular device or component, including device 100.

An Auto Focus (AF) 133 analyzes the image frames and determines a focus position for moving a lens of a corresponding image sensor. The focus position determined by the AF 133 affects the image quality because the formed image is blurred if a wrong focus position is set for the lens. One example technique for AF 133 is Contrast Autofocus (CAF), where a focus position (FV) may be generated for several lens aperture positions and the lens is positioned to a focus position at the peak of the focus position versus lens aperture position curve to capture an in-focus image of the scene. Another example technique for AF 133 is Phase Detection Autofocus (PDAF), where AF 133 may estimate a focus position using a phase difference determined from an image frame captured by an image sensor and move a lens to capture an in-focus image of a scene at the focus position.

The AF 133 and AEC 134 operate to allow a user of the image capturing apparatus 100 to perform a "dumb camera" operation. With AF 133 and AEC 134, the user does not need to manually adjust the lens position or manually determine the exposure level. The AF 133 and AEC 134 allow capturing of image frames much faster than by manually adjusting these camera parameters and improve the user experience by reducing the photographic complexity of the user. One example process for automated image capture is shown in fig. 2.

FIG. 2 is a block diagram illustrating operation of an AF and AEC in accordance with one or more aspects. The AF 210 operation begins with a trigger at time 202. The trigger may be, for example, loading a camera application requesting a preview stream of image frames of a scene in view of the image sensor. After triggering at time 202, an Automatic Exposure Control (AEC) 220 operates in time 204 to converge to an exposure level that meets a set of target parameters. During time 204, autofocus (AF) 210 waits for AEC to converge and settle at the exposure level. After AEC 220 converges, AF 210 operates at time 206 to determine a focus position for adjusting the camera lens. However, the converging exposure level after time 204 may create a poor condition for AF 210 to identify the focus position. Such adverse conditions may include low light, low contrast, low PDAF confidence, high sensor gain (which may result in high noise), and so forth. AEC 220 determines an appropriate luminance target for the current region of interest (ROI). However, AEC 220 and AF 210 may have different desired camera parameters, particularly in High Dynamic Range (HDR) scenes. For example, in a scene where a person stands in front of a window, there may be a strong backlight condition, where AEC 220 preferably reduces the target brightness to avoid overexposure of the highlighted background portion. This may lead to poor conditions of the AF 210, as the region of interest of the focus (the face of a person standing in front of the window) is poorly illuminated, which prevents the identification of the focus position for focusing on the region of interest due to poor contrast in the shadow area of the scene.

Embodiments of the present disclosure provide a mechanism in which the AEC can be configurable with different target parameters during operation of the camera. Different target parameters may allow AEC to temporarily control exposure to improve autofocus operation and then resume controlling exposure to improve overall image quality. In some embodiments, the AEC is configured to operate as part of a state machine. For example, during camera operation, AF may measure statistics from targets, AF coordinates with AEC, and performs interlace lock/restore on focus and/or exposure level to achieve better AF results in challenging low light or HDR scenes. Fig. 3 is a block diagram illustrating one example of coordination between Automatic Exposure Control (AEC) and auto-focus (AF), in accordance with one or more aspects.

Communication between Autofocus (AF) 320 and Automatic Exposure Control (AEC) 330 may improve image quality by allowing AEC 330 to operate to improve image conditions in the area of interest for AF, allowing AF to determine focus position, and then allowing AEC 330 to converge to an exposure level that provides high overall image quality. AEC 330 may be configured to determine the exposure level using two or more sets of target parameters, a first set of target parameters determining a first exposure level to improve focus in a region of interest of a scene used during time 304 (such as by improving brightness and/or contrast in the region of interest), and a second set of target parameters determining a second exposure level to improve overall image performance used during time 308.

The AF 320 operation begins with trigger 322. The trigger 322 may be, for example, loading a camera application that requests a preview stream of image frames of a scene in view of an image sensor. After triggering 322, one or more values (e.g., statistics) may be calculated for the region of interest in the image frame. These values may include brightness levels, contrast values, confidence levels of the PDAF focus determination, and/or sensor levels. A region of interest (ROI) may be determined based on one or more of a user touch point on an image frame, a determined saliency value, a gaze of a user observing a display as determined by eye tracking, a center of the image frame, and/or a detected object (e.g., a face). Image conditions in the ROI may be determined based on the values. For example, one or more conditions (e.g., rules) may be evaluated based on the values to determine whether there are adverse conditions in the ROI for AF operation. In one example, luminance statistics for the ROI may be determined to determine whether the average luminance value is below a threshold indicating that a bad condition exists for AF 320. When a poor AF condition is identified during period 302, AF 320 may communicate with AEC 330, such as by an image signal processor, to perform AEC convergence to increase the average luminance value (e.g., brightness) in the ROI.

At time 304, AEC 330 operates using the first set of target parameters to improve the ROI in a manner that improves the determination of the focus position of the ROI by AF 320. During period 304, AF 320 waits for AEC 330 to stabilize. In some embodiments, AF 320 begins the autofocus operation of period 306 after a particular period. In some embodiments, AF 320 starts the autofocus operation of period 306 after AEC 330 reaches the convergence condition. In some embodiments, when the AEC 330 communicates with the AF 320 to begin autofocus operation, the AF 320 begins autofocus operation for the period 306.

During time period 306, AF 320 determines a focus position with auto focus when AEC is locked at the first exposure level determined during time period 304. In some embodiments, AEC 330 starts a new AEC operation after a certain period of time using a second set of target parameters. In some embodiments, after AF 320 reaches the convergence condition, AEC 330 begins a new AEC operation using a second set of target parameters. In some embodiments, after AF 320 communicates with AEC 330 to begin new AEC operation, AEC 330 begins new AEC operation using a second set of target parameters.

During time period 308, AEC 330 operates using the second set of target parameters to determine exposure levels that meet the second set of target parameters to improve image performance. During this AEC operation, AF 320 locks in the in-focus position and AEC 330 continues to converge using the second set of target parameters. The scene may continue to change during time period 308 and AEC 330 continues to adjust in response to convergence using the second set of target parameters. For example, light in the scene may be turned on so that AEC 330 adjusts the exposure level to compensate for the new light. During time period 308, a scene change may be detected at time 310, and the algorithm may be restarted at time 310 and returned to time 302. The in-focus position may remain locked during the period 308, with AF 320 operation disabled. The image frames may continue to be generated from the ISP and output in a preview display of the camera application executing on the image capture device during time 308. Snapshot image frames may be captured during time period 308, such as by pressing a shutter to request image capture in response to user input. If the shutter is pressed during the other time periods 302, 304, 306, or 310, capturing the snapshot image frames in response to the shutter may be delayed until time 308.

The operations described with reference to fig. 3 may be configured in an image signal processor using a lock and restore state machine, where there are states in AF 320 for locking in focus positions in time periods 304 and 308 and restoring focus position determination in time period 306, and for locking in exposure levels in time period 306 and restoring exposure level determination in time periods 304 and 308.

A method for performing image signal processing with two AEC operations to improve AF operation is shown in fig. 4A. Fig. 4A is a flow diagram illustrating a method of image signal processing using two AEC operations, in accordance with one or more aspects. Fig. 4A begins at block 402 where a first auto-exposure control (AEC) operation is performed with a first set of target parameters to determine a first exposure level for improving auto-focus (AF) performance. For example, the target parameter may comprise an increased average luminance value of the identified region of interest in the scene.

At block 404, a first Auto Focus (AF) operation is performed at a first exposure level, e.g., where the first exposure level is locked until the AF operation is complete. The lock-in exposure level determined at block 402 may provide additional detail within a region of interest in a scene. For example, if the region of interest is in a shadow region, the higher exposure level determined at block 402 may increase the detail within the shadow region. As another example, if the region of interest is located in a highlight region, the lower exposure level determined at block 402 may increase detail within the highlight region.

At block 406, a second AEC operation is performed using a second set of target parameters to determine a second exposure level for improving image performance. The second set of target parameters may be intended to converge across the entire scene towards a particular average luminance value even if details are lost in the region of interest within the scene. With the focal length already determined at block 404, details in the region of interest may be sacrificed to improve overall image quality. In some embodiments, the AF operation may continue during the second AEC operation of block 406. For example, small changes in focal length may be made by the PDAF and/or CAF algorithms to preserve the performance of the image frames. If the AF loses tracking of focus, a scene change may be determined and the method 400 returns to block 402 to resume AF and AEC operations. In some embodiments, the AF operation performed during the second AEC operation of block 406 may be performed by a second AF system, e.g., a laser-assisted AF system.

At block 408, an image frame may be captured at the second exposure level determined at block 406 based on the focus position determined by the AF operation of block 404, which was determined at the first exposure level from block 402. The image frames may be used as part of a preview display in a camera application, save and/or transmit still image frames as photographs, and/or save and/or transmit video sequences of still image frames.

In some scenarios, the conditions may be sufficient to determine the in-focus position in the ROI without additional AEC operations, as shown in fig. 3. Under such conditions, the image signal processor may execute an algorithm corresponding to times 202, 204, 206, 208 of fig. 2 after time period 302. A method of image signal processing for switching between one AEC and two AEC operations according to image conditions is shown in fig. 4B.

Fig. 4B is a flow diagram illustrating a method of image signal processing using two AEC operations, in accordance with one or more aspects. At block 410 of method 420, one or more statistics of a region of interest (ROI) in an image frame obtained from an image sensor are determined. At block 412, the statistics are used to determine whether one or more conditions indicative of poor AF operation are satisfied. For example, adverse conditions for AF operation may include one or more of the following: low contrast in the region of interest, low confidence in the focus position, slow convergence in Auto Focus (AF) operation (e.g., a convergence rate below a rate threshold), or low light level in the region of interest (e.g., a light level below a light threshold). If so, method 420 continues to perform operations as in block 402, block 404, block 406, and block 408 of FIG. 4A. If the condition is not met at block 412, the method 420 continues to blocks 414, 416 and 418 to perform image frame capture using a single AEC operation. At block 414, a first AEC operation is performed to determine a first exposure level. At block 416, a first AF operation is performed at a first exposure level. At block 416, while locked at the first exposure level, the image frame is captured at the first exposure level and the focus position determined by the AF operation of block 416.

A number of Autofocus (AF) techniques are available on image capture devices. For example, laser Autofocus (AF), such as a time of flight (TOF) sensor, may be available in an image capture device and used to improve focus position. One advantage of laser assisted AF is that it does not rely on light from the target, but instead emits a beam from which the distance to the object can then be determined. Laser assisted AF may be particularly advantageous in low light conditions. One example of using a second AF system (such as laser assisted AF) in image frame capture with two AEC operations is shown in FIG. 5.

Fig. 5 is a block diagram illustrating one example of coordination between Automatic Exposure Control (AEC) and Autofocus (AF), in accordance with one or more aspects. Time period 502 may operate similar to time period 302. Time period 504 may operate similar to time period 304. Time period 506 may operate similar to time period 306. Time period 508 may operate similar to time period 308. Time period 510 may operate similar to time period 310. AF 520 may be configured similar to AF 320.AEC 530 may be configured similar to AF 320. Additional AF techniques such as laser assisted AF may be applied during time period 508 to provide some focus correction during time period 508, which may allow for a scene change determination that delays returning operation to time period 502 during time period 510. When the shutter is delayed until time 508, allowing time period 508 to extend longer may improve responsiveness to user shutter presses. In some embodiments, the time period 508 may end when the second AF operation detects that the focal length change exceeds a threshold value. Further, focus correction over time period 508 may provide improved image quality by increasing the likelihood that the captured image frame is captured at the correct focus position of the distance of the ROI. In some embodiments, the AF technique for adjusting focus distance during the second AEC operation may remain active during period 508.

Note that one or more blocks (or operations) described with reference to fig. 1-5 may be combined with one or more blocks (or operations) described with reference to another of these figures. For example, one or more blocks (or operations) of fig. 2 may be combined with one or more blocks (or operations) of fig. 3 or 5. As another example, one or more blocks associated with fig. 4A or 4B may be combined with one or more blocks associated with fig. 5. Additionally or alternatively, one or more of the operations described above with reference to fig. 1 may be combined with one or more of the operations described with reference to fig. 2-5.

In one or more aspects, techniques for supporting image signal processing may include additional aspects, such as any single aspect or any combination of aspects described below or in connection with one or more other processes or devices described elsewhere herein. In some aspects, supporting image signal processing may include an apparatus including an image sensor and an image signal processor. The apparatus is further configured to perform steps comprising: performing a first Automatic Exposure Control (AEC) with a first set of target parameters to determine a first exposure level; performing Auto Focus (AF) at the first exposure level; and/or performing a second automatic exposure control (ARC) with a second set of target parameters to determine a second exposure level. Additionally, the apparatus may perform or operate in accordance with one or more aspects as described herein. In some implementations, the apparatus includes a wireless device, such as a UE. In some implementations, the apparatus can include at least one processor, and a memory coupled to the processor. The processor may be configured to perform the operations described herein with respect to the apparatus. In some other implementations, the apparatus can include a non-transitory computer-readable medium having program code recorded thereon, and the program code can be executable by a computer to cause the computer to perform operations described herein with reference to the apparatus. In some implementations, the apparatus can include one or more units configured to perform the operations described herein. In some implementations, a method of wireless communication may include one or more operations described herein with reference to the apparatus.

In one or more aspects, techniques for supporting image capture and/or image processing may include additional aspects, such as any single aspect or any combination of aspects described below or in connection with one or more other processes or devices described elsewhere herein. In a first aspect, supporting image capture and/or image processing may include an apparatus configured to perform operations comprising: performing a first automatic exposure control operation with a first set of target parameters to determine a first exposure level; performing a first autofocus operation at the first exposure level to determine a first focus position; performing a second auto-exposure control operation with a second set of target parameters after performing the first auto-focus operation to determine a second exposure level; and capturing the first image frame at the second exposure level. In addition, the apparatus may perform or operate in accordance with one or more aspects described below. In some implementations, the apparatus includes a wireless device, such as a UE. In some implementations, the apparatus can include at least one processor, and a memory coupled to the processor. The processor may be configured to perform the operations described herein with respect to the apparatus. In some other implementations, the apparatus can include a non-transitory computer-readable medium having program code recorded thereon, and the program code can be executable by a computer to cause the computer to perform operations described herein with reference to the apparatus. In some implementations, the apparatus can include one or more units configured to perform the operations described herein. In some embodiments, a method of image processing and/or image capturing may include one or more operations described herein with reference to the apparatus.

In a second aspect, in combination with the first aspect, a first automatic exposure control operation is performed with a first set of target parameters to determine a first exposure level to improve focus in a region of interest of a scene.

In a third aspect, with reference to one or more of the first or second aspects, performing a first Automatic Exposure Control (AEC) operation includes at least one of: increasing target exposure, decreasing target exposure, increasing sensor gain, decreasing sensor gain, increasing exposure duration, and/or decreasing exposure duration.

In a fourth aspect, in combination with one or more of the first to third aspects, the apparatus is further configured to perform operations comprising: determining a value representative of a characteristic of the region of interest prior to performing a first automatic exposure control operation with the first set of target parameters; and determining whether the value satisfies a condition, wherein performing the first automatic exposure control operation with the first set of target parameters is in response to determining whether the value satisfies the condition.

In a fifth aspect, in combination with one or more of the first to fourth aspects, determining whether the value satisfies a condition includes determining whether the value indicates one or more of: the contrast in the region of interest is below a contrast threshold, the confidence in the focus position is low, the convergence rate of the autofocus operation is below a rate threshold, or the light level in the region of interest is below a light threshold.

In a sixth aspect, in combination with one or more of the first to fifth aspects, determining a value representative of a characteristic of the region of interest includes determining at least one of a brightness, a contrast, a PDAF confidence level, or a sensor level.

In a seventh aspect, in combination with one or more of the first to sixth aspects, the apparatus is further configured to perform operations including performing a second autofocus operation during performing the second autofocus control operation.

In an eighth aspect, in combination with one or more of the first to seventh aspects, performing the second autofocus operation includes: a laser-assisted autofocus operation is performed.

In a ninth aspect, in combination with one or more of the first to eighth aspects, the apparatus is further configured to perform operations comprising: detecting a scene change after performing the second automatic exposure control operation; and, in response to detecting the scene change, repeatedly performing the first auto-exposure control operation and performing the first auto-focus operation.

In a tenth aspect, in combination with one or more of the first to ninth aspects, the apparatus is further configured to perform operations comprising: receiving, by at least one processor, first information regarding a first automatic exposure control operation; controlling, by the at least one processor, performance of a first autofocus operation based on the first information; receiving, by the at least one processor, second information regarding the first autofocus operation; and controlling, by the at least one processor, execution of the second automatic exposure control operation based on the second information.

In an eleventh aspect, in combination with one or more of the first to tenth aspects, the apparatus further comprises a camera.

In one or more aspects, techniques for supporting image capture and/or image processing may include additional aspects, such as any single aspect or any combination of aspects described above or below or in connection with one or more other processes or devices described elsewhere herein. In a twelfth aspect, supporting image capture and/or image processing may include: an apparatus configured to perform operations comprising: receiving a first image frame from a camera; determining statistics about a region of interest of the first image frame; determining whether the statistical data satisfies a condition; and when the statistical data satisfies a condition: performing a first automatic exposure control operation with a first set of target parameters to determine a first exposure level; performing a first autofocus operation at the first exposure level to determine a first focus position; performing a second auto-exposure control operation with a second set of target parameters after performing the first auto-focus operation to determine a second exposure level; receiving a set of image frames while performing the second automatic exposure control operation; detecting a scene change between two image frames in a set of image frames; and, after detecting the scene change: the operations of receiving the first image frame, determining the statistics, and determining whether the statistics satisfy the condition are repeated. In addition, the apparatus may perform or operate in accordance with one or more aspects as described above or below. In some implementations, the apparatus includes a wireless device, such as a UE. In some implementations, the apparatus can include at least one processor, and a memory coupled to the processor. The processor may be configured to perform the operations described herein with respect to the apparatus. In some other implementations, the apparatus can include a non-transitory computer-readable medium having program code recorded thereon, and the program code can be executable by a computer to cause the computer to perform operations described herein with reference to the apparatus. In some implementations, the apparatus can include one or more units configured to perform the operations described herein. In some embodiments, a method of image processing and/or image capturing may include one or more operations described herein with reference to the apparatus.

In a thirteenth aspect, in combination with one or more of the first to twelfth aspects, the apparatus is further configured to perform a second autofocus operation during performance of the second autofocus control operation.

In a fourteenth aspect, in combination with one or more of the first to thirteenth aspects, performing the second autofocus operation includes: a laser-assisted autofocus operation is performed.

In a fifteenth aspect, in combination with one or more of the first to fourteenth aspects, the apparatus is further configured to capture a second image frame at a second exposure level when performing a second autofocus operation.

In a sixteenth aspect, with reference to one or more of the first to fifteenth aspects, detecting a scene change between two image frames in the set of image frames comprises: it is determined that a change in focal length of the laser assisted autofocus operation exceeds a distance threshold.

Those skilled in the art will appreciate that: information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

The components, functional blocks, and modules described herein with reference to fig. 1-5 include processors, electronic devices, hardware devices, electronic components, logic circuits, memories, software code, firmware code, and other examples, or any combination thereof. Software should be construed broadly to mean instructions, instruction sets, code segments, program code, programs, subroutines, software modules, applications, software packages, routines, subroutines, objects, executable programs, threads of execution, procedures and/or functions, and other examples, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. Furthermore, the features discussed herein may be implemented via dedicated processor circuitry, via executable instructions, or a combination thereof.

Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. The skilled artisan will also readily recognize that the order or combination of components, methods, or interactions described herein are merely examples, and that components, methods, or interactions of the various aspects of the disclosure may be combined or performed in ways other than those illustrated and described herein.

The various illustrative logics, logical blocks, modules, circuits, and algorithm processes described in connection with the implementations disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. The interchangeability of hardware and software has been described generally in terms of functionality, and is illustrated in the various illustrative components, blocks, modules, circuits, and processes described above. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system.

Hardware and data processing apparatus for implementing the various illustrative logics, logical blocks, modules, and circuits described in connection with the aspects disclosed herein may be implemented or performed with a general purpose single or multi-die processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller, or state machine. In some implementations, a processor may be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. In some implementations, particular processes and methods may be performed by circuitry that is specific to a given function.

In one or more aspects, the functions described may be in hardware, digital electronic circuitry, computer software, firmware, including: the structures disclosed in this specification and their structural equivalents or in any combination thereof. Implementations of the subject matter described in this specification can also be implemented as one or more computer programs, which are one or more modules of computer program instructions, encoded on a computer storage medium for execution by, or to control the operation of, data processing apparatus.

If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. The processes of the methods or algorithms disclosed herein may be implemented using processor-executable software modules that may reside on a computer-readable medium. The computer readable medium includes: computer storage media and communication media, the communication media comprising: any medium that can be enabled to send a computer program from one place to another. A storage media may be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media may comprise: random Access Memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Further, any connection is properly termed a computer-readable medium. Magnetic and optical disks as used herein include: compact Discs (CDs), laser discs, optical discs, digital Versatile Discs (DVDs), floppy disks, and blu-ray discs where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included: within the scope of computer-readable media. Additionally, the operations of a method or algorithm may reside as one or any combination of code and instruction set on a machine readable medium and computer readable medium, which may be incorporated into a computer program product.

Various modifications to the implementations described in this disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to some other implementations without departing from the spirit or scope of the disclosure. Thus, the claims are not intended to be limited to the implementations shown herein but are to be accorded the widest scope consistent with the disclosure, principles and novel features disclosed herein.

In addition, those skilled in the art will readily recognize that the terms "upper" and "lower" are sometimes used to ease the description of the drawings and indicate relative positions on properly oriented pages corresponding to the orientation of the drawings and may not reflect the proper orientation of any device as implemented.

Certain features that are described in this specification in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple implementations separately or in any suitable subcombination. Furthermore, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Similarly, although operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Further, the figures may schematically depict one or more example processes in the form of a flow diagram intent. However, other operations not depicted may be incorporated into the example process shown schematically. For example, one or more additional operations may be performed before, after, concurrently with, or between any of the illustrated operations. In some circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. In addition, some other implementations are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results.

As used herein, includes: in the claims, the term "or" when used in a list of two or more items means that any one of the listed items may be used alone or any combination of two or more of the listed items may be used. For example, if the composition is described as comprising component A, B or C, the composition may comprise a alone; b alone; c alone; a combination of A and B; a combination of a and C; a combination of B and C; or a combination of A, B and C. Further, as used herein, includes: in the claims, "or" as used in a list of items beginning with "at least one" means a separate list, e.g., "at least one of A, B or C" list refers to a or B or C or AB or AC or BC or ABC (i.e., a and B and C) or any combination of any of these. As understood by those of ordinary skill in the art, the term "substantially" is defined as largely but not necessarily entirely specified content (and includes specified content, e.g., substantially 90 degrees includes 90 degrees and substantially parallel includes parallel). In any disclosed implementation, the term "substantially" may be replaced with "within a specified [ percentage ], wherein the percentage includes 0.1%, 1%, 5%, or 10%.

The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method, comprising:

performing a first automatic exposure control operation using a first set of target parameters to determine a first exposure level;

performing a first autofocus operation at the first exposure level to determine a first focus position;

performing a second auto-exposure control operation to determine a second exposure level using a second set of target parameters after performing the first auto-focus operation; and

the first image frame is captured at the second exposure level.

2. The method of claim 1, wherein the first automatic exposure control operation is performed with the first set of target parameters to determine the first exposure level to improve focus in a region of interest of a scene.

3. The method of claim 2, wherein performing the first automatic exposure control operation comprises at least one of raising a target exposure or reducing a target exposure.

4. The method of claim 1, further comprising:

determining a value representative of a characteristic of a region of interest prior to performing the first automatic exposure control operation using the first set of target parameters; and

it is determined whether the value satisfies a condition,

wherein performing the first automatic exposure control operation with the first set of target parameters is in response to determining whether the value satisfies the condition.

5. The method of claim 4, wherein determining whether the value satisfies the condition comprises determining whether the value indicates one or more of: the contrast in the region of interest is below a contrast threshold, the confidence of the focus position is low, the convergence rate of the autofocus operation is below a rate threshold, or the light level in the region of interest is below a light threshold.

6. The method of claim 4, wherein determining the value representative of the characteristic of the region of interest comprises determining at least one of: brightness, contrast, PDAF confidence level, or sensor level.

7. The method of claim 1, further comprising:

a second auto-focus operation is performed during the execution of the second auto-exposure control operation.

8. The method of claim 7, wherein performing the second autofocus operation comprises performing a laser-assisted autofocus operation.

9. The method of claim 1, further comprising:

detecting a scene change after performing the second automatic exposure control operation; and

in response to detecting the scene change, the first automatic exposure control operation and the first autofocus operation are repeatedly performed.

10. The method of claim 1, further comprising:

receiving, by at least one processor, first information regarding the first automatic exposure control operation;

controlling, by at least one processor, execution of the first autofocus operation based on the first information;

receiving, by at least one processor, second information regarding the first autofocus operation; and

the execution of the second automatic exposure control operation is controlled by at least one processor based on the second information.

11. An apparatus, comprising:

a memory storing processor readable code; and

At least one processor coupled to the memory, the at least one processor configured to:

the first image frame is captured at the second exposure level.

12. The apparatus of claim 11, wherein the first automatic exposure control operation is performed with the first set of target parameters to determine the first exposure level to improve focus in a region of interest of a scene by increasing brightness in the region of interest of the scene.

13. The apparatus of claim 12, in which the at least one processor is further configured:

determining a value representative of a characteristic of the region of interest prior to performing the first automatic exposure control operation using the first set of target parameters; and

it is determined whether the value satisfies a condition,

wherein the at least one processor performs the first automatic exposure control operation with the first set of target parameters in response to determining whether the value satisfies the condition.

14. The apparatus of claim 13, wherein determining whether the value satisfies the condition comprises determining whether the value indicates one or more of: the low contrast in the region of interest, the low confidence in the focus position, the convergence rate in the autofocus operation being below a rate threshold, or the light level in the region of interest being below a light threshold.

15. The apparatus of claim 11, in which the at least one processor is further configured:

16. The apparatus of claim 15, wherein performing the second autofocus operation comprises performing a laser-assisted autofocus operation.

17. The apparatus of claim 11, in which the at least one processor is further configured:

18. The apparatus of claim 11, in which the at least one processor is further configured:

Receiving first information about the first automatic exposure control operation;

controlling execution of the first autofocus operation based on the first information;

receiving second information about the first autofocus operation; and

and controlling execution of the second automatic exposure control operation based on the second information.

19. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform operations comprising:

the first image frame is captured at the second exposure level.

20. The non-transitory computer readable medium of claim 19, wherein performing a first auto-exposure control operation with a first set of target parameters determines a first exposure level to improve focus in a region of interest of a scene by increasing brightness in the region of interest of the scene.

21. The non-transitory computer-readable medium of claim 19, wherein the instructions, when executed by the processor, cause the processor to perform further operations comprising:

it is determined whether the value satisfies a condition,

22. The non-transitory computer-readable medium of claim 19, wherein the instructions, when executed by the processor, cause the processor to perform further operations comprising:

23. The non-transitory computer-readable medium of claim 22, wherein the instructions, when executed by the processor, cause the processor to perform further operations comprising:

detecting a scene change after the second auto-exposure control operation is performed by detecting a focus change exceeding a threshold value during the execution of the second auto-focus operation; and

24. The non-transitory computer-readable medium of claim 19, wherein the instructions, when executed by the processor, cause the processor to perform further operations comprising:

25. The non-transitory computer-readable medium of claim 19, wherein the instructions, when executed by the processor, cause the processor to perform further operations comprising:

receiving second information about the first autofocus operation; and

26. An apparatus, comprising:

A camera;

a memory; and

at least one processor coupled to the memory and configured to:

receiving a first image frame from the camera;

determining statistics about a region of interest of the first image frame;

judging whether the statistical data meets the condition or not;

when the statistical data satisfies the condition:

performing a second auto-exposure control operation to determine a second exposure level using a second set of target parameters after performing the first auto-focus operation;

receiving a set of image frames while performing the second automatic exposure control operation;

detecting a scene change between two image frames in the set of image frames; and

repeating the following operations after detecting a scene change: the method includes receiving a first image frame, determining statistics, and determining whether the statistics satisfy the condition.

27. The apparatus of claim 26, wherein the at least one processor is further configured to:

28. The apparatus of claim 27, wherein performing the second autofocus operation comprises performing a laser-assisted autofocus operation.

29. The apparatus of claim 27, wherein the at least one processor is further configured to:

a second image frame is captured at the second exposure level while the second autofocus operation is performed.

30. The apparatus of claim 27, wherein detecting the scene change between two image frames in the set of image frames comprises determining that a focal length change of the laser assisted autofocus operation exceeds a distance threshold.